Year 2020 has finally ended. It's been so bizarre that in the modern history you cannot name any other period of time that has a bigger impact on the human beings and their daily lives at this scale. Maybe some of you who witnessed all the ups and downs within year 2020 just like we did, have already taken so much effort to adapt to the new normal, may it be a different working schedule, a new lifestyle or whatnot, yet still having problems knowing how to deal with the new challenges in the coming year. As they say, "Well begun is half done." But what makes a good beginning? From Memri's point of view, we believe that an insightful summary of the past would definitely help pave the way to a successful fresh start in the future.
Organization of Memri
This year we started out with just two people doing research towards an idea we had. Earlier in the year we translated that idea into our mission: "Empowering people by unlocking the potential of their data". Then we quickly realized that we, in order to turn it into a successful mission, needed to bundle forces and attract more outside talent. And that's exactly what we did. After recruiting new talents, by the end of 2020 we have grown into a team of 9 people dedicated to the project and a handful of others helping us with specific tasks.
We've transformed what was a project into a separate legal entity with an organizational structure that is rather uncommon and would allow us to ensure our company lives up to its purpose and can't be sold off to the highest bidder, but should also enable us to raise funding from more traditional investors. The challenge was to find a structure that investors know and trust, without losing the democratic values of our organization. We'll dive into how we've done that, the challenges we encountered and how our structure works in a dedicated blog.
Unfortunately this year due to the Covid-19 pandemic we weren't able to organize a team retreat or see each other as much as we would have liked. While being focused on pushing the products further and getting to a place where we could show our work and ideas to investors, we realized that the focus on social activities lacked behind. When we grew our collaboration skills, we sometimes forgot that having fun is something to strive for as well. Just realizing that has made a big difference already, and things like game nights have been adopted with open arms. But just like almost everyone: We can't wait for the pandemic to end and be able to see each other in real life again!
This year started off with a lot of research on securely hosting the pods for our users. We looked at secure multi-party computation and secure enclaves to protect computations from eavesdropping by third parties, investigated hardware virtualization and containerization to isolate users from each other, and considered various cluster solutions like Kubernetes and OpenStack to ensure high availability. In the end we concluded that Kubernetes containers give us the most flexibility in deployment and can still be combined with technologies like hardware virtualization and secure enclaves for optimal privacy. A strong combination for our purposes.
Next up was reducing the footprint of our software: each user will be assigned a private server, so to keep things affordable we need to find the right balance between resource usage and functionality. This issue led into a period of research on unikernels to reduce the overhead of the operation system and experimentation with various database solutions to minimize our memory footprint. We discovered technologies like Kata and Firecracker to create very small virtual machines, and since existing databases turned out to be too resource hungry we ended up designing our own database solution.
The last part of the year was one of designing and starting the implementation of our hosting infrastructure. We had to come up with a payment infrastructure, design solutions for spinning up components on demand, and consider various aspects of monitoring and security in our cluster. While we have only just started putting the various bits and pieces of our hosting infrastructure in place, the end result looks promising and will be something to look forward to in 2021.
Integrators and Indexers
In this spring, we started working on the creation of the pyintegrators repository, a place to store all integrators that are written in Python. Features like tests, documentations, CI etc. are also set up for this repository. We also created a structure in the Pod for it to invoke the integrators. The created integrators can communicate with the pod via a podClient which is part of the client library provided by the pyintegrators repository for easier integrations.
By using the client library, so far we have added several integrators to the pyintegrators repository. For example, one is an Email integrator that authenticates with Gmail server via OAuth2 and then fetches all Gmail messages into the Pod. Another example is the WhatsApp integrator which relies on the Matrix Synapse homeserver and WhatsApp bridge to authenticate with the WhatsApp server via its web service API, and imports all messages including the media files through Matrix's client-server APIs to the Pod. We expect that in the coming year more integrators will be added, for example an iCloud photos integrator is already working in progress, and for social network applications and messengers.
We also created several so-called Indexers to e.g. detect lists in the imported notes, add geographical information to the items etc. One important Indexer can provide state-of-the-art face recognition which contains similar functionality to the face recognition of an iPhone.
The schema is one of Memri’s components that changed quite a lot as our understanding grew over the past year. From the start we knew there’s a balancing act in defining the data types used by Memri. On the one hand, we want to set a standard and a base people can build from. This helps to guide development, make it easy to understand how items are related, and prevent bugs as a result from unexpected changes. On the other hand, we need to provide flexibility for contributors to add whatever types they need to work with the data in a way that suits them best. We want to empower the user after all, and setting strict standards does not provide freedom to work with data however you see fit. We changed several things to find the right balance on this scale. One of the biggest changes is that the schema became data, not code. This means that if the schema is updated to support new data types, the Pod and frontends do not change in their code, they simply load the new schema. This way there’s no need for recompilation, testing, versioning etc. for every schema change. This makes it way faster to develop and provides the freedom to use the schema that fits your use case best.
During the summer, we worked with an external lawyer on the Memri Privacy Preserving License (MPPL), a software license that users of the software to protect people's personal data. The license is based on the Mozilla Public License and adds a privacy clause that aims to protect user's privacy more than the GDPR. Specifically it does this by taking the privacy-by-design architecture of the Memri platform into account and puts constraints on raw data usage. It makes a distinction between encrypted, unencrypted and aggregated data and then specifies different possible uses for the different actors. Specifically someone that is running the software for others can only access encrypted data unless a specific non-click-through agreement has been made with the user and a third-party for usage of data for a specific time period. The license is now at version 0.6 and we are looking for continued interaction with the community to evolve it to version 1.0.
At the beginning of the year we only had a vision of building a front-end that can eventually serve as the display layer of any kind of data and that is usable for non-techies like my mom. We didn't really know yet what that would look like or what the UX paradigm should be. By the middle of the year we knew what we are building is a "data browser" and we developed a programming language called Cascading-Views or CVU for short that enables maximum reuse of interface components while making the system flexible enough to build the many use cases that we envision. We started off building this for iOS using SwiftUI and later the port to the browser application followed from community contributions.
We added many renderers. Renderers are the elemental unit of the CVU language and are configurable using CVU to fit the needs of the micro-app developer. We built the following renderers:
- List renderer
- Grid renderer
- Map renderer
- File renderer
- Calendar renderer
- Timeline renderer
- General editor renderer
- Chart renderer
- Bar chart
- Line chart
- Email renderer
- Photo renderer
We enabled end-to-end encryption for communication of data and encrypted storage of data locally on the device for iOS. The iOS app stores keys in the secure enclave of the device which has hardware security built in and uses bio-metrics to unlock when available.
We also did over 60 use case interviews with potential customers, and learned a lot about the needs for an integrated productivity tool that let you have an overview across all your data. This informed us on where to focus on and what value proposition to work towards in 2021.
We welcomed our first community members this year! Developers who recognize and share the values of Memri have come to us and contributed to various projects: a browser application and an Android application for the front-end, integrators for different social networks and so on. For us, it is always grateful to receive contributions and help from the community members and in reward, we would like to support them as much as we can.
Besides that, we've been working on creating a good contributor's experience: We have setup the new documentation site with consolidated docs from all projects (coming soon), with thorough explanations and tutorials. We've also worked on the contributor's site (memri.io) as the first step into reaching out to the community and welcoming new enthusiasts with all the guidance they need. Although both items are still in progress, stay tuned!