This week sees a concerted effort to choose the right database for our backend. We have worked on the architecture of our plugin system, while starting development on more plugins of our own, and made good progress on the legal framework required for our organisation.
Our latest choices for a database have resulted in two main contenders: Dgraph and SQLite. To make an informed decision we spent a large part of the week optimising and testing the performance of these two databases. The main problem here is the support of heterogeneous types: the ability to query the database without expecting the result to be of a specific type.
While Dgraph natively supports heterogeneous types, a relational database like SQLite does not. The support in Dgraph comes at a cost however, since Dgraph is designed for larger cluster deployments and comes with a memory footprint to match. SQLite is aimed at smaller installations and can be made to support heterogeneous types with some trickery, but this could have a negative influence on performance for large datasets.
The results we got so far are encouraging. We sufficiently brought down the memory footprint of Dgraph, and SQLite seems to be performant enough for our purposes. We are now in the luxury position that both databases are good candidates for our final pick.
We improved on our architecture for plugin support, both for our own plugins and those contributed by external developers. Plugins should be properly isolated and only have the minimal amount of information necessary to do their job. Compromised plugins can not become a stepping stone to break into other components of the backend.
In particular we are using OAuth to not expose user credentials to plugins. And while many services use the newest version 2 of OAuth, some still use the old version 1 which will require more work to support.
Additional plugins for downloading and importing data into Memri are now in development, and we started writing our first indexer plugin to analyse and enrich our database using existing data.
To ensure a proper balance of power between our stakeholders, we are investigating different ways to incorporate the Memri project. In particular we want to formalise our vision on the relationship between e.g. our employees, community and investors.
What is actually possible depends on the country we incorporate in, and by going back and forth on ideas we have really improved our understanding on what is possible.
In addition to incorporation is our work on an open source license that takes into account the privacy of the project's users. Currently based on the MPL 2.0 license, we have a first draft versioning nearing completion, which we plan to share with you soon.
Work on the first front-end application for iOS has reached a point where we are fixing lots of small issues so that its ready for the first alpha testers and for community members to write CVU for. We also ported the first part of the app (the parsers to be specific) to the web. These will be the basis for tooling that make it easy to write CVU. The app feels more and more stable and ready to be used on a daily basis!
If you want to go fast, go alone. If you want to go far, go together.
We started the preparation for building a lively community around Memri. Right now we are working on a developer's guide as well as other documentations to make sure they will be ready for access. Hopefully sooner we will be welcoming our first community members. Stay tuned!