When you're working on a lot of interrelated ideas for a long time, I think there's this natural point where, meiosis-like, that body of ideas starts to split and boundaries form around different groups of ideas.
In Earthstar's case, these groups would be:
- Formatting rules for document paths and 'cinn25519' keypairs.
- A sound, multi-writer, mutable data model and replication scheme for distributed systems.
Willow is the name for a new project which will formalise this latter set of ideas. We think it's a compelling new primitive for building distributed systems, and one which we think can supplant the dominant paradigm — the append-only log — for many use-cases.
And I'm delighted to say that I can use the pronoun 'we', as I'll be joined by my esteemed friend Aljoscha Meyer on this project. We'll be working towards three outcomes:
- A specification describing Willow’s data model, storage and replication.
- A reference implementation written in TypeScript.
- Using this implementation as the new core of Earthstar.
Willow won't just be a reshuffling of Earthstar's existing functionality. It will bring new features (like tombstone-less deletion) as well as new efficiencies. It will bring the best things about Earthstar to new use-cases and environments.
I'm very happy and excited to be working on this. I'm particularly happy to be working with Aljoscha again, as we had a lot of fun collaborating on implementing range-based set reconciliation for Earthstar.
I'm also very grateful to have funding dedicated specifically towards writing a specification — many thanks to NLnet for providing that funding and making this kind of foundational work possible.