Gwil's garden

Keeping myself busy.

Earthstar × NLnet

December 30, 2021

It's been a year and a half since I began contributing to the Earthstar project. The project's aim is to provide a toolbox for users who want to operate their own online utilities: their own personal file sync; a blog publishing system; graphical chatrooms; a bespoke social network.

Earthstar sits on a relatively unexplored band of the spectrum of new web technologies: ephemerality in place of permanence, privacy in place of discovery, trust in place of trustlessness, small where everything else is big.

All of this gets me excited. When you're donating your spare hours to a project that excites you, you naturally start asking yourself: how can I work on this full-time?

NLnet Foundation has been funding projects building towards an open, fairer web since 1997. They've funded projects like Manyverse, Briar, MNT Reform, PeerTube, and Wireguard to name just a few.

And now Earthstar can be added to that list!

<mark>This funding means I'll be able to work full-time on Earthstar for an extended period of time</mark>.

What will I be working on?

The planned work has been broken down into five milestones.


The first milestone will deliver a complete reworking of Earthstar that is nearing completion. This new version has support for resumable synchronisation (e.g. over intermittent connections), support for multiple JS runtimes (e.g. browsers, Node, Deno), a synchronous cache abstraction for replicas, and a channel-based API for subscribing to replica events. The major remaining part is implementing synchronisation over a swappable transport (e.g. HTTP, WebSocket, BroadcastChannel).

Collaborative knowledge publishing

The second milestone focuses on delivering the benefits of the new version of Earthstar (and its future additions within this project) via a CLI tool, as well as a compelling use case: collaborative knowledgebase editing and publishing. Users will be able to edit and view a knowledgebase as a simple directory of files on their filesystem, editable with their existing tools. Using the Earthstar CLI they will be able to sync these documents to their own pub servers, which will be able to serve a read-only version of this knowledgebase as a website.

Replica servers

Earthstar promotes the use of small, user-operated ‘replica servers’ for networks to propagate their data through. In order for this to be a viable approach, we need a replica server that is trivial to deploy and configure. We also need ways for early adopters of this to support the general Earthstar ecosystem.

Restricting read / write access

Obtaining a share’s public key currently grants you read/write access to its contents. In this milestone, an optional private key required for write access will be added. This opens the door to being able to host a share’s data without being able to write to it. Additionally, helpers for encrypting and decrypting the contents of a document will be added.

Large blob support

Finally, we want to add support for large blobs to Earthstar. This would mean support for internally representing a document’s contents as Uint8array, and improved methods for storing this data in different replica implementations, especially the IndexedDB replica.

There's a lot to be done, and thankfully much of it is already underway. Wish me luck! I'll be sharing monthly development updates here, on Earthstar's Open Collective, and on Secure Scuttlebutt (@PEqxT8YRvrkf2fyjAgTl8pmdo8aR4KVsKLyh0b5JyOE=.ed25519)


This marks the end of a bumpy transition from commercial to public work for me. I want to make this the beginning of something lasting, rather than a flash in the pan, and I'm honoured that I can even try. What a way to end one year and begin another.

My thanks to: