TL;DR: Efficient sync implementation is done. I published a new generic library for range-based set reconciliation. I have finished a GUI for modifying a server’s settings. I am exploring using Earthstar servers to serve Earthstar browser apps which are stored and distributed on Earthstar shares (Earthstar). The next major version of Earthstar will be released January 2023.

Last month I wrote about range-based set reconciliation, the method powering Earthstar’s new efficient sync. Since then, the work of integrating this into Earthstar has been completed, and yielded a bonus.

That bonus is a module is called range-reconcile. This is a generic TypeScript implementation of the range-based reconciliation method, and has a pretty nice message protocol attached to it. It’s definitely on the ‘low-level building block’ end of the library spectrum, but it should serve as a good reference to some, and a handy tool for the motivated. It’s used inside of Earthstar so I expect it to be well maintained.

And that finishes milestone 4 of 5 for this year. This was also the last milestone which makes major changes to Earthstar’s implementation.

For the last milestone, I can start working on more user-facing problems again.

Servers serving web UIs

Running a server requires a lot of technical literacy, and there’s not a lot that can be done about it. But the truth about internet-connected p2p is that someone is going to have to run a server somewhere, if only to let peers know of each others existence.

With Earthstar I don’t want to obscure this, but emphasise it: if you want to sync over the internet, you will need to know someone with a server. And that is useful! An always online peer who can also serve requests in a browser? There’s a lot you can do with that.

One example: serving a web interface to data stored on the peer. This is what the blog you’re reading now is. This blog is an Earthstar peer I can sync data to, and when it receives browser requests it queries data from its replica and uses it to build a HTML response, i.e. the page you’re reading now.

It’s one thing serving static HTML, but what if a replica server could also serve full-blown browser applets?

And what if those applets had access to the same settings and share data?

And what if we stored those applets in a share, so we could sync around client code the same way we do other data?

This is something I wanted to begin exploring while creating a new web based UI for server settings.

Server settings web UI

This is an applet for modifying which shares a server will replicate with peers. Rather than using authentication and persistence specific to the server, these settings are stored on an Earthstar share.

This system has two components:

• A web based client which creates configuration documents and syncs them to server(s).
• A server extension which reads these documents and uses them to configure itself. The server chooses a specific share to read its settings from.

I’m pretty happy with this arrangement.

Firstly, we don’t need to complicate servers with their own authentication and settings persistence.

Secondly, we can specify settings once and have them sync to many servers.

Finally, we can use some of Earthstar’s built-in capabilities to provide unique features. For instance, we can create temporary settings using ephemeral documents. This lets server operators do things like host a specific share for only a week.

The final piece needed to make this work is easy access to the web client. Access which the server can provide!

Servers serving clients

Then I start asking myself how the server knows where to get the assets for the settings app. Where is all this client code stored? Am I building it on the server…?

And then I start thinking awful, terrible, wonderful thoughts. With Earthstar supporting binary attachments, why not store client code in an Earthstar share too, and read it off there?

Then a single client can be synced between many servers, and — oh! — why stop at a settings app? Let’s store loads of clients on a share. Chat apps. Notepads. Shared bookmarks.

So I’ve started working on a kind of desktop UI for servers to present, which reads available applets off of an Earthstar share and serves them to users.

Because all of these applets are served from the same server, they fall under the same origin rule, which means they have access to a common pool of data.

The desktop takes advantage of this by providing a UI to manage user keypairs, shares, and servers, which any client can then read the configuration of (this has been exposed via a new ClientSettings API exported by the JS library).

The client apps don’t need to be built with any specific frameworks, they just need to be able to run in the browser.

This desktop UI will be made available by a single server extension users can add to their Earthstar servers. At some point. A lot of work to be done yet.

Speaking of which…

Wrapping up the year

There is a huge amount of work done this year which has yet to be released. This includes attachments, protected shares, and efficient sync, and lots more. I will be using the last month of 2022 to prepare this release, writing more documentation, updating READMEs, tutorials, updating the website, and getting everything ready for Earthstar’s biggest release ever. I expect this release to be out by January of 2023.