Shared storage new API

[fwsGonzo]

Currently the storage VM is a completely separate VM from the main VM, which runs through main() separately, with a different type argument. Instead of "request" it's "storage".

This is fine, however it takes quite a bit of redundant memory and page tables to have two separate VMs with a large (sometimes gigantic) address space.

Instead, I would like to either directly use the main VM as storage, or make a fork of the main VM and use that as storage. I will most likely implement the fork variant, as that will be the most robust solution. It's also safe to take down storage, re-initialize it if it crashes etc. Can't do that with the main VM.

But, what will the API look like now? All forks must have the same parent VM, and they must all be forked from the same state. So, how do you turn that into a good API? How do you diverge from the main VM into a storage VM in a sane way?

Initially I thought:

if (enable_storage()) {
    // Inside this we have diverged from requests and are now a storage VM
}

... but that's not possible, because forks can not start from a different main VM state. It introduces potential ghosts in the machine and lots of scary behavior.
The simplest way could be to introduce a new callback function, which is simply what happens after storage is forked:

set_storage_initializer(my_storage_function);

Not sure yet.

[lrowe]

I hope you don't mind me commenting a little more broadly about storage here.

Putting everything in a single program is a great for prototyping and has the nice property of making in place upgrades easier, but it is limiting for more complex use cases where you might want to mix and match different components built with different frameworks or languages which might be built and maintained by other people.

Perhaps someone ports Redis/Valkey to TinyKVM which you use for storage, your UI code is rendered with Deno and you build a small dedicated Rust or C++ program to handle websocket traffic. So I'd love to see the various methods of communication broken out to allow communication between different instances.

For non-ephemeral instances it seems like sockets and/or shared memory could handle most things. It should be possible to use these with some sort of async reactor within the guest. It would be amazing if there was some way to get SQLite WAL mode (which relies on a small amount of shared memory) to work between instances/programs.

Things are a little trickier for ephemeral instances since you don't want to inadvertently leave non-ephemeral state around in shared memory or an open connection to a storage server which might retain session state.

[fwsGonzo]

The intention behind storage is to have agreement and a way to execute logic serially across requests/time, not to embed container-ware like Redis. Like the Redis architect himself writes https://news.ycombinator.com/item?id=19464144, many have tried it and failed for a variety of reasons. Redis is large, doesn't need the "best latency", and it gives a lot of guarantees. But we can certainly try to add a Redis client.

I believe the Enterprise-version of Varnish will have a global kv-store soon with a Redis backend. It will be embedded a as a library in VC Enterprise (correct me @scance) and so I believe TinyKVM can just access it through custom system calls. That doesn't really cover the "being able to port anything" question you had, though, but I have no answer other than we'll see.

As for SQLite I have some experience with it from before. It's very easy to add and doesn't really depend on much. I actually added it to IncludeOS back in the day and made it store the database image in high memory, and added a way to store the database state to my live-update mechanism so that when IncludeOS (the entire OS) was hot-reloaded the database could survive the update. I think adding a SQLite program makes perfect sense. But how do you intend to use it? There are usually language-specific bindings that greatly simplify interaction. To me adding SQLite is a 5min job, so I wouldn't add it unless I needed it right now.