Can we use thread-local storage (as implemented by e.g. LLVM) as a guiding precedent here? I.e. can we teach LLVM to manage task-local storage using an analogous mechanism to how it does thread-local storage and build abstractions on top of that for wasi-libc, Rust, etc.?

That'd help keep, in the example above, A/B's state separate for exports yeah. That wouldn't solve C's problem though where C may not be in control of exports but it still wants to register a waitable in theory. In essence C's problem is "ok root of whatever async computation I'm in, add this thing to your waitable set and re-poll me when it's ready". That's out of the scope of data storage and more about runtime semantics. There'll be other management intrinsics as well such as removing waitables, etc.

Yeah, it seems analogous to the challenge of writing runtime-agnostic async libraries in Rust; in absence of a stable, "official" abstraction which runtimes can implement and libraries can target, you end up having to "pick one and stick with it" and then hope nobody tries to compose your library with another one that picked a different runtime (or different major version of the same runtime).

Given that wasi-libc is the common denominator for most language toolchains these days (or at least the ones that want to support shared-everything static or dynamic linking with C), perhaps the core runtime bits in wit-bindgen-rt need to move there, with each higher-level language building on top of that.

I think that's what I'm concluding as well though. Ideally though something really really small is all that goes in wasi-libc, all of wit-bindgen-rt is quite a lot to write in C and stabilize forever...

Trying to think this through, what I think we sort of want is something like:

struct cm32p3_task {
  // theoretical future-compat? (e.g. v2 means more fields for some future
  // definition of this structure).
  int version;

  // passed to callbacks below
  void *ptr;

  // For the current exported task, register `callback(ptr)` to happen when `waitable`
  // gets signaled. Basically add `waitable` to a waitable set and arrange for the
  // callback to get invoked when the current exported task gets the notification
  // the waitable is ready.
  //
  // If already registered this removes the previous callback.
  //
  // TODO: how to manage the lifetime of `callback` and `ptr`? Second function
  // for dtor? Unsure.
  void (*waitable_register)(void *cm32p3_task_ptr, uint32_t waitable, void(*callback)(void*), void *ptr);

  // Dual of the above, but removes it from the waitable set and/or tracking.
  void (*waitable_unregister)(void *cm32p3_task_ptr, uint32_t waitable);
};

// Set the current task (runtimes call this when an async task is entered)
//
// Basically sets some global to this value.
void cm32p3_task_set(struct cm32p3_task *task);

// Get the current task (runtimes call this when a new waitable is being
// registered, such as when an async import blocked).
struct cm32p3_task *cm32p3_task_get(void);

or... something along those lines.

Ok thinking through this some more. Here's what I'd like to propose:

The context.{get,set} intrinsics

Slot 0 will be reserved for "exported task" use. Slot 1 will be reserved for future use with the stack pointer itself, and used for stack-switching runtimes as well. Slot 0 is expected to be a pointer into linear memory.

Langauges which define the root of an exported tasks (e.g. a function that's canon lift'd) get to work with slot 0. The management of this slot is recommend to:

• The first 4 bytes in memory is a "magic" which is language/runtime specific. This is to help catch bugs of crossing the wires by accident.
• When a canon lift'd function or callback runs then NULL is stored to slot 0. When the function or callback returns then it's replaced with the actual pointer value.
• When accessing slot 0 runtime should assert the returned value is non-null and then assert the magic is as-expected.

Libc conventions

This header would be added to wasi-libc: https://gist.github.com/alexcrichton/99c9d63cfd15673cfe2fa34a7993e20b. Notable parts of this header are:

void p3_task_set(struct p3_task_v1 *task);
struct p3_task_v1 *p3_task_get(void);

This does NOT manage slot 0 with context.{get,set}, this is entirely different. Instead this mutates a 4-byte location in linear memory (e.g. static p3_task_ptr* PTR = NULL; or something like that).

Runtimes which define exports would call p3_task_set. Runtimes which call imports would call p3_task_get. This is the connection point between exports/imports where it's how an import requests its notification gets registered/routed.

typedef struct p3_task_v1 {
  int version;
  void *ptr;
  void* (*waitable_register)(void *p3_task_ptr, uint32_t waitable, void(*complete)(uint32_t code, void *ptr), void *ptr);
  void *(*waitable_unregister)(void *p3_task_ptr, uint32_t waitable);
} p3_task_v1;

This is the structure that runtimes defining exports must provide. This is effectively a vtable of sorts which provides register/unregister functions to add a waitable to the exported task's waitable set. How exactly the exported task manages this is left up to the runtime providing the export, but it's probably managing a waitable set.

Usage in wit-bindgen

Let's say you define an exported function my_export with wit-bindgen 0.30 which calls a function my_import defined with wit-bindgen 0.40. The sequence of events here when calling this is:

1. Runtime-generated glue for my_export is called first
2. This allocates wit_bindgen_30::TaskState which internally contains p3_task_v1.
3. The p3_task_set function is called with this pointer
4. The my_export function is called
5. The my_import function is called and it returns "blocked" as its status code
6. The p3_task_get function is called, and waitable_register is invoked
   with the subtask and a pointer specific to wit_bindgen_40 is passed in.
7. The wit_bindgen_30 runtime adds this waitable to a waitable set and then
   returns.
8. Control flow eventually reaches my_export and the root export.
9. p3_task_set(NULL) is called
10. context.set 0 is used to store wit_bindgen_30::TaskState.
11. Eventually the callback for my_export is called.
12. context.get 0 is called and validated
13. context.set 0 is called with NULL
14. p3_task_set is called with the wit_bindgen_30::TaskState
15. The notification is processed, invoking the callback provided in step (6)
    which is defined in wit_bindgen_40.
16. The Rust-level Future for my_export is polled and it completes.

The tl;dr is that imports/exports are separated by p3_task_v1 at the ABI level and the Rust-language-level async is what glues things together at the source level.

How to implement this

My thinking is that because changing wasi-libc is hard we'd start out providing weak definitions of the above functions in wit-bindgen. That way wit-bindgen would refer to the C-level symbols and would provide definitions as well, but the definitions would get satisfied by any version of wit-bindgen (e.g. any weak definition is taken). That would enable implementing this today as-is and in time this could move to wasi-libc.

cc @vados-cosmonic from our discussion this morning too

SGTM. I like the idea of starting with this in wit-bindgen and then migrating it to wasi-libc later as needed.