[WasmFS] Create public wasmfs_mount function

[sbc100]

We already had wasmfs_unmount so it seems logical we should have this one too.

This is designed to replace wasmfs_create_directory which has a rather misleading name.

I have left wasmfs_create_directory around for now but is simply combination of mkdir + mount (which is essentially what is was doing already).

This change ends up simplifying doMkdir which no longer needs to take a backend argument.

[sbc100]

Still a work-in-progress, but seems to pass all tests.

I'd like to deprecate wasmfs_create_file too.

[tlively]

This looks good to me so far, but I think we need to keep the ability to mount individual files. Do you have a plan to deprecate wasmfs_create_file but keep that ability?

[sbc100]

This looks good to me so far, but I think we need to keep the ability to mount individual files. Do you have a plan to deprecate wasmfs_create_file but keep that ability?

I think I'd be ok with that, but I want to be clear what the use case is. What kind of backend would be single file mount? Presumably such a backend could only ever contains a single file, right? Since we cannot mount backends twice?

Would you be ok with renaming wasmfs_create_file to wasmfs_mount_single_file?

[tlively]

What kind of backend would be single file mount? Presumably such a backend could only ever contains a single file, right? Since we cannot mount backends twice?

Right. I think we have fetch backend that fetches a single file from a given URL, for example.

Would you be ok with renaming wasmfs_create_file to wasmfs_mount_single_file?

Sounds fine, but we probably need to keep the old name around for a bit to avoid breakage.

[kripken]

How does this work? Does it delete the existing directory if it exists? Or if the directory exists and has contents, are they converted to the new backend somehow?

[sbc100]

The child directory gets replaced. At least that is the idea.

Anything that was there before is no longer accessible. Ideally it would become visibile again if/when the filesystem is unmounted, but I guess that would only really work for things like OPFS that are persistent.

[tlively]

If the "process" stays alive and the mounted filesystem is unmounted, then even if the parent backend is something ephemeral like MEMFS, the previous contents should become visible again, right? As long as the parent backend is concerned, those contents were there the whole time.

[sbc100]

Yes, I agree that would be ideal. Is that how it works today though? I think we would need to add more tests to confirm.

[tlively]

That's not how wasmfs_create_directory works, no, because it's not allowed to overwrite or shadow an existing directory. But if we're adding a proper mount, then that's the only reasonable behavior.

[kripken]

I think it may be simpler to just leave aside the concept of "mounting".

Starting from scratch: all we have here is a system with files and directories, and each file and directory is implemented in some manner. For example, the root is typically implemented using a simple in-memory approach. Some files may be enormous, larger than can fit in wasm memory, so we want to implement them on the JS side (using a Typed Array or Blob or such), so we also have JS-backed files, and so forth.

And, when one creates a file or directory, the default is to create it using the same implementation as the parent. To change that one must override.

So if you have a game with a folder of files, some of which need to use JS, you would createFile most normally, but createFile the JS-backed ones by specifying the JSFile implementation.

Does that make sense? It seems unambiguous to me. This isn't posix.

For example, having wasmfs_create_file sometimes create a file and sometimes not, depending on the backend seems odd to me.

createFile always creates a file, I think?

Its was not clear to me if the file or directory being created lives in backend itself, or in the FS into which the backend is being mounted.

I wouldn't think in terms of things living in backends or being mounted. Instead, all the files lives as files in WasmFS. The technical details of how they are implemented can vary between files (is their data in wasm memory, or in JS), but WasmFS sees them all as a whole. WasmFS will do operations like read() which are then virtual methods that implement the right behavior for each file.

[sbc100]

I think it may be simpler to just leave aside the concept of "mounting".

Starting from scratch: all we have here is a system with files and directories, and each file and directory is implemented in some manner. For example, the root is typically implemented using a simple in-memory approach. Some files may be enormous, larger than can fit in wasm memory, so we want to implement them on the JS side (using a Typed Array or Blob or such), so we also have JS-backed files, and so forth.

This makes some sense to me when you are talking about creating new, ephemeral files.

It makes a little less sense when you are dealing with external, pre-existing files, such as the case for hostfs/nodefs or OPFS. For example, can I decide that I want certain specific files to be implemented using nodefs or OPFS?

And, when one creates a file or directory, the default is to create it using the same implementation as the parent. To change that one must override.

So if you have a game with a folder of files, some of which need to use JS, you would createFile most normally, but createFile the JS-backed ones by specifying the JSFile implementation.

Does that make sense? It seems unambiguous to me. This isn't posix.

Why stray from POSIX though? Its something that is well understood by many folks. The concept of mounting directories that is. I'm not sure I see the benefit of straying. I think you are saying that fine grained mounting of files is the benefit but I'm not sure I see that. Do we have game developers (or anyone) asking for per-file backends?

For example, having wasmfs_create_file sometimes create a file and sometimes not, depending on the backend seems odd to me.

createFile always creates a file, I think?

So I can't use createFile to add an existing file from nodefs or wasmfs? I agree it would be confusing to do this, but its seems like the API wants to allow this.

[kripken]

It makes a little less sense when you are dealing with external, pre-existing files, such as the case for hostfs/nodefs or OPFS. For example, can I decide that I want certain specific files to be implemented using nodefs or OPFS?

Yes, you can. And what that means is left to the backend.

So if you have a normal root and then add a NODEFS directory under /my_nodefs then accessing /my_nodefs will go through that backend, which will then use node in some manner. Typically when creating such a directory you tell the backend what to do, in this case, which real files to refer to. Say, /my_nodefs in WasmFS may map to /home/foo/sandbox/ in the outer filesystem. So accessing /my_nodefs/file.txt will go through NODEFS to access /home/foo/sandbox/file.txt.

Why stray from POSIX though? Its something that is well understood by many folks. The concept of mounting directories that is. I'm not sure I see the benefit of straying.

I would say that the model of WasmFS is simpler than posix... if you are not very used to posix, I suppose. I have really summarized all of it in that one paragraph before, while posix has a lot more detail to it.

I am not opposed to adding full posix support on top, if we want that. But I think the basic model of "WasmFS has files and directories; each of those is implemented in a customizable manner" is a really simple baseline.

I think you are saying that fine grained mounting of files is the benefit but I'm not sure I see that. Do we have game developers (or anyone) asking for per-file backends?

Yes, that feature request came from @juj .

And it is simple in the WasmFS model. Any file and directory can be created in any backend. That is, disallowing files would be more complex.

So I can't use createFile to add an existing file from nodefs or wasmfs? I agree it would be confusing to do this, but its seems like the API wants to allow this.

I think you can use createFile to add an existing file from nodefs. Why would you not be able to?

Perhaps we could decide that nodefs can only create directories and not files, if that feels simpler. I don't recall if we discussed that. But in principle there is no reason not to create files as well. The file would just track the underlying file it is backed by, just like a JSFile tracks the JS side data it is backed by.

Put another way, why would nodefs be different than JSFile? That nodefs files are persistent is a difference, but not a problem (when creating the file as backed by nodefs, you can tell it what it is backed by).

[sbc100]

I think you can use createFile to add an existing file from nodefs. Why would you not be able to?

Doesn't that directly contradict what you just said about "createFile always creates a file".

[kripken]

Oh, I think I see our misunderstanding.

createFile always creates a WasmFS file.

And a WasmFS file may be backed in various ways. For example, if we create a WasmFS file that is backed by memory, then we allocate memory for it. Or, if we back it by JS, we call into JS to prepare the data structures there. In each case the backend decides what to do.

And what the backend does, can include persistent storage. createFile might create a WasmFS file that is backed by nodefs, and nodefs might be given a path that already exists. In that case we create a WasmFS file but not a file in the underlying host filesystem.

In other words, whether the backend's implementation also creates anything is left to the backend. Likewise, whether it persists anything afterwards, etc. From the WasmFS perspective all those behaviors are abstracted away and left to the backends, leaving WasmFS simpler.