Support creating files and sockets from file descriptors

[robert-ancell]

On Unix systems (e.g. Linux) file descriptors are the handle used for files and sockets. These may be passed between processes, e.g. using Unix domain sockets. This gives the case where Dart code might receive such a file descriptor, but is not able to make use of it using standard Dart APIs.

A proposed API would be something like:

import 'dart:io'; 
var fd = get_fd_from_somewhere();
var file = File.fromFileDescriptor(fd);
var data = await file.readAsString();

This new API would throw an exception on platforms that didn't support it. I'm aware this might be considered a bit special case, so it may be more desirable to have an API like File.fromNativeHandle() that could support other platforms such as Windows (as long as we can make a common handle type).

Another option is to use an FFI based library to wrap the file descriptors, but this makes it much harder to cleanly and efficiently implement asynchronous reads from the file descriptors.

A specific case where this feature would be used is the IPC mechanism DBus, which allows file descriptors to be sent in messages. This is used in cases where large amounts of data need to be transferred between processes, and this is inefficient to do using IPC.

[zanderso]

I would recommend against doing this. At the moment, it is by design that an isolate can't access file descriptors owned by other isolates. However with this change, an isolate could manipulate files owned by another isolate by forging the integer file descriptor. /cc @aam

[robert-ancell]

@zanderso is the intention of isolates that code cannot access resources of another isolate (i.e. a security requirement) or shouldn't access resources (i.e. a code structure requirement)?

Some thoughts on this:

• We definitely would not want to encourage Dart applications to pass file descriptors between isolates. This could be helped by not having a getFileDescriptor() method, which makes it hard to pass between isolates.
• Files and sockets created in an isolate clearly belong to that isolate, however there are other file descriptors that are created before the process starts, for example stdin/stdout/stderr and other pipes that might have been created. These aren't clearly owned by any isolate.

[aam]

an isolate could manipulate files owned by another isolate by forging the integer file descriptor

I think the intent would be keep file descriptors(perhaps FileDescriptionReference) opaque objects, rather than forgeable integers.
User would be able to only get these references in and out of domain sockets messages, would be able to get Sockets or RandomAccessFiles out of them.

[mraleph]

We could consider adding int to File/Socket functionality to dart:ffi rather than dart:io (or yet another library dart:io_ffi or something like that). This would, I think, address a concern that dart:io provides certain level of safety and isolation.

[robert-ancell]

A particular case might be a Dart process has been started with a pipe using file descriptor 5. This number might have been passed using an environment variable (e.g. PIPE_FD=5) or a command line argument (--pipe-fd 5). Perhaps we should get Dart to scan open file descriptors on startup and allow these to be accessed by number, but not allow any new file descriptors (e.g. created by Dart code) to be accessed.

Note that if any of these file descriptors were closed during the process lifetime they could be re-used. Not sure if there is a suitable method to guard against that.

[Piero512]

Hi, today I have realized about an interesting usecase for this feature.

While I was trying to create a Dart-only wrapper of dns_sd.h through FFI, I noticed that all of the glue C code i'm using to abstract creating a service broadcast could be written in dart if I could just use RawSocket stream of RawSocketEvent notifications and a minimal amount of FFI glue code to send the fds to Dart. Also, that saves me from having to poll those fds constantly.

I am also welcoming in that the Native API be expanded to be able to send a socket from C through a NativePort, but I imagine that one would be harder as there's an assumption that isolates can't share file descriptors.

This is my POC of bonjour ffi

[Piero512]

I would recommend against doing this. At the moment, it is by design that an isolate can't access file descriptors owned by other isolates. However with this change, an isolate could manipulate files owned by another isolate by forging the integer file descriptor. /cc @aam

About this issue. Well, on dart:io the integrated http server supports opening servers in the same listening address and the same port so it's not like it has never happened before that isolates share (at the very least) duplicated file descriptors.

It would also enable integration of other event loops into Dart (at least on Unix systems).

[simolus3]

I have started with a WIP CL for this: https://dart-review.googlesource.com/c/sdk/+/360863 It adds support for constructing sockets from file descriptors. For me, a use case I'd like to see supported in Dart is using file descriptors inherited from a parent process. With systemd for instance, we can have a privileged daemon listen on sockets and start an unprivileged server inheriting that socket on the first connection. This improves startup time and helps with sandboxing, but it's a pain to use in Dart if we can't go from FDs to ServerSockets.

Obviously there's the question of whether this should be in dart:io or dart:ffi (or perhaps even a dart:io-unsafe?). I've only implemented it on Linux for now, but I hope this sparks a discussion around suitable APIs for this - I'm happy to help with the implementation on all supported platforms.

[Piero512]

Hello

I reviewed your code and I don't know, maybe it is better to try to extend the ResourceHandle API instead? That one would allow you more flexibility rather than just ServerSockets.

Something like a ResourceHandle.fromFileDescriptor that would allow it to convert to most IO classes in dart:io, like server sockets, files, etc...

[Piero512]

@brianquinlan Hello, sorry for the direct ping, but I noticed that you're responsible for dart:io and I think this would be a very nice feature to have, and we want to know if you need anything else to be defined.

[kingwill101]

I got around this issue using dart ffi. Far from adequate but for now it works

see https://github.com/kingwill101/dart_native_socket