add hook to notify user services when inputs have been pulled

[GitHK]

Detecting when inputs are done pulling at runtime is hard fo the users for different reasons.

• input ports are pulled in parallel (so file changes arrive in a non ordered manner)
• if an input port is zipped, it gets unzipped in the input's port folder (this has the effect of files appearing one by one)

In order to simplify the user's life the following proposal is made.
Services can define two new additional "hooks":
- inputs_pulling_started an event which signals the start of inputs pulling to the users
- inputs_pulling_finished an event which signals the end of inputs pulling to the users

With these two events the user can implement their require logic in much more reliable way. An issue still remains with the simplicity of doing so.

in order to further aid the users the following additional proposal is made.

The oSPARC team creates a utility package to help the user. It will be used like this:

• inside the docker image the user will install some scripts inside /osparc-utils/inputs-monitor
• user will connect define the inputs_pulling_started "hook" on the service label and connect it to /osparc-utils/inputs-monitor/inputs_pulling_started.sh script
• user will connect define the inputs_pulling_finished "hook" on the service label and connect it to /osparc-utils/inputs-monitor/inputs_pulling_finisheds.sh script
• user has access to two new bash command /osparc-utils/inputs-monitor/get_version.sh which returns a string repressing the last version
• the user can continuously query call this endpoint to figure out if new inputs are available. When the inputs have finished downloading, /osparc-utils/inputs-monitor/get_version.sh will return a different version string.
• additional, for ease of use, above functionality can be packaged in a python module as well.

[elisabettai]

Thanks for this proposal. It sounds lot of work from your side and not easy at all from the users' perspective.

[GitHK]

Please note this is intended to be used when you expect inputs to change at runtime. Not before service start. For that there is a different proposal #6406

This is actually the "easy" solution for the user. Also form a backend perspective.

[sanderegg]

I also find this to be much less urgent than the other proposal. And also a service has multiple input ports, so you cannot have a global hook, it would need to be per port. Also, stupid question, could we not download the zip file into a temporary folder and move all its contents in "almost" one shot in the final folder?
Also the automation with the hook is nice to have as I think in the case where you change the inputs a user is probably doing manual work, so he/she knows when the inputs are in.

[GitHK]

I also find this to be much less urgent than the other proposal. And also a service has multiple input ports, so you cannot have a global hook, it would need to be per port.

per port hooks are tricky with our current system, as our hook system does not allow to push arguments to the container, it's more like a "signal". Currently only: "some inputs started downloading" and "some inputs finished downloading" can be provided.

Also, stupid question, could we not download the zip file into a temporary folder and move all its contents in "almost" one shot in the final folder?

File is already downloaded in a temporary folder, it is unzipped in the target folder though. Since /tmp and /some/custom/path/inputs are on different volumes. Now they should reside on the same file system. I'm not sure how docker handles move commands across volumes (might not be atomic).

Also the automation with the hook is nice to have as I think in the case where you change the inputs a user is probably doing manual work, so he/she knows when the inputs are in.

Yes users at this point know exactly where they have the files, so they just need to know when it is safe to open them. I also agree, that this case is not high priority

[mrnicegyu11]

I would like to revise your proposed API a bit before actually moving towards implementing this. I think there can be some improvements if we think this through deeply in a group.

I specifically think that first a visual indication in the e.g. osparc-gui around the iframe to show when inputs have changed or are changing (yello light or so or loading bar whatever) is much more useful for most users, all non-power-users, and power-users like werner have anyway already built functionality to handle this client-side.

[GitHK]

Team should decide if this should be continued or not. I have detailed my implementation already.
Also this issue #6276 and this pr #6286 might make this obsolete