Support legacy dream.py API

[CapableWeb]

Is your feature request related to a problem? Please describe.

Currently building my own UI using the HTTP API provided by dream.py. It's being deprecated now in favor of a socket-based API, which makes it slightly harder to use for my purposes.

Describe the solution you'd like

Move the old dream.py HTTP API into a legacy directory I can help maintain. It'll focus on providing backwards compatibility without any new features unless it makes it still backwards compatible. Discussed as a possible solution with @psychedelicious

Describe alternatives you've considered

• Forking invoke-ai/invokeAI and maintaining it myself
• Use a HTTP API from a different SD fork

Additional context

Initially raised the conversation via Discord (starting here: https://discord.com/channels/1020123559063990373/1020123559831539744/1025351359446388787)

Message Log:

@CapableWeb — Today at 12:20 PM

is there any particular reason the dream.py API changed from being a simple HTTP API to a WebSocket API? Usually you'd use WS for high-traffic endpoints, or when you really need stateful communication. But most people use dream.py over a local connection and the amount of data sent/received is really tiny, is there some other use case that justifies the added complexity that I missed?
@psychedelicious — Today at 12:21 PM

@CapableWeb It allows for very simple bidirectional communication
@CapableWeb — Today at 12:22 PM

guess we have different definitions of simple 😄 Old API was pretty simply, do a POST request, a streaming response body is returned. Now you need to manage the state of a socket and everything that comes with that
@psychedelicious — Today at 12:22 PM

Simple as in implementation

Hehe yeah it's true. Are you building on top of the API?
@CapableWeb — Today at 12:23 PM

ok, so basically for developer ergonomics reason?

yeah, the dream.py HTTP API. Just checked out latest development branch in order to add the --device parameter but noticed that the entire API now changed so... Rethinking next steps
@psychedelicious — Today at 12:25 PM

Soooo I wrote the server in its current state, mainly as a temporary solution. It just needed to be good enough to support a modern UI. Socketio was a very straightforward choice, and the events are all closely linked to the UI - it's not intended to provide an API to other clients.

We have a much better server and API in the works which will replace the current socketio one. That will be worth building off of.
@CapableWeb — Today at 12:26 PM

hm, yeah, not sure I can just hold off on updating until anything new is in place. Definitely hear what you're saying though. I guess it'd be a far ask to ask the old API to remain along-side the new socket-based one?
@psychedelicious — Today at 12:27 PM

Personally I think that is reasonable, it's still there, just no easy way to run it.
@CapableWeb — Today at 12:28 PM

ah I see, it's no longer in dream.py?
@psychedelicious — Today at 12:29 PM

Correct. One option could be to put the old dream.py in a legacy folder and just leave it as-is with no promises for future support.
@CapableWeb — Today at 12:29 PM

yeah, I could live with that, can even commit to maintaining that if needed in the future
@psychedelicious — Today at 12:30 PM

It would be nice to support both protocols in the current server but the whole server is basically re-written with a bunch of new/improved features and not feasible to retain the old API.

Can you please raise an issue on github?

Definitely using the current socketio API is a no-go - but once we have the more permanent server in place, it will be far more capable and worth migrating to, I think.
@CapableWeb — Today at 12:32 PM

yeah, I saw the new jobs thing and all that. Not super useful for my own use case but I can definitely see how it's useful for invoke-ai's own UI
@CapableWeb — Today at 12:32 PM

will do. Thanks a lot for the reasoning 🙂
@psychedelicious — Today at 12:33 PM

No worries, while we have our own UI project we also want to support other projects where possible

[hipsterusername]

As mentioned in discord - I do think it's worth establishing expectations that new features may not receive support in the old api unless we have an "Old API maintainer". And, if we do have that (perhaps CapableWeb?), we should formalize it as REST API support.

[tildebyte]

Could also have a separate repo[1] in this org for "Legacy API" or maybe even "Simplified API" or some such.

[1] Which would provide an opportunity to delete all of the bits which aren't directly related to the HTTP API, making it easier to maintain

[CapableWeb]

As mentioned in discord - I do think it's worth establishing expectations that new features may not receive support in the old api unless we have an "Old API maintainer". And, if we do have that (perhaps CapableWeb?), we should formalize it as REST API support.

Yeah, that makes sense I think. Having legacy/dream.py would also signal that it is "legacy" and might not receive new functionality. Model changes/optimizations would happen on the layer beneath, but the API interface would still remain the same.

I'd be fine with maintaining this legacy interface, as I'm depending on it myself.

Could also have a separate repo[1] in this org for "Legacy API" or maybe even "Simplified API" or some such.

We could, but then I might just fork the repository into my own org and maintain it that way. It's simpler for maintenance to keep it in the same repository, won't add burden to anyone else if I'm the one maintaining it and as mentioned above, changes related to anything else but the API interface would benefit the legacy interface without further changes too, which is beneficial for the "legacy API" users.

[psychedelicious]

I think there are a number of projects which depend on HTTP API, so it'd be great for all if you were willing to keep it maintained and if we keep it in this repo or at least this org. As time goes on and we refine the backend with which the API interfaces, it will hopefully become easier and easier to maintain as well.

[tildebyte]

My point about creating a new repo under this org really works best if what's being taken from here and put in the new repo is significantly decoupled from this repo. If the dependencies are pretty tight, then yeah, you're just buying trouble.

[lstein]

@CapableWeb Are you volunteering to be the “legacy maintainer?” I’m happy to have to take that on.

[CapableWeb]

@psychedelicious I think there are a number of projects which depend on HTTP API, so it'd be great for all if you were willing to keep it maintained

@lstein Are you volunteering to be the “legacy maintainer?” I’m happy to have to take that on.

Yes, I'm volunteering for this, granted the file can be kept in this repository in order to make incompatibilities easier to spot and lessen the complexity compared to having the "legacy API" being in a separate repository.

My point about creating a new repo under this org really works best if what's being taken from here and put in the new repo is significantly decoupled from this repo. If the dependencies are pretty tight, then yeah, you're just buying trouble.

Yeah, agreed. Maybe in the future it won't, but for now the only realistic option would be to keep it in this repository.

So if everyone is fine with it, I'll base my work from 803a51d5adca7e6e28491fc414fd3937bee7cb79 which is the last commit I personally locked my application too, and make sure the API is compatible with that in the future, will send PR to development when that's in place.

[lstein]

That’s wonderful. Would you be willing to be added to the team as a developer or triager? The latter can assign bug reports and the like but doesn’t have commit privileges. Whatever you’re comfortable with.

[lstein]

Just following up. Do you want to be added to the invoke-ai organization?

[CapableWeb]

@lstein sorry, sure thing! Being added as a developer makes sense so I could apply patches from PRs to the legacy API interface.

[damian0815]

i would like to push back on this being a "legacy" feature, as i have been in multiple situations now where i've got the WebUI running on a remote service (runpod.io) and want to be able to script a batch of tests with different parameters without logging in the runpod.io box, manually killing the web ui, and starting dream.py by hand.

i have commented on this elsewhere, and i'm willing to be the maintainer of such an API - but only a minimal version, eg while dream.py --web --port 9090 is running i can curl https://<remoteaddress>:9090/invoke?cmd="mountain+man"+-s10+-S12345+-W384 etc..., if that would be suitable for your purposes @CapableWeb

[CapableWeb]

@damian0815 the name legacy is simply to mark that it's different than from what a normal user would use, as I'm basing a service/software package I'm authoring on the dream.py API from version 803a51d5adca7e6e28491fc414fd3937bee7cb79. So the idea is to maintain exact API compatibility until "forever", with no incompatible changes at all moving forward from there.

[damian0815]

ahha gotcha

[CapableWeb]

For whoever is interested, opened up a PR to recover the old dream.py HTTP API (which basically accepted simple HTTP POST requests): #1070

@damian0815 you might be interested to check that out, give it a try and see if it works for your purposes too. It works for me, but very possible I've missed something you're relying on.

[Kyle0654]

I've spent several weeks now working on a new API/workflow that should eventually be feature-compatible with the old API (though not functionally identical, especially regarding streamed responses): #1047. It does support RESTful usage though, produces an OpenAPI spec, etc.

[CapableWeb]

@Kyle0654 sounds great for a lot of use cases, but seems the API interface would be fundamentally different, at least judging by the example in static/dream_web/test.html

The "legacy" API (old dream.py) was very simple and stateless. Do a POST request, receive streamed response about progress and finally a result event is returned and it's done.

In contrast, the new API seems to require 1) create a session 2) subscribe to the session via WebSockets to receive the events and 3) call "invoke" to actually start the response. This adds another protocol on top of already using HTTP + it is now stateful.

I guess in the future the "legacy" API could do those things but still expose the good'ol POST request < get response interface, handling the stateness for the API users.

(Bit OT, but not sure I share your definition of REST as being "stateless" would be a big part of it [in my opinion], and I didn't see a way of using the new API in a stateless manner, it quite literally requires a session to be kept in mind for future requests)

[Kyle0654]

I had worked really hard to rebuild the old API in Flask a while ago, but the streaming response blocked execution, which made request queueing not work. It's probably solvable, but was also tightly coupled to other parts of the code, and was tough to pull apart/maintain. A lot of the code accidentally made it in (this PR deletes it): https://github.com/invoke-ai/InvokeAI/blob/main/server/application.py

It's probably possible to build a streaming response API in FastAPI for compatibility. I think the big challenge is the async/sync boundary, which I'm not as familiar with yet in Python. This PR is draft to get feedback like this, and solicit contribution =). I'd also worry about long-running requests timing out in different hosting scenarios. =/

I also regret using the word "session", but don't have a better name in mind x.x. It's really just a resource representing the node graph and execution history. There shouldn't be any implication that it's a stateful application session on a server - the client isn't remembered at all for the HTTP interface (and the socket.io interface is just for signaling changes - you could poll to accomplish the same thing). I just couldn't come up with a better name for "graph+history".

The "create session from graph" API has a query parameter to "invoke now". It might be possible to add another parameter for "wait for results" or "stream progress and results", though that feels like it'd become a large API to maintain.

[xrd]

I'm searching for documentation on how to use an API with InvokeAI. Is there a PR which summarizes the current state of the "new" API, namely what is described here:

In contrast, the new API seems to require 1) create a session 2) subscribe to the session via WebSockets to receive the events and 3) call "invoke" to actually start the response. This adds another protocol on top of already using HTTP + it is now stateful.

I see some issues listed here, but not a working PR which indicates the work that is being done to support the legacy dream.py code, or the code which is for the "new" API. I would benefit from having those added to this issue report if that's possible.

edit: I see the new work here (though the PR does not yet have, IMHO, documentation on how to use it, which is what @lstein asked for): #1070. But, I'm unclear how to investigate the "new" API using websockets; does this require reading through the code?

[psychedelicious]

@xrd ,

We have 3 APIs - let's call them "legacy", "current" and "nodes".

"Legacy" is HTTP only. There was a volunteer (not a core contributor IIRC) who was going to maintain this but I'm not sure what its state is. It is not documented AFAIK.

"Current" is what the web UI uses now. It is mostly socket.io, with some HTTP for transferring images and potentially larger files. It is also not documented because it was not built to be consumed by anything other than the web UI, and it has been changed constantly since it was written.

"Nodes" is a future API, mostly built, but to which we have not yet migrated. It is intended to be consumed by external services and is the one you described where processing is described by a session node-link graph. You can find more information about it here: #1047

So at this moment, we do not provide an API that is really suitable for consumption by an external service, certainly not one that is supported.

However, we definitely want to support other services by providing a robust web API. We will be migrating the UI over fairly soon - after the upcoming v2.2 release is the plan - and the nodes API will be released alongside the mgirated web UI.

Hope that helps and sorry we do not have a good solution at the moment.

[xrd]

@psychedelicious This really helps a lot. Thank you.