[New Pipeline]: UniControl for Unified Controllable Visual Generation

[canqin001]

Model/Pipeline/Scheduler description

Github: https://github.com/salesforce/UniControl
Paper: https://arxiv.org/abs/2305.11147

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

@canqin001 @yihaocs @shugerdou

[sayakpaul]

Thanks for opening this issue, @canqin001. The main components seem similar to ControlNet to me.

Would you maybe be willing to contribute this as a pipeline to Diffusers?

[canqin001]

@sayakpaul Yes. We'd like to contribute this as a pipeline to diffusers. Could you provide more details on how to incorporate it into diffusers? Thank you!

[sayakpaul]

Sure!

As a starting point, I think referring to the following documents will help you get some context:

• https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md
• https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md

And finally, the ControlNet PR would serve as a great reference for you:
#2407 (contributed by the awesome @takuma104).

[patrickvonplaten]

Exciting!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

I think adding this model is still very much relevant - they have recently adding some new very cool checkpoints: https://github.com/salesforce/UniControl

Anybody interested in picking this up?

[canqin001]

Thanks @patrickvonplaten. We really want to integrate UniControl to Diffuser. I just emailed @Abhinay1997 who will help us. Also my colleague @chiachihchen will participate. BTW, we have separated the inference code and released the HF space (https://huggingface.co/spaces/Robert001/UniControl-Demo).

[Abhinay1997]

@patrickvonplaten @sayakpaul there are minor differences in the ControlNet for UniControl. We'll need to add a few layers ( for the task aware hypernet and the MOE adapter.). The approach is to modify the existing ControlNet while maintaining backward compatibility correct ?

[sayakpaul]

Yes that is correct.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Abhinay1997]

Working on colab to port the weights and modify the controlnet. Expect lot of progress this week

[Abhinay1997]

We have gotten the pipeline running end to end. There is a bug somewhere that results in gaussian noise as output. Similar to an issue I had with initial port of TuneAVideo where an op was incorrect. Hoping thats the case here. Once this is fixed. Will update the PR :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Abhinay1997]

Working poc in PR. Adding tests, batching, etc

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Abhinay1997]

PR is still being worked on. Tests pending. ETA this week

[Sierkinhane]

Any updates? When will the unicontrol be available in diffusers?