[WIP] NewPipeline - UniControlDiffusionPipeline #4378
Conversation
|
@sayakpaul is there anyway to disable CI/CD tests while PR is still in draft ? |
|
Okay if they are failing now :-) |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
Working on this on colab. Will push the changes here soon :) |
|
Working poc code to be pushed by end of week. |
|
Can you write short snippet on how to use the pipeline? |
|
@adhikjoshi will add it as soon. Fixing some issues. |
…uity with Uni-ControlNet (Zhao et al)
* remove default value for task. * `check_inputs` validates task as well.
* Update `check_inputs` to validate task as a list. * Update `__call__` to merge conditioning samples from multiple tasks * Update all parameters that work with MultiControlNetModel also work with multi-task inputs.
|
Fixing multi task input. a.k.a MultiControlNet equivalent for UniControl. |
|
Adding tests and docs. Currently focused on getting the annotator working. |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
Apologies for the delay on this, pipeline works without issues. Need to add tests for each of the supported tasks. Was fully occupied this last few weeks. |
|
Waiting for PR to test dataset to be merged so I can fix the links. |
|
@sayakpaul @patrickvonplaten need the dataset PR to be merged for the final part: https://huggingface.co/datasets/hf-internal-testing/diffusers-images/discussions/32 |
|
Very nice progress here - I'm wondering though whether we should maybe move this PR to the research_projects folder? It's been on-going for long sadly so I'm not sure how much the community would benefit from an integration into core at this point. cc @yiyixuxu |
|
Hey Patrick, I'd still think there's value in adding it to core as there's no moe style control net in core right now. Is there something in diffusers that can handle multiple control net conditionings with the same weights ? Trying to understand as I believe it's still relevant |
|
Hi Patrick, thank you for replying to us. Sorry for the delay in the PR dev, and I totally understand your concern. The UniControl (https://github.com/salesforce/UniControl) has received over 550+ stars and was presented in NeurIPS 23 last month, attracting volumes of researchers to its poster. This is the first paper to unify different conditions in one model, which has a unique value in the research for its pioneering exploration. I believe these unified models will receive increasing popularity in the near future due to their compactness and convenience. Incorporating it into the Diffusers would be greatly beneficial for the community. @patrickvonplaten @Abhinay1997 |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
PR that integrates UniControl with diffusers. Co-authored by @canqin001
Discussed under #3788
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
@patrickvonplaten and @sayakpaul