Add T2I-Adapter model and pipeline

[HimariO]

This PR implements the T2I-Adapter, related pipeline, and model sideloading mechanism discussed in #2390.

Model/Pipeline description

T2I-Adapter by @TencentARC is

... a simple and small (~70M parameters, ~300M storage space) network that can provide extra guidance to pre-trained text-to-image models while freezing the original large text-to-image models.
T2I-Adapter aligns internal knowledge in T2I models with external control signals. We can train various adapters according to different conditions, and achieve rich control and editing effects.

Usage Examples

import torch
from diffusers import StableDiffusionAdapterPipeline, Adapter

adapter = Adapter.from_pretrained("RzZ/sd-v1-4-adapter-color")
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "RzZ/sd-v1-4-adapter",
    adapter=adapter,
    torch_dtype=torch.float16,
).to("cuda")

out_images = pipe(prompts, images).images

import torch
from diffusers import StableDiffusionAdapterPipeline, Adapter, MultiAdapter

adapters = [
    Adapter.from_pretrained("RzZ/sd-v1-4-adapter-keypose"),
    Adapter.from_pretrained("RzZ/sd-v1-4-adapter-depth"),
]

pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "RzZ/sd-v1-4-adapter",
    adapter=adapter,
    torch_dtype=torch.float16,
).to("cuda")

out_images = pipe(
    ["A man waling in an office room with nice view"],
    [[cond_image_keypose, cond_image_depth]],
).images

TODO

• Implment Adapter model
• Implment StableDiffusionAdapterPipeline
• Create test cases for StableDiffusionAdapterPipeline
• Support for multi-adapter (adapter composition)
• Add support for the style and color adapter that the author just released yesterday
• Create document for StableDiffusionAdapterPipeline
• Refine model & pipeline doc strings
• Cleaning up development scripts & apply code style fixes

Discussion

• According to the author some of the adapter models is still work in progress(depth and style adapter), although those adapters seem to be working reasonably well. Not sure it is the right time to include them in this PR.
• Due to the vastly different architecture and decency of CLIPFeatureExtractor of the newly released style adapter, I think it's better to exclude it from this PR for now, so we don't keep expanding the scope of this PR
  @sayakpaul @wfng92

[patrickvonplaten]

@williamberman has probably the best XP to help here :-)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[HimariO]

@williamberman @patrickvonplaten @sayakpaul I think this PR is ready for review. 🙌

[patrickvonplaten]

Hey @HimariO,

Sorry that we're a bit slow here. I'll try to review today (or Monday the latest)

[williamberman]

hey @HimariO thank you so much for your work on this :)

My initial reaction is that the sideloading mechanism adds a bit too much "magic". Could we have a description on why we need to add a mixin as such? From my initial reading here my understanding was t2i follows the same pattern as controlnet which should just require passing the inputs to the unet's forward

[HimariO]

Hi @williamberman, although Adapter and ControlNet have a lot of similarities, there are some differences in their feature fusing schema. I've included two diagrams that illustrate these differences below. The rectangles and trapezoids that are paired with the same color represent the CrossAttnDownBlock2D or CrossAttnUpBlock2D:

The main difference will be:

1. Adapter needs to fuse its control signal with hidden states from the layer inside the CrossAttnDownBlock2D(one layer deeper into UNet).
2. The fused hidden state will be passed on to the following downsample blocks instead of upsample blocks through residual connections like ControlNet.

As we previously discussed in #2331, I too believe that more research like ControlNet will be happening in the near future. Therefore, providing a convenient way to experiment/integrate new ideas like controlling different modules or different fusing methods through a sideload approach could be valuable and more scalable compared to the straightforward method of passing the control signal in a top-down fashion.

Let me know if you have any questions or concerns!

[patrickvonplaten]

Hey @HimariO,

Thanks a lot for your design that is super useful! I'm with @williamberman here - I think we should try to simplify the code and make the design more similar to controlnet.

I've made a quick design proposal PR here: #2708 - would this design choice be ok for you?

We don't want to add a whole new design pattern for a new SD controlling algorithm. Instead we should try to make the fewest possible changes given what we already have - so we should try to strongly adapt this PR to the ControlNet one.

@HimariO could you maybe try to adapt your PR to conform a bit more to the design in #2708 . I think this could work nicely no?

[patrickvonplaten]

Feel free to copy anything you need from #2708 - it's just there as a design proposal for you

[HimariO]

@patrickvonplaten Thanks for the proposal, I will look into it later, probably after I finish reading the source code of the new CoAdapter @TencentARC release not too long ago.

[williamberman]

Hi @williamberman, although Adapter and ControlNet have a lot of similarities, there are some differences in their feature fusing schema. I've included two diagrams that illustrate these differences below. The rectangles and trapezoids that are paired with the same color represent the CrossAttnDownBlock2D or CrossAttnUpBlock2D:

The main difference will be:

1. Adapter needs to fuse its control signal with hidden states from the layer inside the CrossAttnDownBlock2D(one layer deeper into UNet).
2. The fused hidden state will be passed on to the following downsample blocks instead of upsample blocks through residual connections like ControlNet.

As we previously discussed in #2331, I too believe that more research like ControlNet will be happening in the near future. Therefore, providing a convenient way to experiment/integrate new ideas like controlling different modules or different fusing methods through a sideload approach could be valuable and more scalable compared to the straight_forward_ method of passing the control signal in a top-down fashion.

Let me know if you have any questions or concerns!

Nice! Thanks for the diagram super helpful :) Looks like can still be handled by directly passing the values through the forward methods. I think patrick's proposal makes a lot of sense.

[HimariO]

@patrickvonplaten 's is indeed very clean and keeps the change to a minimum. One use case #2708 may have some trouble handling will be using Adapter and ControlNet at the same time(ex: color-adapter + canny-controlnet). But I think we can just stick with #2708 for now(?).

[patrickvonplaten]

n and keeps the change to a minimum. One use case #2708 may have some trouble handling will be using Adapter and ControlNet at the same time(ex: color-adapter + canny-controlnet). But I think we can just stick with #2708 for now(?).

Ah I see, yes could we maybe try to stick to #2708 for now and see in a follow-up how we could adapt things?
Let me know if you need any help :-)

[HimariO]

@patrickvonplaten @williamberman this PR is updated with #2708.

[williamberman]

Would it be possible to use our existing ResnetBlock2D? It looks very similar

[HimariO]

I did think of that before, but the existence of ResnetBlock2D.norm1, ResnetBlock2D.norm2, and the time step embedding argument in the forward method prevented me from using it to implement the adapter,

[williamberman]

the time embedding is optional, feel free to make a small change to the block definition where time_embedding_norm can take the value "no_norm" which when set will set norm1 and norm2 to None and then they're skipped in the forward :)

[williamberman]

@HimariO any progress on replacing the resnet block? happy to help if needed

[HimariO]

I later spot some more differences:

1. Adapter sometimes uses kernel size 1 in the second conv2d module of the ResNet block, so we will need to add one(or two, since there are two conv2d in the ResNet block) more parameters to the ResnetBlock2D
2. Adapter use the "conv2d -> activation -> conv2d" pattern, and the ResnetBlock2D use "activation -> conv2d -> activation -> conv2d" pattern. Not sure there is any good way to bypass ResnetBlock2D's first activation function.

Not sure it is a good idea to make those changes to ResnetBlock2D or not, some help/suggestions here will be great :)

[patrickvonplaten]

I'll try to look into it tomorrow! I think it'd be totally fine though to slighly adapt the existing ResnetBlock

[williamberman]

TODO - upload weights to https://huggingface.co/TencentARC

[williamberman]

TODO update hub repos

[patrickvonplaten]

this line and the two following lines is different to what we have currently in the UNet and why it can be called "BottleNeck". Nevertheless the rest is exactly the same as far as I can see

[patrickvonplaten]

cc @williamberman it'd be great if you could try to unblock this PR by helping on the resnet refactor :-)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[williamberman]

sorry for the delay here @HimariO I've had a few too many things on my plate and haven't been able to get to the resnet refactor :)

[HimariO]

No worries @williamberman , take your time. Let me know if you need help or if you have an idea on how to approach the factor. In the meantime, I'll try to keep this pull request up to date with the main branch.

[adhikjoshi]

I think, having one pipeline for controlnet and T2I makes sense here, both are identical and can be used same way. With current rate of new AI projects coming, i think interoperability is what we should go for.

Or thing like "plugin pipelines" is what will be needed, where instead of updating diffusers, anyone can make pipeline which can be loaded as plugin, i think extending community pipelines the way its loaded, will help in it.

pipe = DiffusionPipeline.from_pretrained("stablediffusionapi/edge-of-realism").to("cuda")
pipe.load_custom_pipeline("t2i")

Right now, new diffusers version upgrade is needed for any new pipeline we want to use. Then, just side loading loading pipeline will do the trick. Also lots of overhead will be removed from diffusers team to support never ending list of projects in diffusers.

Then diffusers team can launch new pipelines instead of new diffusers. There can be official pipelines and community one's.

@williamberman @patrickvonplaten

[patrickvonplaten]

I think, having one pipeline for controlnet and T2I makes sense here, both are identical and can be used same way. With current rate of new AI projects coming, i think interoperability is what we should go for.

Or thing like "plugin pipelines" is what will be needed, where instead of updating diffusers, anyone can make pipeline which can be loaded as plugin, i think extending community pipelines the way its loaded, will help in it.

pipe = DiffusionPipeline.from_pretrained("stablediffusionapi/edge-of-realism").to("cuda")
pipe.load_custom_pipeline("t2i")

Right now, new diffusers version upgrade is needed for any new pipeline we want to use. Then, just side loading loading pipeline will do the trick. Also lots of overhead will be removed from diffusers team to support never ending list of projects in diffusers.

Then diffusers team can launch new pipelines instead of new diffusers. There can be official pipelines and community one's.

@williamberman @patrickvonplaten

We don't want to entangle different concepts here. ControlNet and T2I should be different pipelines. @williamberman if you're too busy this week, I can pick up this issue here

[bonlime]

@HimariO
The implementation above is not correct. you also need to update the output_states with new hidden_states, so the code should look like this to match original implementation. Without it the results do not match original version

if additional_residuals is not None:
    hidden_states += additional_residuals
    output_states = output_states[:-1] + (hidden_states,)

[williamberman]

hey @HimariO sorry for the repeated delays here, I was taking a look today at getting this running on my machine and I couldn't get the conversion script to work for the canny model. Could you add some examples on how to convert the original models?

specifically the conversion script will write out the state dict but the statedict isn't compatible with the model config in your repos

[sayakpaul]

Okay to close this in favor of #3932 no?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.