IPAdapter style transfer

[cubiq]

In SDXL by applying the weights only to the transformer index 6 it is possible to get a very powerful style transfer tool guided by IPAdapter. I don't know why I haven't thought about it before... I implemented it in ComfyUI and I guess it would be a cool feature for diffusers.

Similarly I believe it's possible to do the same with the composition only (haven't worked on this yet)

In the corner you can see the reference image

"cyberpunk sports car"

"peaceful landscape in japan"

"woman daydreaming at the window"

"cyberpunk alley"

[asomoza]

Really interesting, I did some test in diffusers and it works really good and sometimes not that much, still need to test it more when I have more time.

I found that if I send as a negative the same image (with a little noise) it uses only the colors which is also nice.

Because I'm lazy, I used your car image as an input:

"a cat"

normal	with negative image

For example for this image taken from Visual Style Prompting

prompt: " a tiger"

source	IP Adapter	Visual Style Prompting

It doesn't capture the style as good but probably I can play with something to get to that point.

Thank you for this @cubiq , I will use it and it's really cool.

As a side note, My knowledge of the unet is not that deep yet, why is that you say:

I don't know why I haven't thought about it before

Is it that obvious that the transformers index 6 is the one responsible of the style? In that case which one(s) would be responsible for the composition?

Now I probably want a granular control over all of them if that's the case, the same as we have with LoRAs now.

[cubiq]

the prompt is very important, if you add just a few tokens you can get as close or as far as you want from the reference

[asomoza]

oh ok, that's one difference between the visual style prompting, still is close enough and at least in diffusers we don't need to load another pipeline.

Also:

Now I probably want a granular control over all of them if that's the case, the same as we have with LoRAs now.

I couldn't resist so I just did it, so now I know how to control the composition:

prompt: "a dog"

source	generated

or how to make the dog more tigerish:

without style	with style

Really cool that we can make all this with just one ip adapter

[yiyixuxu]

super cool! can we add this as a community script for now? we can figure out if we can better support all these experimental IP-adpter features later when we have more of the examples

you can make a custom attention processor like this, basically just copy-paste IPAdapterAttnProcessor2_0 and make the modification needed

class StyleIPAdapterAttnProcessor2_0(torch.nn.Module):
    def __init__(self):
        ...
    def forward(self, ...):
        ...

and then you can do something like this to set the processor

from diffusers import DiffusionPipeline
from diffusers.models.attention_processor import IPAdapterAttnProcessor, IPAdapterAttnProcessor2_0
import torch

pipe = DiffusionPipeline.from_pretrained(...)
pipe.load_ip_adapter(...)

your_attn_processor = pipe.unet.attn_processors
for k, w in your_attn_processor.items():
    if isinstance(w, (IPAdapterAttnProcessor, IPAdapterAttnProcessor2_0)):
        your_attn_processor[k] = StyleIPAdapterAttnProcessor2_0()

pipe.unet.set_attn_processor(your_attn_processor)

[cubiq]

I've done the same for composition instead of style and you can mix the two. It's pretty fun

[darshats]

About this question - "Is it that obvious that the transformers index 6 is the one responsible of the style? In that case which one(s) would be responsible for the composition?"

I think this paper might explain? its the fifth transformer block index skipping the first block.

@yardenfren1996

[Podidiving]

👋

@asomoza can you share the code please? 🙏
I tried to reproduce it and disabled IP-adapter for all that are not *up_blocks.0.attentions.1* blocks but it didn't work as I was expecting

[asomoza]

Hi, this is old though, I did my own implementation for those generations. After that, what was added was InstantStyle which it should be the same. What result are you expecting?

[Podidiving]

I did my own implementation for those generations

Could you share this implementation? I'm interested in the implementation you did for this discussion. I did mine, but I'm not sure it's the correct one, since I can't achieve quite the same results

[asomoza]

That implementation was a really quick hack I did back then, I don't have that code anymore. But it's basicaly what I have in the app I was doing back then:

The first one is without noise, the second one has a little gaussian noise. You can look at the code but that app uses a very heavy modification of diffusers with a node system and on top PyQT, so don't know if it's going to help you. Here's the commit with that code.

[Podidiving]

ty ❤️