[Community] Hypernetworks

[dblunk88]

Is your feature request related to a problem? Please describe.
Support for hypernetworks since they seem to yield great results. An example can be seen here:
https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/hypernetworks/hypernetwork.py

info: AUTOMATIC1111/stable-diffusion-webui#2670

Describe the solution you'd like
A diffusers hypernetwork implementation that is compatible with Automatic's implementation

Describe alternatives you've considered
Switching back and forth between different repos, but a one-stop shop is preferred.

[WASasquatch]

This is a really cool feature, and would be awesome in diffusers.

[patrickvonplaten]

Very much agree - we should definitely add this. In general there are multiple features that come out recently that require to modify the attentions for each layer - maybe we could support all of them in one go:

• https://github.com/cloneofsimo/paint-with-words-sd
• Hypernetworks
• am I missing one more @patil-suraj ?

[patrickvonplaten]

@patil-suraj maybe if you already have a good idea here on how to implement hypernetworks you could share here?

[aungkhant0911]

@patrickvonplaten unrelated: would be nice to have Google's Prompt2Prompt feature. https://prompt-to-prompt.github.io/

[patil-suraj]

This one also https://github.com/google/prompt-to-prompt

It would cool to have these, haven't looked deeply into it, will assign some time for it soon and reply here :)

[patrickvonplaten]

@aungkhant0911 - we're also always happy about community contributions :-)

[duongna21]

Happy to work on prompt-to-prompt. Maybe next week.

21/11 update: Sorry, I haven't made time for this yet. Will try in the next 1-2 weeks.
14/12 update: Sorry, still can't assign a time for this.

[dblunk88]

Any updates on this and/or any help needed?

[patrickvonplaten]

Hey @dblunk88,

In short we would really like to support prompt-to-prompt! So far we haven't found time to dive into it.

The best approach IMO would be to add the functionality of the paper: https://github.com/google/prompt-to-prompt/tree/main by providing a nice API to change the attention_weights and then creating a prompt-to-prompt example pipeline. Would this be interesting for you maybe?

We'll eventually look into this, but I can currently not say whether it will take 1 or 4 weeks - it depends also a bit on how much the community asks for it

[jinwonkim93]

Hi @patrickvonplaten . Are you planning to implement only prompt-to-prompt or general method for modifying layer? I think https://github.com/cloneofsimo/paint-with-words-sd is also very cool feature and does not require much of modification.

[patrickvonplaten]

Yes in a first step the plan was to look into prompt-to-prompt. I'm a bit behind on this. Hoping to get to it by next week

[dblunk88]

any updates on this?

[patil-suraj]

Not yet. We recently added a feature to be able to easily tweak cross attention (cf #1639), which would make it easy to implement. Let us know if anyone from the community is interested in this, happy to help :)

[patrickvonplaten]

We're currently adding LoRA: #1884 which should be very similar to how hypernetworks are done.

[ballenvironment]

Please add this, and hypernetwork training as well!