t2i pipeline #3932

williamberman · 2023-07-03T17:10:50Z

rebased and refactored from #2555

tl;dr

I refactored the main adapter class into the separate "light adapter" and "full adapters" from the original implementation and wrote separate block types for each. The result has little to no conditionals and avoids the large resnet class.
I removed all un-used arguments that could just be left as defaults.
I fixed up the conversion script so it directly writes out the full diffusers style checkpoint - as was it was writing out torch weights and required some additional leg work to get the right config for the main class

TODO

Upload new checkpoints to the diffusers org
Go over tests
Fix docs

Converting

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_canny_sd14v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_canny_sd14v1.converted \
    --in_channels=1

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_canny_sd15v2.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_canny_sd15v2.converted \
    --in_channels=1

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_depth_sd14v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_depth_sd14v1.converted \
    --in_channels=3

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_depth_sd15v2.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_depth_sd15v2.converted \
    --in_channels=3

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_keypose_sd14v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_keypose_sd14v1.converted \
    --in_channels=3

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_openpose_sd14v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_openpose_sd14v1.converted \
    --in_channels=3

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_seg_sd14v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_seg_sd14v1.converted \
    --in_channels=3

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_sketch_sd14v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_sketch_sd14v1.converted \
    --in_channels=1

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_sketch_sd15v2.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_sketch_sd15v2.converted \
    --in_channels=1

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_zoedepth_sd15v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_zoedepth_sd15v1.converted \
    --in_channels=3

python scripts/convert_original_t2i_adapter.py \
    --checkpoint_path ../T2I-Adapter-src/models/t2iadapter_color_sd14v1.pth \
    --output_path ../T2I-Adapter-converted/t2iadapter_color_sd14v1.converted \
    --is_adapter_light

uploaded models

williamberman · 2023-07-04T03:08:25Z

@HimariO would love your review on this if possible, I tried requesting as a reviewer formally but it wouldn't let me

patrickvonplaten · 2023-07-04T17:28:46Z

@yiyixuxu @sayakpaul could you try to give this PR a review?

docs/source/en/api/pipelines/overview.mdx

docs/source/en/api/pipelines/stable_diffusion/adapter.mdx

docs/source/en/using-diffusers/controlling_generation.mdx

src/diffusers/models/adapter.py

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py

sayakpaul

PR looks great to me! Great job!

patrickvonplaten · 2023-07-12T19:47:17Z

Happy to merge and once the authors get back to us we can transfer the checkpoints

sayakpaul · 2023-07-14T08:10:18Z

Happy to merge and once the authors get back to us we can transfer the checkpoints

Since we have approval we can proceed. But I would first recommend adding model cards to all the pipelines and then moving forward. Cc: @williamberman

yiyixuxu

cool!

yiyixuxu · 2023-07-15T02:02:42Z

src/diffusers/models/adapter.py

+        for x, w, adapter in zip(x_list, adapter_weights, self.adapters):
+            features = adapter(x)
+            if accume_state is None:
+                accume_state = features


weights don't apply to the first adapter output?

src/diffusers/models/unet_2d_blocks.py

src/diffusers/models/unet_2d_condition.py

amrakm · 2023-07-15T15:05:28Z

Is this example code still the latest?

from PIL import Image
from diffusers.utils import load_image
import torch
from diffusers import StableDiffusionAdapterPipeline, T2IAdapter

# Load the image
image = load_image("https://huggingface.co/RzZ/sd-v1-4-adapter-color/resolve/main/color_ref.png")

# Create the color palette
color_palette = image.resize((8, 8))
color_palette = color_palette.resize((512, 512), resample=Image.Resampling.NEAREST)

# Initialize the T2IAdapter
adapter = T2IAdapter.from_pretrained("RzZ/sd-v1-4-adapter-color")

# Initialize the StableDiffusionAdapterPipeline
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    adapter=adapter,
    torch_dtype=torch.float16,
)

# Move the pipeline to the GPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
pipe.to(device)

# Generate the output image
out_image = pipe(
    "At night, glowing cubes in front of the beach",
    image=color_palette,
).images[0]

I'm getting this error when I run it:

Cell In[3], line 14
     11 color_palette = color_palette.resize((512, 512), resample=Image.Resampling.NEAREST)
     13 # Initialize the T2IAdapter
---> 14 adapter = T2IAdapter.from_pretrained("RzZ/sd-v1-4-adapter-color")
     16 # Initialize the StableDiffusionAdapterPipeline
     17 pipe = StableDiffusionAdapterPipeline.from_pretrained(
     18     "CompVis/stable-diffusion-v1-4",
     19     adapter=adapter,
     20     torch_dtype=torch.float16,
     21 )

File ~/anaconda3/envs/diff-t2i/lib/python3.10/site-packages/diffusers/models/modeling_utils.py:604, in ModelMixin.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    602 missing_keys = set(model.state_dict().keys()) - set(state_dict.keys())
    603 if len(missing_keys) > 0:
--> 604     raise ValueError(
    605         f"Cannot load {cls} from {pretrained_model_name_or_path} because the following keys are"
    606         f" missing: \n {', '.join(missing_keys)}. \n Please make sure to pass"
    607         " `low_cpu_mem_usage=False` and `device_map=None` if you want to randomly initialize"
    608         " those weights or else make sure your checkpoint file is correct."
    609     )
    610 unexpected_keys = []
    612 empty_state_dict = model.state_dict()

ValueError: Cannot load  from RzZ/sd-v1-4-adapter-color because the following keys are missing: 
 adapter.conv_in.weight, adapter.conv_in.bias, adapter.body.1.resnets.2.block1.bias, adapter.body.2.resnets.3.block1.weight, adapter.body.1.resnets.3.block1.bias, adapter.body.2.resnets.3.block2.weight, adapter.body.1.resnets.0.block1.weight, adapter.body.1.resnets.0.block2.weight, adapter.body.2.in_conv.weight, adapter.body.2.resnets.2.block2.bias, adapter.body.3.resnets.0.block2.bias, adapter.body.3.resnets.3.block1.bias, adapter.body.1.resnets.3.block1.weight, adapter.body.1.resnets.3.block2.weight, adapter.body.3.resnets.0.block2.weight, adapter.body.1.resnets.3.block2.bias, adapter.body.3.resnets.2.block1.weight, adapter.body.1.resnets.2.block1.weight, adapter.body.2.resnets.2.block1.bias, adapter.body.2.resnets.0.block2.weight, adapter.body.1.resnets.1.block2.bias, adapter.body.0.resnets.3.block1.weight, adapter.body.0.resnets.3.block2.bias, adapter.body.2.resnets.0.block2.bias, adapter.body.1.resnets.1.block1.weight, adapter.body.2.in_conv.bias, adapter.body.2.resnets.2.block2.weight, adapter.body.1.resnets.1.block2.weight, adapter.body.3.resnets.2.block1.bias, adapter.body.1.in_conv.weight, adapter.body.3.resnets.3.block1.weight, adapter.body.3.resnets.3.block2.weight, adapter.body.2.resnets.1.block1.bias, adapter.body.2.resnets.3.block2.bias, adapter.body.1.resnets.2.block2.weight, adapter.body.3.resnets.0.block1.weight, adapter.body.3.resnets.1.block1.weight, adapter.body.2.resnets.1.block1.weight, adapter.body.1.resnets.0.block2.bias, adapter.body.0.resnets.2.block1.bias, adapter.body.0.resnets.2.block1.weight, adapter.body.2.resnets.0.block1.weight, adapter.body.3.resnets.0.block1.bias, adapter.body.3.resnets.1.block2.bias, adapter.body.0.resnets.2.block2.weight, adapter.body.1.resnets.0.block1.bias, adapter.body.1.resnets.2.block2.bias, adapter.body.0.resnets.1.block1.bias, adapter.body.0.resnets.0.block2.bias, adapter.body.2.resnets.2.block1.weight, adapter.body.0.resnets.3.block2.weight, adapter.body.0.resnets.1.block2.bias, adapter.body.3.resnets.1.block1.bias, adapter.body.3.resnets.2.block2.bias, adapter.body.2.resnets.1.block2.bias, adapter.body.0.resnets.2.block2.bias, adapter.body.3.resnets.3.block2.bias, adapter.body.0.resnets.0.block2.weight, adapter.body.2.resnets.1.block2.weight, adapter.body.0.resnets.3.block1.bias, adapter.body.0.resnets.0.block1.weight, adapter.body.3.resnets.1.block2.weight, adapter.body.2.resnets.0.block1.bias, adapter.body.1.resnets.1.block1.bias, adapter.body.3.resnets.2.block2.weight, adapter.body.0.resnets.1.block1.weight, adapter.body.0.resnets.0.block1.bias, adapter.body.1.in_conv.bias, adapter.body.0.resnets.1.block2.weight, adapter.body.2.resnets.3.block1.bias. 
 Please make sure to pass `low_cpu_mem_usage=False` and `device_map=None` if you want to randomly initialize those weights or else make sure your checkpoint file is correct.```

patrickvonplaten · 2023-07-17T08:37:52Z

cc @williamberman can we make sure to add @HimariO here as a co-author as this PR superseeds: #2555

Load adapter module with from_pretrained Prototyping generalized adapter framework Writeup doc string for sideload framework(WIP) + some minor update on implementation Update adapter models Remove old adapter optional args in UNet Add StableDiffusionAdapterPipeline unit test Handle cpu offload in StableDiffusionAdapterPipeline Auto correct coding style Update model repo name to "RzZ/sd-v1-4-adapter-pipeline" Refactor MultiAdapter to better compatible with config system Export MultiAdapter Create pipeline document template from controlnet Create dummy objects Supproting new AdapterLight model Fix StableDiffusionAdapterPipeline common pipeline test [WIP] Update adapter pipeline document Handle num_inference_steps in StableDiffusionAdapterPipeline Update definition of Adapter "channels_in" Update documents Apply code style Fix doc typo and merge error Update doc string and example Quality of life improvement Remove redundant code and file from prototyping Remove unused pageage Remove comments Fix title Fix typo Add conditioning scale arg Bring back old implmentation Offload sideload Add supply info on document Update src/diffusers/models/adapter.py Co-authored-by: Will Berman <[email protected]> Update MultiAdapter constructor Swap out custom checkpoint and update pipeline constructor Update docment Apply suggestions from code review Co-authored-by: Will Berman <[email protected]> Correcting style Following single-file policy Update auto size in image preprocess func Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py Co-authored-by: Will Berman <[email protected]> fix copies Update adapter pipeline behavior Add adapter_conditioning_scale doc string Add the missing doc string Apply suggestions from code review Co-authored-by: Patrick von Platen <[email protected]> Fix few bugs from suggestion Handle L-mode PIL image as control image Rename to differentiate adapter resblock Update src/diffusers/models/adapter.py Co-authored-by: Sayak Paul <[email protected]> Fix typo Update adapter parameter name Update test case and code style Fix copies Fix typo Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py Co-authored-by: Will Berman <[email protected]> Update Adapter class name Add checkpoint converting script Fix style Fix-copies Remove dev script Apply suggestions from code review Co-authored-by: Patrick von Platen <[email protected]> Updates for parameter rename Fix convert_adapter remove main fix diff more refactoring more more small fixes refactor tests more slow tests more tests Update docs/source/en/api/pipelines/overview.mdx Co-authored-by: Sayak Paul <[email protected]> add community contributor to docs Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> fix remove from_adapters license paper link docs more url fixes more docs fix fixes fix fix

williamberman · 2023-07-17T18:35:58Z

Is this example code still the latest?

from PIL import Image
from diffusers.utils import load_image
import torch
from diffusers import StableDiffusionAdapterPipeline, T2IAdapter

# Load the image
image = load_image("https://huggingface.co/RzZ/sd-v1-4-adapter-color/resolve/main/color_ref.png")

# Create the color palette
color_palette = image.resize((8, 8))
color_palette = color_palette.resize((512, 512), resample=Image.Resampling.NEAREST)

# Initialize the T2IAdapter
adapter = T2IAdapter.from_pretrained("RzZ/sd-v1-4-adapter-color")

# Initialize the StableDiffusionAdapterPipeline
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    adapter=adapter,
    torch_dtype=torch.float16,
)

# Move the pipeline to the GPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
pipe.to(device)

# Generate the output image
out_image = pipe(
    "At night, glowing cubes in front of the beach",
    image=color_palette,
).images[0]

I'm getting this error when I run it:

Cell In[3], line 14
     11 color_palette = color_palette.resize((512, 512), resample=Image.Resampling.NEAREST)
     13 # Initialize the T2IAdapter
---> 14 adapter = T2IAdapter.from_pretrained("RzZ/sd-v1-4-adapter-color")
     16 # Initialize the StableDiffusionAdapterPipeline
     17 pipe = StableDiffusionAdapterPipeline.from_pretrained(
     18     "CompVis/stable-diffusion-v1-4",
     19     adapter=adapter,
     20     torch_dtype=torch.float16,
     21 )

File ~/anaconda3/envs/diff-t2i/lib/python3.10/site-packages/diffusers/models/modeling_utils.py:604, in ModelMixin.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    602 missing_keys = set(model.state_dict().keys()) - set(state_dict.keys())
    603 if len(missing_keys) > 0:
--> 604     raise ValueError(
    605         f"Cannot load {cls} from {pretrained_model_name_or_path} because the following keys are"
    606         f" missing: \n {', '.join(missing_keys)}. \n Please make sure to pass"
    607         " `low_cpu_mem_usage=False` and `device_map=None` if you want to randomly initialize"
    608         " those weights or else make sure your checkpoint file is correct."
    609     )
    610 unexpected_keys = []
    612 empty_state_dict = model.state_dict()

ValueError: Cannot load  from RzZ/sd-v1-4-adapter-color because the following keys are missing: 
 adapter.conv_in.weight, adapter.conv_in.bias, adapter.body.1.resnets.2.block1.bias, adapter.body.2.resnets.3.block1.weight, adapter.body.1.resnets.3.block1.bias, adapter.body.2.resnets.3.block2.weight, adapter.body.1.resnets.0.block1.weight, adapter.body.1.resnets.0.block2.weight, adapter.body.2.in_conv.weight, adapter.body.2.resnets.2.block2.bias, adapter.body.3.resnets.0.block2.bias, adapter.body.3.resnets.3.block1.bias, adapter.body.1.resnets.3.block1.weight, adapter.body.1.resnets.3.block2.weight, adapter.body.3.resnets.0.block2.weight, adapter.body.1.resnets.3.block2.bias, adapter.body.3.resnets.2.block1.weight, adapter.body.1.resnets.2.block1.weight, adapter.body.2.resnets.2.block1.bias, adapter.body.2.resnets.0.block2.weight, adapter.body.1.resnets.1.block2.bias, adapter.body.0.resnets.3.block1.weight, adapter.body.0.resnets.3.block2.bias, adapter.body.2.resnets.0.block2.bias, adapter.body.1.resnets.1.block1.weight, adapter.body.2.in_conv.bias, adapter.body.2.resnets.2.block2.weight, adapter.body.1.resnets.1.block2.weight, adapter.body.3.resnets.2.block1.bias, adapter.body.1.in_conv.weight, adapter.body.3.resnets.3.block1.weight, adapter.body.3.resnets.3.block2.weight, adapter.body.2.resnets.1.block1.bias, adapter.body.2.resnets.3.block2.bias, adapter.body.1.resnets.2.block2.weight, adapter.body.3.resnets.0.block1.weight, adapter.body.3.resnets.1.block1.weight, adapter.body.2.resnets.1.block1.weight, adapter.body.1.resnets.0.block2.bias, adapter.body.0.resnets.2.block1.bias, adapter.body.0.resnets.2.block1.weight, adapter.body.2.resnets.0.block1.weight, adapter.body.3.resnets.0.block1.bias, adapter.body.3.resnets.1.block2.bias, adapter.body.0.resnets.2.block2.weight, adapter.body.1.resnets.0.block1.bias, adapter.body.1.resnets.2.block2.bias, adapter.body.0.resnets.1.block1.bias, adapter.body.0.resnets.0.block2.bias, adapter.body.2.resnets.2.block1.weight, adapter.body.0.resnets.3.block2.weight, adapter.body.0.resnets.1.block2.bias, adapter.body.3.resnets.1.block1.bias, adapter.body.3.resnets.2.block2.bias, adapter.body.2.resnets.1.block2.bias, adapter.body.0.resnets.2.block2.bias, adapter.body.3.resnets.3.block2.bias, adapter.body.0.resnets.0.block2.weight, adapter.body.2.resnets.1.block2.weight, adapter.body.0.resnets.3.block1.bias, adapter.body.0.resnets.0.block1.weight, adapter.body.3.resnets.1.block2.weight, adapter.body.2.resnets.0.block1.bias, adapter.body.1.resnets.1.block1.bias, adapter.body.3.resnets.2.block2.weight, adapter.body.0.resnets.1.block1.weight, adapter.body.0.resnets.0.block1.bias, adapter.body.1.in_conv.bias, adapter.body.0.resnets.1.block2.weight, adapter.body.2.resnets.3.block1.bias. 
 Please make sure to pass `low_cpu_mem_usage=False` and `device_map=None` if you want to randomly initialize those weights or else make sure your checkpoint file is correct.```

Ah thank you for catching, updated snippet :)

williamberman · 2023-07-17T18:38:08Z

cc @williamberman can we make sure to add @HimariO here as a co-author as this PR superseeds: #2555

Yep! Already a co-author in the squashed commit history and thanked in docs :)

williamberman · 2023-07-17T19:55:40Z

test failure unrelated

* Quick implementation of t2i-adapter Load adapter module with from_pretrained Prototyping generalized adapter framework Writeup doc string for sideload framework(WIP) + some minor update on implementation Update adapter models Remove old adapter optional args in UNet Add StableDiffusionAdapterPipeline unit test Handle cpu offload in StableDiffusionAdapterPipeline Auto correct coding style Update model repo name to "RzZ/sd-v1-4-adapter-pipeline" Refactor MultiAdapter to better compatible with config system Export MultiAdapter Create pipeline document template from controlnet Create dummy objects Supproting new AdapterLight model Fix StableDiffusionAdapterPipeline common pipeline test [WIP] Update adapter pipeline document Handle num_inference_steps in StableDiffusionAdapterPipeline Update definition of Adapter "channels_in" Update documents Apply code style Fix doc typo and merge error Update doc string and example Quality of life improvement Remove redundant code and file from prototyping Remove unused pageage Remove comments Fix title Fix typo Add conditioning scale arg Bring back old implmentation Offload sideload Add supply info on document Update src/diffusers/models/adapter.py Co-authored-by: Will Berman <[email protected]> Update MultiAdapter constructor Swap out custom checkpoint and update pipeline constructor Update docment Apply suggestions from code review Co-authored-by: Will Berman <[email protected]> Correcting style Following single-file policy Update auto size in image preprocess func Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py Co-authored-by: Will Berman <[email protected]> fix copies Update adapter pipeline behavior Add adapter_conditioning_scale doc string Add the missing doc string Apply suggestions from code review Co-authored-by: Patrick von Platen <[email protected]> Fix few bugs from suggestion Handle L-mode PIL image as control image Rename to differentiate adapter resblock Update src/diffusers/models/adapter.py Co-authored-by: Sayak Paul <[email protected]> Fix typo Update adapter parameter name Update test case and code style Fix copies Fix typo Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py Co-authored-by: Will Berman <[email protected]> Update Adapter class name Add checkpoint converting script Fix style Fix-copies Remove dev script Apply suggestions from code review Co-authored-by: Patrick von Platen <[email protected]> Updates for parameter rename Fix convert_adapter remove main fix diff more refactoring more more small fixes refactor tests more slow tests more tests Update docs/source/en/api/pipelines/overview.mdx Co-authored-by: Sayak Paul <[email protected]> add community contributor to docs Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx Co-authored-by: Sayak Paul <[email protected]> fix remove from_adapters license paper link docs more url fixes more docs fix fixes fix fix * fix sample inplace add * additional_kwargs -> additional_residuals * move t2i adapter pipeline to own module * preprocess -> _preprocess_adapter_image * add TencentArc to license * fix example code links * add image converter and fix example doc string * fix links * clearer additional residual application --------- Co-authored-by: HimariO <[email protected]>

williamberman force-pushed the will/general-adapter_ branch 3 times, most recently from f61bfdc to 104b077 Compare July 4, 2023 03:02

williamberman changed the title ~~Quick implementation of t2i-adapter~~ t2i pipeline Jul 4, 2023

williamberman force-pushed the will/general-adapter_ branch from 104b077 to a10e41c Compare July 4, 2023 03:05

williamberman marked this pull request as ready for review July 4, 2023 03:07

williamberman requested review from sayakpaul and patrickvonplaten July 4, 2023 03:07

williamberman requested review from pcuenca and yiyixuxu July 4, 2023 03:13