StableDiffusionInpaintPipeline.from_single_file()

[danilodelucio]

Hi there, this is my first time creating a discussion here in Github. 🥹

I'm trying to generate an image using the inpainting pipeline, and I would like to use a local checkpoint but I'm getting the following error:

ValueError: Trying to set a tensor of shape torch.Size([320, 4, 3, 3]) in "weight" (which has shape torch.Size([320, 9, 3, 3])), this look incorrect.

import torch
from diffusers import StableDiffusionInpaintPipeline
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid

checkpoint = "D:\\_StableDiffusion\\ComfyUI\\ComfyUI_windows_portable\\ComfyUI\\models\\checkpoints\\dreamshaper_8.safetensors"

pipeline = StableDiffusionInpaintPipeline.from_single_file(
    checkpoint, 
    torch_dtype=torch.float16, 
    use_safetensors=True).to("cuda")

# pipeline = AutoPipelineForInpainting.from_pretrained(
#     "runwayml/stable-diffusion-inpainting", 
#     torch_dtype=torch.float16,
#     variant="fp16" 
# ).to("cuda")

pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()

input_mask = load_image("D:/Dropbox/VFX_PERSONAL_PROJECTS/ActionVFX/Soldier Running Through Field/footage/mask_0029.jpg")
input_image = load_image("D:/Dropbox/VFX_PERSONAL_PROJECTS/ActionVFX/Soldier Running Through Field/footage/PF0039-04_Clip-4_REC709_HD_jpg/PF0039-04_Clip-4_REC709_HD_0029.jpg")

image = pipeline(
                prompt = "remove person and rebuild the background", 
                negative_prompt = "", 
                image = input_image,
                mask_image = input_mask,
                # generator = setSeed(seed),
                num_inference_steps = 20,
                guidance_scale = 7,
                strength = 0.7
                ).images[0]

# make_image_grid([init_image, image], rows=1, cols=2)
output = r"C:\Users\danil\.nuke\Python\NukeDiffusion\_output\test_inpainting_11.png"
image.save(output)

So I have 3 quick questions about this approach.

1. How to solve the error above;
2. If I enable the second pipeline block using the AutoPipelineForInpainting.from_pretrained(), my image is generating fine, but I saw that the final resolution is not the same as the original (my original image is 1920x1080 and the output is being 512x512). When I use the img2img workflow, the resolution of the input image is passed in automatically. So, should I use the arguments width and height or there is another method to pass the resolution automatically?
3. Depending on the answer, does this apply to XL models too?

Thanks in advance! 🙏

[sayakpaul]

Can you try installing diffusers from source and try again?

pip install git+https://github.com/huggingface/diffusers

[danilodelucio]

Thanks for the reply! I did it and still not working.

[yiyixuxu]

hi!
you need to pass num_in_channels=4

pipeline = StableDiffusionInpaintPipeline.from_single_file(
    checkpoint, 
    torch_dtype=torch.float16, 
    num_in_channels=4).to("cuda")

cc @stevhliu @DN6 here, do we need to add this to the doc somewhere?

[stevhliu]

I think we have something similar to this in the docs here. Does that doc need to be updated and maybe we can include this there?

[yiyixuxu]

@stevhliu
I think this argument is not listed in the signature of from_single_file. I will do a PR to try to list all the arguments.

we can then decide if we want to talk more about them on doc or not, is this the only place we talk about from_single_file() - I think we should probably update it if it's the case, no? (this might be helpful for your reference https://huggingface.slack.com/archives/C05L70FUSKV/p1708298982391669 )

[stevhliu]

Yeah that's the only place we focus on from_single_file() (there are some other one-off references to it for using with specific pipelines). I'll update it to be a little more detailed :)

[danilodelucio]

Hey guys, thanks for the reply!

I was about to ask about the num_in_channels=4 argument because I didn't find it in the document (I would appreciate if someone give me a quick explanation).

Thank you!