New inpaint pipeline with parameter 'strength' generates poor results when using EulerAncestralDiscreteScheduler

[Magicalppj]

Describe the bug

Hi, I recently found the update for inpainting with strength, it's really cool. code
But when using EulerAncestralDiscreteScheduler, just simply set strength as 0.5 or other values<1, the inpaint pipeline fails.

I'm not familiar with the math stuff in diffusion, but it seems that it's the scale process that leads to the problem.

When I rescale the latents back after the preparing latents function with latents = latents / self.scheduler.init_noise_sigma before the denoise process by unet, the results seems to be good.

    # for EulerAncestralDiscreteScheduler, the init_noise_sigma is 14.6146
    # for DPMSolverMultistepScheduler, the init_noise_sigma is 1.0
    latents = latents * self.scheduler.init_noise_sigma 

    return latents

input and mask

result

Reproduction

import diffusers
import torch
from PIL import Image
from diffusers import StableDiffusionInpaintPipeline
pipe = StableDiffusionInpaintPipeline.from_pretrained('runwayml/stable-diffusion-inpainting', torch_dtype=torch.float16).to('cuda')
pipe.safety_checker = None
pipe.scheduler = diffusers.EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
image_pil = Image.open('./test/images/girl.png')
target_mask = Image.open('./test/images/girl_mask.png')
inpainted_image = pipe(
prompt="1girl, holding flowers, best quality, masterpiece, realistic, highres, ultral high details", negative_prompt="nsfw, breasts, nude, lowres, bad anatomy, bad hands, cropped, worst quality", image=image_pil, mask_image=target_mask,
height=image_pil.size[1], width=image_pil.size[0],strength=0.5,
num_inference_steps=30, guidance_scale=9,
).images[0]

Logs

No response

System Info

• diffusers version: 0.17.0.dev0
• Platform: Linux-5.19.0-35-generic-x86_64-with-glibc2.35
• Python version: 3.10.9
• PyTorch version (GPU?): 2.0.0+cu117 (True)
• Huggingface_hub version: 0.13.3
• Transformers version: 4.26.0
• Accelerate version: 0.18.0
• xFormers version: 0.0.15.dev+0bad001.d20230403
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[rupertmenneer]

@Magicalppj thanks for catching this! you are quite right it looks like the init_noise_sigma scaling factor for the latents breaks things when the initialised latents aren't pure noise (e.g. strength <1.) and the sigma is not 1. See:

After this small fix which means only the 'pure noise' when strength = 1 is scaled by the init_noise_sigma, e.g.
For line 639 in pipeline_stable_diffusion_inpaint.py

if is_strength_max:
    # if strength is 100% then simply initialise the latents to noise
    latents = noise
    # scale the initial noise by the standard deviation required by the scheduler
    latents = latents * self.scheduler.init_noise_sigma

The results look like this for various samplers:

Problem solved! I will submit a PR soon with this fix. Possibly after the weekend.

@patrickvonplaten I also noticed that a strength of 0 can break things for other samplers and in general, the whole pipeline doesn't really make sense with inference steps < 1. Should I throw an error if this is the case to prompt the user to increase the strength so inference steps are more than or equal to 1 or should I just return the original image and skip the sampling loop altogether? Let me know the preference and I can put into a PR with the above when I find some time.

[Magicalppj]

@Magicalppj thanks for catching this! you are quite right it looks like the init_noise_sigma scaling factor for the latents breaks things when the initialised latents aren't pure noise (e.g. strength <1.) and the sigma is not 1. See:

After this small fix which means only the 'pure noise' when strength = 1 is scaled by the init_noise_sigma, e.g. For line 639 in pipeline_stable_diffusion_inpaint.py

if is_strength_max:
    # if strength is 100% then simply initialise the latents to noise
    latents = noise
    # scale the initial noise by the standard deviation required by the scheduler
    latents = latents * self.scheduler.init_noise_sigma

Thanks for reply! It's clear now the scale for latents should only be applied to pure noise, that indeed solves this problem, great!

[patrickvonplaten]

@rupertmenneer, it'd be great if you could send a PR! Agree that it doesn't make much sense if we have < 1 inference steps - happy to throw a nice error here. Also, it'd be very nice if you could open a PR to fix the strength :-)

[rupertmenneer]

@patrickvonplaten attached the PR - put both changes discussed into it as they are both very minor, hope that works! Thanks 👍