Skip to content

Commit c470274

Browse files
elismasilvaelismasilva
authored
feat: new community mixture_tiling_sdxl pipeline for SDXL (#10759)
* feat: new community mixture_tiling_sdxl pipeline for SDXL mixture-of-diffusers support * fix use of variable latents to tile_latents * removed references to modules that are not being used in this pipeline * make style, make quality
1 parent 798e171 commit c470274

File tree

2 files changed

+1278
-36
lines changed

2 files changed

+1278
-36
lines changed

‎examples/community/README.md

+93-36
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@ Please also check out our [Community Scripts](https://github.com/huggingface/dif
5050
| IADB Pipeline | Implementation of [Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model](https://arxiv.org/abs/2305.03486) | [IADB Pipeline](#iadb-pipeline) | - | [Thomas Chambon](https://github.com/tchambon)
5151
| Zero1to3 Pipeline | Implementation of [Zero-1-to-3: Zero-shot One Image to 3D Object](https://arxiv.org/abs/2303.11328) | [Zero1to3 Pipeline](#zero1to3-pipeline) | - | [Xin Kong](https://github.com/kxhit) |
5252
| Stable Diffusion XL Long Weighted Prompt Pipeline | A pipeline support unlimited length of prompt and negative prompt, use A1111 style of prompt weighting | [Stable Diffusion XL Long Weighted Prompt Pipeline](#stable-diffusion-xl-long-weighted-prompt-pipeline) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1LsqilswLR40XLLcp6XFOl5nKb_wOe26W?usp=sharing) | [Andrew Zhu](https://xhinker.medium.com/) |
53+
| Stable Diffusion Mixture Tiling Pipeline SD 1.5 | A pipeline generates cohesive images by integrating multiple diffusion processes, each focused on a specific image region and considering boundary effects for smooth blending | [Stable Diffusion Mixture Tiling Pipeline SD 1.5](#stable-diffusion-mixture-tiling-sd-15) | [![Hugging Face Space](https://img.shields.io/badge/🤗%20Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/albarji/mixture-of-diffusers) | [Álvaro B Jiménez](https://github.com/albarji/) |
54+
| Stable Diffusion Mixture Tiling Pipeline SDXL | A pipeline generates cohesive images by integrating multiple diffusion processes, each focused on a specific image region and considering boundary effects for smooth blending | [Stable Diffusion Mixture Tiling Pipeline SDXL](#stable-diffusion-mixture-tiling-sdxl) | [![Hugging Face Space](https://img.shields.io/badge/🤗%20Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/elismasilva/mixture-of-diffusers-sdxl-tiling) | [Eliseu Silva](https://github.com/DEVAIEXP/) |
5355
| FABRIC - Stable Diffusion with feedback Pipeline | pipeline supports feedback from liked and disliked images | [Stable Diffusion Fabric Pipeline](#stable-diffusion-fabric-pipeline) | [Notebook](https://github.com/huggingface/notebooks/blob/main/diffusers/stable_diffusion_fabric.ipynb)| [Shauray Singh](https://shauray8.github.io/about_shauray/) |
5456
| sketch inpaint - Inpainting with non-inpaint Stable Diffusion | sketch inpaint much like in automatic1111 | [Masked Im2Im Stable Diffusion Pipeline](#stable-diffusion-masked-im2im) | - | [Anatoly Belikov](https://github.com/noskill) |
5557
| sketch inpaint xl - Inpainting with non-inpaint Stable Diffusion | sketch inpaint much like in automatic1111 | [Masked Im2Im Stable Diffusion XL Pipeline](#stable-diffusion-xl-masked-im2im) | - | [Anatoly Belikov](https://github.com/noskill) |
@@ -2402,7 +2404,7 @@ pipe_images = mixing_pipeline(
24022404

24032405
![image_mixing_result](https://huggingface.co/datasets/TheDenk/images_mixing/resolve/main/boromir_gigachad.png)
24042406

2405-
### Stable Diffusion Mixture Tiling
2407+
### Stable Diffusion Mixture Tiling SD 1.5
24062408

24072409
This pipeline uses the Mixture. Refer to the [Mixture](https://arxiv.org/abs/2302.02412) paper for more details.
24082410

@@ -2433,6 +2435,96 @@ image = pipeline(
24332435

24342436
![mixture_tiling_results](https://huggingface.co/datasets/kadirnar/diffusers_readme_images/resolve/main/mixture_tiling.png)
24352437

2438+
### Stable Diffusion Mixture Canvas
2439+
2440+
This pipeline uses the Mixture. Refer to the [Mixture](https://arxiv.org/abs/2302.02412) paper for more details.
2441+
2442+
```python
2443+
from PIL import Image
2444+
from diffusers import LMSDiscreteScheduler, DiffusionPipeline
2445+
from diffusers.pipelines.pipeline_utils import Image2ImageRegion, Text2ImageRegion, preprocess_image
2446+
2447+
2448+
# Load and preprocess guide image
2449+
iic_image = preprocess_image(Image.open("input_image.png").convert("RGB"))
2450+
2451+
# Create scheduler and model (similar to StableDiffusionPipeline)
2452+
scheduler = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)
2453+
pipeline = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", scheduler=scheduler).to("cuda:0", custom_pipeline="mixture_canvas")
2454+
pipeline.to("cuda")
2455+
2456+
# Mixture of Diffusers generation
2457+
output = pipeline(
2458+
canvas_height=800,
2459+
canvas_width=352,
2460+
regions=[
2461+
Text2ImageRegion(0, 800, 0, 352, guidance_scale=8,
2462+
prompt=f"best quality, masterpiece, WLOP, sakimichan, art contest winner on pixiv, 8K, intricate details, wet effects, rain drops, ethereal, mysterious, futuristic, UHD, HDR, cinematic lighting, in a beautiful forest, rainy day, award winning, trending on artstation, beautiful confident cheerful young woman, wearing a futuristic sleeveless dress, ultra beautiful detailed eyes, hyper-detailed face, complex, perfect, model, textured, chiaroscuro, professional make-up, realistic, figure in frame, "),
2463+
Image2ImageRegion(352-800, 352, 0, 352, reference_image=iic_image, strength=1.0),
2464+
],
2465+
num_inference_steps=100,
2466+
seed=5525475061,
2467+
)["images"][0]
2468+
```
2469+
2470+
![Input_Image](https://huggingface.co/datasets/kadirnar/diffusers_readme_images/resolve/main/input_image.png)
2471+
![mixture_canvas_results](https://huggingface.co/datasets/kadirnar/diffusers_readme_images/resolve/main/canvas.png)
2472+
2473+
### Stable Diffusion Mixture Tiling SDXL
2474+
2475+
This pipeline uses the Mixture. Refer to the [Mixture](https://arxiv.org/abs/2302.02412) paper for more details.
2476+
2477+
```python
2478+
import torch
2479+
from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler, AutoencoderKL
2480+
2481+
device="cuda"
2482+
2483+
# Load fixed vae (optional)
2484+
vae = AutoencoderKL.from_pretrained(
2485+
"madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16
2486+
).to(device)
2487+
2488+
# Create scheduler and model (similar to StableDiffusionPipeline)
2489+
model_id="stablediffusionapi/yamermix-v8-vae"
2490+
scheduler = DPMSolverMultistepScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)
2491+
pipe = DiffusionPipeline.from_pretrained(
2492+
model_id,
2493+
torch_dtype=torch.float16,
2494+
vae=vae,
2495+
custom_pipeline="mixture_tiling_sdxl",
2496+
scheduler=scheduler,
2497+
use_safetensors=False
2498+
).to(device)
2499+
2500+
pipe.enable_model_cpu_offload()
2501+
pipe.enable_vae_tiling()
2502+
pipe.enable_vae_slicing()
2503+
2504+
generator = torch.Generator(device).manual_seed(297984183)
2505+
2506+
# Mixture of Diffusers generation
2507+
image = pipe(
2508+
prompt=[[
2509+
"A charming house in the countryside, by jakub rozalski, sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece",
2510+
"A dirt road in the countryside crossing pastures, by jakub rozalski, sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece",
2511+
"An old and rusty giant robot lying on a dirt road, by jakub rozalski, dark sunset lighting, elegant, highly detailed, smooth, sharp focus, artstation, stunning masterpiece"
2512+
]],
2513+
tile_height=1024,
2514+
tile_width=1280,
2515+
tile_row_overlap=0,
2516+
tile_col_overlap=256,
2517+
guidance_scale_tiles=[[7, 7, 7]], # or guidance_scale=7 if is the same for all prompts
2518+
height=1024,
2519+
width=3840,
2520+
target_size=(1024, 3840),
2521+
generator=generator,
2522+
num_inference_steps=30,
2523+
)["images"][0]
2524+
```
2525+
2526+
![mixture_tiling_results](https://huggingface.co/datasets/elismasilva/results/resolve/main/mixture_sdxl.png)
2527+
24362528
### TensorRT Inpainting Stable Diffusion Pipeline
24372529

24382530
The TensorRT Pipeline can be used to accelerate the Inpainting Stable Diffusion Inference run.
@@ -2475,41 +2567,6 @@ image = pipe(prompt, image=input_image, mask_image=mask_image, strength=0.75,).i
24752567
image.save('tensorrt_inpaint_mecha_robot.png')
24762568
```
24772569

2478-
### Stable Diffusion Mixture Canvas
2479-
2480-
This pipeline uses the Mixture. Refer to the [Mixture](https://arxiv.org/abs/2302.02412) paper for more details.
2481-
2482-
```python
2483-
from PIL import Image
2484-
from diffusers import LMSDiscreteScheduler, DiffusionPipeline
2485-
from diffusers.pipelines.pipeline_utils import Image2ImageRegion, Text2ImageRegion, preprocess_image
2486-
2487-
2488-
# Load and preprocess guide image
2489-
iic_image = preprocess_image(Image.open("input_image.png").convert("RGB"))
2490-
2491-
# Create scheduler and model (similar to StableDiffusionPipeline)
2492-
scheduler = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)
2493-
pipeline = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", scheduler=scheduler).to("cuda:0", custom_pipeline="mixture_canvas")
2494-
pipeline.to("cuda")
2495-
2496-
# Mixture of Diffusers generation
2497-
output = pipeline(
2498-
canvas_height=800,
2499-
canvas_width=352,
2500-
regions=[
2501-
Text2ImageRegion(0, 800, 0, 352, guidance_scale=8,
2502-
prompt=f"best quality, masterpiece, WLOP, sakimichan, art contest winner on pixiv, 8K, intricate details, wet effects, rain drops, ethereal, mysterious, futuristic, UHD, HDR, cinematic lighting, in a beautiful forest, rainy day, award winning, trending on artstation, beautiful confident cheerful young woman, wearing a futuristic sleeveless dress, ultra beautiful detailed eyes, hyper-detailed face, complex, perfect, model, textured, chiaroscuro, professional make-up, realistic, figure in frame, "),
2503-
Image2ImageRegion(352-800, 352, 0, 352, reference_image=iic_image, strength=1.0),
2504-
],
2505-
num_inference_steps=100,
2506-
seed=5525475061,
2507-
)["images"][0]
2508-
```
2509-
2510-
![Input_Image](https://huggingface.co/datasets/kadirnar/diffusers_readme_images/resolve/main/input_image.png)
2511-
![mixture_canvas_results](https://huggingface.co/datasets/kadirnar/diffusers_readme_images/resolve/main/canvas.png)
2512-
25132570
### IADB pipeline
25142571

25152572
This pipeline is the implementation of the [α-(de)Blending: a Minimalist Deterministic Diffusion Model](https://arxiv.org/abs/2305.03486) paper.

0 commit comments

Comments
 (0)