Unable to run inference with provided 🤗 example scripts

[jndietz]

This is tangentially related to my issue #2326. I believe I have successfully generated a LoRA, and would like to use it in a DiffusionPipeline for inference. Full disclosure: I am trying to run this on a 1050ti with 4GB VRAM. I am able to run inference from within the automatic1111 webui. Following some of the docs on 🤗, I think this should be optimized to run on under 4GB VRAM.

import os
from diffusers import StableDiffusionPipeline
import torch
    
model_path = os.getcwd() + "/laskajavids" # 👈 my lora output is in here
pipe = StableDiffusionPipeline.from_pretrained('runwayml/stable-diffusion-v1-5', torch_dtype=torch.float16)
pipe.unet.load_attn_procs(model_path)
pipe.to("cuda")

prompt = "a photo of laskajavids"
pipe.enable_attention_slicing()
pipe.enable_xformers_memory_efficient_attention()
image = pipe(prompt, width=512, height=768, num_inference_steps=30, guidance_scale=7.5).images[0]
image.save("laskajavids.png")

It seems like it is about to do something cool, then I get the following error:

(Python310) PS C:\projects\python-test> python .\pipeline.py    
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
safety_checker\model.safetensors not found
Fetching 15 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 5000.36it/s]
Traceback (most recent call last):
    pipe.enable_attention_slicing()
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 916, in enable_attention_slicing
    self.set_attention_slice(slice_size)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 931, in set_attention_slice
    module.set_attention_slice(slice_size)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\unet_2d_condition.py", line 387, in set_attention_slice
    fn_recursive_set_attention_slice(module, reversed_slice_size)
    fn_recursive_set_attention_slice(child, slice_size)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\unet_2d_condition.py", line 383, in fn_recursive_set_attention_slice
    fn_recursive_set_attention_slice(child, slice_size)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\unet_2d_condition.py", line 383, in fn_recursive_set_attention_slice
    fn_recursive_set_attention_slice(child, slice_size)
  [Previous line repeated 3 more times]
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\unet_2d_condition.py", line 380, in fn_recursive_set_attention_slice
    module.set_attention_slice(slice_size.pop())
    self.set_processor(processor)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\cross_attention.py", line 154, in set_processor
    self.processor = processor
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1304, in __setattr__
    raise TypeError("cannot assign '{}' as child module '{}' "
TypeError: cannot assign 'diffusers.models.cross_attention.SlicedAttnProcessor' as child module 'processor' (torch.nn.Module or None expected)

I'm not sure if triton is related to this issue. I did make an attempt at running pip install triton but got a not found error, even though it is on pypi 🤷‍♂️

[patrickvonplaten]

Hey @jndietz,

What diffusers version are you using? Think this might be solved on "main" diffusers :-)

[jndietz]

I'll double check, but pretty sure I'm using the latest from main. Will report back.

[jndietz]

@patrickvonplaten Sorry, I misspoke. I am using version 0.12.1, which is the latest as of today.

[jndietz]

@patrickvonplaten I checked out the latest from main and ran pip install -e . from the directory, and it installed the dev version.

Now I am getting this error -- is it due to my configuration in the above script?

(Python310) PS C:\projects\python-test> python .\pipeline.py
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
text_encoder\model.safetensors not found
Fetching 15 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 4994.80it/s]
Traceback (most recent call last):
  File "C:\projects\python-test\pipeline.py", line 13, in <module>
    image = pipe(prompt, width=512, height=768, num_inference_steps=30, guidance_scale=7.5).images[0]
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\github\huggingface\diffusers\src\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 571, in __call__
    prompt_embeds = self._encode_prompt(
  File "C:\github\huggingface\diffusers\src\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 287, in _encode_prompt
    prompt_embeds = self.text_encoder(
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\models\clip\modeling_clip.py", line 722, in forward
    return self.text_model(
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\models\clip\modeling_clip.py", line 643, in forward
    encoder_outputs = self.encoder(
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\models\clip\modeling_clip.py", line 574, in forward
    layer_outputs = encoder_layer(
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\models\clip\modeling_clip.py", line 317, in forward
    hidden_states, attn_weights = self.self_attn(
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Jared\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\models\clip\modeling_clip.py", line 257, in forward
    attn_output = torch.bmm(attn_probs, value_states)
RuntimeError: expected scalar type Half but found Float

[jndietz]

My LoRA was trained with the following script:

accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path=$1  \
  --instance_data_dir=$2 \
  --output_dir=$3 \
  --instance_prompt="a photo of laskajavids" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --checkpointing_steps=200 \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=1000 \
  --validation_prompt="A photo of laskajavids by the pool" \
  --validation_epochs=50 \
  --seed="0" \
  --mixed_precision="fp16" \
  --use_8bit_adam

[jndietz]

@patrickvonplaten I got it - I had to also upgrade transformers to the latest. I found a similar issue here:

https://discuss.huggingface.co/t/error-expected-scalar-type-half-but-found-float/25685

Is there some way of knowing which version of diffusers and transformers are compatible? Or is it safe to assume "use the latest from both?

[patrickvonplaten]

Hey @jndietz,

Yes, so the minimum requirement for transformers can be checked here:

diffusers/setup.py

Line 110 in e3ddbe2

"transformers>=4.25.1",