Error converting LoRA to safetensors/ckpt #2326

jndietz · 2023-02-13T02:27:39Z

Generally speaking, I am not clear on what to do with the output of these LoRA python scripts. I don't think the output can be natively used by the webuis. Other LoRAs I've seen online are usually safetensors. Here is what I did...

I used the provided python script in examples to generate a LoRA:

#!/bin/sh

accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path=$1  \
  --instance_data_dir=$2 \
  --output_dir=$3 \
  --instance_prompt="a photo of laskajavids" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --checkpointing_steps=200 \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=1000 \
  --validation_prompt="A photo of laskajavids by the pool" \
  --validation_epochs=50 \
  --seed="0" \
  --mixed_precision="fp16" \
  --use_8bit_adam

I successfully (or not?) created a LoRA, and it output the following to /output:

checkpoint-1000
checkpoint-200
checkpoint-400
checkpoint-600
checkpoint-800
pytorch_lora_weights.bin

Then, I tried to run scripts\convert_diffusers_to_original_stable_diffusion.py, like so:

 python /diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py --model_path /output --checkpoint_path /test.ckpt --use_safetensors

I received the following error:

Traceback (most recent call last):
  File "/diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py", line 290, in <module>
    unet_state_dict = torch.load(unet_path, map_location="cpu")
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 771, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 270, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 251, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/output/unet/diffusion_pytorch_model.bin'

Did I miss something when creating the LoRA?

The text was updated successfully, but these errors were encountered:

jndietz · 2023-02-13T03:40:24Z

I just tried running the newer script examples\text_to_image\train_text_to_image_lora.py, with the following

export MODEL_NAME=$1
export TRAIN_DIR=$2

accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$TRAIN_DIR --caption_column="additional_feature" \
  --resolution=512 --random_flip \
  --train_batch_size=1 \
  --validation_epochs=120 \
  --max_train_steps=1000 \
  --checkpointing_steps=100 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --output_dir=$3 \
  --validation_prompt="photo of laskajavids" \
  --enable_xformers_memory_efficient_attention

And the output appeared to be the same as examples/train_dreambooth_lora.py. So, I got the same error:

python /diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py --model_path /output --checkpoint_path /test.ckpt --use_safetensors
Traceback (most recent call last):
  File "/diffusers/scripts/convert_diffusers_to_original_stable_diffusion.py", line 290, in <module>
    unet_state_dict = torch.load(unet_path, map_location="cpu")
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 771, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 270, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/ubuntu/miniconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 251, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/output/unet/diffusion_pytorch_model.bin'

jndietz · 2023-02-13T05:00:50Z

Looking at the script itself, I'm seeing why this isn't working. My output directory isn't a model. What do I do with the output in there currently?

checkpoint-1000
checkpoint-200
checkpoint-400
checkpoint-600
checkpoint-800
pytorch_lora_weights.bin

patrickvonplaten · 2023-02-13T13:09:20Z

@williamberman could you take a look here?

NicholasKao1029 · 2023-02-13T19:48:17Z

Following

patrickvonplaten · 2023-02-14T22:06:43Z

By other WebUIs was mostly Automatic1111 meant?

We should really think about whether we can somehow integrate diffusers into AUTO1111

jndietz · 2023-02-14T22:11:51Z

@patrickvonplaten Yes, I am speaking to automatic1111's webui specifically. Though, I want to be able to do this via a python script without having to use a webui.

I'll have to go back and make sure I followed the 🤗 docs. I do feel like I followed them closely, which is why I am confused as to how to use the outputted LoRAs and convert them into a safetensors or ckpt.

patrickvonplaten · 2023-02-16T09:54:08Z

You can use LoRA trained embeddings easily with diffusers, see: https://huggingface.co/docs/diffusers/v0.12.0/en/training/lora#lora-support-in-diffusers but I think it's not so easy to convert them to A1111.

haofanwang · 2023-02-16T13:00:36Z

Hi, @patrickvonplaten, does diffusers support convert or directly load from Lora+safetensor? I have tried with the converting scripts, but failed.

The weight is from civitai. Any help would be appreciated.

patrickvonplaten · 2023-02-16T18:07:42Z

I'll try to allocate time for this. Otherwise maybe @williamberman or @sayakpaul could check it out as well. I'm not 100% sure whether we can convert civitai lora weights

sayakpaul · 2023-02-17T03:42:05Z

W.r.t #2363 I think there are a couple of different conversion pathways we're talking about for completness:

Diffusers LoRA weights to safetensors
civitai weights to diffusers
civitai LoRA weights to diffusers

@patrickvonplaten am I missing out on something?

haofanwang · 2023-02-17T03:50:07Z

I think the second one has already been ready with ./scripts/convert_original_stable_diffusion_to_diffusers.py, it can convert civitai weights (in safetensors but without lora) into diffusers format.

The third one should be civitai LoRA weights (in safetensors format) to diffusers. I'm actually working on it by diving into stable-diffusion-webui which supports loading from lora+safetensors format, I will provide my script for your reference once I finish.

@sayakpaul

sayakpaul · 2023-02-17T04:08:20Z

That is amazing! It would be amazing to contribute a PR for that as well!

garyhxfang · 2023-02-17T06:04:52Z

haofanwang

@haofanwang
Oh, I really need that master. I am also stuck in coverting civitai LoRA weights to diffusers. Do you mind add me as friend on Wechat? Hope that I can have some communication with you on stable diffusion. I have sent you a email to this email address [email protected].

haofanwang · 2023-02-17T16:40:11Z

Should we add Lora related code to research_projects? @sayakpaul

sayakpaul · 2023-02-17T16:41:40Z

If it's about conversion, okay to add them to scripts/. WDYT @williamberman?

haofanwang · 2023-02-18T05:27:42Z

@sayakpaul @williamberman I have made a PR for this, if it looks good to you, it should be fine to merge.

jndietz · 2023-02-19T16:20:09Z

@haofanwang That is awesome - now how can I convert my 🤗 diffuser to a safetensors/ckpt? Is it just a matter if mapping keys?

haofanwang · 2023-02-20T07:57:35Z

@jndietz Yes. Once the PR merged into diffuser, you can just run the convert script!

harrywang · 2023-02-20T16:51:23Z

@haofanwang thanks a lot for the PR. I have a question related to the discussion AUTOMATIC1111/stable-diffusion-webui#7387 - after we trained lora with dreambooth using diffusers - the .bin file is only about ~3m while the lora models on Civitai.com are around ~150m - how can we convert the 3m file to the file we can use in Automatic1111? I am still learning about this and sorry for asking stupid questions. Thanks again.

jndietz · 2023-02-20T16:55:29Z

@haofanwang Isn't that script for converting safetensors to diffusers?

haofanwang · 2023-02-20T17:54:15Z

@harrywang Don't worry about the file size, you are on the right way, it can work just as other weights in civitai. This is because the default setting for dimension lora layer is quite small. You can find more info at the end of this tutorial, give us a star if it is helpful.

haofanwang · 2023-02-20T17:56:21Z

@jndietz No, it is not a general converting script. For now, we only handle LoRA weights stored in safetensors format.

harrywang · 2023-02-20T22:38:29Z

@harrywang Don't worry about the file size, you are on the right way, it can work just as other weights in civitai. This is because the default setting for dimension lora layer is quite small. You can find more info at the end of this tutorial, give us a star if it is helpful.

@haofanwang Thanks a lot for the nice reply and tutorial! - I have shared that with our team.

harrywang · 2023-02-21T00:11:14Z

@haofanwang Another question: convert_lora_safetensor_to_diffusers.py converts safetensors to diffusers format. After I trained LoRA model, I have the following in the output folder and checkpoint subfolder:

How to convert them into safetensors like the ones I downloaded from civitai or huggingface so that I can use this via Automatica1111?

Thanks a lot!!

haofanwang · 2023-02-21T04:59:02Z

@harrywang could you open a new issue here? as this issue becomes just like a Q&A, I will take a look soon.

harrywang · 2023-02-21T15:53:12Z

@harrywang could you open a new issue here? as this issue becomes just like a Q&A, I will take a look soon.

No problem. I have created an issue haofanwang/Lora-for-Diffusers#1 Thanks!

ignacfetser · 2023-02-24T17:36:58Z

@haofanwang Another question: convert_lora_safetensor_to_diffusers.py converts safetensors to diffusers format. After I trained LoRA model, I have the following in the output folder and checkpoint subfolder:

How to convert them into safetensors like the ones I downloaded from civitai or huggingface so that I can use this via Automatica1111?

Thanks a lot!!

@harrywang check out my comment in this discussion, it might help you until an official solution gets released

harrywang · 2023-02-25T16:25:57Z

@haofanwang Another question: convert_lora_safetensor_to_diffusers.py converts safetensors to diffusers format. After I trained LoRA model, I have the following in the output folder and checkpoint subfolder:
How to convert them into safetensors like the ones I downloaded from civitai or huggingface so that I can use this via Automatica1111?
Thanks a lot!!

@harrywang check out my comment in this discussion, it might help you until an official solution gets released

Thanks. But when I use https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py to train a model, there is no custom_checkpoint_0.pkl file in the checkpoint folders:

any idea?

ignacfetser · 2023-02-25T20:35:40Z

@harrywang I've just realized that I used a different version of the lora training script which explains the missing file:
https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py
I'll check out the one you've linked if I can make the converter work on it as well.

ignacfetser · 2023-02-26T19:38:01Z

@harrywang so I have just replaced the 'custom_checkpoint_0.pkl' with 'pytorch_model.bin' in the converter script and it works just fine using in automatic1111

jndietz · 2023-02-27T00:32:17Z

@ignacfetser I'll give this a shot later, thanks.

jndietz · 2023-02-27T05:36:41Z

@ignacfetser Worked perfectly. I will close this issue.

I used train_dreambooth_lora.py to train, then used this script from @ignacfetser to convert the cwd from diffusers to safetensors.

import os;
import re;
import torch;
from safetensors.torch import save_file;

loraName = os.path.split(os.getcwd())[1];

for root, dirs, files in os.walk('.'):
  for dir in dirs:
    ckptIndex = re.search('^checkpoint\-(\d+)$', dir);
    if ckptIndex:
      newDict = dict();
      checkpoint = torch.load(os.path.join(dir, 'custom_checkpoint_0.pkl'));
      for idx, key in enumerate(checkpoint):

        newKey = re.sub('\.processor\.', '_', key);
        newKey = re.sub('mid_block\.', 'mid_block_', newKey);
        newKey = re.sub('_lora.up.', '.lora_up.', newKey);
        newKey = re.sub('_lora.down.', '.lora_down.', newKey);
        newKey = re.sub('\.(\d+)\.', '_\\1_', newKey);
        newKey = re.sub('to_out', 'to_out_0', newKey);
        newKey = 'lora_unet_'+newKey;

        newDict[newKey] = checkpoint[key];

      newLoraName = loraName + '-' + ckptIndex.group(1) + '.safetensors';
      print("Saving " + newLoraName);
      save_file(newDict, newLoraName);

@ignacfetser To satisfy my own curiosity, how did you know what keys needed map, and why? Was there some documentation you read, or did you load the models and inspect the keys manually?

harrywang · 2023-02-27T19:59:58Z

@jndietz the conversion worked but when I loaded and used the converted lora weights in webui it does not have any effect - do you have the same issue?

ignacfetser · 2023-02-27T20:23:03Z

@jndietz I have downloaded an auto1111 'compatible' lora model from civitai and cross referenced the keys, also the auto1111 ui logs out the wrong keys when loading the model and that made it easier.

jndietz · 2023-02-27T20:53:10Z

@harrywang I'm not seeing any issues with my LoRAs that I created with diffusers. I put them in the automatic1111 models/Lora folder, and refresh in the UI, and add it to the prompt. It is working.

harrywang · 2023-02-27T21:43:58Z

@jndietz thanks! I tried another lora I trained and it worked - I guess the Lora I used before was not trained well.

harrywang · 2023-02-28T00:34:26Z

I adapted the script by @ignacfetser by adding the CPU support and a simple argparse： https://github.com/harrywang/finetune-sd/blob/main/convert-to-safetensors.py Thanks again @ignacfetser @jndietz @haofanwang for your guidance and help. Now, I can train the models using diffusers and use them in WebUI, cheers!

ProGamerGov · 2023-03-17T18:37:09Z

Do these Diffusers to safetensors scripts work for both SD 1.5 and SD 2.1?

tony109060581 · 2023-04-24T11:47:34Z

harrywang

Hi harry, which sd network you using work fine?

capntrips · 2023-05-02T21:23:20Z

In case anyone else ends up here with a "Failed to match keys when loading Lora" from Automatic1111 WebUI after using one of the above conversion scripts, see #3284. I was able to work around it temporarily by storing the good keys and referencing those when saving the file.

modulator · 2023-05-29T13:58:10Z

Related Question:

I recently created multiple LoRAs via "LyCORIS extraction from Trained Dreambooth models" (I used "Kohya SS" Utilities tab)...

Everything went smoothly (they are the correct 90K size, etc). But I accidentally saved all the LoRAs as .CKPT instead of .safetensors... And they don't seem to be working as .ckpt format...

Questions:
A) Can I just change their extensions to .safetensors to make them work?
B) If NOT, is there any easy way to convert them from .ckpt to .safetensors?

If not, I could re-extract them from scratch via LyCORIS, but I'd rather NOT if I can avoid that...

Any thoughts? :)

slessans · 2023-08-07T05:03:08Z

For stable diffusion 2 the following script works:

https://gist.github.com/slessans/ac1045b6d4627753743bc2081546733a

StanislawKarnacky · 2023-08-11T20:39:49Z

@slessans Maybe you got a solution on how to convert LoRA trained on SDXL Dreambooth .bin to .safetensors? Tried a few scripts, and all of them ended with a key mismatch.

jndietz changed the title ~~Error converting LoRA to safetensors~~ Error converting LoRA to safetensors/ckpt Feb 13, 2023

patrickvonplaten assigned williamberman Feb 13, 2023

jndietz mentioned this issue Feb 14, 2023

Unable to run inference with provided 🤗 example scripts #2344

Closed

patrickvonplaten self-assigned this Feb 16, 2023

patrickvonplaten mentioned this issue Feb 16, 2023

How could I convert a LoRA .safetensors or .ckpt file into the format that diffusers can process? #2363

Closed

haofanwang mentioned this issue Feb 17, 2023

Support convert LoRA safetensors into diffusers format #2403

Merged

pcuenca mentioned this issue Feb 21, 2023

Adding support for safetensors and LoRa. #2448

Merged

jndietz closed this as completed Feb 27, 2023

This was referenced Mar 1, 2023

convert lora .bin weights to ckpt or safetensors haofanwang/Lora-for-Diffusers#1

Open

the trained safetensors does not work in Automatic1111 WebUI cloneofsimo/lora#212

Open

Error converting LoRA to safetensors/ckpt #2326

Error converting LoRA to safetensors/ckpt #2326

Comments

jndietz commented Feb 13, 2023 • edited Loading

jndietz commented Feb 13, 2023

jndietz commented Feb 13, 2023

patrickvonplaten commented Feb 13, 2023

NicholasKao1029 commented Feb 13, 2023

patrickvonplaten commented Feb 14, 2023

jndietz commented Feb 14, 2023 • edited Loading

patrickvonplaten commented Feb 16, 2023

haofanwang commented Feb 16, 2023

patrickvonplaten commented Feb 16, 2023

sayakpaul commented Feb 17, 2023

haofanwang commented Feb 17, 2023

sayakpaul commented Feb 17, 2023

garyhxfang commented Feb 17, 2023 • edited Loading

haofanwang commented Feb 17, 2023

sayakpaul commented Feb 17, 2023

haofanwang commented Feb 18, 2023 • edited Loading

jndietz commented Feb 19, 2023

haofanwang commented Feb 20, 2023

harrywang commented Feb 20, 2023

jndietz commented Feb 20, 2023

haofanwang commented Feb 20, 2023

haofanwang commented Feb 20, 2023

harrywang commented Feb 20, 2023

harrywang commented Feb 21, 2023 • edited Loading

haofanwang commented Feb 21, 2023

harrywang commented Feb 21, 2023

ignacfetser commented Feb 24, 2023

harrywang commented Feb 25, 2023

ignacfetser commented Feb 25, 2023

ignacfetser commented Feb 26, 2023

jndietz commented Feb 27, 2023

jndietz commented Feb 27, 2023 • edited Loading

harrywang commented Feb 27, 2023

ignacfetser commented Feb 27, 2023

jndietz commented Feb 27, 2023

harrywang commented Feb 27, 2023

harrywang commented Feb 28, 2023 • edited Loading

ProGamerGov commented Mar 17, 2023

tony109060581 commented Apr 24, 2023 • edited Loading

capntrips commented May 2, 2023 • edited Loading

modulator commented May 29, 2023

slessans commented Aug 7, 2023

StanislawKarnacky commented Aug 11, 2023

jndietz commented Feb 13, 2023 •

edited

Loading

jndietz commented Feb 14, 2023 •

edited

Loading

garyhxfang commented Feb 17, 2023 •

edited

Loading

haofanwang commented Feb 18, 2023 •

edited

Loading

harrywang commented Feb 21, 2023 •

edited

Loading

jndietz commented Feb 27, 2023 •

edited

Loading

harrywang commented Feb 28, 2023 •

edited

Loading

tony109060581 commented Apr 24, 2023 •

edited

Loading

capntrips commented May 2, 2023 •

edited

Loading