[Safetensors] Make safetensors the default way of saving weights #4235

patrickvonplaten · 2023-07-24T14:09:12Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2023-07-24T14:18:47Z

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten · 2023-08-07T12:36:42Z

@DN6 let me know if you need any help here

patrickvonplaten · 2023-08-14T14:39:45Z

@DN6 we should try to get this one merged - it has been stale now for three weeks

DN6 · 2023-08-14T16:51:04Z

I'll take a look this week @patrickvonplaten

…huggingface/diffusers into make_safe_serialization_default

examples/custom_diffusion/train_custom_diffusion.py

sayakpaul · 2023-08-17T04:39:43Z

examples/dreambooth/train_dreambooth_lora.py

@@ -1374,7 +1374,7 @@ def compute_text_embeddings(prompt):
        pipeline = pipeline.to(accelerator.device)

        # load attention processors
-        pipeline.load_lora_weights(args.output_dir, weight_name="pytorch_lora_weights.bin")
+        pipeline.load_lora_weights(args.output_dir, weight_name="pytorch_lora_weights.safetensors")


Does save_lora_weights() have safe_serialization set to True default? Otherwise, this will fail.

Also, we don't need to explicitly specify the weight_name actually since the naming is in the diffusers format.

safe_serialization defaults to True in the LoraLoaderMixin

diffusers/src/diffusers/loaders.py

Line 1418 in 1dd3441

safe_serialization: bool = True,

Regarding, weight_name. Not setting it while saving is fine, however, loading looks like it needs a path to the weights

diffusers/src/diffusers/loaders.py

Line 906 in 1dd3441

def load_lora_weights(self, pretrained_model_name_or_path_or_dict: Union[str, Dict[str, torch.Tensor]], **kwargs):

Yeah, you only need pipeline.load_lora_weights(args.output_dir).

sayakpaul · 2023-08-17T04:41:46Z

examples/test_examples.py

@@ -437,6 +437,7 @@ def test_custom_diffusion(self):
                --lr_scheduler constant
                --lr_warmup_steps 0
                --modifier_token <new1>
+                --no_safe_serialization


Ah I see. Cool.

sayakpaul

Looks good. Thanks for tackling this very important component!

I think we're still missing a couple examples no?

I would have expected to see safe_serialization to be set to True in all the officially supported training examples. But didn't see that. What am I missing out on?

DN6 · 2023-08-17T05:12:46Z

@sayakpaul It's possible that a couple have slipped by me. Do you happen to have links for training examples where safe_serialization isn't enabled by default?

sayakpaul · 2023-08-17T05:16:16Z

DreamBooth

diffusers/examples/dreambooth/train_dreambooth.py

Line 1363 in 852dc76

pipeline.save_pretrained(args.output_dir)
diffusers/examples/dreambooth/train_dreambooth_lora.py

Line 913 in 852dc76

LoraLoaderMixin.save_lora_weights(
diffusers/examples/dreambooth/train_dreambooth_lora_sdxl.py

Line 820 in 852dc76

StableDiffusionXLPipeline.save_lora_weights(

ControlNet

diffusers/examples/controlnet/train_controlnet.py

Line 1106 in 852dc76

controlnet.save_pretrained(args.output_dir)
diffusers/examples/controlnet/train_controlnet_sdxl.py

Line 1215 in 852dc76

controlnet.save_pretrained(args.output_dir)

Some examples.

sayakpaul · 2023-08-17T05:24:22Z

Overlook on my part. This PR is ready to merge!

patrickvonplaten · 2023-08-22T16:01:05Z

tests/pipelines/test_pipelines.py

@@ -884,7 +884,7 @@ def test_custom_model_and_pipeline(self):
        )

        with tempfile.TemporaryDirectory() as tmpdirname:
-            pipe.save_pretrained(tmpdirname)
+            pipe.save_pretrained(tmpdirname, safe_serialization=False)


Do we also test that saving with safe serialization works fine? The default is now that dicts are saved with safe_serialization -> we should test this as well no?

We have these tests here for saving.

diffusers/tests/pipelines/test_pipelines.py

Line 126 in 1dd3441

error = f"{traceback.format_exc()}"

diffusers/tests/pipelines/test_pipelines.py

Lines 578 to 603 in 1dd3441

def test_local_save_load_index(self):

prompt = "hello"

for variant in [None, "fp16"]:

for use_safe in [True, False]:

pipe = StableDiffusionPipeline.from_pretrained(

"hf-internal-testing/tiny-stable-diffusion-pipe-indexes",

variant=variant,

use_safetensors=use_safe,

safety_checker=None,

)

pipe = pipe.to(torch_device)

generator = torch.manual_seed(0)

out = pipe(prompt, num_inference_steps=2, generator=generator, output_type="numpy").images

with tempfile.TemporaryDirectory() as tmpdirname:

pipe.save_pretrained(tmpdirname)

pipe_2 = StableDiffusionPipeline.from_pretrained(

tmpdirname, safe_serialization=use_safe, variant=variant

)

pipe_2 = pipe_2.to(torch_device)

generator = torch.manual_seed(0)

out_2 = pipe_2(prompt, num_inference_steps=2, generator=generator, output_type="numpy").images

assert np.max(np.abs(out - out_2)) < 1e-3

Are there some specific things we should also be testing here?

…gingface#4235) * make safetensors default * set default save method as safetensors * update tests * update to support saving safetensors * update test to account for safetensors default * update example tests to use safetensors * update example to support safetensors * update unet tests for safetensors * fix failing loader tests * fix qc issues * fix pipeline tests * fix example test --------- Co-authored-by: Dhruv Nair <[email protected]>

make safetensors default

690f8a6

patrickvonplaten marked this pull request as draft July 24, 2023 14:15

DN6 self-assigned this Jul 25, 2023

DN6 added 3 commits August 15, 2023 04:44

merge main into make_safe_serialization_default

d366af4

set default save method as safetensors

35273fd

Merge branch 'main' into make_safe_serialization_default

948db23

DN6 requested a review from yiyixuxu August 15, 2023 06:05

DN6 added 13 commits August 15, 2023 06:29

update tests

f663696

update to support saving safetensors

e513a41

update test to account for safetensors default

5043f07

update example tests to use safetensors

f63f7a6

update example to support safetensors

7e63f22

Merge branch 'main' into make_safe_serialization_default

3f9a7bb

update unet tests for safetensors

48a3d55

Merge branch 'make_safe_serialization_default' of https://github.com/…

848e982

…huggingface/diffusers into make_safe_serialization_default

fix failing loader tests

9c456b1

Merge branch 'main' into make_safe_serialization_default

ed15f87

fix qc issues

0954946

fix pipeline tests

981fbcd

fix example test

1dd3441

DN6 marked this pull request as ready for review August 16, 2023 15:41

DN6 requested a review from sayakpaul August 16, 2023 15:43

DN6 mentioned this pull request Aug 16, 2023

Safe Serialization does not work for Custom Diffusion #4634

Closed

sayakpaul reviewed Aug 17, 2023

View reviewed changes

examples/custom_diffusion/train_custom_diffusion.py Show resolved Hide resolved

sayakpaul reviewed Aug 17, 2023

View reviewed changes

sayakpaul approved these changes Aug 17, 2023

View reviewed changes

sayakpaul merged commit 029fb41 into main Aug 17, 2023

sayakpaul deleted the make_safe_serialization_default branch August 17, 2023 05:24

DN6 mentioned this pull request Aug 18, 2023

Remove all safetensors import mocking from tests #4244

Closed

canberk17 mentioned this pull request Aug 22, 2023

Custom Diffusion : run train_custom_diffusion.py fails #4704

Closed

patrickvonplaten commented Aug 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Safetensors] Make safetensors the default way of saving weights #4235

[Safetensors] Make safetensors the default way of saving weights #4235

patrickvonplaten commented Jul 24, 2023

HuggingFaceDocBuilderDev commented Jul 24, 2023 •

edited

Loading

patrickvonplaten commented Aug 7, 2023

patrickvonplaten commented Aug 14, 2023

DN6 commented Aug 14, 2023

sayakpaul Aug 17, 2023

sayakpaul Aug 17, 2023

DN6 Aug 17, 2023

DN6 Aug 17, 2023

sayakpaul Aug 17, 2023

sayakpaul Aug 17, 2023

sayakpaul left a comment

DN6 commented Aug 17, 2023

sayakpaul commented Aug 17, 2023

sayakpaul commented Aug 17, 2023

patrickvonplaten Aug 22, 2023

DN6 Aug 23, 2023

	def test_local_save_load_index(self):
	prompt = "hello"
	for variant in [None, "fp16"]:
	for use_safe in [True, False]:
	pipe = StableDiffusionPipeline.from_pretrained(
	"hf-internal-testing/tiny-stable-diffusion-pipe-indexes",
	variant=variant,
	use_safetensors=use_safe,
	safety_checker=None,
	)
	pipe = pipe.to(torch_device)
	generator = torch.manual_seed(0)
	out = pipe(prompt, num_inference_steps=2, generator=generator, output_type="numpy").images

	with tempfile.TemporaryDirectory() as tmpdirname:
	pipe.save_pretrained(tmpdirname)
	pipe_2 = StableDiffusionPipeline.from_pretrained(
	tmpdirname, safe_serialization=use_safe, variant=variant
	)
	pipe_2 = pipe_2.to(torch_device)

	generator = torch.manual_seed(0)

	out_2 = pipe_2(prompt, num_inference_steps=2, generator=generator, output_type="numpy").images

	assert np.max(np.abs(out - out_2)) < 1e-3

[Safetensors] Make safetensors the default way of saving weights #4235

[Safetensors] Make safetensors the default way of saving weights #4235

Conversation

patrickvonplaten commented Jul 24, 2023

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Jul 24, 2023 • edited Loading

patrickvonplaten commented Aug 7, 2023

patrickvonplaten commented Aug 14, 2023

DN6 commented Aug 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

DN6 commented Aug 17, 2023

sayakpaul commented Aug 17, 2023

DreamBooth

ControlNet

sayakpaul commented Aug 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jul 24, 2023 •

edited

Loading