Skip to content

Model offload #3889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jun 30, 2023
Merged

Model offload #3889

merged 4 commits into from
Jun 30, 2023

Conversation

pcuenca
Copy link
Member

@pcuenca pcuenca commented Jun 28, 2023 •
edited
Loading

< 8GB with model offload (without refiner)

@pcuenca pcuenca marked this pull request as draft June 28, 2023 10:44
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jun 28, 2023 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@pcuenca
Copy link
Member Author

pcuenca commented Jun 28, 2023

When running the refiner we have a peak of ~9 GB because the call sequence is:

text_encoder_2 -> vae.encode -> [unet loop] -> vae.decode

However the offload chain goes text_encoder_2 -> unet -> vae. Therefore, the vae.encode call does not automatically offload the text encoder. If it did, we'd have < 8GB memory consumption.

Not sure if it makes sense to hardcode a .to("cpu") in this case. How do you feel about it @patrickvonplaten @sayakpaul @williamberman

pcuenca and others added 2 commits June 28, 2023 18:52
@pcuenca
Hardcode encoder offload on img2img vae encode
6fd4aaa 
Saves some GPU RAM in img2img / refiner tasks so it remains below 8 GB.
@patrickvonplaten
Merge branch 'sd_xl' into sd_xl_offload
fe3464f
@patrickvonplaten patrickvonplaten marked this pull request as ready for review June 30, 2023 10:37
@patrickvonplaten patrickvonplaten merged commit 558ef96 into sd_xl Jun 30, 2023
@patrickvonplaten patrickvonplaten deleted the sd_xl_offload branch June 30, 2023 10:38
@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

patrickvonplaten added a commit that referenced this pull request Jul 6, 2023
@patrickvonplaten @pcuenca @sayakpaul
[SD-XL] Add new pipelines (#3859)
bc9a8ce 
* Add new text encoder

* add transformers depth

* More

* Correct conversion script

* Fix more

* Fix more

* Correct more

* correct text encoder

* Finish all

* proof that in works in run local xl

* clean up

* Get refiner to work

* Add red castle

* Fix batch size

* Improve pipelines more

* Finish text2image tests

* Add img2img test

* Fix more

* fix import

* Fix embeddings for classic models (#3888)

Fix embeddings for classic SD models.

* Allow multiple prompts to be passed to the refiner (#3895)

* finish more

* Apply suggestions from code review

* add watermarker

* Model offload (#3889)

* Model offload.

* Model offload for refiner / img2img

* Hardcode encoder offload on img2img vae encode

Saves some GPU RAM in img2img / refiner tasks so it remains below 8 GB.

---------

Co-authored-by: Patrick von Platen <[email protected]>

* correct

* fix

* clean print

* Update install warning for `invisible-watermark`

* add: missing docstrings.

* fix and simplify the usage example in img2img.

* fix setup for watermarking.

* Revert "fix setup for watermarking."

This reverts commit 491bc9f.

* fix: watermarking setup.

* fix: op.

* run make fix-copies.

* make sure tests pass

* improve convert

* make tests pass

* make tests pass

* better error message

* fiinsh

* finish

* Fix final test

---------

Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Sayak Paul <[email protected]>
yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request Dec 25, 2023
@patrickvonplaten @pcuenca @sayakpaul
[SD-XL] Add new pipelines (huggingface#3859)
a4967f4 
* Add new text encoder

* add transformers depth

* More

* Correct conversion script

* Fix more

* Fix more

* Correct more

* correct text encoder

* Finish all

* proof that in works in run local xl

* clean up

* Get refiner to work

* Add red castle

* Fix batch size

* Improve pipelines more

* Finish text2image tests

* Add img2img test

* Fix more

* fix import

* Fix embeddings for classic models (huggingface#3888)

Fix embeddings for classic SD models.

* Allow multiple prompts to be passed to the refiner (huggingface#3895)

* finish more

* Apply suggestions from code review

* add watermarker

* Model offload (huggingface#3889)

* Model offload.

* Model offload for refiner / img2img

* Hardcode encoder offload on img2img vae encode

Saves some GPU RAM in img2img / refiner tasks so it remains below 8 GB.

---------

Co-authored-by: Patrick von Platen <[email protected]>

* correct

* fix

* clean print

* Update install warning for `invisible-watermark`

* add: missing docstrings.

* fix and simplify the usage example in img2img.

* fix setup for watermarking.

* Revert "fix setup for watermarking."

This reverts commit 491bc9f.

* fix: watermarking setup.

* fix: op.

* run make fix-copies.

* make sure tests pass

* improve convert

* make tests pass

* make tests pass

* better error message

* fiinsh

* finish

* Fix final test

---------

Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Sayak Paul <[email protected]>
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
@patrickvonplaten @pcuenca @sayakpaul
[SD-XL] Add new pipelines (huggingface#3859)
7aaf47a 
* Add new text encoder

* add transformers depth

* More

* Correct conversion script

* Fix more

* Fix more

* Correct more

* correct text encoder

* Finish all

* proof that in works in run local xl

* clean up

* Get refiner to work

* Add red castle

* Fix batch size

* Improve pipelines more

* Finish text2image tests

* Add img2img test

* Fix more

* fix import

* Fix embeddings for classic models (huggingface#3888)

Fix embeddings for classic SD models.

* Allow multiple prompts to be passed to the refiner (huggingface#3895)

* finish more

* Apply suggestions from code review

* add watermarker

* Model offload (huggingface#3889)

* Model offload.

* Model offload for refiner / img2img

* Hardcode encoder offload on img2img vae encode

Saves some GPU RAM in img2img / refiner tasks so it remains below 8 GB.

---------

Co-authored-by: Patrick von Platen <[email protected]>

* correct

* fix

* clean print

* Update install warning for `invisible-watermark`

* add: missing docstrings.

* fix and simplify the usage example in img2img.

* fix setup for watermarking.

* Revert "fix setup for watermarking."

This reverts commit 491bc9f.

* fix: watermarking setup.

* fix: op.

* run make fix-copies.

* make sure tests pass

* improve convert

* make tests pass

* make tests pass

* better error message

* fiinsh

* finish

* Fix final test

---------

Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Sayak Paul <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pcuenca @HuggingFaceDocBuilderDev @patrickvonplaten