-
Notifications
You must be signed in to change notification settings - Fork 5.9k
Add UniDiffuser model and pipeline #2963
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 7 commits
Commits
Show all changes
334 commits
Select commit
Hold shift + click to select a range
115e382
Fix a bug of pano when not doing CFG (#3030)
ernestchu 10c54cb
Text2video zero refinements (#3070)
19and99 945f300
Release: v0.15.0
patrickvonplaten 322b5cb
[Tests] Speed up panorama tests (#3067)
sayakpaul af0c3a7
[Post release] v0.16.0dev (#3072)
patrickvonplaten 7a1d100
Adds profiling flags, computes train metrics average. (#3053)
andsteing bbabf3f
[Pipelines] Make sure that None functions are correctly not saved (#3…
patrickvonplaten 068d6b4
doc string example remove from_pt (#3083)
yiyixuxu 9dd6058
[Tests] parallelize (#3078)
patrickvonplaten 74907ee
Throw deprecation warning for return_cached_folder (#3092)
patrickvonplaten 0be9f8b
Allow SD attend and excite pipeline to work with any size output imag…
jcoffland d6ae0ae
[docs] Update community pipeline docs (#2989)
stevhliu 7f3cb6d
Add to support Guess Mode for StableDiffusionControlnetPipleline (#2998)
takuma104 1184b36
fix default value for attend-and-excite (#3099)
yiyixuxu fa6a6b4
remvoe one line as requested by gc team (#3077)
yiyixuxu a256f84
ddpm custom timesteps (#3007)
williamberman e9cb03e
Fix breaking change in `pipeline_stable_diffusion_controlnet.py` (#3118)
remorses c98e41d
Add global pooling to controlnet (#3121)
patrickvonplaten 653b3c1
[Bug fix] Fix img2img processor with safety checker (#3127)
patrickvonplaten 7fa3b6c
[Bug fix] Make sure correct timesteps are chosen for img2img (#3128)
patrickvonplaten cc14690
Improve deprecation warnings (#3131)
patrickvonplaten 6f12a36
Fix config deprecation (#3129)
patrickvonplaten 16ddd8b
feat: verfication of multi-gpu support for select examples. (#3126)
sayakpaul 07731e9
speed up attend-and-excite fast tests (#3079)
yiyixuxu 7a39b0f
Optimize log_validation in train_controlnet_flax (#3110)
cgarciae 7ae597f
make style
patrickvonplaten 00a5e55
Correct textual inversion readme (#3145)
patrickvonplaten ff5b99b
Add unet act fn to other model components (#3136)
williamberman c8eaea5
class labels timestep embeddings projection dtype cast (#3137)
williamberman 1fac211
[ckpt loader] Allow loading the Inpaint and Img2Img pipelines, while …
cmdr2 f3300a8
add from_ckpt method as Mixin (#2318)
1lint fc3760d
Add TensorRT SD/txt2img Community Pipeline to diffusers along with Te…
asfiyab-nvidia 6058378
Correct `Transformer2DModel.forward` docstring (#3074)
offchan42 e5335f3
Update pipeline_stable_diffusion_inpaint_legacy.py (#2903)
hwuebben 4afb911
Modified altdiffusion pipline to support altdiffusion-m18 (#2993)
superhero-7 b320c6b
controlnet training resize inputs to multiple of 8 (#3135)
williamberman 3417b1f
adding custom diffusion training to diffusers examples (#3031)
nupurkmr9 4a6aee9
make style
patrickvonplaten 3306b04
Update custom_diffusion.mdx (#3165)
abd21da
Added distillation for quantization example on textual inversion. (#2…
XinyuYe-Intel f99a9ff
Update Noise Autocorrelation Loss Function for Pix2PixZero Pipeline (…
clarencechen 206b9b6
[DreamBooth] add text encoder LoRA support in the DreamBooth training…
sayakpaul 6ad4392
Update Habana Gaudi documentation (#3169)
regisss 4366b0c
Add model offload to x4 upscaler (#3187)
patrickvonplaten e21784e
[docs] Deterministic algorithms (#3172)
stevhliu 006ae03
Update custom_diffusion.mdx to credit the author (#3163)
sayakpaul dac4d4a
Fix TensorRT community pipeline device set function (#3157)
asfiyab-nvidia c98a055
make `from_flax` work for controlnet (#3161)
yiyixuxu 15a90e2
[docs] Clarify training args (#3146)
stevhliu cf35763
Multi Vector Textual Inversion (#3144)
patrickvonplaten c729403
Add `Karras sigmas` to HeunDiscreteScheduler (#3160)
youssefadr 43c90b0
[AudioLDM] Fix dtype of returned waveform (#3189)
sanchit-gandhi 49c9b4c
Fix bug in train_dreambooth_lora (#3183)
crywang a69502f
[Community Pipelines] Update lpw_stable_diffusion pipeline (#3197)
SkyTNT 3267649
Make sure VAE attention works with Torch 2_0 (#3200)
patrickvonplaten de05ea0
Revert "[Community Pipelines] Update lpw_stable_diffusion pipeline" (…
williamberman 8953209
[Bug fix] Fix batch size attention head size mismatch (#3214)
patrickvonplaten 4e03663
fix mixed precision training on train_dreambooth_inpaint_lora (#3138)
themrzmaster 167cb7a
adding enable_vae_tiling and disable_vae_tiling functions (#3225)
init-22 0431637
Add ControlNet v1.1 docs (#3226)
patrickvonplaten 9e2f445
Fix issue in maybe_convert_prompt (#3188)
pdoane 81950af
Sync cache version check from transformers (#3179)
ychfan 711119a
Fix docs text inversion (#3166)
patrickvonplaten 416f31a
add model (#3230)
patrickvonplaten 7ad77dd
Allow return pt x4 (#3236)
patrickvonplaten 3acc879
Allow fp16 attn for x4 upscaler (#3239)
patrickvonplaten 4c73947
fix fast test (#3241)
patrickvonplaten 81d7eba
Adds a document on token merging (#3208)
sayakpaul f83fbbd
[AudioLDM] Update docs to use updated ckpt (#3240)
sanchit-gandhi 4cc60b5
Release: v0.16.0
patrickvonplaten 163c33b
Post release for 0.16.0 (#3244)
patrickvonplaten 59986b6
[docs] only mention one stage (#3246)
pcuenca a640f1b
Write model card in controlnet training script (#3229)
pcuenca 7880ed7
[2064]: Add stochastic sampler (sample_dpmpp_sde) (#3020)
nipunjindal 8def721
[Stochastic Sampler][Slow Test]: Cuda test fixes (#3257)
nipunjindal 97cf386
Remove required from tracker_project_name (#3260)
pcuenca cf2bf70
adding required parameters while calling the get_up_block and get_dow…
init-22 76e5941
[docs] Update interface in repaint.mdx (#3119)
ernestchu 1147c76
Update IF name to XL (#3262)
apolinario cd13b10
fix typo in score sde pipeline (#3132)
fecet 5f3b10a
Fix typo in textual inversion JAX training script (#3123)
jairtrejo 9585b23
AudioDiffusionPipeline - fix encode method after config changes (#3114)
teticio 79706a7
Revert "Revert "[Community Pipelines] Update lpw_stable_diffusion pip…
patrickvonplaten 12868b1
Fix community pipelines (#3266)
patrickvonplaten 2c87f65
update notebook (#3259)
yiyixuxu a80f696
[docs] add notes for stateful model changes (#3252)
williamberman 72a8467
[LoRA] quality of life improvements in the loading semantics and docs…
sayakpaul 716c255
[Community Pipelines] EDICT pipeline implementation (#3153)
Joqsan c1dce20
[Docs]zh translated docs update (#3245)
DrDavidS 5151f21
Update logging.mdx (#2863)
tolgacangoz ffe6e92
Add multiple conditions to StableDiffusionControlNetInpaintPipeline (…
timegate 10d856a
Let's make sure that dreambooth always uploads to the Hub (#3272)
patrickvonplaten 029a28f
Diffedit Zero-Shot Inpainting Pipeline (#2837)
clarencechen fd47d7c
add constant learning rate with custom rule (#3133)
jason9075 220657b
Allow disabling torch 2_0 attention (#3273)
patrickvonplaten 1b95720
[doc] add link to training script (#3271)
yiyixuxu 08fbaaf
temp disable spectogram diffusion tests (#3278)
williamberman 799015c
Changed sample[0] to images[0] (#3304)
IliaLarchenko c38d004
Typo in tutorial (#3295)
IliaLarchenko 6a84a74
Torch compile graph fix (#3286)
patrickvonplaten 863bb75
Postprocessing refactor img2img (#3268)
yiyixuxu c8cc4f0
[Torch 2.0 compile] Fix more torch compile breaks (#3313)
patrickvonplaten 6e8d065
fix: scale_lr and sync example readme and docs. (#3299)
sayakpaul d38b4d9
Update stable_diffusion.mdx (#3310)
mu94-csl 0d22064
Fix missing variable assign in DeepFloyd-IF-II (#3315)
gitmylo 5a75a8a
Correct doc build for patch releases (#3316)
patrickvonplaten 5ea3424
Add Stable Diffusion RePaint to community pipelines (#3320)
Markus-Pobitzer 7815c41
Fix multistep dpmsolver for cosine schedule (suitable for deepfloyd-i…
LuChengTHU 0e8f4f0
[docs] Improve LoRA docs (#3311)
stevhliu 7929587
Added input pretubation (#3292)
isamu-isozaki 3e8d3d8
Update write_own_pipeline.mdx (#3323)
csaybar b4aa419
update controlling generation doc with latest goodies. (#3321)
sayakpaul 458847e
[Quality] Make style (#3341)
patrickvonplaten 140ab74
Fix config dpm (#3343)
patrickvonplaten 652dbaa
Add the SDE variant of DPM-Solver and DPM-Solver++ (#3344)
LuChengTHU 1d213de
Add upsample_size to AttnUpBlock2D, AttnDownBlock2D (#3275)
will-rice 434b255
Add UniDiffuser classes to __init__ files, modify transformer block t…
dg845 7097dd7
Update fast tests to use test checkpoints stored on the hub and to be…
dg845 fc85263
Fix code with make style.
dg845 9d39bef
Revert "Fix code style with make style."
dg845 1cb726a
Merge branch 'main' into unidiffuser-pipeline
dg845 e62b32a
Add self.image_encoder, self.text_decoder to list of models to offloa…
dg845 fc540b5
Fix code quality with make style.
dg845 54c495f
Support using a data type embedding for UniDiffuser-v1.
dg845 8dd7b0b
Add fast test for checking UniDiffuser-v1 sampling.
dg845 34a40ad
Make changes so that the repository consistency tests pass.
dg845 0cddc3c
Add UniDiffuser dummy objects via make fix-copies.
dg845 16fd515
Fix bugs and make improvements to the UniDiffuser pipeline:
dg845 5728328
Fix code style with make style.
dg845 abd6fca
Add/edit docstrings for added classes and public pipeline methods. Al…
dg845 ae7d549
Add documentation for UniDiffuser and fix some typos/formatting in do…
dg845 2b92111
Fix code with make style.
dg845 a46e1ec
Refactor and improve the UniDiffuser convert_from_ckpt.py script.
dg845 a7f50f4
Move the UniDiffusers convert_from_ckpy.py script to diffusers/script…
dg845 8a57342
Fix code quality via make style.
dg845 006ab49
Improve UniDiffuser slow tests.
dg845 8f2d325
make style
dg845 a54d631
Fix some typos in the UniDiffuser docs.
dg845 fa9e387
Remove outdated logic based on transformers version in UniDiffuser pi…
dg845 19a20a5
Remove dependency on einops by refactoring einops operations to pure …
dg845 28dda62
make style
dg845 de8794c
Add slow test on full checkpoint for joint mode and correct expected …
dg845 7242f1b
make style
dg845 1a58958
Fix mixed precision issue by wrapping the offending code with the tor…
dg845 f36df41
Revert "Fix mixed precision issue by wrapping the offending code with…
dg845 1bc2b91
Add fast test for CUDA/fp16 model behavior (currently failing).
dg845 5341450
Fix the mixed precision issue and add additional tests of the pipelin…
dg845 b1a6f22
make style
dg845 54cfa3b
Use a CLIPVisionModelWithProjection instead of CLIPVisionModel for im…
dg845 10e3774
Make style and remove some testing code.
dg845 4d656b5
Fix shape errors for the 'joint' and 'img2text' modes.
dg845 be4abff
Fix tests and remove some testing code.
dg845 848b7e6
Add option to use fixed latents for UniDiffuserPipelineSlowTests and …
dg845 e56fab2
Improve UniDiffuser docs, particularly the usage examples, and improv…
dg845 ecaf07f
make style
dg845 c161e29
Fix examples to load model in float16.
dg845 926c7fb
In image-to-text mode, sample from the autoencoder moment distributio…
dg845 edbadcc
make style
dg845 6b35c03
When encoding the image using the VAE, scale the image latents by the…
dg845 f46593e
make style
dg845 ec7fb87
Clean up code and make slow tests pass.
dg845 029c96c
make fix-copies
patrickvonplaten 6644d11
[docs] Fix docstring (#3334)
stevhliu c221086
if dreambooth lora (#3360)
williamberman f670e08
Postprocessing refactor all others (#3337)
yiyixuxu 7266fc1
[docs] Improve safetensors docstring (#3368)
stevhliu 4b76097
add: a warning message when using xformers in a PT 2.0 env. (#3365)
sayakpaul 6e297b4
StableDiffusionInpaintingPipeline - resize image w.r.t height and wid…
rupertmenneer fec7bd1
make style
patrickvonplaten e162d49
[docs] Adapt a model (#3326)
stevhliu caa080c
[docs] Load safetensors (#3333)
stevhliu 75c2f75
make style
patrickvonplaten f0c0f00
[Docs] Fix stable_diffusion.mdx typo (#3398)
sudowind 42eabb8
Support ControlNet v1.1 shuffle properly (#3340)
takuma104 1965acf
[Tests] better determinism (#3374)
sayakpaul 7b7b6bf
[docs] Add transformers to install (#3388)
stevhliu c998614
[deepspeed] partial ZeRO-3 support (#3076)
stas00 1085f3e
Add omegaconf for tests (#3400)
patrickvonplaten 188de89
Fix various bugs with LoRA Dreambooth and Dreambooth script (#3353)
patrickvonplaten 89a8f73
Fix docker file (#3402)
patrickvonplaten cb4016d
fix: deepseepd_plugin retrieval from accelerate state (#3410)
sayakpaul 41763f6
[Docs] Add `sigmoid` beta_scheduler to docstrings of relevant Schedul…
Laurent2916 622c3c6
Don't install accelerate and transformers from source (#3415)
patrickvonplaten f001e07
Don't install transformers and accelerate from source (#3414)
patrickvonplaten 80c2e55
Improve fast tests (#3416)
patrickvonplaten d749d57
attention refactor: the trilogy (#3387)
williamberman 6ce7f8f
[Docs] update the PT 2.0 optimization doc with latest findings (#3370)
sayakpaul 480b525
Fix style rendering (#3433)
pcuenca d3b3855
unCLIP scheduler do not use note (#3417)
williamberman 9a31cce
Replace deprecated command with environment file (#3409)
jongwooo df625f4
fix warning message pipeline loading (#3446)
patrickvonplaten 8065462
add stable diffusion tensorrt img2img pipeline (#3419)
asfiyab-nvidia d5f65dc
Refactor controlnet and add img2img and inpaint (#3386)
patrickvonplaten 2b11926
[Scheduler] DPM-Solver (++) Inverse Scheduler (#3335)
clarencechen 63abfce
[Docs] Fix incomplete docstring for resnet.py (#3438)
Laurent2916 32162aa
fix tiled vae blend extent range (#3384)
superlabs-dev 3f5a176
Small update to "Next steps" section (#3443)
pcuenca 3019e08
Allow arbitrary aspect ratio in IFSuperResolutionPipeline (#3298)
devxpy bb1172b
Adding 'strength' parameter to StableDiffusionInpaintingPipeline (#3…
rupertmenneer 2a16062
[WIP] Bugfix - Pipeline.from_pretrained is broken when the pipeline i…
vimarshc 68a97bd
Fix gradient checkpointing bugs in freezing part of models (requires_…
IrisRainbowNeko ce072e0
Make dreambooth lora more robust to orig unet (#3462)
patrickvonplaten ee10c71
Reduce peak VRAM by releasing large attention tensors (as soon as the…
cmdr2 9388b3a
Add min snr to text2img lora training script (#3459)
wfng92 2ef1b00
Add inpaint lora scale support (#3460)
Glaceon-Hyy 62d9c72
[From ckpt] Fix from_ckpt (#3466)
patrickvonplaten 368f9ad
Update full dreambooth script to work with IF (#3425)
williamberman 68441bf
Add IF dreambooth docs (#3470)
williamberman eb7ae28
parameterize pass single args through tuple (#3477)
williamberman bb1e25a
attend and excite tests disable determinism on the class level (#3478)
williamberman 9a195d7
dreambooth docs torch.compile note (#3471)
williamberman 09ddb88
add: if entry in the dreambooth training docs. (#3472)
sayakpaul e36596c
[docs] Textual inversion inference (#3473)
stevhliu 147da83
[docs] Distributed inference (#3376)
stevhliu 2d8e089
[{Up,Down}sample1d] explicit view kernel size as number elements in f…
williamberman 53e37b8
mps & onnx tests rework (#3449)
pcuenca 8eae86d
[Attention processor] Better warning message when shifting to `AttnPr…
sayakpaul 55ca69b
[Docs] add note on local directory path. (#3397)
sayakpaul a8219e8
Refactor full determinism (#3485)
patrickvonplaten a3e1153
Fix DPM single (#3413)
patrickvonplaten 01b42e4
Add `use_Karras_sigmas` to DPMSolverSinglestepScheduler (#3476)
Isotr0py d22535a
Adds local_files_only bool to prevent forced online connection (#3486)
w4ffl35 b78e854
make style
patrickvonplaten a9ac5a8
[Docs] Korean translation (optimization, training) (#3488)
tjdtnsu 634cf1f
DataLoader respecting EXIF data in Training Images (#3465)
Ambrosiussen 5782887
make style
patrickvonplaten f61028f
feat: allow disk offload for diffuser models (#3285)
hari10599 30329a2
[Community] reference only control (#3435)
okotaku ca87f4d
Support for cross-attention bias / mask (#2634)
Birch-san cdf38f1
do not scale the initial global step by gradient accumulation steps w…
williamberman 51f0951
Remove CPU latents logic for UniDiffuserPipelineFastTests.
dg845 4ccb2b5
make style
dg845 97e8eef
Revert "Clean up code and make slow tests pass."
dg845 9f7247c
Revert bad commit and clean up code.
dg845 302fde9
add: contributor note.
sayakpaul 9f84416
Batched load of textual inversions (#3277)
pdoane 6326cb7
Revert "add: contributor note."
dg845 6d0f321
Re-add contributor note and refactored fast tests fixed latents code …
dg845 73504c4
make style
dg845 0ed1857
Refactored the code:
dg845 d53026d
make style
dg845 0adb0a8
Remove padding logic from UniDiffuserTextDecoder.generate_beam since …
dg845 43b8894
Update checkpoint id for small test v1 checkpoint to hf-internal-test…
dg845 a5a9dac
make style
dg845 d4b11aa
Make improvements to the documentation.
dg845 98ce17d
Move ImageTextPipelineOutput documentation from /api/pipelines/unidif…
dg845 f8c325a
Change order of arguments for UniDiffuserTextDecoder.generate_beam.
dg845 b4feac8
make style
dg845 4f21661
Merge branch 'main' into unidiffuser-pipeline
dg845 07d68d7
Update docs/source/en/api/pipelines/unidiffuser.mdx
sayakpaul File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
from ...utils import ( | ||
OptionalDependencyNotAvailable, | ||
is_torch_available, | ||
is_transformers_available, | ||
is_transformers_version, | ||
) | ||
|
||
|
||
try: | ||
if not (is_transformers_available() and is_torch_available() and is_transformers_version(">=", "4.25.0")): | ||
raise OptionalDependencyNotAvailable() | ||
except OptionalDependencyNotAvailable: | ||
from ...utils.dummy_torch_and_transformers_objects import ( | ||
ImageTextPipelineOutput, | ||
UniDiffuserPipeline, | ||
) | ||
else: | ||
from .modeling_text_decoder import UniDiffuserTextDecoder | ||
from .modeling_uvit import UniDiffuserModel, UTransformer2DModel | ||
from .pipeline_unidiffuser import ImageTextPipelineOutput, UniDiffuserPipeline |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed.