Skip to content

Adding better way to define multiple concepts and also validation capabilities. #3807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Jul 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
92814a3
- Added validation parameters
mauricio-repetto Jun 14, 2023
728f28b
- Fixed bad logic for class data root directories
mauricio-repetto Jun 14, 2023
2f34eb6
Defaulting validation_steps to None for an easier logic
mauricio-repetto Jun 15, 2023
1f007a7
Fixed multiple validation prompts
mauricio-repetto Jun 15, 2023
c5756c4
Fixed bug on validation negative prompt
mauricio-repetto Jun 15, 2023
f171686
Changed validation logic for tracker.
mauricio-repetto Jun 15, 2023
c23faef
Added uuid for validation image labeling
mauricio-repetto Jun 15, 2023
d1885dc
Fix error when comparing validation prompts and validation negative p…
mauricio-repetto Jun 15, 2023
f1a5f8f
Improved error message when negative prompts for validation are more …
mauricio-repetto Jun 15, 2023
6eb2587
- Changed image tracking number from epoch to global_step
mauricio-repetto Jun 15, 2023
e2e7f25
Added some validations more when using concept_list parameter and the…
mauricio-repetto Jun 15, 2023
5144944
Fixed error message
mauricio-repetto Jun 15, 2023
04f4518
Added more validations for validation parameters
mauricio-repetto Jun 15, 2023
aa5bb74
Improved messaging for errors
mauricio-repetto Jun 15, 2023
d01502b
Fixed validation error for parameters with default values
mauricio-repetto Jun 15, 2023
29be786
- Added train step to image name for validation
mauricio-repetto Jun 15, 2023
e325150
- Added train step to image's name for validation
mauricio-repetto Jun 15, 2023
1338dcd
Updated README.md file.
mauricio-repetto Jun 16, 2023
e0dfaa4
reverted back original script of train_dreambooth.py
mauricio-repetto Jun 16, 2023
22a202d
reverted back original script of train_dreambooth.py
mauricio-repetto Jun 16, 2023
3c75d22
left one blank line at the eof
mauricio-repetto Jun 16, 2023
f7aa125
reverted back setup.py
mauricio-repetto Jun 16, 2023
3767a62
reverted back setup.py
mauricio-repetto Jun 16, 2023
6f0d794
added same logic for when parameters for prior preservation are used …
mauricio-repetto Jun 16, 2023
e84cac8
Merge branch 'main' into main
patrickvonplaten Jun 28, 2023
ef99e82
Ran black formatter.
mauricio-repetto Jun 28, 2023
0485253
Merge branch 'main' of github.com:mauricio-repetto/diffusers
mauricio-repetto Jun 28, 2023
42e3d11
fixed a few strings
mauricio-repetto Jun 28, 2023
5c775ca
fixed import sort with isort and removed fstrings without placeholder
mauricio-repetto Jun 28, 2023
2753220
fixed import order with ruff (since with isort wasn't ok)
mauricio-repetto Jun 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions examples/research_projects/multi_subject_dreambooth/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,53 @@ This example shows training for 2 subjects, but please note that the model can b

Note also that in this script, `sks` and `t@y` were used as tokens to learn the new subjects ([this thread](https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/issues/71) inspired the use of `t@y` as our second identifier). However, there may be better rare tokens to experiment with, and results also seemed to be good when more intuitive words are used.

**Important**: New parameters are added to the script, making possible to validate the progress of the training by
generating images at specified steps. Taking also into account that a comma separated list in a text field for a prompt
it's never a good idea (simply because it is very common in prompts to have them as part of a regular text) we
introduce the `concept_list` parameter: allowing to specify a json-like file where you can define the different
configuration for each subject that you want to train.

An example of how to generate the file:
```python
import json

# here we are using parameters for prior-preservation and validation as well.
concepts_list = [
{
"instance_prompt": "drawing of a t@y meme",
"class_prompt": "drawing of a meme",
"instance_data_dir": "/some_folder/meme_toy",
"class_data_dir": "/data/meme",
"validation_prompt": "drawing of a t@y meme about football in Uruguay",
"validation_negative_prompt": "black and white"
},
{
"instance_prompt": "drawing of a sks sir",
"class_prompt": "drawing of a sir",
"instance_data_dir": "/some_other_folder/sir_sks",
"class_data_dir": "/data/sir",
"validation_prompt": "drawing of a sks sir with the Uruguayan sun in his chest",
"validation_negative_prompt": "an old man",
"validation_guidance_scale": 20,
"validation_number_images": 3,
"validation_inference_steps": 10
}
]

with open("concepts_list.json", "w") as f:
json.dump(concepts_list, f, indent=4)
```
And then just point to the file when executing the script:

```bash
# exports...
accelerate launch train_multi_subject_dreambooth.py \
# more parameters...
--concepts_list="concepts_list.json"
```

You can use the helper from the script to get a better sense of each parameter.

### Inference

Once you have trained a model using above command, the inference can be done simply using the `StableDiffusionPipeline`. Make sure to include the `identifier`(e.g. sks in above example) in your prompt.
Expand Down
Loading