From b875668b66f952928bce89c5dbb7d6d1afa47410 Mon Sep 17 00:00:00 2001 From: Nathan Lambert Date: Wed, 9 Nov 2022 13:32:35 -0800 Subject: [PATCH 1/5] add docs for multi-modal --- README.md | 9 +++++++-- docs/source/_toctree.yml | 2 ++ docs/source/using-diffusers/multi_modal.mdx | 21 +++++++++++++++++++++ 3 files changed, 30 insertions(+), 2 deletions(-) create mode 100644 docs/source/using-diffusers/multi_modal.mdx diff --git a/README.md b/README.md index 5c7b911c2ede..58dc5be8d0d5 100644 --- a/README.md +++ b/README.md @@ -353,7 +353,8 @@ Textual Inversion is a technique for capturing novel concepts from a small numbe ## Stable Diffusion Community Pipelines -The release of Stable Diffusion as an open source model has fostered a lot of interesting ideas and experimentation. Our [Community Examples folder](https://github.com/huggingface/diffusers/tree/main/examples/community) contains many ideas worth exploring, like interpolating to create animated videos, using CLIP Guidance for additional prompt fidelity, term weighting, and much more! [Take a look](https://huggingface.co/docs/diffusers/using-diffusers/custom_pipeline_overview) and [contribute your own](https://huggingface.co/docs/diffusers/using-diffusers/contribute_pipeline). +The release of Stable Diffusion as an open source model has fostered a lot of interesting ideas and experimentation. +Our [Community Examples folder](https://github.com/huggingface/diffusers/tree/main/examples/community) contains many ideas worth exploring, like interpolating to create animated videos, using CLIP Guidance for additional prompt fidelity, term weighting, and much more! [Take a look](https://huggingface.co/docs/diffusers/using-diffusers/custom_pipeline_overview) and [contribute your own](https://huggingface.co/docs/diffusers/using-diffusers/contribute_pipeline). ## Other Examples @@ -402,10 +403,14 @@ image.save("ddpm_generated_image.png") - [Unconditional Latent Diffusion](https://huggingface.co/CompVis/ldm-celebahq-256) - [Unconditional Diffusion with continuous scheduler](https://huggingface.co/google/ncsnpp-ffhq-1024) -**Other Notebooks**: +**Other Image Notebooks**: * [image-to-image generation with Stable Diffusion](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/image_2_image_using_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), * [tweak images via repeated Stable Diffusion seeds](https://colab.research.google.com/github/pcuenca/diffusers-examples/blob/main/notebooks/stable-diffusion-seeds.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), +**Diffusers for Other Modalities**: +* [Molecule conformation generation](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/geodiff_molecule_conformation.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), +* [Model-based reinforcement learning](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), + ### Web Demos If you just want to play around with some web demos, you can try out the following 🚀 Spaces: | Model | Hugging Face Spaces | diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index d8efb5eee3d3..84b8b423c5bb 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -29,6 +29,8 @@ - local: using-diffusers/contribute_pipeline title: "How to contribute a Pipeline" title: "Pipelines for Inference" + - sections: using-diffusers/multi_modal + title: "Other Modalities" title: "Using Diffusers" - sections: - local: optimization/fp16 diff --git a/docs/source/using-diffusers/multi_modal.mdx b/docs/source/using-diffusers/multi_modal.mdx new file mode 100644 index 000000000000..93cfb23b1c65 --- /dev/null +++ b/docs/source/using-diffusers/multi_modal.mdx @@ -0,0 +1,21 @@ + + +# Using Diffusers with other modalities + +Diffusers is in the process of expanding to modalities other than images. + +To try some of these in colab, please look at the following examples: +* [Molecule conformation generation](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/geodiff_molecule_conformation.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), +* [Model-based reinforcement learning](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), + +Alternatively, the [`DanceDiffusionPipeline`] can be used to generate audio rapidly! \ No newline at end of file From 4ec17aad0e527f19527df38a5f9232d00ac068e3 Mon Sep 17 00:00:00 2001 From: Nathan Lambert Date: Wed, 9 Nov 2022 13:47:27 -0800 Subject: [PATCH 2/5] many changes --- docs/source/_toctree.yml | 14 ++++++++++++-- docs/source/api/experimental/rl.mdx | 13 +++++++++++++ docs/source/using-diffusers/audio.mdx | 16 ++++++++++++++++ .../{multi_modal.mdx => other-modalities.mdx} | 5 ++--- docs/source/using-diffusers/rl.mdx | 19 +++++++++++++++++++ 5 files changed, 62 insertions(+), 5 deletions(-) create mode 100644 docs/source/api/experimental/rl.mdx create mode 100644 docs/source/using-diffusers/audio.mdx rename docs/source/using-diffusers/{multi_modal.mdx => other-modalities.mdx} (71%) create mode 100644 docs/source/using-diffusers/rl.mdx diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 84b8b423c5bb..20ecce4b8b98 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -29,8 +29,14 @@ - local: using-diffusers/contribute_pipeline title: "How to contribute a Pipeline" title: "Pipelines for Inference" - - sections: using-diffusers/multi_modal - title: "Other Modalities" + - sections: + - local: using-diffusers/rl + title: "Reinforcement Learning" + - local: using-diffusers/audio + title: "Audio" + - local: using-diffusers/other + title: "Other Modalities" + title: "Beyond Images" title: "Using Diffusers" - sections: - local: optimization/fp16 @@ -105,4 +111,8 @@ - local: api/pipelines/repaint title: "RePaint" title: "Pipelines" + - sections: + - local: api/experimental/rl + title: "RL Planning" + title: "Experimental Features" title: "API" diff --git a/docs/source/api/experimental/rl.mdx b/docs/source/api/experimental/rl.mdx new file mode 100644 index 000000000000..b127b4752573 --- /dev/null +++ b/docs/source/api/experimental/rl.mdx @@ -0,0 +1,13 @@ + + +# TODO \ No newline at end of file diff --git a/docs/source/using-diffusers/audio.mdx b/docs/source/using-diffusers/audio.mdx new file mode 100644 index 000000000000..5a5c2241ca75 --- /dev/null +++ b/docs/source/using-diffusers/audio.mdx @@ -0,0 +1,16 @@ + + +# Using Diffusers for audio + +The [`DanceDiffusionPipeline`] can be used to generate audio rapidly! +More coming soon! \ No newline at end of file diff --git a/docs/source/using-diffusers/multi_modal.mdx b/docs/source/using-diffusers/other-modalities.mdx similarity index 71% rename from docs/source/using-diffusers/multi_modal.mdx rename to docs/source/using-diffusers/other-modalities.mdx index 93cfb23b1c65..046f91b54ce7 100644 --- a/docs/source/using-diffusers/multi_modal.mdx +++ b/docs/source/using-diffusers/other-modalities.mdx @@ -14,8 +14,7 @@ specific language governing permissions and limitations under the License. Diffusers is in the process of expanding to modalities other than images. -To try some of these in colab, please look at the following examples: -* [Molecule conformation generation](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/geodiff_molecule_conformation.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), +Currently, the only example is for [molecule conformation](https://www.nature.com/subjects/molecular-conformation#:~:text=Definition,to%20changes%20in%20their%20environment.) generation. * [Model-based reinforcement learning](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), -Alternatively, the [`DanceDiffusionPipeline`] can be used to generate audio rapidly! \ No newline at end of file +More coming soon! \ No newline at end of file diff --git a/docs/source/using-diffusers/rl.mdx b/docs/source/using-diffusers/rl.mdx new file mode 100644 index 000000000000..c6136ada4034 --- /dev/null +++ b/docs/source/using-diffusers/rl.mdx @@ -0,0 +1,19 @@ + + +# Using Diffusers for reinforcement learning + +Support for one RL model and related pipelines is included in the `experimental` source of diffusers. + +To try some of this in colab, please look at the following example: +* [Model-based reinforcement learning](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), + From 794fd0e6a023a755e2246a423c393ab5bccfe02d Mon Sep 17 00:00:00 2001 From: Nathan Lambert Date: Wed, 9 Nov 2022 13:52:41 -0800 Subject: [PATCH 3/5] fix docs build --- docs/source/_toctree.yml | 2 +- docs/source/api/experimental/rl.mdx | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 20ecce4b8b98..75569ce8b231 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -34,7 +34,7 @@ title: "Reinforcement Learning" - local: using-diffusers/audio title: "Audio" - - local: using-diffusers/other + - local: using-diffusers/other-modalities title: "Other Modalities" title: "Beyond Images" title: "Using Diffusers" diff --git a/docs/source/api/experimental/rl.mdx b/docs/source/api/experimental/rl.mdx index b127b4752573..65abb06e7523 100644 --- a/docs/source/api/experimental/rl.mdx +++ b/docs/source/api/experimental/rl.mdx @@ -10,4 +10,6 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -# TODO \ No newline at end of file +# TODO + +Coming soon! \ No newline at end of file From ff54a4d791bc2ccc41c90230f85b7462e4422e77 Mon Sep 17 00:00:00 2001 From: Nathan Lambert Date: Wed, 9 Nov 2022 14:04:20 -0800 Subject: [PATCH 4/5] fix links --- docs/source/_toctree.yml | 2 +- docs/source/using-diffusers/other-modalities.mdx | 2 +- docs/source/using-diffusers/rl.mdx | 3 +-- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 75569ce8b231..efbcbf04822d 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -36,7 +36,7 @@ title: "Audio" - local: using-diffusers/other-modalities title: "Other Modalities" - title: "Beyond Images" + title: "Taking Diffusers Beyond Images" title: "Using Diffusers" - sections: - local: optimization/fp16 diff --git a/docs/source/using-diffusers/other-modalities.mdx b/docs/source/using-diffusers/other-modalities.mdx index 046f91b54ce7..b3a4e041dab0 100644 --- a/docs/source/using-diffusers/other-modalities.mdx +++ b/docs/source/using-diffusers/other-modalities.mdx @@ -15,6 +15,6 @@ specific language governing permissions and limitations under the License. Diffusers is in the process of expanding to modalities other than images. Currently, the only example is for [molecule conformation](https://www.nature.com/subjects/molecular-conformation#:~:text=Definition,to%20changes%20in%20their%20environment.) generation. -* [Model-based reinforcement learning](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), +* Generate conformations in Colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/geodiff_molecule_conformation.ipynb) More coming soon! \ No newline at end of file diff --git a/docs/source/using-diffusers/rl.mdx b/docs/source/using-diffusers/rl.mdx index c6136ada4034..6e18e07001b6 100644 --- a/docs/source/using-diffusers/rl.mdx +++ b/docs/source/using-diffusers/rl.mdx @@ -15,5 +15,4 @@ specific language governing permissions and limitations under the License. Support for one RL model and related pipelines is included in the `experimental` source of diffusers. To try some of this in colab, please look at the following example: -* [Model-based reinforcement learning](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg), - +* Model-based reinforcement learning on Colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/reinforcement_learning_with_diffusers.ipynb) ![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg) From 477578a19e7fdd90e0beb9e594261476bd9e81e5 Mon Sep 17 00:00:00 2001 From: Nathan Lambert Date: Thu, 17 Nov 2022 10:20:29 -0800 Subject: [PATCH 5/5] Update docs/source/using-diffusers/other-modalities.mdx Co-authored-by: Pedro Cuenca --- docs/source/using-diffusers/other-modalities.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/using-diffusers/other-modalities.mdx b/docs/source/using-diffusers/other-modalities.mdx index b3a4e041dab0..1dc0877adb24 100644 --- a/docs/source/using-diffusers/other-modalities.mdx +++ b/docs/source/using-diffusers/other-modalities.mdx @@ -14,7 +14,7 @@ specific language governing permissions and limitations under the License. Diffusers is in the process of expanding to modalities other than images. -Currently, the only example is for [molecule conformation](https://www.nature.com/subjects/molecular-conformation#:~:text=Definition,to%20changes%20in%20their%20environment.) generation. +Currently, one example is for [molecule conformation](https://www.nature.com/subjects/molecular-conformation#:~:text=Definition,to%20changes%20in%20their%20environment.) generation. * Generate conformations in Colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/geodiff_molecule_conformation.ipynb) More coming soon! \ No newline at end of file