Make SDXL treat the "optional" components as really optional

[ppbrown]

What does this PR do?

This makes the "optional" text modules in StableDiffusionXLPipeline actually function as optional, instead of mandatory.
(There was already code present that suggests you can optionally skip modules: unfortunately, the code does not actually work that way, so this fixes the unfiled bug)

The patch can be verified by running the code mentioned in the target model in the README.md at
https://huggingface.co/opendiffusionai/sdxlone

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x ] Did you read the contributor guideline?
• [ x] Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Core library:

• Pipelines and pipeline callbacks: @yiyixuxu and @asomoza

[asomoza]

@ppbrown thanks for your contribution.

We usually only add additional code to the official pipelines and the core when there's a fully working and released model that it's also popular, to avoid adding more complexity and making it a huge library with unused models.

For what I can see in your model repo:

So, I created this version to experimentally prove the better way.

and in the other model there is this part:

This raw version of the model seems to work great with 3-5 word prompts... but then decays after that. So I'm working on a finetuned version.

both texts suggest that this is just and experiment that needs more finetuning, also it's not that popular right now.

I suggest that we add this as a community pipeline for the moment so people can test your model and we can see then how this goes for the community, also it would make it a lot faster to merge.

[ppbrown]

In discord it was suggested I point out in original code where it has partial support for this concept, to justify this being classed as a bugfix.
Right at the top of the diffs, line 374 in original file,

        text_encoders = (
            [self.text_encoder, self.text_encoder_2] if self.text_encoder is not None else [self.text_encoder_2]
        )

(And then it has the opposite for when text_encoder_2 is None)

[asomoza]

thanks, pinging @DN6 or @sayakpaul. Was the intention with the original SDXL implementation to make the second text encoder optional and skippable?