[RFC] Better support using multiple loggers simultaneously by deprecating LoggerCollection

[ananthsub]

Proposed refactor

Deprecate LoggerCollection:
https://github.com/PyTorchLightning/pytorch-lightning/blob/a6a28e08d22f59f2468ff1049c67507df7519e7c/pytorch_lightning/loggers/base.py#L370-L464

Motivation

The logger collection implements the base logger interface and wraps a sequence of loggers. This way, the trainer always behaves as though a single logger is being used. Such a collection is somewhat handy for writing data, but is much less helpful when reading state from loggers, especially when individual loggers have different state/property values. For instance, It's not clear at all what LoggerCollection.name should be. The concatenation of all its instance loggers names? The first logger's name? This also came up here: #10954

This leads to inconsistencies across components. Namely, Lightning offers no such LoggerCollection-equivalent for Callbacks. The trainer treats them as a flat list, and when hooks need to be called, the Trainer iterates over this list, and calls the callbacks' hooks successively. Why do we not do the same for loggers?

This leads to leaks in the user-facing API. If users pass a list of loggers to the Trainer's logger flag, accessing trainer.logger returns a single instance of type LoggerCollection, which is different than what users specified, which means users have to learn about this abstraction and likely update their code to handle cases where a single logger is used vs multiple.

Pitch

1. Introduce a loggers property on the Trainer which returns a list of the passed in Loggers objects, so users have an option to not use the LoggerCollection

@property
def loggers(self) -> List[LightningLoggerBase]:

2. Deprecate LoggerCollection in favor of this loggers property. Wherever we call the Logger APIs from within the Trainer, iterate over the loggers list and call the hooks instead.

Example:

Current
https://github.com/PyTorchLightning/pytorch-lightning/blob/a6a28e08d22f59f2468ff1049c67507df7519e7c/pytorch_lightning/trainer/trainer.py#L1444-L1445

New

for logger in self.loggers:
    logger.finalize("success")

3. After LoggerCollection is removed, we could redefine logger on the Trainer to be something like this. Or we could deprecate logger entirely in favor of loggers to avoid any potential misuse/missed expectations around supporting multiple loggers.

@property
def logger(self) -> Optional[LightningLoggerBase]:
    loggers = self.loggers
    return loggers[0] if len(loggers) > 0 else None

Originally posted by @ananthsub in #11209 (comment)

Pitch

cc @justusschock @awaelchli @akihironitta @edward-io @Borda @ananthsub @tchaton

[ananthsub]

See this issue as an area where we need to access each individual logger's group separator: #11254

This is annoying to do with LoggerCollections today since it would require iterating over the iterables available inside of the logger collection. It would also force my code to be aware of LoggerCollection vs using a single logger, which defeats the whole point of having the LoggerCollection interface.

It'd be much much easier to iterate over the available loggers like this:

for logger in trainer.loggers:
    converted_metrics = _prefix_metrics(metrics_dict, prefix, logger.group_separator)
    logger.log_metrics(converted_metrics, trainer.global_step)

As this code will work for a single logger or multiple.

[tchaton]

Hey @ananthsub,

Sounds like a good suggestion. This could lead to some breaking change.

[wilson100hong]

@ananthsub this is better than LoggerCollection! I want to address a scenario that user might want to have a dict of Logger so they can choose the logger using key instead of index, which might be convenient in code management? WDYT?

[ananthsub]

Currently, the Trainer accepts an iterable of loggers, so passing a dict is "supported" as long as the keys of the dict are the LightningLoggerBase objects, not the values, since the LoggerCollection today simply iterates over this to do the logging. Passing a Dict[str, LightningLoggerBase] would break in this case.

so they can choose the logger using key instead of index, which might be convenient in code management?

Where would the user be accessing the logger from? trainer.loggers ? What would the type of loggers be based on this input? What specific dict type would the Trainer accept, to limit how much handling it'd need to do for iterating over the loggers to log values.

I proposed using loggers: List[LightningLoggerBase] for consistency with callbacks: List[Callback] . Otherwise we'd questions as to why the Trainer supports a dict of loggers, but only a list of Callbacks

[carmocca]

3. After LoggerCollection is removed, we could redefine logger on the Trainer to be something like this

Do you plan to add a deprecation path for the setter?

Also, do you plan to rename Trainer(logger=...) to Trainer(loggers=...) after this issue is resolved?

[ananthsub]

@carmocca good questions.

Do you plan to add a deprecation path for the setter?

trainer.logger = ....

This update should also be mirrored in trainer.loggers to avoid staleness fyi @akashkw

EDIT: I don't see how we can manage this if someone does this:

trainer.loggers = ...

what's the expectation for trainer.logger to be in that case after LoggerCollection is deprecated? trainer.loggers[0]?
I think this points that trainer.logger has to be deprecated. Otherwise we very quickly will fall out of sync between the two.

Also, do you plan to rename Trainer(logger=...) to Trainer(loggers=...) after this issue is resolved?

One option is to go all in with loggers and do the following:

• Deprecate trainer.logger in favor of trainer.loggers
• Deprecate LightningModule.logger in favor of LightningModule.loggers: https://github.com/PyTorchLightning/pytorch-lightning/blob/86b177ebe5427725b35fde1a8808a7b59b8a277a/pytorch_lightning/core/lightning.py#L257-L260
• Change Trainer(logger=...) to Trainer(loggers=...)

from an API standpoint, this is the cleanest and most consistent. but it also introduces more breaking changes to the Trainer and LightningModule, which I understand users are sensitive to. Over time it reduces the risk of bugs though, like keeping the attributes in sync

[daniellepintz]

Do we have an estimate of what percentage of users use multiple loggers vs one logger?

[daniellepintz]

I vote for going all in with loggers.

For the cases where users set trainer.logger = ... or trainer.loggers = ..., I think until we remove trainer.logger in v1.8 we will have to create setters for logger and loggers such that when one is set we update the other.

If someone sets trainer.loggers = ... I'd say we set trainer.logger to a LoggerCollection of those loggers.

[akashkw]

I think the setters for trainer.logger and trainer.loggers belong in this PR #11683, so I'll go ahead and add them.

[awaelchli]

For the cases where users set trainer.logger = ... or trainer.loggers = ..., I think until we remove

This is not allowed. We need to move away from this in the future anyway. The logger has to be owned by the logger connector and not be set on the Trainer indirectly. It is what we wanted to do for a while but never got to it.

[awaelchli]

The logger vs. loggers proposal here has some open questions which I believe we should address first before we set the api changes in motion.

If we previously had

trainer = Trainer(logger=Logger())

and in my LightningModule:

self.logger.log_image(...)

Then now with the new change, if I change to trainer = Trainer(logger=[Logger1(), Logger2()]) then that means I now also need to change the LightningModule code to a for loop iterating over loggers. This is not very well aligned with the design goals of Lightning (remember, we moved automatic_optimization to the LM because of this).

Furthermore, there is the question of whether to go "all in" with the loggers property. But what that means for the user is that they would have to write self.loggers[0] when 99% of time you just want to use a single logger which is silly imo. Having to write self.loggers[0] every time just screams after a self.logger property.

Do we have an estimate of what percentage of users use multiple loggers vs one logger?

99% use one logger. 1% uses 2 loggers. nobody uses more than 2.

Motivation by @ananthsub:

This leads to leaks in the user-facing API. If users pass a list of loggers to the Trainer's logger flag, accessing trainer.logger returns a single instance of type LoggerCollection, which is different than what users specified, which means users have to learn about this abstraction and likely update their code to handle cases where a single logger is used vs multiple.

To emphasize, the last part here "users have to [...] update their code to handle cases where a single logger is used vs multiple." is neither handled well by the current LoggerCollection nor does it get solved by the proposals here.

Shouldn't we investigate a design that lets us switch out the loggers without being forced to change the LM code?

[ananthsub]

@awaelchli - one solution for that is to pick an arbitrary logger from the loggers list as a convenience. we do this with model checkpointing callbacks here: https://github.com/PyTorchLightning/pytorch-lightning/blob/1203094a201bd38f0b8b77d93bc39fc95f06d8ae/pytorch_lightning/trainer/trainer.py#L2199-L2205

What we could do is redefine logger in place, such that, in the interim period we do this:

if len(loggers) == 0:
    return None
if len(loggers) == 1:
    return loggers[0]
else:
    # emit warning of using logger when multiple loggers are configured, with this behavior changing in v1.8 to return the first logger in the list
    return LoggerCollection(loggers)

and afterward we do this:

if len(loggers) == 0:
    return None
if len(loggers) > 1:
    # emit warning of using logger when multiple loggers are configured
return loggers[0]

In this case, we don't have to deprecate the single logger, but indicate that it's implementation will change in v1.8 such that LoggerCollection will be removed

[akashkw]

@ananthsub I was in the middle of suggesting a similar solution right as you posted it.

I agree with @awaelchli that deprecating trainer.logger is going to be clunkier for most users, and I noticed while refactoring the codebase that having a for loop just didn't make sense in many places so I had to use trainer.loggers[0].

I think maintaining both trainer.logger and trainer.loggers is still much cleaner than having LoggerCollection. Users will still have to update their code when going from a single logger to multiple, but learning to use trainer.loggers is much simpler than learning to use LoggerCollection.

[ananthsub]

To summarize:

• the focus of this issue is deprecating LoggerCollection in favor of loggers
• We don't need to remove the fastpath option of logger because it's convenient for the overwhelmingly common case of using a single logger

Regarding the trainer API, I also don't see a strong need to do this migration: Trainer(logger=...) to Trainer(loggers=...)
The documentation around this can clearly cite that multiple loggers are supported. But changing the argument incurs another migration.& breaking change eventually

[daniellepintz]

That sounds good. @awaelchli please confirm if this makes sense to you, so that @akashkw can finish his PR

[awaelchli]

I am ok with that, we just need to be fully aware that if we remove the LoggerCollection abstraction, we won't be able to improve the situation where the user needs to change their code to make the LM compatible with the trainer settings, at least not with the current logger reference, and unless we recommend to always add that for loop.

In summary, the implications are:

Before:

# agnostic to number of loggers in trainer (except 0)
self.logger.log_image(...)

After:

# have to change to:
for logger in self.loggers:
    logger.log_image(...)

This becomes the standard of accessing custom logging methods, the for loop will always be required to make it compatible with 0, 1, and multiple loggers.

The only way I see to avoid the "boilerplate" for loop without a logger collection abstraction is to add the methods log_text, log_image, etc. to the LM itself, which internally would loop over the loggers.

[ananthsub]

The only way I see to avoid the "boilerplate" for loop without a logger collection abstraction is to add the methods log_text, log_image, etc. to the LM itself, which internally would loop over the loggers.

I'm not a fan of this for a few reasons:

• I don't think a 1-line for-loop is boilerplate for users
• People may want to log only certain data to specific loggers, which loggers allows, and a method like LightningModule.log_image doesn't
• The schemas, argument inputs and return types for loggers around text & image are not standardized at all. each one adopts different conventions. log_text log_image doesn't unify that
• It sets a precedent that the LightningModule will re-expose APIs that other components already offer and specialize in. To me this is bloat. For example, would the LIghtningModule also then have a profile method, that calls the profiler attached to it? Would someone need to know the difference between LightningModule.profile and Profiler.profile and when to use what?

[akashkw]

I've outlined some of the use cases before and after we deprecate LoggerCollection
There are three types of logger methods to consider for each scenario.

Methods that:

• Can be called on every logger (such as log_metrics)
• Can be called on some loggers (such as log_image)
• Only make sense for one logger (such as version)

Before deprecating LoggerCollection:

• Single Logger

# Most common use case, which is very straightforward
self.logger.log_metrics(...)
self.logger.log_image(...)
self.logger.version(...)

• Multiple Loggers

self.logger.log_metrics(...)

# Can't control which loggers have log_image called
# Other than NeptuneLogger and WandbLogger, all loggers raise NotImplementedError
self.logger.log_image(...)

# Concatenates the versions of each logger, which probably isn't that useful
self.logger.version(...)

After deprecating LoggerCollection:

• Single Logger

# Most common use case, which is still very straightforward
self.logger.log_metrics(...)
self.logger.log_image(...)
self.logger.version(...)

• Multiple Loggers

for logger in self.loggers:
    logger.log_metrics(...)

# Can control which loggers have log_image called
for logger in self.loggers:
    if isinstance(logger, NeptuneLogger):
        logger.log_image(...)

# User can do whatever they want with logger.version
for logger in self.loggers:
    if logger.version != required_version:
        raise Exception()