Single-process multi-node CPU training

[borchero]

What does this PR do?

Whenever a model is small, it is not necessarily useful to use a GPU. However, one might still benefit from distributing computations for model training across nodes -- especially when you consider that containers are also "nodes".

This PR sets DistributedType.DDP instead of DistributedType.DDP_SPAWN whenever DistributedType.DDP_CPU is set and only a single process or a single process per node is used.

Does your PR introduce any breaking changes? If yes, please list them.

• Minor breaking change: whenever one requests a num_nodes > 1 and sets num_processes = None on the Trainer, num_processes now defaults to 1 and DistributedType.DDP is used. Before, num_processes was set to os.cpu_count() and DistributedType.DDP_SPAWN was used. However, it is unlikely that people rely on that behavior, especially since num_processes defaults to 1.

One could decide to keep the old behavior, however, I would argue that it is far more likely to only want a single process when distributing across nodes (think containers).

Fixes #9877

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

[awaelchli]

Thanks for the PR. LGTM, just a few changes requested.

[borchero]

Implemented the changes, thanks for the pointers @awaelchli :)

[awaelchli]

Thanks @borchero <3

[borchero]

Don't think the failing test is caused by my changes.

[codecov]

Codecov Report

Merging #9603 (146123b) into master (6feda08) will decrease coverage by 4%.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #9603    +/-   ##
=======================================
- Coverage      93%     89%    -4%     
=======================================
  Files         179     179            
  Lines       15805   15807     +2     
=======================================
- Hits        14648   14029   -619     
- Misses       1157    1778   +621     

[rohitgr7]

nice work!