guide: Modeling Pipelines

[jorgeorpinel]

From #144 (comment)

• What are data pipelines, including DAG concept (extract from https://dvc.org/doc/command-reference/dag)

Structure:

See #2857 for the original structure ideas.

• Defining Pipelines
  • Focus on the pipelining process and codification link to dvc.yaml ref. for formalities
  • Defining stages (extract from run and stage refs.)
    • Dependencies and (general) parameters
    • Outputs and metrics/plots
  • Should provide some actual dvc.yaml example/templates to copy&paste.
  • Multiple dvc.yaml files vs. multiple pipelines (repro --recursive)
  • Describing formally outs, deps, stage doesn't make sense here
  • Mention exp init? (Link to appropriate page/section) Maybe
• Pipeline reproduction
  • Extract some details from https://dvc.org/doc/command-reference/repro ?
• Experimental pipelines (discussion)
• Operationalizing pipelines?

Other tasks

• ML Pipelines: recommend push=false for intermediate outputs #4267
• Mention VS Code "as an editor that supports schema definition, etc."
• Update some of the [stage] links around this term that currently go to run/stage refs (some should link to the concept page instead)
• Recover some info deleted in this change to Running Experiments ?
• guide: Extract templating and foreach features from Pipeline Files (dvc.yaml) #3670