Tracking issue for rustdoc --output-format=doctest command-line flag

[GuillaumeGomez]

This option is needed by the Rust-for-Linux project. The goal is to allow them to extract doctests so they can modify them and run them however they want to simplify their testing pipeline.

Steps

• Implement the feature ([rustdoc] Add --output-format=doctest command-line flag #134531)
• Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

None currently.

[ojeda]

Thanks a lot!

@rustbot label A-rust-for-linux

[epage]

Starting to play with this to see how we can use this in Cargo.

If I'm understanding correctly, doctests are not scraped from re-exported APIs. Does that mean I can run rustdoc src/lib.rs --output-format json without a care for dependencies? If so, that means we can have cargo run this ASAP within the unit graph which would be great for overall build times (we'd still need to wait for the lib to be built to compile the doctest).

In that case, is there a way we can prevent the compile errors from showing up and have the rustdoc process return success?

$ rustdoc +nightly --output-format doctest -Zunstable-options src/lib.rs
error[E0432]: unresolved import `clap_builder`
  --> src/lib.rs:88:9
   |
88 | pub use clap_builder::*;
   |         ^^^^^^^^^^^^ use of unresolved module or unlinked crate `clap_builder`
   |
help: you might be missing a crate named `clap_builder`, add it to your project and import it in your code
   |
88 + extern crate clap_builder;
   |

{"format_version":1,"doctests":[{"file":"src/lib.rs","line":46,"doctest_attributes":{"original":"rust","should_panic":
false,"no_run":false,"ignore":"None","rust":true,"test_harness":false,"compile_fail":false,"standalone_crate":false,"e
rror_codes":[],"edition":null,"added_css_classes":[],"unknown":[]},"original_code":"#[cfg(feature = \"derive\")] {\nus
e clap::Parser;\n\n/// Simple program to greet a person\n#[derive(Parser, Debug)]\n#[command(version, about, long_abou
t = None)]\nstruct Args {\n    /// Name of the person to greet\n    #[arg(short, long)]\n    name: String,\n\n    ///
Number of times to greet\n    #[arg(short, long, default_value_t = 1)]\n    count: u8,\n}\n\nfn main() {\n    let args
 = Args::parse();\n\n    for _ in 0..args.count {\n        println!(\"Hello {}!\", args.name);\n    }\n}\n}","doctest_
code":"#![allow(unused)]\nfn main() {\n#[cfg(feature = \"derive\")] {\nuse clap::Parser;\n\n/// Simple program to gree
t a person\n#[derive(Parser, Debug)]\nerror: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0432`.
#[command(version, about, long_about = None)]\nstruct Args {\n    /// Name of the person to greet\n    #[arg(short, lo
ng)]\n    name: String,\n\n    /// Number of times to greet\n    #[arg(short, long, default_value_t = 1)]\n    count:
u8,\n}\n\nfn main() {\n    let args = Args::parse();\n\n    for _ in 0..args.count {\n        println!(\"Hello {}!\",
args.name);\n    }\n}\n}\n}","name":"src/lib.rs - (line 46)"}]}

It looks like --output-format doctest prevents --error-format json

What are your thoughts on using jsonlines, like --error-format json? Seems a natural fit with the array (minus the format which could be inside each element within the array). I'm feeling the pain of cargo having used single-json blobs in some cargo output and feel like jsonlines offers a lot more flexibility. I was concerned this was going to be a blocker when its mixed with --error-format json but it appears rustc/rustdoc outputs to stderr (rather than stdout like cargo), so the two sets of messages can be distinguished.

As for message content:

• Shouldn't standalone_crate for 2021 be true? I'm getting false
• I feel like it would be good to future proof standalone_crate as a tristate. Currently, its not false as in "never" but as in "auto" / null since there is a fallback to running standalone
• Could we get start/end span information? I'd love spans for each line of the original code but I'm guessing thats not available.
• At least for myself, I feel like the name isn't needed. I'd love (likely requiring an Edition if at all) to transition to using the path to the documented item (foo::bar) like other tests
• Can we get the path to the documented item (foo::bar)?
• What does ignore: "None" mean? What other values are supported and what do they mean?
• Can we get it documented what the expected procedure is for combining, compiling, and running these is?

[GuillaumeGomez]

The doctests are still extracted from compiler information, which might include dependencies (because if you inline a foreign item, then you need to get its attributes).

• Shouldn't standalone_crate for 2021 be true? I'm getting false

Sounds like a bug, please open an issue. :)

• Could we get start/end span information? I'd love spans for each line of the original code but I'm guessing thats not available.

In most cases it should be possible. If it's generated from a (proc-)macro, it becomes a bit tricky, we can add a from_expansion boolean I suppose to go around that (like rustc's span).

• At least for myself, I feel like the name isn't needed. I'd love (likely requiring an Edition if at all) to transition to using the path to the documented item (foo::bar) like other tests

Can't find it but I remember there was an issue about adding the possibility to name a doctest.

• Can we get the path to the documented item (foo::bar)?

I suppose so.

• What does ignore: "None" mean? What other values are supported and what do they mean?

Terrible naming and no docs, I'm ashamed.

So:

• Ignore::All means "ignore this test all the time".
• Ignore::None means "never ignore this test".
• Ignore::Some(condition) means "ignore this test if target doesn't contain condition".

• Can we get it documented what the expected procedure is for combining, compiling, and running these is?

Not sure to understand what you mean there.

Please open issues for bug and requested features and add details for the last point. :)