Tracking issue for proc_macro::Span inspection APIs
#54725
Comments
alexcrichton
added
A-macros
|
I wanted to shed some pain this is causing with tarpaulin. Tarpaulin has worked amazingly well for me and my company as a replacement for kcov which historically been a pain to get accurate, reliable and correct results. At the moment tarpaulin stands as the most promising goto option for codecoverage in rust and just feels more like the only first class tooling option. Having one of those when choosing to invest in a technology is important for many company's adoption story for checking off code quantity checkmarks. When they see that rust doesn't have a reasonable code quality story that works on stable rust, that's can result in a "pass" rather than "I'll take it". There are currently some work arounds for making this work-ish on stable but it feels very much like the story serde was in a year or so ago when you wanted to show all your friends how amazing serde was but then was embarrassed to show what it took to make work on stable because of a macro stabilization blocker. |
|
With procedural macros having reached a point where they're very useful on stable, I expect many users will find themselves needing access to this information. Would it be reasonable to only stabilize parts of the |
This unstable feature (rust-lang/rust#54725) is the last thing that we require in Nightly. Removing it will cause a significant regression in error messages, but that can be improved after switching to parsing the grammar as tokens rather than as a string literal.
|
I have a concern about exposing My understanding is that, if one adds a blank line to the start of the file, the line_column information of the input spans to proc macro changes. That means that IDE would have to re-expand procedural macros even after insignificant white space changes. I would feel more confident if proc-macros were strictly a pure function from the input token stream to output token stream. This can be achieved, for example, by making line-column infocation relative to the start of macro invocation (as opposed to relative to the start of the file). I don't feel that exposing absolute position is necessary the end of the world: IDE can track if a macro actually makes use of the info, or it can supply macro with some fake line/columns. But it's hard to tell if such hacks would work well in practice, and it's hard to experiment with them for the lack of IDE which handled proc-macros incrementally.... |
|
If the parser allocated IDs to every AST node, then, and this is the hard part, when an edit was made to the source, the parser tried to keep those IDs the same in all the non-edited code and only allocate new IDs for new nodes, that would allow spans to be kept completely separate from the AST. Those IDs could be passed through macro expansion without causing unnecessary invalidations. If something needed a span later on, it could then go back and ask the parser for the span for that particular AST node ID. I feel like having an incremental parser is important, not because parsing is the bottleneck, but because it underpins everything else. |
|
@davidlattimore this is fascinating, but slightly off-topic for the issue. I've created a thread on internals: https://internals.rust-lang.org/t/macros-vs-incremental-parsing/9323 |
|
The column!() macro as well as std::panic::Location::column are returning 1-based columns while the span available from the proc-macro crate is 0-based according to its docs. Is this inconsistency intended? |
|
This thread has more discussion about 1-based columns: #46762 (comment) |
|
Another open question is how this API relates to #47389 which is about minimizing span information throughout the compiler. Should stabilization be blocked until a design for #47389 is found? Is it too late already as we have |
|
#47389 is mostly concerned about data types that are used later in the compilation pipeline, such as type information and MIR. Exposing things at the proc-macro level should not be too bad. |
|
But rust-analyzer might one day expand the scope of incremental compilation to the parsing stage, right? |
Templates can now be placed directly next to the source file that they are defined in as a default. This relies on an unstable rust compiler feature, which exposes the source file to proc macros. See <rust-lang/rust#54725> for more info. This requires the nightly compiler to run, and enabling the proc_macro_span and procmacro2_semver_exempt cfg flags. To build / test: ```shell RUSTFLAGS='--cfg proc_macro_span --cfg procmacro2_semver_exempt' \ cargo +nightly build ``` Fixes: <askama-rs#877>
Templates can now be placed directly next to the source file that they are defined in as a default. This relies on an unstable rust compiler feature, which exposes the source file to proc macros. See <rust-lang/rust#54725> for more info. This requires the nightly compiler to run, and enabling the proc_macro_span and procmacro2_semver_exempt cfg flags. To build / test: ```shell RUSTFLAGS='--cfg proc_macro_span --cfg procmacro2_semver_exempt' \ cargo +nightly build ``` Fixes: <askama-rs#877>
|
I’d like to see progress on this issue. How can we help move it forward? Source FileMy main use case is to locate file references in macros, similar to how It seems the main blocker for stabilizing Line/Column InformationMany use cases have been mentioned already. The main one i have in Slint is to know if an identifier token and a punctuation token are separated by space or not. Other usecases are using it with tools like Regarding concerns about incremental compilation, I think this is not a real issue:
Given this, I don’t see why line/column APIs couldn’t move toward stabilization. Are there other blockers preventing this? |
… r=m-ou-se Stabilize span_open() and span_close(). This proposes to stabilize `Group::span_open()` and `Group::span_close()`. These are part of the `proc_macro_span` feature gate tracked in rust-lang#54725 Most of the features gated behind `proc_macro_span` are about source location information (file path, line and column information), expansion information (parent()), source_text(), etc. Those are not ready for stabilizaiton. However, getting the span of the `(` and `)` separately instead of only of the entire `(...)` can be very useful in proc macros, and doesn't seem blocked on anything that all the other parts of `proc_macro_span` are blocked on. So, this renames the feature gate for those two functions to `proc_macro_group_span` and stabilizes them.
|
An open question is what to do with Do we need |
|
I believe the best thing we can do is to remove the impl Span {
pub fn file(&self) -> String; // Mapped file name, for display purposes.
pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk.
}PR: #139671 |
|
Yes I agree this would be preferable, people are almost certainly going to be converting it to |
Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}
Simplification/redesign of the unstable proc macro span API, tracked in rust-lang#54725:
Before:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn source_file(&self) -> SourceFile;
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SourceFile { .. }
impl !Send for SourceFile {}
impl !Sync for SourceFile {}
impl SourceFile {
pub fn path(&self) -> PathBuf;
pub fn is_real(&self) -> bool;
}
```
After:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn file(&self) -> String; // Mapped file name, for display purposes.
pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk.
}
```
This resolves the last blocker for stabilizing these methods. (Stabilizing will be a separate PR with FCP.)
Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}
Simplification/redesign of the unstable proc macro span API, tracked in rust-lang#54725:
Before:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn source_file(&self) -> SourceFile;
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SourceFile { .. }
impl !Send for SourceFile {}
impl !Sync for SourceFile {}
impl SourceFile {
pub fn path(&self) -> PathBuf;
pub fn is_real(&self) -> bool;
}
```
After:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn file(&self) -> String; // Mapped file name, for display purposes.
pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk.
}
```
This resolves the last blocker for stabilizing these methods. (Stabilizing will be a separate PR with FCP.)
Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}
Simplification/redesign of the unstable proc macro span API, tracked in rust-lang#54725:
Before:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn source_file(&self) -> SourceFile;
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SourceFile { .. }
impl !Send for SourceFile {}
impl !Sync for SourceFile {}
impl SourceFile {
pub fn path(&self) -> PathBuf;
pub fn is_real(&self) -> bool;
}
```
After:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn file(&self) -> String; // Mapped file name, for display purposes.
pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk.
}
```
This resolves the last blocker for stabilizing these methods. (Stabilizing will be a separate PR with FCP.)
Rollup merge of rust-lang#139671 - m-ou-se:proc-macro-span, r=dtolnay Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file} Simplification/redesign of the unstable proc macro span API, tracked in rust-lang#54725: Before: ```rust impl Span { pub fn line(&self) -> usize; pub fn column(&self) -> usize; pub fn source_file(&self) -> SourceFile; } #[derive(Clone, Debug, PartialEq, Eq)] pub struct SourceFile { .. } impl !Send for SourceFile {} impl !Sync for SourceFile {} impl SourceFile { pub fn path(&self) -> PathBuf; pub fn is_real(&self) -> bool; } ``` After: ```rust impl Span { pub fn line(&self) -> usize; pub fn column(&self) -> usize; pub fn file(&self) -> String; // Mapped file name, for display purposes. pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk. } ``` This resolves the last blocker for stabilizing these methods. (Stabilizing will be a separate PR with FCP.)
|
PR for stabilizing start+end+line+column |
|
After discussion in the libs-api meeting, I've split out the |
|
Opened #139901 for the open question on byte offsets. Feel free to vote 👍 or 👎 on that PR, or leave a comment there if you have more thoughts. |
|
Opened #139903 for the open question on the name of |
Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}
Simplification/redesign of the unstable proc macro span API, tracked in rust-lang/rust#54725:
Before:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn source_file(&self) -> SourceFile;
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SourceFile { .. }
impl !Send for SourceFile {}
impl !Sync for SourceFile {}
impl SourceFile {
pub fn path(&self) -> PathBuf;
pub fn is_real(&self) -> bool;
}
```
After:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn file(&self) -> String; // Mapped file name, for display purposes.
pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk.
}
```
This resolves the last blocker for stabilizing these methods. (Stabilizing will be a separate PR with FCP.)
Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}
Simplification/redesign of the unstable proc macro span API, tracked in rust-lang/rust#54725:
Before:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn source_file(&self) -> SourceFile;
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SourceFile { .. }
impl !Send for SourceFile {}
impl !Sync for SourceFile {}
impl SourceFile {
pub fn path(&self) -> PathBuf;
pub fn is_real(&self) -> bool;
}
```
After:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn file(&self) -> String; // Mapped file name, for display purposes.
pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk.
}
```
This resolves the last blocker for stabilizing these methods. (Stabilizing will be a separate PR with FCP.)
Proc macro span API redesign: Replace proc_macro::SourceFile by Span::{file, local_file}
Simplification/redesign of the unstable proc macro span API, tracked in rust-lang#54725:
Before:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn source_file(&self) -> SourceFile;
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SourceFile { .. }
impl !Send for SourceFile {}
impl !Sync for SourceFile {}
impl SourceFile {
pub fn path(&self) -> PathBuf;
pub fn is_real(&self) -> bool;
}
```
After:
```rust
impl Span {
pub fn line(&self) -> usize;
pub fn column(&self) -> usize;
pub fn file(&self) -> String; // Mapped file name, for display purposes.
pub fn local_file(&self) -> Option<PathBuf>; // Real file name as it exists on disk.
}
```
This resolves the last blocker for stabilizing these methods. (Stabilizing will be a separate PR with FCP.)
and others
This issue is intended to track a number of unstable APIs which are used to inspect the contents of a
Spanfor information like the file name, byte position, manufacturing new spans, combining them, etc.This issue tracks the
proc_macro_spanunstable feature.Public API
Already stabilized:
Up for stabilization:
Close to stabilization, pending some bikeshedding:
Things that require more discussion:
The text was updated successfully, but these errors were encountered: