Skip to content

Add @source not support #17255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Mar 25, 2025
Merged

Add @source not support #17255

merged 26 commits into from
Mar 25, 2025

Conversation

RobinMalfait
Copy link
Member

@RobinMalfait RobinMalfait commented Mar 17, 2025

This PR adds a new source detection feature: @source not "…". It can be used to exclude files specifically from your source configuration without having to think about creating a rule that matches all but the requested file:

@import "tailwindcss";
@source not "../src/my-tailwind-js-plugin.js";

While working on this feature, we noticed that there are multiple places with different heuristics we used to scan the file system. These are:

  • Auto source detection (so the default configuration or an @source "./my-dir")
  • Custom sources ( e.g. @source "./**/*.bin" — these contain file extensions)
  • The code to detect updates on the file system

Because of the different heuristics, we were able to construct failing cases (e.g. when you create a new file into my-dir that would be thrown out by auto-source detection, it'd would actually be scanned). We were also leaving a lot of performance on the table as the file system is traversed multiple times for certain problems.

To resolve these issues, we're now unifying all of these systems into one ignore crate walker setup. We also implemented features like auto-source-detection and the not flag as additional gitignore rules only, avoid the need for a lot of custom code needed to make decisions.

High level, this is what happens after the now:

  • We collect all non-negative @source rules into a list of roots (that is the source directory for this rule) and optional globs (that is the actual rules for files in this file). For custom sources (i.e with a custom glob), we add an allowlist rule to the gitignore setup, so that we can be sure these files are always included.
  • For every negative @source rule, we create respective ignore rules.
  • Furthermore we have a custom filter that ensures files are only read if they have been changed since the last time they were read.

So, consider the following setup:

/* packages/web/src/index.css */
@import "tailwindcss";
@source "../../lib/ui/**/*.bin";
@source not "../../lib/ui/expensive.bin";

This creates a git ignore file that (simplified) looks like this:

# Auto-source rules
*.{exe,node,bin,…}
*.{css,scss,sass,…}
{node_modules,git}/

# Custom sources can overwrite auto-source rules
!lib/ui/**/*.bin

# Negative rules
lib/ui/expensive.bin

We then use this information on top of your existing .gitignore setup to resolve files (i.e so if your .gitignore contains rules e.g. dist/ this line is going to be added before any of the rules lined out in the example above. This allows negative rules to allow-list your .gitignore rules.

To implement this, we're rely on the ignore crate but we had to make various changes, very specific, to it so we decided to fork the crate. All changes are prefixed with a // CHANGED: block but here are the most-important ones:

  • We added a way to add custom ignore rules that extend (rather than overwrite) your existing .gitignore rules
  • We updated the order in which files are resolved and made it so that more-specific files can allow-list more generic ignore rules.
  • We resolved various issues related to adding more than one base path to the traversal and ensured it works consistent for Linux, macOS, and Windows.

Behavioral changes

  1. Any custom glob defined via @source now wins over your .gitignore file and the auto-content rules.
  2. The node_modules and .git folders as well as the .gitignore file are now ignored by default (but can be overridden by an explicit @source rule).
  3. Source paths into ignored-by-default folders (like node_modules) now also win over your .gitignore configuration and auto-content rules.
  4. Introduced @source not "…" to negate any previous rules.
  5. Negative content rules in your legacy JavaScript configuration (e.g. content: ['!./src']) now work with v4.
  6. The order of @source definitions matter now, because you can technically include or negate previous rules. This is similar to your .gitingore file.
  7. Rebuilds in watch mode now take the @source configuration into account

Combining with other features

Note that the not flag is also already compatible with @source inline(…) added in an earlier commit:

@import "tailwindcss";
@source not inline("container");

Test plan

  • We added a bunch of oxide unit tests to ensure that the right files are scanned
  • We updated the existing integration tests with new @source not "…" specific examples and updated the existing tests to match the subtle behavior changes
  • We also added a new special tag [ci-all] that, when added to the description of a PR, causes the PR to run unit and integration tests on all operating systems.

[ci-all]

@RobinMalfait RobinMalfait force-pushed the feat/source-not branch 4 times, most recently from 41102c9 to 39588cf Compare March 18, 2025 11:36
@philipp-spiess philipp-spiess force-pushed the feat/source-not branch 12 times, most recently from d3700cc to 41e1ac2 Compare March 21, 2025 14:38
@RobinMalfait RobinMalfait marked this pull request as ready for review March 21, 2025 17:26
@RobinMalfait RobinMalfait requested a review from a team as a code owner March 21, 2025 17:26
RobinMalfait and others added 17 commits March 25, 2025 13:01
This isn't strictly necessary for this PR, but noticed missing values
when updating the `globs` -> `sources` and made TypeScript happy by
adding this.
These tests will reflect 2 big changes:
1. New tests in general
2. Tests where we used auto source detection and `.gitignore` files,
   will now favor the `@source '…'` rules over the `.gitignore` rules.
   Note: If you _don't_ want this behavior, then you can use `@source
   not '…'` to override rules.
We still emit `globs` for `@tailwindcss/postcss`, but we also added
better normalized globs based on your `@source` directives. We returned
them as a new property to stay backward compatible, but this also means
that the `@tailwindcss/cli` can make use of this.
Some people don't have `node_modules` in the `.gitignore` file so in
some scenario's this will always be scanned.

Another example is if you use the Vercel CLI to deploy your project
because `.gitignore` files are not pushed...

This is technically a breaking change, but now that we have `@source not
'…'` you can include files/folders from `node_modules` even though the
folder is ignored by default.
@philipp-spiess philipp-spiess force-pushed the feat/source-not branch 2 times, most recently from 3d2fc5c to 5e9cdf8 Compare March 25, 2025 12:04
@RobinMalfait RobinMalfait merged commit 1ef9775 into main Mar 25, 2025
18 checks passed
@RobinMalfait RobinMalfait deleted the feat/source-not branch March 25, 2025 14:54
RobinMalfait added a commit that referenced this pull request Mar 28, 2025
We didn't add a changelog for the PR
(#17255) and there are a
few things we need to talk about. So opened this PR to get everything in
and once we're happy we can merge it in one go instead of changing on
`main` directly.

---------

Co-authored-by: Philipp Spiess <[email protected]>
philipp-spiess added a commit that referenced this pull request Mar 28, 2025
We didn't add a changelog for the PR
(#17255) and there are a
few things we need to talk about. So opened this PR to get everything in
and once we're happy we can merge it in one go instead of changing on
`main` directly.

---------

Co-authored-by: Philipp Spiess <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment