-
Notifications
You must be signed in to change notification settings - Fork 12.8k
Proposal: compute module format based on package.json visible to declarationDir
/outDir
#54546
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -3508,11 +3526,58 @@ export function createProgram(rootNamesOrOptions: readonly string[] | CreateProg | |||
return result; | |||
} | |||
|
|||
function getCreateSourceFileOptions(fileName: string, moduleResolutionCache: ModuleResolutionCache | undefined, host: CompilerHost, options: CompilerOptions): CreateSourceFileOptions { | |||
function getImpliedNodeFormatForFile(fileName: string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’m open to suggestions for a different name to differentiate from ts.getImpliedNodeFormatForFile
which is public API, but I intend to expose this one as program.getImpliedNodeFormatForFile
, and add JSDoc to the top-level standalone one that says you should use the program
version if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think functionality needs to be moved under getImpliedNodeFormatForFileWorker with optional new methods that are required since we expose that method publicly and we want it to be same as what program does to figure out file's implied format.
getOutputDeclarationFileNameWithoutConfigFile( | ||
fileNameForModuleFormatDetection, | ||
options, | ||
!host.useCaseSensitiveFileNames(), | ||
currentDirectory, | ||
getAssumedCommonSourceDirectory) || | ||
getOutputJSFileNameWithoutConfigFile( | ||
fileNameForModuleFormatDetection, | ||
options, | ||
!host.useCaseSensitiveFileNames(), | ||
currentDirectory, | ||
getAssumedCommonSourceDirectory) || | ||
fileNameForModuleFormatDetection; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I considered checking to see if a result computed from declarationDir
disagrees with a result computed from outDir
and issue an error, but that seems like such an edge case that it wouldn’t be worth spending time doing that for every source file that could be influenced by it. I decided to prefer declarationDir
since that seems most likely to give a library author the same analysis that their users will see. But if someone tries really hard, they can make an invalid situation by using separate outDir
and declarationDir
with conflicting package.json files in each.
declarationDir
/outDir
declarationDir
/outDir
@typescript-bot pack this |
Heya @andrewbranch, I've started to run the tarball bundle task on this PR at 8a7c262. You can monitor the build here. |
@typescript-bot test top100 |
Heya @andrewbranch, I'm starting to run the diff-based top-repos suite on this PR at 8a7c262. Hold tight - I'll update this comment with the log link once the build has been queued. Update: The results are in! |
Hey @andrewbranch, I've packed this into an installable tgz. You can install it for testing by referencing it in your
and then running There is also a playground for this build and an npm module you can use via |
@andrewbranch Here are the results of running the top-repos suite comparing Something interesting changed - please have a look. Details
|
As you know - I'm not a fan of dual packages so I'm not really advocating here anyhow on their behalf, I'm not invested in the game. I have some thoughts on the DX of this improvement though.
|
@@ -596,7 +596,21 @@ export function getOutputDeclarationFileName(inputFileName: string, configFile: | |||
); | |||
} | |||
|
|||
function getOutputJSFileName(inputFileName: string, configFile: ParsedCommandLine, ignoreCase: boolean, getCommonSourceDirectory?: () => string) { | |||
/** @internal */ | |||
export function getOutputDeclarationFileNameWithoutConfigFile(inputFileName: string, options: CompilerOptions, ignoreCase: boolean, currentDirectory: string, getCommonSourceDirectory: () => string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need separate functions for these. whats different. i couldnt make out from the function,
configFile used in the helper method only needs options and filenames (root file names) so we have that in program so we could change to Pick<ParsedCommandLine, "options" | "fileNames">
and that should work right. or am i missing something.
One thing I think I'd like to see (which maybe overlaps with @Andarist's commentary) is some way to do this that doesn't involve pre-seeding a That being said, I do like the proposal, given it effectively makes safe the packaging method that @isaacs is now using for his packages (e.g. https://unpkg.com/browse/[email protected]/dist/). |
The existence of https://unpkg.com/browse/[email protected]/dist/cjs/package.json in this form... so confusing 🙈 Can't wait for some resolvers to be tripped over non-root |
I don't have any strong feelings about the details in this approach, but my overall response to the idea is that it feels like y'all are going out of your way to address my niche use case in particular without me even having asked, and I rarely feel so loved in OSS, so thank you very much for making my day ❤️ Shipping hybrid TS modules is indeed a bit of a house of cards currently, but enough people are using CJS (and will continue for the foreseeable future) that it is a worthwhile endeavor imo, whether it's ill-advised or not. It would be really awesome if I could have some set of configs so that I just run For example, https://github.com/tapjs/processinfo is a hybrid module, but some of the files are cjs only, and some are esm only, so they have different include/exclude lists. Anything that uses
Hah, oof. Yeah, that's a bug. When I added a bin script, I made it import from |
'npm publish' runs 'npm prepare', but does _not_ run postprepare, leading to a 'works on my machine' problem. Via: microsoft/TypeScript#54546 (comment)
In this case, it was likely just creating a CJS package scope and nothing else.
Nice! ❤️ When re-reading my comment now - I think that it could be taken as snarky or smth, it definitely was not my intention, just noting down something I noticed ;p It's super cool that you addressed this |
No worries, I usually try to assume positive intent. Turns out I'd already had a fix in place, but the fix was in |
Thanks for the feedback. I take all these points, but to clarify my suggested workflow, I assume that the outDir package.json files will be committed, while the rest of the outDir contents would typically gitignored. This could be done with scripts, but there’s little reason not to just commit them IMO. So if using other tools in combination with tsc, they don’t need to add package.jsons, they just need to not delete the non-gitignored one that’s already there.
The only version of this I could see is something that would emit the appropriate package.json into the |
I can't see any tests around this so I'm curious if a |
In theory, all module resolution should happen from output files, but in practice, it never does. #37378 is essentially a feature request for that. |
That would also imply that this should be allowed too then right? // tsconfig.json
{
"compilerOptions": {
"module": "nodenext",
"outDir": "./dist"
}
}
// src/index.ts
import { a } from "./a";
console.log(a);
// dist/a.d.ts
export declare const a: string; Which seems quite unintuitive and frustrating if e.g. there used to be a file at (I'm leaving this comment here rather than on the issue you linked since from my (maybe incorrect?) reading of that, it seems like the consensus there was to not change module resolution like this PR is doing?) |
This PR doesn’t change module resolution at all. The only outDir-based logic applied is finding package.jsons for purposes of module format detection. |
There might be little reason to not commit them but it's usually just way simpler to not commit them which also extends to...
It's just way easier to just So overall, I think that it's a somewhat big ask for other tools to tweak how they work today so those files don't get removed.
Playing devil's advocate - if emitting them is not the table here, what exact value the current proposed approach would have (or rather, under what circumstances one would like to use/it would kick in)? I wonder if you need it if you'd go with emitting those |
The value here is mostly for people who won’t use a third-party tool at all, or will only use a third-party tool to post-process what tsc builds.
I don’t really expect other tools to change, which reinforces why I’m making this PR. If tsc makes it reasonably easy to do type-accurate and runtime-safe dual emit, and it can be shown that every third party tool is runtime-unsafe and/or produces broken/unchecked type declarations, then I think new projects will adopt this, and a small number of existing projects with fairly simple builds that only leverage other tools to do dual emit might switch.
I don’t really see a problem with that. It’s not hard to say the package.json that matters is Is there something in particular you would suggest instead? |
FWIW isn't it kind of annoying to have to .gitignore "entire |
Yeah, it’s kind of annoying for 30 seconds when it doesn’t work the first time and you have to ask Copilot how to do it and then never look at your .gitignore again. |
I think the most significant problem with this approach is that it ignores the possibility of TS bundlers and runtimes that consume input files and also care about package.json |
Maybe I’m just not using my imagination enough but this constraint sounds like it would prevent you from doing dual packages from the same source entirely |
@fatcerberus yeah kind of—the main thing I’m thinking is we need to not make agreement a huge pain for projects that aren’t doing dual emit. |
I'm a fan of the intent here for sure. I'm not a fan of the idea of needing two package.json files, especially when that's what I'd love to see this happen, one way or another. |
I might be misunderstanding what you’re saying, but if not, I think you may have the wrong idea about what {
"exports": {
"import": "./esm.js",
"require": "./cjs.js"
}
} No matter how they are accessed, Node will interpret both |
To my understanding, conditional I've added a test to In short, my approach does update file extensions and associated specifiers. The package.json file in test looks like this: {
"version": "0.0.0",
"type": "commonjs",
"exports": {
".": {
"import": {
"types": "./dist/mjs/index.d.mts",
"default": "./dist/index.mjs"
},
"require": {
"types": "./dist/index.d.ts",
"default": "./dist/index.js"
},
"default": "./dist/index.js"
},
"./package.json": "./package.json"
}
} |
“supposed to” by convention, not by any magic in the implementation. Your example works only because the file extensions agree. Conditions are only used during module resolution; they have zero impact on module format detection. Conditions are really just arbitrary strings, but Node applies the |
You're right, it does work. I will invoke your stated design goals 4 and 7 here. Whether you want to support this or not is up to you. I don't particularly need it 👋 |
If anything Andrew is preserving the runtime behavior of all JavaScript code here (goal 7). Non-conventional packages might exist out there and they should be supported by TS, it should be able to reflect their runtime behavior. Therefore the assumption about the module format based on the condition alone is just something that TS can't do. |
Note: there are good reasons not to publish dual CJS/ESM npm packages. This is not a blanket endorsement of the practice. However, we’ve noticed that many existing dual packages ship types that don’t correctly describe the JS contents of the package. I believe this is in no small part because doing dual CJS/ESM emit with tsc is a cumbersome UX (and completely undocumented). This is a small change that significantly improves the experience.
In
--module nodenext
, we decide whether a fileindex.ts
is ESM or CJS by looking up through its containing directories for the nearest package.json file so we can see whether it includes"type": "module"
. With this PR, we instead begin that search with the file’sdeclarationDir
oroutDir
. This allows the developer to pre-seed anoutDir
per module format with a package.json, and run a compilation into each:dist/esm/package.json
:dist/cjs/package.json
:These builds can be linked with a solution-style tsconfig.json:
Note that in this configuration, where the input files are covered by two different tsconfigs, TS Server picks uses the second one in the array for errors and intellisense. #54476 discusses ideas for improvements to the editor and orchestration experience.