Skip to content

redo #28564 #36665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 7, 2020
Merged

redo #28564 #36665

merged 1 commit into from
Feb 7, 2020

Conversation

sandersn
Copy link
Member

@sandersn sandersn commented Feb 7, 2020

Redo #28564

I haven't looked at the baseline changes; maybe they're horrible.

@sandersn
Copy link
Member Author

sandersn commented Feb 7, 2020

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Feb 7, 2020

Heya @sandersn, I've started to run the perf test suite on this PR at 4e3c2c5. You can monitor the build here. It should now contribute to this PR's status checks.

Update: The results are in!

@sandersn sandersn added the For Milestone Bug PRs that fix a bug with a specific milestone label Feb 7, 2020
@typescript-bot
Copy link
Collaborator

@sandersn
The results of the perf run you requested are in!

Here they are:

Comparison Report - master..36665

Metric master 36665 Delta Best Worst
Angular - node (v10.16.3, x64)
Memory used 356,858k (± 0.01%) 358,602k (± 0.03%) +1,745k (+ 0.49%) 358,432k 358,954k
Parse Time 1.61s (± 0.50%) 1.62s (± 0.58%) +0.01s (+ 0.68%) 1.60s 1.64s
Bind Time 0.89s (± 0.95%) 0.88s (± 1.01%) -0.01s (- 0.56%) 0.86s 0.90s
Check Time 4.67s (± 0.44%) 4.69s (± 0.68%) +0.01s (+ 0.28%) 4.62s 4.77s
Emit Time 5.22s (± 0.25%) 5.20s (± 0.36%) -0.02s (- 0.40%) 5.15s 5.23s
Total Time 12.40s (± 0.23%) 12.39s (± 0.35%) -0.00s (- 0.02%) 12.31s 12.52s
Monaco - node (v10.16.3, x64)
Memory used 364,682k (± 0.02%) 364,676k (± 0.02%) -6k (- 0.00%) 364,503k 364,824k
Parse Time 1.25s (± 0.66%) 1.25s (± 0.27%) -0.00s (- 0.08%) 1.25s 1.26s
Bind Time 0.78s (± 0.48%) 0.77s (± 0.58%) -0.00s (- 0.39%) 0.76s 0.78s
Check Time 4.68s (± 0.70%) 4.68s (± 0.46%) -0.00s (- 0.04%) 4.64s 4.74s
Emit Time 2.90s (± 0.77%) 2.90s (± 0.69%) -0.00s (- 0.14%) 2.86s 2.95s
Total Time 9.61s (± 0.45%) 9.61s (± 0.35%) -0.01s (- 0.06%) 9.54s 9.69s
TFS - node (v10.16.3, x64)
Memory used 324,299k (± 0.07%) 324,182k (± 0.02%) -117k (- 0.04%) 324,068k 324,468k
Parse Time 0.94s (± 0.52%) 0.95s (± 0.50%) +0.00s (+ 0.32%) 0.94s 0.96s
Bind Time 0.75s (± 1.54%) 0.75s (± 0.64%) +0.00s (+ 0.00%) 0.74s 0.76s
Check Time 4.23s (± 0.38%) 4.22s (± 0.65%) -0.01s (- 0.21%) 4.16s 4.28s
Emit Time 3.01s (± 0.76%) 3.00s (± 1.58%) -0.01s (- 0.46%) 2.90s 3.10s
Total Time 8.93s (± 0.43%) 8.91s (± 0.68%) -0.02s (- 0.22%) 8.75s 9.03s
Angular - node (v12.1.0, x64)
Memory used 332,632k (± 0.07%) 334,315k (± 0.02%) +1,683k (+ 0.51%) 334,123k 334,442k
Parse Time 1.57s (± 0.54%) 1.57s (± 0.45%) -0.00s (- 0.00%) 1.55s 1.58s
Bind Time 0.86s (± 1.10%) 0.87s (± 1.04%) +0.01s (+ 0.69%) 0.85s 0.89s
Check Time 4.62s (± 0.80%) 4.56s (± 0.54%) -0.06s (- 1.26%) 4.49s 4.61s
Emit Time 5.41s (± 0.57%) 5.38s (± 0.49%) -0.03s (- 0.48%) 5.32s 5.44s
Total Time 12.46s (± 0.47%) 12.38s (± 0.31%) -0.08s (- 0.63%) 12.33s 12.50s
Monaco - node (v12.1.0, x64)
Memory used 344,551k (± 0.02%) 344,532k (± 0.02%) -19k (- 0.01%) 344,450k 344,693k
Parse Time 1.22s (± 0.67%) 1.21s (± 0.80%) -0.01s (- 0.90%) 1.19s 1.23s
Bind Time 0.75s (± 0.80%) 0.75s (± 0.94%) +0.00s (+ 0.54%) 0.74s 0.77s
Check Time 4.54s (± 0.35%) 4.54s (± 0.56%) +0.00s (+ 0.07%) 4.48s 4.59s
Emit Time 2.94s (± 0.61%) 2.96s (± 0.84%) +0.02s (+ 0.71%) 2.93s 3.04s
Total Time 9.44s (± 0.31%) 9.46s (± 0.38%) +0.02s (+ 0.18%) 9.35s 9.51s
TFS - node (v12.1.0, x64)
Memory used 306,384k (± 0.02%) 306,445k (± 0.02%) +62k (+ 0.02%) 306,357k 306,565k
Parse Time 0.94s (± 0.66%) 0.94s (± 0.51%) -0.00s (- 0.21%) 0.93s 0.95s
Bind Time 0.70s (± 0.53%) 0.71s (± 1.06%) +0.00s (+ 0.28%) 0.69s 0.72s
Check Time 4.17s (± 0.44%) 4.17s (± 0.62%) +0.01s (+ 0.14%) 4.12s 4.22s
Emit Time 3.07s (± 0.63%) 3.08s (± 0.52%) +0.02s (+ 0.55%) 3.05s 3.12s
Total Time 8.88s (± 0.38%) 8.91s (± 0.25%) +0.03s (+ 0.30%) 8.85s 8.95s
Angular - node (v8.9.0, x64)
Memory used 351,823k (± 0.01%) 353,454k (± 0.01%) +1,630k (+ 0.46%) 353,329k 353,549k
Parse Time 2.10s (± 0.54%) 2.13s (± 0.83%) +0.03s (+ 1.29%) 2.10s 2.19s
Bind Time 0.92s (± 0.64%) 0.93s (± 1.01%) +0.01s (+ 0.65%) 0.91s 0.95s
Check Time 5.46s (± 0.66%) 5.45s (± 0.57%) -0.01s (- 0.11%) 5.36s 5.50s
Emit Time 6.24s (± 0.36%) 6.25s (± 0.86%) +0.01s (+ 0.24%) 6.14s 6.38s
Total Time 14.71s (± 0.25%) 14.76s (± 0.52%) +0.05s (+ 0.33%) 14.56s 14.91s
Monaco - node (v8.9.0, x64)
Memory used 362,939k (± 0.01%) 362,914k (± 0.01%) -25k (- 0.01%) 362,768k 363,015k
Parse Time 1.56s (± 0.43%) 1.57s (± 0.68%) +0.01s (+ 0.38%) 1.55s 1.59s
Bind Time 0.95s (± 0.49%) 0.94s (± 0.63%) -0.00s (- 0.53%) 0.93s 0.96s
Check Time 5.39s (± 1.25%) 5.39s (± 1.77%) -0.00s (- 0.07%) 5.27s 5.61s
Emit Time 3.30s (± 3.17%) 3.31s (± 4.02%) +0.00s (+ 0.12%) 3.03s 3.47s
Total Time 11.21s (± 0.55%) 11.21s (± 0.40%) +0.00s (+ 0.00%) 11.11s 11.32s
TFS - node (v8.9.0, x64)
Memory used 323,541k (± 0.02%) 323,479k (± 0.01%) -62k (- 0.02%) 323,386k 323,563k
Parse Time 1.26s (± 0.44%) 1.27s (± 0.65%) +0.00s (+ 0.32%) 1.25s 1.29s
Bind Time 0.75s (± 0.65%) 0.75s (± 0.69%) -0.00s (- 0.40%) 0.74s 0.76s
Check Time 4.82s (± 0.56%) 4.80s (± 0.59%) -0.02s (- 0.44%) 4.76s 4.89s
Emit Time 3.20s (± 0.73%) 3.19s (± 0.58%) -0.01s (- 0.22%) 3.14s 3.22s
Total Time 10.04s (± 0.41%) 10.01s (± 0.47%) -0.03s (- 0.26%) 9.94s 10.16s
Angular - node (v8.9.0, x86)
Memory used 200,058k (± 0.02%) 200,951k (± 0.02%) +893k (+ 0.45%) 200,864k 201,062k
Parse Time 2.04s (± 1.23%) 2.04s (± 0.40%) +0.00s (+ 0.20%) 2.02s 2.06s
Bind Time 1.05s (± 0.85%) 1.05s (± 0.59%) +0.00s (+ 0.29%) 1.04s 1.06s
Check Time 4.97s (± 0.70%) 4.94s (± 0.59%) -0.03s (- 0.58%) 4.86s 4.97s
Emit Time 6.06s (± 1.14%) 6.14s (± 1.39%) +0.08s (+ 1.35%) 6.00s 6.33s
Total Time 14.11s (± 0.61%) 14.17s (± 0.64%) +0.06s (+ 0.43%) 13.95s 14.37s
Monaco - node (v8.9.0, x86)
Memory used 203,746k (± 0.02%) 203,796k (± 0.02%) +50k (+ 0.02%) 203,702k 203,876k
Parse Time 1.60s (± 0.55%) 1.60s (± 0.55%) -0.00s (- 0.19%) 1.59s 1.63s
Bind Time 0.77s (± 1.35%) 0.77s (± 0.52%) -0.00s (- 0.39%) 0.76s 0.78s
Check Time 5.19s (± 1.98%) 5.18s (± 1.83%) -0.00s (- 0.04%) 5.04s 5.50s
Emit Time 3.13s (± 2.72%) 3.15s (± 2.33%) +0.02s (+ 0.51%) 2.86s 3.23s
Total Time 10.69s (± 0.42%) 10.70s (± 0.47%) +0.00s (+ 0.04%) 10.56s 10.79s
TFS - node (v8.9.0, x86)
Memory used 182,635k (± 0.03%) 182,671k (± 0.02%) +36k (+ 0.02%) 182,625k 182,723k
Parse Time 1.30s (± 0.77%) 1.30s (± 0.80%) -0.00s (- 0.15%) 1.29s 1.34s
Bind Time 0.71s (± 0.87%) 0.71s (± 0.87%) 0.00s ( 0.00%) 0.70s 0.73s
Check Time 4.58s (± 0.77%) 4.59s (± 0.53%) +0.00s (+ 0.11%) 4.55s 4.66s
Emit Time 2.96s (± 1.44%) 2.95s (± 0.48%) -0.01s (- 0.47%) 2.92s 2.98s
Total Time 9.57s (± 0.68%) 9.55s (± 0.27%) -0.01s (- 0.15%) 9.51s 9.64s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-166-generic
Architecturex64
Available Memory16 GB
Available Memory4 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v10.16.3, x64)
  • node (v12.1.0, x64)
  • node (v8.9.0, x64)
  • node (v8.9.0, x86)
Scenarios
  • Angular - node (v10.16.3, x64)
  • Angular - node (v12.1.0, x64)
  • Angular - node (v8.9.0, x64)
  • Angular - node (v8.9.0, x86)
  • Monaco - node (v10.16.3, x64)
  • Monaco - node (v12.1.0, x64)
  • Monaco - node (v8.9.0, x64)
  • Monaco - node (v8.9.0, x86)
  • TFS - node (v10.16.3, x64)
  • TFS - node (v12.1.0, x64)
  • TFS - node (v8.9.0, x64)
  • TFS - node (v8.9.0, x86)
Benchmark Name Iterations
Current 36665 10
Baseline master 10

@sandersn
Copy link
Member Author

sandersn commented Feb 7, 2020

No impact to check time, but it doesn't mean much without knowing how many overload errors are in the perf test suite. That's the next thing I need to look at.

@weswigham
Copy link
Member

Hey, it at least means that the hypothesis of "this should only affect compilations with errors" is probably true~

@sandersn
Copy link
Member Author

sandersn commented Feb 7, 2020

So, uh, here are the numbers of getCandidateForOverloadFailure calls. They are small. The number in parenthesis is the total number of resolveErrorCalls; that function is called a number of places.

Before After
Angular 0 (4415) 8 (4407)
TFS 0 (25) 0 (25)
Monaco 0 (17) 2 (14)

I'm still leaning toward yes since it appears (1) to be pay-to-use (2) angular, the project that uses it a little, on the whole checks at the same speed.

@weswigham if you agree, sign off and you or I can merge this.

@sandersn
Copy link
Member Author

sandersn commented Feb 7, 2020

Actually, I want to look over the baseline changes before I sign off. So far the look benign, but there are a lot of them.

Copy link
Member

@weswigham weswigham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty much always fine with slight degredation to error-case performance in exchange for either better performance elsewhere or better errors, since "constant and extensive errors" is not the steady state for a codebase, and this should allow us to have much better API and real-world services performance (since we'll be able to reuse the hot diagnostics-producing checker from the constantly made err event for quick info). As for the baselines, it looked to me like a lot were pretty expected - many type baselines got return types when calls had errors, many error baselines got another entry or two as a result of downstream errors exposed by this.

@sandersn sandersn merged commit 71c1da0 into master Feb 7, 2020
@sandersn sandersn deleted the getCandidateForOverloadFailure-redux branch February 7, 2020 17:55
@sandersn
Copy link
Member Author

Note: breaks jest types in the following call:

const spy3Mock = spy3
    .mockImplementation(() => '')
    .mockImplementation()
    // $ExpectError
    .mockImplementation((arg: {}) => arg)
    .mockImplementation((...args: string[]) => args.join(''))
    .mockImplementationOnce(() => '')
    .mockName('name')
    .mockReturnThis()
    .mockReturnValue('value')
    .mockReturnValueOnce('value')
    .mockResolvedValue('value')
    .mockResolvedValueOnce('value')
    .mockRejectedValue('value')
    .mockRejectedValueOnce('value');

The last four calls expect an argument of type never now instead of string. This seems wrong, or at least unintended.

@sandersn
Copy link
Member Author

sandersn commented Feb 10, 2020

Oh, the tests were ALWAYS an error, it's just the we now report downstream errors after the expected error on mockImplemention((arg: {}) => arg).
If you move the mockResolveValue calls above that line, you get the error even before this change.

The test should be making those calls on a mock of a promise, so that's probably the right fix.

@weswigham
Copy link
Member

weswigham commented Feb 11, 2020

@sandersn Somehow this has caused some concerning regressions in rwc (lol we didn't run rwc) to do with elided imports (namely we're failing to mark some imports as used and they're mistakenly elided). I don't think it's a fundamental flaw with this, but it's definitely something we need to look into. The affected projects include Azure_FrameworkTests and immutable. (You can see the diffs on the rwc diff PRs)

The immutable case is something like

//@filename: Traversable.ts
export interface ITraversable<T> {}

export function isTraversable(a: any): boolean {
    return isList(a) || isOption(a);
}

export function isList(a: any) {
    return (a instanceof _list.Cons) || (a instanceof _list.Nil);
}

export function isOption(a: any) {
    return (a instanceof _option.Some) || (a instanceof _option.None);
}
// @filename: List.ts
import _tr = require('./Traversable');
export class Cons<T> extends IList<T> {

    flatten<U>(): IList<U> {
        return this.foldLeft<IList<U>>(new Nil<U>(), (acc, t) => {
            if(_tr.isList(t)) { // usage here
                var l = <IList<U>><any> t;
                return acc.append(l);
            } else if(_tr.isOption(t)) { // and here
                var o = <_option.IOption<U>><any> t;
                if(o.isDefined()) {
                    return acc.appendOne(o.get());
                } else return acc;
            } else {
                return acc.appendOne(<U><any> t);
            }
        });
    }

}

I think we're failing to check a child function body or something?

@sandersn
Copy link
Member Author

Probably when there is an error in the parent call expression. I observed this in the user tests too.

@sandersn
Copy link
Member Author

Makes sense that we wouldn't have noticed this since our test coverage of the language service is so much worse, and getCandidateForOverloadFailure was only called from there.

@sandersn
Copy link
Member Author

After discussion with @weswigham, the fix is to make getCandidateForOverloadFailure call resolveUntypedCall the way resolveErrorCall does. resolveUntypedCall is just a barebones set of check* calls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Milestone Bug PRs that fix a bug with a specific milestone
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants