Skip to content

Perf test PR for "Optimize source mapping into external source map sources" #40126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 19, 2020
Merged

Perf test PR for "Optimize source mapping into external source map sources" #40126

merged 1 commit into from
Aug 19, 2020

Conversation

DanielRosenwasser
Copy link
Member

Reverts #40055

@typescript-bot typescript-bot added the For Uncommitted Bug PR for untriaged, rejected, closed or missing bug label Aug 19, 2020
@DanielRosenwasser
Copy link
Member Author

@typescript-bot test this
@typescript-bot user test this
@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 19, 2020 •
edited
Loading

Heya @DanielRosenwasser, I've started to run the parallelized community code test suite on this PR at 364a232. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 19, 2020 •
edited
Loading

Heya @DanielRosenwasser, I've started to run the extended test suite on this PR at 364a232. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 19, 2020 •
edited
Loading

Heya @DanielRosenwasser, I've started to run the perf test suite on this PR at 364a232. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

Comparison Report - master..40126

Metric master 40126 Delta Best Worst
Angular - node (v10.16.3, x64)
Memory used 343,680k (± 0.02%) 343,186k (± 0.01%) -495k (- 0.14%) 343,091k 343,307k
Parse Time 2.01s (± 0.58%) 2.00s (± 0.61%) -0.01s (- 0.50%) 1.98s 2.03s
Bind Time 0.82s (± 0.61%) 0.82s (± 1.48%) +0.01s (+ 0.74%) 0.80s 0.86s
Check Time 4.77s (± 0.68%) 4.78s (± 0.62%) +0.01s (+ 0.19%) 4.74s 4.88s
Emit Time 5.18s (± 0.78%) 5.19s (± 0.69%) +0.01s (+ 0.19%) 5.13s 5.28s
Total Time 12.77s (± 0.53%) 12.79s (± 0.50%) +0.02s (+ 0.16%) 12.69s 13.00s
Monaco - node (v10.16.3, x64)
Memory used 339,340k (± 0.02%) 339,267k (± 0.04%) -73k (- 0.02%) 339,061k 339,623k
Parse Time 1.56s (± 0.54%) 1.55s (± 0.44%) -0.02s (- 1.02%) 1.53s 1.56s
Bind Time 0.71s (± 0.67%) 0.71s (± 0.56%) -0.00s (- 0.56%) 0.70s 0.72s
Check Time 4.97s (± 0.37%) 4.92s (± 0.41%) -0.05s (- 1.07%) 4.89s 4.96s
Emit Time 2.75s (± 0.85%) 2.74s (± 0.82%) -0.01s (- 0.36%) 2.71s 2.81s
Total Time 10.00s (± 0.38%) 9.91s (± 0.26%) -0.09s (- 0.85%) 9.86s 9.96s
TFS - node (v10.16.3, x64)
Memory used 302,055k (± 0.02%) 302,146k (± 0.09%) +91k (+ 0.03%) 301,833k 303,220k
Parse Time 1.21s (± 0.84%) 1.21s (± 0.55%) -0.00s (- 0.17%) 1.19s 1.22s
Bind Time 0.67s (± 0.54%) 0.67s (± 0.71%) -0.00s (- 0.15%) 0.66s 0.68s
Check Time 4.45s (± 0.64%) 4.44s (± 0.55%) -0.02s (- 0.38%) 4.38s 4.48s
Emit Time 2.90s (± 0.94%) 2.88s (± 1.37%) -0.02s (- 0.59%) 2.76s 2.97s
Total Time 9.24s (± 0.59%) 9.20s (± 0.69%) -0.04s (- 0.42%) 9.01s 9.28s
material-ui - node (v10.16.3, x64)
Memory used 460,548k (± 0.01%) 460,193k (± 0.01%) -355k (- 0.08%) 460,034k 460,341k
Parse Time 1.96s (± 0.58%) 1.95s (± 0.82%) -0.01s (- 0.41%) 1.92s 1.99s
Bind Time 0.66s (± 1.13%) 0.65s (± 1.16%) -0.01s (- 0.91%) 0.64s 0.67s
Check Time 13.40s (± 0.49%) 13.36s (± 0.56%) -0.04s (- 0.31%) 13.15s 13.51s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 16.02s (± 0.46%) 15.96s (± 0.45%) -0.05s (- 0.34%) 15.77s 16.10s
Angular - node (v12.1.0, x64)
Memory used 320,664k (± 0.14%) 320,405k (± 0.03%) -259k (- 0.08%) 320,172k 320,661k
Parse Time 1.99s (± 0.65%) 1.98s (± 0.53%) -0.01s (- 0.50%) 1.96s 2.01s
Bind Time 0.80s (± 0.37%) 0.81s (± 0.49%) +0.01s (+ 0.87%) 0.80s 0.82s
Check Time 4.66s (± 0.44%) 4.63s (± 0.35%) -0.02s (- 0.47%) 4.60s 4.66s
Emit Time 5.38s (± 1.14%) 5.35s (± 0.70%) -0.03s (- 0.50%) 5.30s 5.46s
Total Time 12.83s (± 0.53%) 12.78s (± 0.29%) -0.05s (- 0.36%) 12.72s 12.87s
Monaco - node (v12.1.0, x64)
Memory used 321,639k (± 0.01%) 321,537k (± 0.01%) -103k (- 0.03%) 321,402k 321,615k
Parse Time 1.53s (± 0.93%) 1.53s (± 0.65%) +0.00s (+ 0.13%) 1.50s 1.55s
Bind Time 0.69s (± 0.52%) 0.69s (± 0.75%) -0.00s (- 0.43%) 0.68s 0.70s
Check Time 4.74s (± 0.30%) 4.73s (± 0.30%) -0.01s (- 0.17%) 4.70s 4.77s
Emit Time 2.81s (± 0.79%) 2.80s (± 0.56%) -0.01s (- 0.50%) 2.77s 2.83s
Total Time 9.78s (± 0.46%) 9.75s (± 0.18%) -0.03s (- 0.28%) 9.72s 9.79s
TFS - node (v12.1.0, x64)
Memory used 286,562k (± 0.02%) 286,597k (± 0.03%) +35k (+ 0.01%) 286,429k 286,805k
Parse Time 1.23s (± 0.74%) 1.24s (± 0.77%) +0.01s (+ 0.49%) 1.21s 1.26s
Bind Time 0.64s (± 1.06%) 0.64s (± 0.62%) -0.00s (- 0.47%) 0.63s 0.65s
Check Time 4.34s (± 0.56%) 4.36s (± 0.67%) +0.02s (+ 0.41%) 4.30s 4.43s
Emit Time 2.92s (± 0.98%) 2.91s (± 0.73%) -0.01s (- 0.17%) 2.85s 2.95s
Total Time 9.13s (± 0.58%) 9.15s (± 0.45%) +0.02s (+ 0.18%) 9.06s 9.23s
material-ui - node (v12.1.0, x64)
Memory used 438,819k (± 0.01%) 438,610k (± 0.01%) -209k (- 0.05%) 438,466k 438,733k
Parse Time 1.98s (± 0.69%) 1.98s (± 0.56%) -0.00s (- 0.10%) 1.96s 2.01s
Bind Time 0.63s (± 0.98%) 0.63s (± 1.19%) -0.00s (- 0.32%) 0.61s 0.64s
Check Time 11.96s (± 0.97%) 11.97s (± 0.94%) +0.01s (+ 0.08%) 11.80s 12.22s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 14.57s (± 0.80%) 14.57s (± 0.83%) +0.00s (+ 0.03%) 14.39s 14.87s
Angular - node (v8.9.0, x64)
Memory used 340,243k (± 0.02%) 339,804k (± 0.01%) -438k (- 0.13%) 339,708k 339,913k
Parse Time 2.55s (± 0.84%) 2.53s (± 0.47%) -0.02s (- 0.74%) 2.51s 2.57s
Bind Time 0.85s (± 1.07%) 0.85s (± 0.26%) 0.00s ( 0.00%) 0.84s 0.85s
Check Time 5.41s (± 0.59%) 5.40s (± 0.47%) -0.01s (- 0.20%) 5.33s 5.45s
Emit Time 5.92s (± 1.53%) 5.89s (± 1.38%) -0.03s (- 0.49%) 5.71s 6.04s
Total Time 14.72s (± 0.74%) 14.67s (± 0.58%) -0.06s (- 0.39%) 14.47s 14.84s
Monaco - node (v8.9.0, x64)
Memory used 340,574k (± 0.01%) 340,581k (± 0.01%) +7k (+ 0.00%) 340,530k 340,666k
Parse Time 1.88s (± 0.75%) 1.87s (± 0.66%) -0.01s (- 0.27%) 1.85s 1.91s
Bind Time 0.88s (± 0.56%) 0.88s (± 0.54%) -0.00s (- 0.11%) 0.88s 0.90s
Check Time 5.46s (± 0.64%) 5.48s (± 0.49%) +0.02s (+ 0.31%) 5.43s 5.56s
Emit Time 3.23s (± 0.59%) 3.23s (± 0.37%) +0.00s (+ 0.12%) 3.20s 3.26s
Total Time 11.45s (± 0.40%) 11.46s (± 0.33%) +0.01s (+ 0.09%) 11.38s 11.57s
TFS - node (v8.9.0, x64)
Memory used 303,862k (± 0.01%) 303,883k (± 0.02%) +20k (+ 0.01%) 303,815k 304,005k
Parse Time 1.55s (± 0.57%) 1.55s (± 0.64%) +0.00s (+ 0.06%) 1.53s 1.58s
Bind Time 0.68s (± 0.73%) 0.68s (± 0.66%) +0.00s (+ 0.30%) 0.67s 0.69s
Check Time 5.18s (± 1.04%) 5.20s (± 0.47%) +0.03s (+ 0.50%) 5.13s 5.26s
Emit Time 2.97s (± 1.55%) 2.94s (± 1.32%) -0.03s (- 0.91%) 2.85s 3.03s
Total Time 10.36s (± 0.36%) 10.37s (± 0.50%) +0.00s (+ 0.04%) 10.25s 10.49s
material-ui - node (v8.9.0, x64)
Memory used 464,884k (± 0.01%) 464,525k (± 0.01%) -359k (- 0.08%) 464,418k 464,665k
Parse Time 2.39s (± 0.43%) 2.38s (± 0.71%) -0.01s (- 0.25%) 2.35s 2.43s
Bind Time 0.79s (± 1.06%) 0.78s (± 1.56%) -0.01s (- 0.76%) 0.76s 0.81s
Check Time 18.03s (± 1.06%) 17.97s (± 0.69%) -0.06s (- 0.33%) 17.57s 18.23s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 21.20s (± 0.93%) 21.13s (± 0.59%) -0.07s (- 0.34%) 20.73s 21.37s
Angular - node (v8.9.0, x86)
Memory used 195,224k (± 0.03%) 195,043k (± 0.03%) -181k (- 0.09%) 194,903k 195,165k
Parse Time 2.45s (± 0.34%) 2.45s (± 0.65%) -0.00s (- 0.04%) 2.42s 2.50s
Bind Time 0.99s (± 0.75%) 0.98s (± 0.84%) -0.01s (- 0.91%) 0.97s 1.00s
Check Time 4.88s (± 0.52%) 4.87s (± 0.61%) -0.01s (- 0.16%) 4.81s 4.94s
Emit Time 5.94s (± 1.02%) 5.93s (± 0.85%) -0.00s (- 0.08%) 5.84s 6.07s
Total Time 14.26s (± 0.49%) 14.23s (± 0.44%) -0.03s (- 0.18%) 14.09s 14.38s
Monaco - node (v8.9.0, x86)
Memory used 193,571k (± 0.02%) 193,593k (± 0.02%) +22k (+ 0.01%) 193,510k 193,684k
Parse Time 1.90s (± 0.91%) 1.91s (± 0.47%) +0.01s (+ 0.26%) 1.90s 1.94s
Bind Time 0.70s (± 0.47%) 0.70s (± 1.17%) +0.00s (+ 0.57%) 0.69s 0.73s
Check Time 5.58s (± 0.54%) 5.61s (± 0.76%) +0.03s (+ 0.54%) 5.53s 5.75s
Emit Time 2.68s (± 1.14%) 2.69s (± 0.89%) +0.01s (+ 0.30%) 2.64s 2.76s
Total Time 10.86s (± 0.50%) 10.90s (± 0.48%) +0.04s (+ 0.41%) 10.80s 11.03s
TFS - node (v8.9.0, x86)
Memory used 173,818k (± 0.02%) 173,790k (± 0.02%) -28k (- 0.02%) 173,735k 173,850k
Parse Time 1.58s (± 0.70%) 1.59s (± 1.49%) +0.01s (+ 0.51%) 1.56s 1.67s
Bind Time 0.64s (± 1.28%) 0.65s (± 1.08%) +0.01s (+ 0.78%) 0.64s 0.67s
Check Time 4.71s (± 0.63%) 4.71s (± 0.57%) -0.00s (- 0.06%) 4.64s 4.76s
Emit Time 2.80s (± 1.01%) 2.81s (± 0.59%) +0.01s (+ 0.29%) 2.78s 2.85s
Total Time 9.73s (± 0.36%) 9.75s (± 0.48%) +0.01s (+ 0.14%) 9.68s 9.86s
material-ui - node (v8.9.0, x86)
Memory used 263,211k (± 0.02%) 263,053k (± 0.02%) -158k (- 0.06%) 262,959k 263,138k
Parse Time 2.45s (± 0.73%) 2.44s (± 0.39%) -0.01s (- 0.29%) 2.41s 2.46s
Bind Time 0.68s (± 1.86%) 0.68s (± 1.24%) +0.00s (+ 0.00%) 0.66s 0.70s
Check Time 16.46s (± 0.57%) 16.51s (± 0.65%) +0.05s (+ 0.29%) 16.33s 16.73s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 19.59s (± 0.45%) 19.63s (± 0.54%) +0.04s (+ 0.19%) 19.44s 19.86s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-166-generic
Architecturex64
Available Memory16 GB
Available Memory1 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v10.16.3, x64)
  • node (v12.1.0, x64)
  • node (v8.9.0, x64)
  • node (v8.9.0, x86)
Scenarios
  • Angular - node (v10.16.3, x64)
  • Angular - node (v12.1.0, x64)
  • Angular - node (v8.9.0, x64)
  • Angular - node (v8.9.0, x86)
  • Monaco - node (v10.16.3, x64)
  • Monaco - node (v12.1.0, x64)
  • Monaco - node (v8.9.0, x64)
  • Monaco - node (v8.9.0, x86)
  • TFS - node (v10.16.3, x64)
  • TFS - node (v12.1.0, x64)
  • TFS - node (v8.9.0, x64)
  • TFS - node (v8.9.0, x86)
  • material-ui - node (v10.16.3, x64)
  • material-ui - node (v12.1.0, x64)
  • material-ui - node (v8.9.0, x64)
  • material-ui - node (v8.9.0, x86)
Benchmark Name Iterations
Current 40126 10
Baseline master 10

@DanielRosenwasser
Copy link
Member Author

@sheetalkamat looks like this may have actually been a (very minor) regression. Do you think that's just variance?

@JoostK JoostK mentioned this pull request Aug 19, 2020
Merged
@JoostK
Copy link
Contributor

JoostK commented Aug 19, 2020

@DanielRosenwasser thanks for running the perf tests, I do agree that it looks like a slight regression.

I opened #40130 that implements the alternative I mentioned in #40055 (comment), which does away with the full cache and fully removes the overhead from the source map generator (I actually like this approach much better than my original PR, looking at it now). Could you please trigger a perf run for that one as well?

Unrelated: I am being requested to sign the CLA in #40130 (comment), which I wasn't in #40055. Maybe something you'll want to look into.

@sheetalkamat
Copy link
Member

I think it would be better to merge this revert and then rebase #40130 and then run perf test on this.

@DanielRosenwasser DanielRosenwasser merged commit 31f4957 into master Aug 19, 2020
@DanielRosenwasser DanielRosenwasser deleted the revert-40055-optimize-external-source-map-source branch August 19, 2020 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@DanielRosenwasser @typescript-bot @JoostK @sheetalkamat