[benchmark] Janitor Duty: Sweep Quadratic #22673

palimondo · 2019-02-16T22:46:06Z

To enable robust performance measurements by minimizing the accumulated error, this splits the composite tests from DictionatyCopy and DictionaryFilter into smaller individual benchmarks by dictionary size. These benchmarks guard against the quadratic behavior (SR-3268). Also removes the older disabled HashQuadratic benchmark that was covering the same issue.

This was obsoleted by DictionaryCopy.

Split the composite tests from `DictionatyCopy` and `DictionaryFilter` into individual benchmarks by dictionary size. Lowered the workloads to run faster (more stable results).

palimondo · 2019-02-17T06:12:26Z

@swift-ci please benchmark

Running with 8a19a985c61cce4dd1964b522049b6204f0c41e4 ([DNM] Disable randomized hashSeed (test Quadratic)) that turns off the randomized hashSeed to demonstrate that the rewritten tests are still able to catch the regression.

swift-ci · 2019-02-17T07:00:03Z

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
SetSubtractingInt100	151	167	+10.6%	0.90x (?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToString	74000	80500	+8.8%	0.92x (?)
StringHashing_ascii	39	42	+7.7%	0.93x (?)
Improvement
ObjectiveCBridgeStubFromNSDateRef	5050	4190	-17.0%	1.21x (?)
ObjectiveCBridgeStubFromNSString	1020	903	-11.5%	1.13x (?)
ObjectiveCBridgeStubFromArrayOfNSString2	3730	3410	-8.6%	1.09x (?)
DictionaryRemove	3280	3030	-7.6%	1.08x (?)
Added
Dict.CopyKeyValue.16k	1137	1529	1268	—
Dict.CopyKeyValue.20k	3644	3717	3679	—
Dict.CopyKeyValue.24k	1252	1252	1252	—
Dict.CopyKeyValue.28k	1863	2376	2034	—
Dict.FilterAllMatch.16k	711	711	711	—
Dict.FilterAllMatch.20k	849	851	850	—
Dict.FilterAllMatch.24k	843	861	853	—
Dict.FilterAllMatch.28k	1333	1337	1335	—
Removed
DictionaryCopy	53497	58134	55079	—
DictionaryFilter	44472	44768	44581	—

Code size: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
DictionaryCopy.o	7885	10182	+29.1%	0.77x

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
SetIsSubsetBox0	362	462	+27.6%	0.78x
ObjectiveCBridgeFromNSSetAnyObjectToStringForced	96000	110500	+15.1%	0.87x (?)
SetSubtractingInt100	159	183	+15.1%	0.87x
ObjectiveCBridgeFromNSDictionaryAnyObject	40300	45700	+13.4%	0.88x (?)
SetSubtractingInt0	73	81	+11.0%	0.90x (?)
ObjectiveCBridgeFromNSSetAnyObject	53400	59200	+10.9%	0.90x (?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToString	73500	80500	+9.5%	0.91x (?)
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced	80000	87500	+9.4%	0.91x (?)
SetSubtractingInt25	101	110	+8.9%	0.92x (?)
SetSymmetricDifferenceInt100	220	239	+8.6%	0.92x (?)
ObjectiveCBridgeStubNSDateRefAccess	343	371	+8.2%	0.92x (?)
Improvement
Set.isDisjoint.Box25	713	610	-14.4%	1.17x
DictionaryRemove	5790	5340	-7.8%	1.08x (?)
Added
Dict.CopyKeyValue.16k	1695	2069	1820	—
Dict.CopyKeyValue.20k	4302	4365	4329	—
Dict.CopyKeyValue.24k	2057	2074	2067	—
Dict.CopyKeyValue.28k	2920	3426	3102	—
Dict.FilterAllMatch.16k	733	734	734	—
Dict.FilterAllMatch.20k	871	873	872	—
Dict.FilterAllMatch.24k	871	888	881	—
Dict.FilterAllMatch.28k	1380	1382	1381	—
Removed
DictionaryCopy	73998	79288	75766	—
DictionaryFilter	46360	46391	46373	—

Code size: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
DictionaryCopy.o	6177	7958	+28.8%	0.78x

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Regression
SetIsSubsetBox0	879	1715	+95.1%	0.51x
SetIsSubsetInt0	633	733	+15.8%	0.86x (?)
Improvement
Set.isDisjoint.Box25	2611	2040	-21.9%	1.28x
ObjectiveCBridgeStubFromNSDateRef	5480	4510	-17.7%	1.22x (?)
DictionarySwapOfObjects	22200	18800	-15.3%	1.18x (?)
DictionarySwapAtOfObjects	17600	14920	-15.2%	1.18x (?)
ObjectiveCBridgeStubFromNSDate	7470	6450	-13.7%	1.16x (?)
SetExclusiveOr_OfObjects	37210	33090	-11.1%	1.12x
SetSymmetricDifferenceBox0	3721	3311	-11.0%	1.12x
DictionarySwap	4780	4352	-9.0%	1.10x (?)
SetUnion_OfObjects	25940	23910	-7.8%	1.08x (?)
SetUnionBox0	2594	2392	-7.8%	1.08x (?)
Added
Dict.CopyKeyValue.16k	6090	6692	6294	—
Dict.CopyKeyValue.20k	26754	27239	26962	—
Dict.CopyKeyValue.24k	6917	7033	6956	—
Dict.CopyKeyValue.28k	8536	9281	8821	—
Dict.FilterAllMatch.16k	4733	4862	4777	—
Dict.FilterAllMatch.20k	24734	24899	24808	—
Dict.FilterAllMatch.24k	4824	4901	4853	—
Dict.FilterAllMatch.28k	6166	6198	6178	—
Removed
DictionaryCopy	266292	271700	268214	—
DictionaryFilter	214044	216822	214972	—

✅	Benchmark Check Report
⚠️⏱	`Dict.FilterAllMatch.28k` execution took at least 1333 μs. _{Decrease the workload of Dict.FilterAllMatch.28k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.FilterAllMatch.28k` has very wide range of memory used between independent, repeated measurements. _{Dict.FilterAllMatch.28k mem_pages [i1, i2]: min=[757, 756] 𝚫=1 R=[39, 41]}
⚠️⏱	`Dict.CopyKeyValue.20k` execution took at least 3640 μs. _{Decrease the workload of Dict.CopyKeyValue.20k by a factor of 4 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.CopyKeyValue.20k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.20k mem_pages [i1, i2]: min=[368, 368] 𝚫=0 R=[78, 78]}
⚠️⏱	`Dict.CopyKeyValue.16k` execution took at least 1119 μs. _{Decrease the workload of Dict.CopyKeyValue.16k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.CopyKeyValue.16k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.16k mem_pages [i1, i2]: min=[369, 369] 𝚫=0 R=[39, 40]}
⚠️⏱	`Dict.CopyKeyValue.28k` execution took at least 1860 μs. _{Decrease the workload of Dict.CopyKeyValue.28k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.CopyKeyValue.28k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.28k mem_pages [i1, i2]: min=[756, 756] 𝚫=0 R=[39, 0]}
⚠️Ⓜ️	`Dict.FilterAllMatch.24k` has very wide range of memory used between independent, repeated measurements. _{Dict.FilterAllMatch.24k mem_pages [i1, i2]: min=[369, 369] 𝚫=0 R=[39, 0]}
⚠️⏱	`Dict.CopyKeyValue.24k` execution took at least 1247 μs. _{Decrease the workload of Dict.CopyKeyValue.24k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.CopyKeyValue.24k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.24k mem_pages [i1, i2]: min=[368, 368] 𝚫=0 R=[39, 39]}
⚠️Ⓜ️	`Dict.FilterAllMatch.20k` has very wide range of memory used between independent, repeated measurements. _{Dict.FilterAllMatch.20k mem_pages [i1, i2]: min=[369, 369] 𝚫=0 R=[39, 39]}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

palimondo · 2019-02-17T07:13:25Z

@swift-ci benchmark

palimondo · 2019-02-17T07:14:01Z

@swift-ci please smoke test

swift-ci · 2019-02-17T07:59:57Z

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
DataAppendDataLargeToLarge	38600	51200	+32.6%	0.75x (?)
Improvement
StringBuilderLong	1370	1230	-10.2%	1.11x (?)
Added
Dict.CopyKeyValue.16k	990	1387	1123	—
Dict.CopyKeyValue.20k	1106	1106	1106	—
Dict.CopyKeyValue.24k	1245	1246	1246	—
Dict.CopyKeyValue.28k	1893	2403	2064	—
Dict.FilterAllMatch.16k	715	715	715	—
Dict.FilterAllMatch.20k	781	782	781	—
Dict.FilterAllMatch.24k	848	849	848	—
Dict.FilterAllMatch.28k	1394	1440	1410	—
Removed
DictionaryCopy	53472	58353	55193	—
DictionaryFilter	44538	44639	44583	—

Code size: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
DictionaryCopy.o	7885	10182	+29.1%	0.77x

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
SetSubtractingInt25	102	111	+8.8%	0.92x (?)
ObjectiveCBridgeStubNSDateRefAccess	343	371	+8.2%	0.92x (?)
Improvement
DataAppendDataLargeToLarge	51000	37400	-26.7%	1.36x (?)
Added
Dict.CopyKeyValue.16k	1587	1961	1717	—
Dict.CopyKeyValue.20k	1745	1762	1754	—
Dict.CopyKeyValue.24k	1946	1967	1960	—
Dict.CopyKeyValue.28k	3050	3572	3225	—
Dict.FilterAllMatch.16k	742	745	744	—
Dict.FilterAllMatch.20k	811	812	811	—
Dict.FilterAllMatch.24k	879	887	882	—
Dict.FilterAllMatch.28k	1449	1464	1455	—
Removed
DictionaryCopy	73938	78856	75645	—
DictionaryFilter	46272	46457	46371	—

Code size: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
DictionaryCopy.o	6177	7958	+28.8%	0.78x

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Improvement
ObjectiveCBridgeStubFromNSDateRef	5480	4510	-17.7%	1.22x (?)
ObjectiveCBridgeStubFromNSDate	7460	6450	-13.5%	1.16x (?)
Added
Dict.CopyKeyValue.16k	4662	5314	4881	—
Dict.CopyKeyValue.20k	5548	5683	5636	—
Dict.CopyKeyValue.24k	6502	6699	6570	—
Dict.CopyKeyValue.28k	8619	9410	8884	—
Dict.FilterAllMatch.16k	3235	3280	3252	—
Dict.FilterAllMatch.20k	3774	3816	3788	—
Dict.FilterAllMatch.24k	4376	4424	4392	—
Dict.FilterAllMatch.28k	6096	6120	6110	—
Removed
DictionaryCopy	297362	305285	300035	—
DictionaryFilter	214145	214664	214334	—

✅	Benchmark Check Report
⚠️⏱	`Dict.FilterAllMatch.28k` execution took at least 1392 μs. _{Decrease the workload of Dict.FilterAllMatch.28k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.FilterAllMatch.28k` has very wide range of memory used between independent, repeated measurements. _{Dict.FilterAllMatch.28k mem_pages [i1, i2]: min=[757, 757] 𝚫=0 R=[39, 39]}
⚠️⏱	`Dict.CopyKeyValue.20k` execution took at least 1105 μs. _{Decrease the workload of Dict.CopyKeyValue.20k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.CopyKeyValue.20k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.20k mem_pages [i1, i2]: min=[368, 368] 𝚫=0 R=[39, 40]}
⚠️Ⓜ️	`Dict.CopyKeyValue.16k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.16k mem_pages [i1, i2]: min=[369, 368] 𝚫=1 R=[39, 40]}
⚠️⏱	`Dict.CopyKeyValue.28k` execution took at least 1888 μs. _{Decrease the workload of Dict.CopyKeyValue.28k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.CopyKeyValue.28k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.28k mem_pages [i1, i2]: min=[756, 755] 𝚫=1 R=[39, 40]}
⚠️Ⓜ️	`Dict.FilterAllMatch.24k` has very wide range of memory used between independent, repeated measurements. _{Dict.FilterAllMatch.24k mem_pages [i1, i2]: min=[369, 369] 𝚫=0 R=[0, 39]}
⚠️Ⓜ️	`Dict.FilterAllMatch.16k` has very wide range of memory used between independent, repeated measurements. _{Dict.FilterAllMatch.16k mem_pages [i1, i2]: min=[369, 369] 𝚫=0 R=[39, 0]}
⚠️⏱	`Dict.CopyKeyValue.24k` execution took at least 1244 μs. _{Decrease the workload of Dict.CopyKeyValue.24k by a factor of 2 (10), to be less than 1000 μs.}
⚠️Ⓜ️	`Dict.CopyKeyValue.24k` has very wide range of memory used between independent, repeated measurements. _{Dict.CopyKeyValue.24k mem_pages [i1, i2]: min=[368, 368] 𝚫=0 R=[39, 39]}
⚠️Ⓜ️	`Dict.FilterAllMatch.20k` has very wide range of memory used between independent, repeated measurements. _{Dict.FilterAllMatch.20k mem_pages [i1, i2]: min=[369, 369] 𝚫=0 R=[0, 39]}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

Change the test to work after removal of `HashQuadratic`.

palimondo · 2019-02-18T09:09:54Z

@swift-ci please test

Running full tests to verify the adjusted Benchmark_O tests pass after the removal of HashQuadratic.

palimondo · 2019-02-18T09:52:01Z

@eeckstein @lorentey Please review 🙏

palimondo · 2019-02-18T20:57:44Z

@swift-ci smoke test linux platform

palimondo · 2019-02-18T22:09:40Z

@eeckstein I believe this is now mergeable: the full test that runs Benchmark_O.test.md passed on mac os and the previously broken Linux smoke test (unrelated to this PR) now also passed (full test there is not required as the benchmark validation does not run on Linux).

As for the new benchmarks, the first benchmark run with disabled hashSeed randomization demonstrates how the 20k variants (in particular) successfully catch the quadratic behavior even though they run in substantially shorter time then the 2 removed benchmarks. Many of these did not make it under 1000 μs, but that's OK, they are all below 2000 μs — 5 times lower than the 10k scheduler quantum. These are much more robust against system noise than before.

eeckstein · 2019-02-18T23:16:38Z

I'll let @lorentey review this

lorentey

The code reorganization looks good to me! I don't much mind losing the old variants here -- although I guess they did have some value in release-to-release comparisons, I expect we have other benchmarks for insertion at least.

I have one question about the results, though: If I read the benchmark results correctly, the Quadratic Surprise doesn't show up in the filter benchmarks. I can see the bump in Dict.CopyKeyValue.20k, but there is no corresponding increase for Dict.FilterAllMatch.20k -- all the filter results seem to be roughly the same.

The implementation of filter will change at some point soon, but I'm wondering if 16k--28k is large enough to trigger the issue.

benchmark/single-source/DictionaryCopy.swift

palimondo · 2019-02-19T00:24:58Z

I have also noticed the filter has been much flatter in response, but the quadratic behavior is still fully visible in the -Onone build. The filter in optimized builds had less pronounced peak at the low end since PR #19213 when you switched it to work directly with native dictionary.

lorentey · 2019-02-20T00:38:03Z

Ah, that makes sense 👍

palimondo · 2019-02-20T00:42:50Z

@lorentey Thank you!

palimondo added 2 commits February 17, 2019 07:09

[benchmark] Remove HashQuadratic

16af31a

This was obsoleted by DictionaryCopy.

[benchmark] Dict.[CopyKeyValue, FilterAllMatch]

8e67642

Split the composite tests from `DictionatyCopy` and `DictionaryFilter` into individual benchmarks by dictionary size. Lowered the workloads to run faster (more stable results).

palimondo force-pushed the a-tall-white-fountain-played branch from bfd3c74 to 8a19a98 Compare February 17, 2019 06:11

swiftlang deleted a comment from swift-ci Feb 17, 2019

palimondo force-pushed the a-tall-white-fountain-played branch from 8a19a98 to 8e67642 Compare February 17, 2019 07:12

[benchmark] Adjust lit tests for Benchmark_O

e13e90b

Change the test to work after removal of `HashQuadratic`.

palimondo changed the title ~~[benchmark] Janitor Duty: Swipe Quadratic~~ [benchmark] Janitor Duty: Sweep Quadratic Feb 18, 2019

palimondo requested review from lorentey and eeckstein February 18, 2019 09:51

This comment has been minimized.

Sign in to view

palimondo mentioned this pull request Feb 18, 2019

[benchmark] Add dataUsingUTF8Encoding() #22648

Merged

lorentey requested changes Feb 18, 2019

View reviewed changes

benchmark/single-source/DictionaryCopy.swift Show resolved Hide resolved

lorentey approved these changes Feb 20, 2019

View reviewed changes

palimondo merged commit 4ca08a5 into swiftlang:master Feb 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[benchmark] Janitor Duty: Sweep Quadratic #22673

[benchmark] Janitor Duty: Sweep Quadratic #22673

Uh oh!

palimondo commented Feb 16, 2019 •

edited

Loading

Uh oh!

palimondo commented Feb 17, 2019 •

edited

Loading

Uh oh!

swift-ci commented Feb 17, 2019

Uh oh!

palimondo commented Feb 17, 2019

Uh oh!

palimondo commented Feb 17, 2019

Uh oh!

swift-ci commented Feb 17, 2019

Uh oh!

palimondo commented Feb 18, 2019 •

edited

Loading

Uh oh!

palimondo commented Feb 18, 2019

Uh oh!

This comment has been minimized.

palimondo commented Feb 18, 2019

Uh oh!

palimondo commented Feb 18, 2019 •

edited

Loading

Uh oh!

eeckstein commented Feb 18, 2019

Uh oh!

lorentey left a comment

Uh oh!

Uh oh!

palimondo commented Feb 19, 2019

Uh oh!

lorentey commented Feb 20, 2019

Uh oh!

palimondo commented Feb 20, 2019

Uh oh!

Uh oh!

[benchmark] Janitor Duty: Sweep Quadratic #22673

[benchmark] Janitor Duty: Sweep Quadratic #22673

Uh oh!

Conversation

palimondo commented Feb 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

palimondo commented Feb 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swift-ci commented Feb 17, 2019

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Uh oh!

palimondo commented Feb 17, 2019

Uh oh!

palimondo commented Feb 17, 2019

Uh oh!

swift-ci commented Feb 17, 2019

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Uh oh!

palimondo commented Feb 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

palimondo commented Feb 18, 2019

Uh oh!

This comment has been minimized.

palimondo commented Feb 18, 2019

Uh oh!

palimondo commented Feb 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eeckstein commented Feb 18, 2019

Uh oh!

lorentey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

palimondo commented Feb 19, 2019

Uh oh!

lorentey commented Feb 20, 2019

Uh oh!

palimondo commented Feb 20, 2019

Uh oh!

Uh oh!

palimondo commented Feb 16, 2019 •

edited

Loading

palimondo commented Feb 17, 2019 •

edited

Loading

palimondo commented Feb 18, 2019 •

edited

Loading

palimondo commented Feb 18, 2019 •

edited

Loading