Rally benchmark #1522

aspacca · 2023-10-24T01:29:20Z

adding command to run rally benchmark

some repetition of the code between system and rally: it is no 100% the same, not sure if trying to factorize it and move to common
require esrally installed on the host, available from $PATH
missing docs

$ pwd
elastic-package/test/packages/benchmarks/rally_benchmark
$ elastic-package benchmark rally --benchmark logs-benchmark
Run rally benchmarks for the package
--- Benchmark results for package: rally_benchmarks - START ---
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                                             │
├────────────────────────┬─────────────────────────────────────────────────────────────────────────┤
│ benchmark              │                                                          logs-benchmark │
│ description            │                                         Benchmark 20000 events ingested │
│ run ID                 │                                    85c5f94b-d53a-4b76-b897-25a5e8df6f22 │
│ package                │                                                        rally_benchmarks │
│ start ts (s)           │                                                              1698117981 │
│ end ts (s)             │                                                              1698118001 │
│ duration               │                                                                     20s │
│ generated corpora file │                   ~/.elastic-package/tmp/rally_corpus/corpus-1222483366 │
╰────────────────────────┴─────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────╮
│ parameters                                                         │
├─────────────────────────────────┬──────────────────────────────────┤
│ package version                 │                      999.999.999 │
│ input                           │                       filestream │
│ data_stream.name                │                           testds │
│ data_stream.vars.paths          │                     [dummy path] │
│ warmup time period              │                              10s │
│ corpora.generator.total_events  │                            20000 │
│ corpora.generator.template.path │ ./logs-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                  │
│ corpora.generator.template.type │                           gotext │
│ corpora.generator.config.path   │      ./logs-benchmark/config.yml │
│ corpora.generator.config.raw    │                            map[] │
│ corpora.generator.fields.path   │      ./logs-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                            map[] │
╰─────────────────────────────────┴──────────────────────────────────╯
╭───────────────────────╮
│ cluster info          │
├───────┬───────────────┤
│ name  │ elasticsearch │
│ nodes │             1 │
╰───────┴───────────────╯
╭──────────────────────────────────────────────────────────────╮
│ data stream stats                                            │
├────────────────────────────┬─────────────────────────────────┤
│ data stream                │ logs-rally_benchmarks.testds-ep │
│ approx total docs ingested │                               0 │
│ backing indices            │                               1 │
│ store size bytes           │                             269 │
│ maximum ts (ms)            │                               0 │
╰────────────────────────────┴─────────────────────────────────╯
╭────────────────────────────────────╮
│ disk usage for index .ds-logs-rall │
│ y_benchmarks.testds-ep-2023.10.23- │
│ 000001 (for all fields)            │
├──────────────────────────────┬─────┤
│ total                        │ 0 B │
│ inverted_index.total         │ 0 B │
│ inverted_index.stored_fields │ 0 B │
│ inverted_index.doc_values    │ 0 B │
│ inverted_index.points        │ 0 B │
│ inverted_index.norms         │ 0 B │
│ inverted_index.term_vectors  │ 0 B │
│ inverted_index.knn_vectors   │ 0 B │
╰──────────────────────────────┴─────╯
╭─────────────────────────────────────────────────────────────────────────────────────────╮
│ pipeline logs-rally_benchmarks.testds-999.999.999 stats in node 0VW3SV6bRRue_LPWFWiOtQ  │
├────────────────────────────────────────────────┬────────────────────────────────────────┤
│ Totals                                         │ Count: 20000 | Failed: 0 | Time: 498ms │
│ grok ()                                        │ Count: 20000 | Failed: 0 | Time: 455ms │
│ user_agent ()                                  │  Count: 20000 | Failed: 0 | Time: 24ms │
│ pipeline (logs-rally_benchmarks.testds@custom) │   Count: 20000 | Failed: 0 | Time: 3ms │
╰────────────────────────────────────────────────┴────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                │
├────────────────────────────────────────────────────────────────┬───────────────────────────┤
│ Cumulative indexing time of primary shards                     │     2.374416666666667 min │
│ Min cumulative indexing time across primary shards             │                     0 min │
│ Median cumulative indexing time across primary shards          │   0.02196666666666667 min │
│ Max cumulative indexing time across primary shards             │   0.35408333333333336 min │
│ Cumulative indexing throttle time of primary shards            │                     0 min │
│ Min cumulative indexing throttle time across primary shards    │                     0 min │
│ Median cumulative indexing throttle time across primary shards │                     0 min │
│ Max cumulative indexing throttle time across primary shards    │                     0 min │
│ Cumulative merge time of primary shards                        │    0.3455833333333333 min │
│ Cumulative merge count of primary shards                       │                       577 │
│ Min cumulative merge time across primary shards                │                     0 min │
│ Median cumulative merge time across primary shards             │ 0.0030833333333333333 min │
│ Max cumulative merge time across primary shards                │   0.06146666666666667 min │
│ Cumulative merge throttle time of primary shards               │                     0 min │
│ Min cumulative merge throttle time across primary shards       │                     0 min │
│ Median cumulative merge throttle time across primary shards    │                     0 min │
│ Max cumulative merge throttle time across primary shards       │                     0 min │
│ Cumulative refresh time of primary shards                      │    0.2704333333333333 min │
│ Cumulative refresh count of primary shards                     │                     18634 │
│ Min cumulative refresh time across primary shards              │                     0 min │
│ Median cumulative refresh time across primary shards           │  0.005966666666666666 min │
│ Max cumulative refresh time across primary shards              │  0.018783333333333332 min │
│ Cumulative flush time of primary shards                        │    11.530216666666666 min │
│ Cumulative flush count of primary shards                       │                     18178 │
│ Min cumulative flush time across primary shards                │ 6.666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │   0.16653333333333334 min │
│ Max cumulative flush time across primary shards                │    0.7956666666666667 min │
│ Total Young Gen GC time                                        │                   0.017 s │
│ Total Young Gen GC count                                       │                         3 │
│ Total Old Gen GC time                                          │                       0 s │
│ Total Old Gen GC count                                         │                         0 │
│ Store size                                                     │    0.06993633136153221 GB │
│ Translog size                                                  │  0.0001253066584467888 GB │
│ Heap used for segments                                         │                      0 MB │
│ Heap used for doc values                                       │                      0 MB │
│ Heap used for terms                                            │                      0 MB │
│ Heap used for norms                                            │                      0 MB │
│ Heap used for points                                           │                      0 MB │
│ Heap used for stored fields                                    │                      0 MB │
│ Segment count                                                  │                       497 │
│ Total Ingest Pipeline count                                    │                     20026 │
│ Total Ingest Pipeline time                                     │                   0.741 s │
│ Total Ingest Pipeline failed                                   │                         0 │
│ Min Throughput                                                 │           62131.30 docs/s │
│ Mean Throughput                                                │           62131.30 docs/s │
│ Median Throughput                                              │           62131.30 docs/s │
│ Max Throughput                                                 │           62131.30 docs/s │
│ 50th percentile latency                                        │     253.73077099999898 ms │
│ 100th percentile latency                                       │     269.62587499999916 ms │
│ 50th percentile service time                                   │     253.73077099999898 ms │
│ 100th percentile service time                                  │     269.62587499999916 ms │
│ error rate                                                     │                  100.00 % │
╰────────────────────────────────────────────────────────────────┴───────────────────────────╯

--- Benchmark results for package: rally_benchmarks - END   ---
Done

@marc-gr: I have a doubt about warm-up period. it is run in a goroutine in the setup method, rally will be executed before that. I'm not sure how does it works for system benchmark. what is it supposed to achieve the warm up period?

aspacca · 2023-10-24T05:09:19Z

closes #1475

ruflin · 2023-10-26T12:46:54Z

README.md

+
+These benchmarks allow you to benchmark an integration corpus with rally.
+
+For details on how to configure rally benchmarks for a package, review the [HOWTO guide](./docs/howto/rally_benchmarking.md).


Is this link correct? Trying to find it in this PR.

the link is correct. but I haven't yet written the docs :)

Can we add these to the PR? I was looking for these as I hit some issues testing the PR. Will comment on more on the issue I hit soon.

yes, it's planned to add to this PR: in the description I listed missing docs :)

Will give it a try when there are docs 🙂

internal/benchrunner/runners/rally/report.go

ruflin · 2023-10-30T08:31:13Z

I'm running these commands to do some testing. Few findings:

elastic-package build fails with package with GA version (999.999.999) is using an unreleased version of the spec (3.0.1-next) (PSR00001). Interestingly enough the rally benchmark runs. Did it install the package?
Keep generated rally track: Is there an option that I can keep the generated rally track file? I checked the directory that was referenced but it seems it is empty after the run. I assume it cleans it up? It would be nice to be able to generate only the rally track so others can run it later or we can run it multiple times for comparison. This would also help to review the rally track itself. I see you have "defer-cleanup" but that only seems temporary?
esrally installation: If rally is not there, can we ask users to run pip3 install esrally or point them to the docs directly?

aspacca · 2023-10-30T08:55:18Z

elastic-package build fails with package with GA version (999.999.999) is using an unreleased version of the spec (3.0.1-next) (PSR00001). Interestingly enough the rally benchmark runs. Did it install the package?

I was not aware of this: during my test I used go run main.go
Before merging the PR the version of the spec will be 3.0.1 (most likely), ie: a released version of the spec

Keep generated rally track: Is there an option that I can keep the generated rally track file? I checked the directory that was referenced but it seems it is empty after the run. I assume it cleans it up? It would be nice to be able to generate only the rally track so others can run it later or we can run it multiple times for comparison. This would also help to review the rally track itself. I see you have "defer-cleanup" but that only seems temporary?

We have a separated command for generating rally track without running the benchmark: elastic-package benchmark generate-corpus --rally-track-output-dir. currently this command still uses the generator assets from the generator repo, but I was planning to refactor it to use the assets in the package. maybe it's better to drop at all the command and add an option to save the track/do not run rally directly in the new command of this PR

esrally installation: If rally is not there, can we ask users to run pip3 install esrally or point them to the docs directly?

👍

ruflin · 2023-10-30T09:07:30Z

We have a separated command for generating rally track without running the benchmark: elastic-package benchmark generate-corpus --rally-track-output-dir. currently this command still uses the generator assets from the generator repo, but I was planning to refactor it to use the assets in the package. maybe it's better to drop at all the command and add an option to save the track/do not run rally directly in the new command of this PR

As a first step, lets make sure it all uses the assets from the package. I agree, unifying the commands might make sense but we could also solve it with docs pointing to it. Key is that the outcome is the same, meaning if I run the rally track "live" or generate the corpus, same data is in.

marc-gr · 2023-10-30T09:59:49Z

@marc-gr: I have a doubt about warm-up period. it is run in a goroutine in the setup method, rally will be executed before that. I'm not sure how does it works for system benchmark. what is it supposed to achieve the warm up period?

In your case I think this comes builtin with rally itself, the intention of this is to defer metric collection until the warm up period ends, so that time is not going to be taken into account for reporting.

aspacca · 2023-10-30T10:23:37Z

As a first step, lets make sure it all uses the assets from the package. I agree, unifying the commands might make sense but we could also solve it with docs pointing to it.

it was in end faster to add saving the rally track and a dry run in this PR than refactor the previous command: I will remove it in a next PR

Key is that the outcome is the same, meaning if I run the rally track "live" or generate the corpus, same data is in.

it will not be exactly the same data: meaning that if I generate the corpus multiple time (running rally) the data will have the same "shape" (cardinality, range, etc), but different randomized value because seeds and time values are based on "now".

the generator tool already has the option to pass a seed and "now" from command line, let me know if you want to add to the command in elastic-package as well

aspacca · 2023-10-30T10:32:28Z

In your case I think this comes builtin with rally itself, the intention of this is to defer metric collection until the warm up period ends, so that time is not going to be taken into account for reporting.

ok, it's a warm up period for metrics collection, that makes sense as it is now then.
I thought it was a warmup for starting the benchmark :)

aspacca · 2023-11-02T08:57:55Z

@ruflin I now refresh the index before collecting stats
also I removed polling the hits while rally is running, since this will compete for resources

docs/howto/rally_benchmarking.md

internal/cobraext/flags.go

ruflin · 2023-11-02T09:09:51Z

If @jsoriano agrees, lets get this PR in rather soonish and then add the corpus templates to some of the integration packages. I expect that this will also provide us some more feedback for iterating on the command itself. Also get it in the hands of the rest of the team to start playing with it.

Can we treat the command "beta" for now so we can still make breaking changes to it?

Co-authored-by: Nicolas Ruflin <[email protected]>

aspacca · 2023-11-02T09:52:57Z

If @jsoriano agrees, lets get this PR in rather soonish

we must merge elastic/package-spec#653 in order to pass CI

…acca/package-spec/v3

jsoriano

Ok to merge this and iterate, once the package-spec change is released.

aspacca · 2023-11-06T08:55:21Z

Ok to merge this and iterate, once the package-spec change is released.

before merging this I have to revert the last two commits, but without package-spec released it won't pass CI.
do I miss anything? :)

ruflin · 2023-11-06T09:13:24Z

I just merged the package-spec PR. My assumption is that we would stay on a commit reference but would move it over to the one in package-spec which is merged now instead of a release. And then as soon as a new package-spec is out, we update.

jsoriano · 2023-11-07T12:08:47Z

Updating package spec in a separate PR #1539

elasticmachine · 2023-11-07T23:48:39Z

💚 Build Succeeded

Buildkite Build
Commit: f406794

History

💔 Build #1843 failed 943367a
💔 Build #1842 failed cc39585
💔 Build #1837 failed 5804ef9
💚 Build #1834 succeeded bb18dc8
💔 Build #1833 failed 9582206

cc @aspacca

Andrea Spacca added 6 commits October 24, 2023 10:12

add rally subcommand in benchmark

4ec6921

add rally corpus output dir

5f14e8c

export GenerateRallyTrack

787276d

add rally runner

9557ce4

fix generator config yaml for system benchmark

c9c3cc7

add rally benchmark test files

8dda99d

aspacca requested review from jsoriano, mrodm and marc-gr October 24, 2023 01:29

aspacca self-assigned this Oct 24, 2023

aspacca mentioned this pull request Oct 24, 2023

Sample data generation #984

Closed

Andrea Spacca added 4 commits October 24, 2023 10:37

fix from CI

dfd63bd

fix from CI

08acf94

fix repeated print of paramters

d320aa5

fix ES host env variable for rally

298b97f

aspacca mentioned this pull request Oct 25, 2023

add input and vars in rally scenario spec, make input and corpora man… elastic/package-spec#653

Merged

2 tasks

Andrea Spacca added 3 commits October 25, 2023 15:19

remove wait_for_data_timeout

373dba8

changelog

918601f

spec reference

dccc3c7

ruflin reviewed Oct 26, 2023

View reviewed changes

ruflin reviewed Oct 30, 2023

View reviewed changes

internal/benchrunner/runners/rally/report.go Show resolved Hide resolved

cr fixes

ddf91d6

fix check-static

45d5659

wait only for warmup

592cdcb

ruflin reviewed Nov 2, 2023

View reviewed changes

docs/howto/rally_benchmarking.md Show resolved Hide resolved

ruflin reviewed Nov 2, 2023

View reviewed changes

docs/howto/rally_benchmarking.md Outdated Show resolved Hide resolved

ruflin reviewed Nov 2, 2023

View reviewed changes

internal/cobraext/flags.go Outdated Show resolved Hide resolved

Andrea Spacca and others added 2 commits November 2, 2023 18:15

remove warmup, include unloaded segment in pipeline stats

671867c

Update internal/cobraext/flags.go

86a6b2f

Co-authored-by: Nicolas Ruflin <[email protected]>

Andrea Spacca added 4 commits November 3, 2023 17:10

docs about replay of rally tracks, bugfixes on metrics

13672e2

temporary: alias github.com/elastic/package-spec/v3 to github.com/asp…

3dc1a58

…acca/package-spec/v3

remove warmup_time_period

9582206

temporary: format version

bb18dc8

jsoriano reviewed Nov 3, 2023

View reviewed changes

aspacca mentioned this pull request Nov 6, 2023

Rally benchmark aws.billing elastic/integrations#8403

Merged

2 tasks

ruflin approved these changes Nov 6, 2023

View reviewed changes

Andrea Spacca added 2 commits November 6, 2023 18:17

remove replace in go mod. update to latest commit

5804ef9

make check-static

cc39585

aspacca mentioned this pull request Nov 7, 2023

Rally benchmark aws.ec2 logs elastic/integrations#8416

Merged

2 tasks

github.com/elastic/package-spec/[email protected]

943367a

Merge branch 'main' into rally-benchmark

f406794

aspacca merged commit 208501b into elastic:main Nov 7, 2023

aspacca mentioned this pull request Nov 7, 2023

Add support for rally benchmark #1475

Closed

ruflin mentioned this pull request Nov 8, 2023

TSDB: document timestamp is outside of ranges of currently writable indices elastic/integrations#8431

Open

aspacca mentioned this pull request Nov 10, 2023

Remove benchmark generate corpus command #1553

Merged

jsoriano mentioned this pull request Nov 30, 2023

fix mode permission on rally track output dir #1575

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rally benchmark #1522

Rally benchmark #1522

aspacca commented Oct 24, 2023 •

edited

Loading

aspacca commented Oct 24, 2023

ruflin Oct 26, 2023

aspacca Oct 26, 2023

ruflin Oct 30, 2023

aspacca Oct 30, 2023

jsoriano Oct 31, 2023

ruflin commented Oct 30, 2023

aspacca commented Oct 30, 2023

ruflin commented Oct 30, 2023

marc-gr commented Oct 30, 2023

aspacca commented Oct 30, 2023

aspacca commented Oct 30, 2023

aspacca commented Nov 2, 2023

ruflin commented Nov 2, 2023

aspacca commented Nov 2, 2023

jsoriano left a comment

aspacca commented Nov 6, 2023

ruflin commented Nov 6, 2023

jsoriano commented Nov 7, 2023

elasticmachine commented Nov 7, 2023


		These benchmarks allow you to benchmark an integration corpus with rally.

		For details on how to configure rally benchmarks for a package, review the [HOWTO guide](./docs/howto/rally_benchmarking.md).

Rally benchmark #1522

Rally benchmark #1522

Conversation

aspacca commented Oct 24, 2023 • edited Loading

aspacca commented Oct 24, 2023

ruflin Oct 26, 2023

Choose a reason for hiding this comment

aspacca Oct 26, 2023

Choose a reason for hiding this comment

ruflin Oct 30, 2023

Choose a reason for hiding this comment

aspacca Oct 30, 2023

Choose a reason for hiding this comment

jsoriano Oct 31, 2023

Choose a reason for hiding this comment

ruflin commented Oct 30, 2023

aspacca commented Oct 30, 2023

ruflin commented Oct 30, 2023

marc-gr commented Oct 30, 2023

aspacca commented Oct 30, 2023

aspacca commented Oct 30, 2023

aspacca commented Nov 2, 2023

ruflin commented Nov 2, 2023

aspacca commented Nov 2, 2023

jsoriano left a comment

Choose a reason for hiding this comment

aspacca commented Nov 6, 2023

ruflin commented Nov 6, 2023

jsoriano commented Nov 7, 2023

elasticmachine commented Nov 7, 2023

💚 Build Succeeded

History

aspacca commented Oct 24, 2023 •

edited

Loading