Choose benchmarking tool

[jrbourbeau]

As part of #148, we should decide on what benchmarking tool we want to use. Currently we're just having pytest print out test run times, but it was always intended for us to switch to using a more proper benchmarking framework at some point. asv is the primary (only?) benchmarking tool I'm aware of that's used in our space (e.g. NumPy, pandas, scikit-image, Xarray all use asv to monitor performance). Because of this I'm inclined to default to using asv unless there is a compelling reason not to do so.

Are folks aware of other benchmarking tools we should consider?

[ian-r-rose]

Airspeed Velocity (asv)

Airspeed velocity is a benchmarking tool that provides a set of tools for tracking speed and finding performance regressions. It's widely used (lots of pydata projects, also rich). Notably, ibis recently moved away from asv to pytest-benchmark, for some of the same reasons as listed below. Until recently, it was quasi-abandoned, but a numfocus grant to pandas has provided some maintenance resources.

Features

1. A CLI for running and inspecting results
2. A static site for visualizing said results, easily hosted on GitHub Pages
3. A JSON database format for storing results

Drawbacks

1. It wants to know a lot of information about the machine on which the benchmark is run. Since the bulk of our computation is remote on well-specified instances, this isn't really relevant for our benchmarks. It's an annoyance, but we could probably find a workaround.
2. It wants to manage your software environment. Again, we don't really need this since we manage our own environment.
3. It doesn't use pytest, instead it has its own setup and teardown functionality, analogous to fixtures, but different. We'd probably need to refactor.

[ian-r-rose]

pytest-benchmark

pytest-benchmark is a pytest plugin that provides some similar functionality to asv. It is used by some projects, including ibis and RAPIDs. It includes a fixture that one can include in regular pytest tests which produces benchmarking statistics, and allows for JSON export.

Features

1. Easy interoperability with existing pytest tests and fixtures
2. JSON database format for storing results
3. Some CLI and local charting tooling

Drawbacks

1. No built-in static site. This is really significant! The rapids team has developed a bridge tool to produce asv JSON from pytest-benchmark results, presumably for this reason: https://github.com/rapidsai/asvdb
2. Seemingly less widely used in the pydata space

[ian-r-rose]

Conbench

Conbench is a multi-language benchmarking tool developed by the arrow team. It includes a schema for benchmark outputs and a server for displaying them on a website. In many ways, it looks like the most featureful option out there.

Features

1. A schema for cross-language benchmarking results
2. A postgres+flask server with a dashboard for reporting results
3. Tooling for running cross-language benchmarks

Drawbacks

1. Perhaps too heavy for what we want right now. I'd rather maintain a static site on GH-pages than a whole server
2. Provides features we likely don't need (we can stick with python-native stuff)

[gjoseph92]

The rapids team has developed a bridge tool to produce asv JSON from pytest-benchmark results

If this is something we could easily drop in and use, then naively to me that makes it sound like pytest-benchmark for running tests + asv for plotting them seems ideal? Given that we've already invested in pytest infrastructure for our cluster fixtures, and for dealing with software environments. The ability to just have to use one tool (pytest) that everyone already knows for both integration tests + benchmarks is appealing.

But I haven't looked at asvdb at all, so I'm probably not understanding the complexity of translating the formats.

[ian-r-rose]

If this is something we could easily drop in and use, then naively to me that makes it sound like pytest-benchmark for running tests + asv for plotting them seems ideal?

It certainly might be a good fit -- I haven't really evaluated yet how easy asvdb is to use, and how well-supported it is.

[fjetter]

I would like to point out that our requirements are a bit more than just time measurement, see #148 (comment)

The list there is probably not exhaustive and I would invite everyone to think about what we need for this to become a good experience.

[hayesgb]

@fjetter -- Thanks for adding this additional set of requirements.

Suggest we make certain we retain the flexibility to add the your referenced (and others if needed) metrics in future iterations, but limit the scope of this initial work to execution time. Thoughts?

[gjoseph92]

I think memory usage needs to be a core requirement. It's often more important than runtime. A change that caused an existing workload to use more memory than before would be a serious regression. A workload that uses less would be a big deal to market and evangelize.

I'd honestly love to have the CSV output of a MemorySampler for every run, and build a simple little Vega-lite app to be able to overlay memory profiles between different runs. But tracking peak memory (which we'd do with a MemorySampler anyway) just like we track runtime should be part of our benchmarking dashboards.