Skip to content

Commit 30d784e

Browse files
kandersolarwholmgren
kandersolar
and
wholmgren
authored
Set up airspeed velocity benchmarks (#1049)
* add airspeed velocity config and tests * add documentation * move whatsnew line to 0.7.2 * update asv.conf.json to most recent asv format * add useful tips to readme * modify asv conf to install optional pvlib reqs (pytables etc) * add more benchmarks for location and solarposition * fix solarposition docstring * add tracking benchmark file * add irradiance benchmark file * fix tracking benchmarks * test python3.7 only, regardless of installed version * switch from conda to virtualenv * restructure * stickler * drop old whatsnew entry * use separate classes for version checking * fix aoi_projection benchmark * do version switching in setup() instead of in global scope * singular space to satisfy stickler * move asv conf to pvlib * move benchmarks to subdirectory * change to local repo instead of cloning from GH * docs * update benchmark readme * create 0.8.1 whatsnew * conda channel ordering * link to readme * fix wrong link * benchmark on minimum and latest dependency versions Co-authored-by: Will Holmgren <[email protected]>
1 parent 72f3783 commit 30d784e

File tree

11 files changed

+582
-0
lines changed

11 files changed

+582
-0
lines changed

‎.gitignore

+5
Original file line numberDiff line numberDiff line change
@@ -92,3 +92,8 @@ coverage.xml
9292

9393
# Ignore Mac DS_store files
9494
*.DS_Store
95+
96+
# airspeed velocity
97+
env
98+
results
99+

‎benchmarks/README.md

+93
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
Benchmarks
2+
==========
3+
4+
pvlib includes a small number of performance benchmarking tests. These
5+
tests are run using
6+
[airspeed velocity](https://asv.readthedocs.io/en/stable/) (ASV).
7+
8+
The basic structure of the tests and how to run them is described below.
9+
We refer readers to the ASV documentation for more details. The AstroPy
10+
[documentation](https://github.com/astropy/astropy-benchmarks/tree/master)
11+
may also be helpful.
12+
13+
The test configuration is described in [asv.conf.json](asv.conf.json).
14+
The performance tests are located in the [benchmarks](benchmarks) directory.
15+
16+
Comparing timings
17+
-----------------
18+
19+
Note that, unlike pytest, the asv tests require changes to be committed
20+
to git before they can be tested. The ``run`` command takes a positional
21+
argument to describe the range of git commits or branches to be tested.
22+
For example, if your feature branch is named ``feature``, a useful asv
23+
run may be (from the same directory as `asv.conf.json`):
24+
25+
```
26+
$ asv run master..feature
27+
```
28+
29+
This will generate timings for every commit between the two specified
30+
revisions. If you only care about certain commits, you can run them by
31+
their git hashes directly like this:
32+
33+
```
34+
$ asv run e42f8d24^!
35+
```
36+
37+
Note: depending on what shell they use, Windows users may need to use
38+
double-carets:
39+
40+
```
41+
$ asv run e42f8d24^^!
42+
```
43+
44+
You can then compare the timing results of two commits:
45+
46+
```
47+
$ asv compare 0ff98b62 e42f8d24
48+
49+
All benchmarks:
50+
51+
before after ratio
52+
[0ff98b62] [e42f8d24]
53+
<asv_setup~1> <asv_setup>
54+
+ 3.90±0.6ms 31.3±5ms 8.03 irradiance.Irradiance.time_aoi
55+
3.12±0.4ms 2.94±0.2ms 0.94 irradiance.Irradiance.time_aoi_projection
56+
256±9ms 267±10ms 1.05 irradiance.Irradiance.time_dirindex
57+
```
58+
59+
The `ratio` column shows the ratio of `after / before` timings. For this
60+
example, the `aoi` function was slowed down on purpose to demonstrate
61+
the comparison.
62+
63+
Generating an HTML report
64+
-------------------------
65+
66+
asv can generate a collection of interactive plots of benchmark timings across
67+
a commit history. First, generate timings for a series of commits, like:
68+
69+
```
70+
$ asv run v0.6.0..v0.8.0
71+
```
72+
73+
Next, generate the HTML report:
74+
75+
```
76+
$ asv publish
77+
```
78+
79+
Finally, start a http server to view the test results:
80+
81+
```
82+
$ asv preview
83+
84+
```
85+
86+
87+
Nightly benchmarking
88+
--------------------
89+
90+
The benchmarks are run nightly for new commits to pvlib-python/master.
91+
92+
- Timing results: https://pvlib-benchmarker.github.io/pvlib-benchmarks/
93+
- Information on the process: https://github.com/pvlib-benchmarker/pvlib-benchmarks

‎benchmarks/asv.conf.json

+186
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
{
2+
// The version of the config file format. Do not change, unless
3+
// you know what you are doing.
4+
"version": 1,
5+
6+
// The name of the project being benchmarked
7+
"project": "pvlib-python",
8+
9+
// The project's homepage
10+
"project_url": "https://pvlib-python.readthedocs.io",
11+
12+
// The URL or local path of the source code repository for the
13+
// project being benchmarked
14+
"repo": "..",
15+
16+
// The Python project's subdirectory in your repo. If missing or
17+
// the empty string, the project is assumed to be located at the root
18+
// of the repository.
19+
// "repo_subdir": "",
20+
21+
// Customizable commands for building, installing, and
22+
// uninstalling the project. See asv.conf.json documentation.
23+
//
24+
// "install_command": ["in-dir={env_dir} python -mpip install {wheel_file}"],
25+
// "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
26+
// "build_command": [
27+
// "python setup.py build",
28+
// "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
29+
// ],
30+
31+
// List of branches to benchmark. If not provided, defaults to "master"
32+
// (for git) or "default" (for mercurial).
33+
// "branches": ["master"], // for git
34+
// "branches": ["default"], // for mercurial
35+
36+
// The DVCS being used. If not set, it will be automatically
37+
// determined from "repo" by looking at the protocol in the URL
38+
// (if remote), or by looking for special directories, such as
39+
// ".git" (if local).
40+
"dvcs": "git",
41+
42+
// The tool to use to create environments. May be "conda",
43+
// "virtualenv" or other value depending on the plugins in use.
44+
// If missing or the empty string, the tool will be automatically
45+
// determined by looking for tools on the PATH environment
46+
// variable.
47+
"environment_type": "conda",
48+
49+
// timeout in seconds for installing any dependencies in environment
50+
// defaults to 10 min
51+
//"install_timeout": 600,
52+
53+
// the base URL to show a commit for the project.
54+
"show_commit_url": "http://github.com/pvlib/pvlib-python/commit/",
55+
56+
// The Pythons you'd like to test against. If not provided, defaults
57+
// to the current version of Python used to run `asv`.
58+
"pythons": ["dummy"], // this gets excluded below
59+
60+
// The list of conda channel names to be searched for benchmark
61+
// dependency packages in the specified order
62+
"conda_channels": ["defaults", "conda-forge"],
63+
64+
// The matrix of dependencies to test. Each key is the name of a
65+
// package (in PyPI) and the values are version numbers. An empty
66+
// list or empty string indicates to just test against the default
67+
// (latest) version. null indicates that the package is to not be
68+
// installed. If the package to be tested is only available from
69+
// PyPi, and the 'environment_type' is conda, then you can preface
70+
// the package name by 'pip+', and the package will be installed via
71+
// pip (with all the conda available packages installed first,
72+
// followed by the pip installed packages).
73+
//
74+
"matrix": {}, // pvlib dependencies specified in the `include` list below
75+
76+
// Combinations of libraries/python versions can be excluded/included
77+
// from the set to test. Each entry is a dictionary containing additional
78+
// key-value pairs to include/exclude.
79+
//
80+
// An exclude entry excludes entries where all values match. The
81+
// values are regexps that should match the whole string.
82+
//
83+
// An include entry adds an environment. Only the packages listed
84+
// are installed. The 'python' key is required. The exclude rules
85+
// do not apply to includes.
86+
//
87+
// In addition to package names, the following keys are available:
88+
//
89+
// - python
90+
// Python version, as in the *pythons* variable above.
91+
// - environment_type
92+
// Environment type, as above.
93+
// - sys_platform
94+
// Platform, as in sys.platform. Possible values for the common
95+
// cases: 'linux2', 'win32', 'cygwin', 'darwin'.
96+
//
97+
// "exclude": [
98+
// {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
99+
// {"environment_type": "conda", "six": null}, // don't run without six on conda
100+
// ],
101+
//
102+
// "include": [
103+
// // additional env for python2.7
104+
// {"python": "2.7", "numpy": "1.8"},
105+
// // additional env if run on windows+conda
106+
// {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""},
107+
// ],
108+
109+
// asv takes the Cartesian product of the 'matrix' version spec, so it's
110+
// no good for specifying sets of versions. Instead we'll use the
111+
// 'include' machinery to specify each set of versions to benchmark.
112+
// However, the 'pythons' key ends up generating an undesirable extra
113+
// environment, so we 'exclude' it.
114+
"include": [
115+
// minimum supported versions
116+
{
117+
"python": "3.6",
118+
"numpy": "1.12.0",
119+
"pandas": "0.22.0",
120+
"scipy": "1.2.0",
121+
// Note: these don't have a minimum in setup.py
122+
"pytables": "3.6.1",
123+
"ephem": "3.7.6.0",
124+
},
125+
// latest versions available
126+
{
127+
"python": "3.8",
128+
"numpy": "",
129+
"pandas": "",
130+
"scipy": "",
131+
"pytables": "",
132+
"ephem": "",
133+
},
134+
],
135+
"exclude": [
136+
{"python": "dummy"}
137+
],
138+
139+
// The directory (relative to the current directory) that benchmarks are
140+
// stored in. If not provided, defaults to "benchmarks"
141+
"benchmark_dir": "benchmarks",
142+
143+
// The directory (relative to the current directory) to cache the Python
144+
// environments in. If not provided, defaults to "env"
145+
// "env_dir": "env",
146+
147+
// The directory (relative to the current directory) that raw benchmark
148+
// results are stored in. If not provided, defaults to "results".
149+
// "results_dir": "results",
150+
151+
// The directory (relative to the current directory) that the html tree
152+
// should be written to. If not provided, defaults to "html".
153+
// "html_dir": "html",
154+
155+
// The number of characters to retain in the commit hashes.
156+
// "hash_length": 8,
157+
158+
// `asv` will cache results of the recent builds in each
159+
// environment, making them faster to install next time. This is
160+
// the number of builds to keep, per environment.
161+
// "build_cache_size": 2,
162+
163+
// The commits after which the regression search in `asv publish`
164+
// should start looking for regressions. Dictionary whose keys are
165+
// regexps matching to benchmark names, and values corresponding to
166+
// the commit (exclusive) after which to start looking for
167+
// regressions. The default is to start from the first commit
168+
// with results. If the commit is `null`, regression detection is
169+
// skipped for the matching benchmark.
170+
//
171+
// "regressions_first_commits": {
172+
// "some_benchmark": "352cdf", // Consider regressions only after this commit
173+
// "another_benchmark": null, // Skip regression detection altogether
174+
// },
175+
176+
// The thresholds for relative change in results, after which `asv
177+
// publish` starts reporting regressions. Dictionary of the same
178+
// form as in ``regressions_first_commits``, with values
179+
// indicating the thresholds. If multiple entries match, the
180+
// maximum is taken. If no entry matches, the default is 5%.
181+
//
182+
// "regressions_thresholds": {
183+
// "some_benchmark": 0.01, // Threshold of 1%
184+
// "another_benchmark": 0.5, // Threshold of 50%
185+
// },
186+
}

‎benchmarks/benchmarks/__init__.py

Whitespace-only changes.

‎benchmarks/benchmarks/irradiance.py

+68
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
"""
2+
ASV benchmarks for irradiance.py
3+
"""
4+
5+
import pandas as pd
6+
from pvlib import irradiance, location
7+
8+
9+
class Irradiance:
10+
11+
def setup(self):
12+
self.times = pd.date_range(start='20180601', freq='1min',
13+
periods=14400)
14+
self.days = pd.date_range(start='20180601', freq='d', periods=30)
15+
self.location = location.Location(40, -80)
16+
self.solar_position = self.location.get_solarposition(self.times)
17+
self.clearsky_irradiance = self.location.get_clearsky(self.times)
18+
self.tilt = 20
19+
self.azimuth = 180
20+
self.aoi = irradiance.aoi(self.tilt, self.azimuth,
21+
self.solar_position.apparent_zenith,
22+
self.solar_position.azimuth)
23+
24+
def time_get_extra_radiation(self):
25+
irradiance.get_extra_radiation(self.days)
26+
27+
def time_aoi(self):
28+
irradiance.aoi(self.tilt, self.azimuth,
29+
self.solar_position.apparent_zenith,
30+
self.solar_position.azimuth)
31+
32+
def time_aoi_projection(self):
33+
irradiance.aoi_projection(self.tilt, self.azimuth,
34+
self.solar_position.apparent_zenith,
35+
self.solar_position.azimuth)
36+
37+
def time_get_ground_diffuse(self):
38+
irradiance.get_ground_diffuse(self.tilt, self.clearsky_irradiance.ghi)
39+
40+
def time_get_total_irradiance(self):
41+
irradiance.get_total_irradiance(self.tilt, self.azimuth,
42+
self.solar_position.apparent_zenith,
43+
self.solar_position.azimuth,
44+
self.clearsky_irradiance.dni,
45+
self.clearsky_irradiance.ghi,
46+
self.clearsky_irradiance.dhi)
47+
48+
def time_disc(self):
49+
irradiance.disc(self.clearsky_irradiance.ghi,
50+
self.solar_position.apparent_zenith,
51+
self.times)
52+
53+
def time_dirint(self):
54+
irradiance.dirint(self.clearsky_irradiance.ghi,
55+
self.solar_position.apparent_zenith,
56+
self.times)
57+
58+
def time_dirindex(self):
59+
irradiance.dirindex(self.clearsky_irradiance.ghi,
60+
self.clearsky_irradiance.ghi,
61+
self.clearsky_irradiance.dni,
62+
self.solar_position.apparent_zenith,
63+
self.times)
64+
65+
def time_erbs(self):
66+
irradiance.erbs(self.clearsky_irradiance.ghi,
67+
self.solar_position.apparent_zenith,
68+
self.times)

0 commit comments

Comments
 (0)