add extras_require complete and test options #566

wholmgren · 2018-09-07T01:04:52Z

Closes clearsky.py requires PyTables but setup.py doesn't list it as install_requires #553
I am familiar with the contributing guidelines.
Fully tested. Added and/or modified tests to ensure correct behavior for all reasonable inputs. Tests (usually) must pass on the TravisCI and Appveyor testing services.
Updates entries to docs/sphinx/source/api.rst for API changes.
Adds description and name entries in the appropriate docs/sphinx/source/whatsnew file for all changes.
Code quality and style is sufficient. Passes LGTM and SticklerCI checks.
New code is fully documented. Includes sphinx/numpydoc compliant docstrings and comments in the code where necessary.
Pull request is nearly complete and ready for detailed review.

I'm not willing to make scipy, tables, etc. requirements for the upcoming 0.6.0 release. But I think we can use the setuptools extras_require option to at least make it a little easier for a few people. So people will be able to install all of the dependencies necessary for any feature by running

edited below here for clarity and future reference

pip install pvlib[optional]  # once released to pypi
pip install -e .[optional]  # works for development too, though escape brackets for zsh

I originally chose "complete" because that's what dask uses. See discussion below for reasons to changing to optional/doc/test/all.

Options can also be combined like this:

pip install -e .[optional,test]

@mikofski you are most qualified to review this!

mikofski · 2018-09-07T03:01:45Z

test `extras_require` using `pip install pvlib[complete]`:

create a virtualenv, install pvlib requirements and build a wheel

markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ virtualenv -p python3 venv

markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ . venv/bin/activate

(venv) markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ pip install numpy scipy siphon netcdf4 matplotlib pytest mock nose pytest-cov pandas ipython jupyter sphinx sphinx-rtd-theme numpydoc numba ephem pytest-mock tables nbsphinx pytz

(venv) markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ python setup.py bdist_wheel sdist

(venv) markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ ls dist/
pvlib-0.5.2+48.gf427a39-py3-none-any.whl
pvlib-0.5.2+48.gf427a39.tar.gz

(venv) markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ deactivate

create another virtualenv and install wheel with extra requirements [complete]

markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ virtualenv -p python3 venv-extras

markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ . venv-extras/bin/activate

(venv-extras) markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ pip list
Package    Version
---------- -------
pip        18.0   
setuptools 40.2.0 
wheel      0.31.1 

(venv-extras) markmikofski@Suzannes-MacBook-Pro:pvlib-python (extras)$ cd

(venv-extras) markmikofski@Suzannes-MacBook-Pro:~$ pip install --find-links ~/Projects/pvlib-python/dist/ pvlib[complete]
Looking in links: /Users/markmikofski/Projects/pvlib-python/dist/
Collecting pvlib[complete]
Collecting pytz (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl
Collecting pandas>=0.14.0 (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/78/78/50ef81a903eccc4e90e278a143c9a0530f05199f6221d2e1b21025852982/pandas-0.23.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting numpy>=1.10.1 (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/a6/0a/2defbdd9bd3a436ee0642bb8d8a66d16e523122d33384a04548dcfab23ba/numpy-1.15.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting six (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting siphon; extra == "complete" (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/56/f5/2be03af9ac2911d4795ac69de86dbd1b48c404a7812a4bf1b00403eafda5/siphon-0.8.0-py2.py3-none-any.whl
Collecting scipy; extra == "complete" (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/a0/b6/70bf61c1badb5fea82d4c558e05e76c2dee5e77bb072fe465d7c7a87287d/scipy-1.1.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting ephem; extra == "complete" (from pvlib[complete])
Collecting tables; extra == "complete" (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/bc/5f/79643d50fcf927dcd4b3bd118062550fdcfbd247d63cad1dfb305c6852b5/tables-3.4.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting numba; extra == "complete" (from pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/fc/f6/1555a24d032c1ebe6db3e92796d5846e7d5b4337eeb53439d95fb4694fc2/numba-0.39.0-cp36-cp36m-macosx_10_9_x86_64.whl
Collecting python-dateutil>=2.5.0 (from pandas>=0.14.0->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl
Collecting requests>=1.2 (from siphon; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl
Collecting beautifulsoup4>=4.6 (from siphon; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/21/0a/47fdf541c97fd9b6a610cb5fd518175308a7cc60569962e776ac52420387/beautifulsoup4-4.6.3-py3-none-any.whl
Collecting protobuf>=3.0.0a3 (from siphon; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/c7/27/133f225035b9539f2dcfebcdf9a69ff0152f56e0120160ec5c972ea7deb9/protobuf-3.6.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting numexpr>=2.5.2 (from tables; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/53/e4/4ea4e519ddbf8b2c26edccfba6a6011419208044c07f8e4f01b3fb3028ba/numexpr-2.6.8-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting llvmlite>=0.24.0dev0 (from numba; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/e7/ee/9e827a16b0f168b4e2da3efc8e3798fc3e4139f3196dafdf127b052c683d/llvmlite-0.24.0-cp36-cp36m-macosx_10_9_x86_64.whl
Collecting idna<2.8,>=2.5 (from requests>=1.2->siphon; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Collecting urllib3<1.24,>=1.21.1 (from requests>=1.2->siphon; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=1.2->siphon; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=1.2->siphon; extra == "complete"->pvlib[complete])
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./Projects/pvlib-python/venv-extras/lib/python3.6/site-packages (from protobuf>=3.0.0a3->siphon; extra == "complete"->pvlib[complete]) (40.2.0)
Installing collected packages: pytz, six, python-dateutil, numpy, pandas, idna, urllib3, certifi, chardet, requests, beautifulsoup4, protobuf, siphon, scipy, ephem, numexpr, tables, llvmlite, numba, pvlib
Successfully installed beautifulsoup4-4.6.3 certifi-2018.8.24 chardet-3.0.4 ephem-3.7.6.0 idna-2.7 llvmlite-0.24.0 numba-0.39.0 numexpr-2.6.8 numpy-1.15.1 pandas-0.23.4 protobuf-3.6.1 pvlib-0.5.2+48.gf427a39 python-dateutil-2.7.3 pytz-2018.5 requests-2.19.1 scipy-1.1.0 siphon-0.8.0 six-1.11.0 tables-3.4.4 urllib3-1.23
(venv-extras) markmikofski@Suzannes-MacBook-Pro:~$ pip list
Package         Version          
--------------- -----------------
beautifulsoup4  4.6.3            
certifi         2018.8.24        
chardet         3.0.4            
ephem           3.7.6.0          
idna            2.7              
llvmlite        0.24.0           
numba           0.39.0           
numexpr         2.6.8            
numpy           1.15.1           
pandas          0.23.4           
pip             18.0             
protobuf        3.6.1            
pvlib           0.5.2+48.gf427a39
python-dateutil 2.7.3            
pytz            2018.5           
requests        2.19.1           
scipy           1.1.0            
setuptools      40.2.0           
siphon          0.8.0            
six             1.11.0           
tables          3.4.4            
urllib3         1.23             
wheel           0.31.1

mikofski

LGTM

mikofski · 2018-09-07T18:58:27Z

odd I don't see netcdf4 in the virtualenv with the "complete" install, but it is a requirement of the building the docs and the forecast.py module

probably best to add it

mikofski

please add netcdf4 to complete extras_require

mikofski · 2018-09-07T18:58:52Z

setup.py

-TESTS_REQUIRE = ['pytest', 'nose']
+TESTS_REQUIRE = ['pytest', 'pytest-cov', 'pytest-mock', 'nose']
+EXTRAS_REQUIRE = {
+    'complete': ['scipy', 'tables', 'numba', 'siphon', 'ephem'],


please add netcdf4

thanks for catching this. working on adding a doc section, too.

mikofski

Is nbsphinx necessary for docs? I think I've had an issue withthout this in the past

mikofski · 2018-09-07T23:15:42Z

setup.py

-    'complete': ['scipy', 'tables', 'numba', 'siphon', 'ephem'],
+    'complete': ['scipy', 'tables', 'numba', 'siphon', 'netcdf4', 'ephem'],
+    'doc': ['sphinx', 'ipython', 'sphinx_rtd_theme', 'numpydoc',
+            'matplotlib', 'jupyter'],


I believe nbsphinx is also necessary for docs

mikofski · 2018-09-07T23:19:23Z

for reference, here's my requirements.txt

stickler-ci · 2018-09-09T22:07:11Z

Your configuration file uses an unknown linter. The flake8 linter is not a supported linter.

mikofski

Since #568 don't need no Sphinx or jupyter anymore. Thanks! Can you paste a link to your rtfd build please?

stickler-ci · 2018-09-09T22:26:51Z

Your configuration file uses an unknown linter. The flake8 linter is not a supported linter.

wholmgren · 2018-09-09T22:28:46Z

Here's a link to the build: https://wholmgren-pvlib-python-new.readthedocs.io/en/extras/index.html

but none of the changes in this PR should effect the documentation build on readthedocs.

I'm wondering if we'd be better off with something like

pvlib[optional]  # scipy, tables, etc.
pvlib[doc]  # sphinx, etc
pvlib[test]  # pytest etc
pvlib[complete] or pvlib[all]  # all of the above, same as pvlib[optional,doc,test]

mikofski · 2018-09-09T22:34:10Z

Yep, sorry fir cross posting
Great idea. Sounds good to me.

stickler-ci · 2018-09-09T22:46:22Z

Your configuration file uses an unknown linter. The flake8 linter is not a supported linter.

wholmgren · 2018-09-10T02:54:00Z

Ok @mikofski, now the changes in this PR do effect the doc build! Between wheels and readthedocs.org including numpy/scipy/matplotlib in its root environment, I think we're now better off using pip installs on RTD than conda envs. This PR consolidates the build requirements in one place (setup.py) and the builds seem to be faster (4-8 minutes instead of 12-15+).

Unfortunately, reconfiguring RTD has not fixed the ipython directive issue. (I think that is an ipython directive issue or an issue with how we're using it rather than an RTD issue.)

mikofski · 2018-09-10T03:00:10Z

Thanks so much for working so hard on this!
At some point does it make sense asking on Google groups? Or searching SO?

wholmgren · 2018-09-10T03:07:01Z

Well I had issues with it locally so not sure what to ask specific to rtd.

…

On Sun, Sep 9, 2018 at 8:00 PM Mark Mikofski ***@***.***> wrote: At some point does it make sense asking on Google groups <https://groups.google.com/forum/m/#!forum/read-the-docs> make sense? Or searching SO <https://stackoverflow.com/questions/tagged/read-the-docs>? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#566 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AELiR0B7HKyYRyVBjcpjyPjBR1cxsbxXks5uZdW7gaJpZM4WeAwk> .

wholmgren · 2018-09-10T16:23:23Z

Filed an issue with IPython: ipython/ipython#11298

wholmgren · 2018-09-11T18:26:45Z

@mikofski are you happy with the latest changes here? You are the active contributor that is most likely to take advantage of this on a regular basis, so want to make sure you're happy with it. I'm ready to merge if you still approve.

The latest doc build (still on conda) now shows the same ipython behavior as in the build of my fork's branch. I think we should go ahead with changing the pvlib RTD configuration to use wheels.

mikofski · 2018-09-11T18:34:42Z

LGTM, if I have a chance, I'll try to figure out what's going on with ipython directive, maybe there's an alterate way to do the doctest, but for now I think this is fine. Thanks again! OK with me to merge ...

add extras_require complete and test options

f427a39

mikofski approved these changes Sep 7, 2018

View reviewed changes

wholmgren added installation Release labels Sep 7, 2018

wholmgren added this to the 0.6.0 milestone Sep 7, 2018

mikofski requested changes Sep 7, 2018

View reviewed changes

wholmgren added 2 commits September 7, 2018 13:12

Merge remote-tracking branch 'pvlib/master' into extras

0666699

update extras

b0dc0f8

mikofski requested changes Sep 7, 2018

View reviewed changes

wholmgren added 2 commits September 9, 2018 15:02

Merge remote-tracking branch 'pvlib/master' into extras

af328bd

remove jupyter

3c105e6

mikofski approved these changes Sep 9, 2018

View reviewed changes

update doc notes

cd0bac6

wholmgren added 2 commits September 9, 2018 15:39

replace conda with pip in readthedocs.yml

4846e0b

fix doc/docs typo

38ecc70

add all, replace complete with optional

c7b0a16

wholmgren merged commit 3e629d2 into pvlib:master Sep 11, 2018

wholmgren deleted the extras branch September 11, 2018 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add extras_require complete and test options #566

add extras_require complete and test options #566

wholmgren commented Sep 7, 2018 •

edited

Loading

mikofski commented Sep 7, 2018 •

edited

Loading

mikofski left a comment

mikofski commented Sep 7, 2018

mikofski left a comment

mikofski Sep 7, 2018

wholmgren Sep 7, 2018

mikofski left a comment

mikofski Sep 7, 2018

mikofski commented Sep 7, 2018

stickler-ci bot commented Sep 9, 2018

mikofski left a comment •

edited

Loading

stickler-ci bot commented Sep 9, 2018

wholmgren commented Sep 9, 2018

mikofski commented Sep 9, 2018

stickler-ci bot commented Sep 9, 2018

wholmgren commented Sep 10, 2018 •

edited

Loading

mikofski commented Sep 10, 2018 •

edited

Loading

wholmgren commented Sep 10, 2018 via email

wholmgren commented Sep 10, 2018

wholmgren commented Sep 11, 2018

mikofski commented Sep 11, 2018

add extras_require complete and test options #566

add extras_require complete and test options #566

Conversation

wholmgren commented Sep 7, 2018 • edited Loading

mikofski commented Sep 7, 2018 • edited Loading

test extras_require using pip install pvlib[complete]:

mikofski left a comment

Choose a reason for hiding this comment

mikofski commented Sep 7, 2018

mikofski left a comment

Choose a reason for hiding this comment

mikofski Sep 7, 2018

Choose a reason for hiding this comment

wholmgren Sep 7, 2018

Choose a reason for hiding this comment

mikofski left a comment

Choose a reason for hiding this comment

mikofski Sep 7, 2018

Choose a reason for hiding this comment

mikofski commented Sep 7, 2018

stickler-ci bot commented Sep 9, 2018

mikofski left a comment • edited Loading

Choose a reason for hiding this comment

stickler-ci bot commented Sep 9, 2018

wholmgren commented Sep 9, 2018

mikofski commented Sep 9, 2018

stickler-ci bot commented Sep 9, 2018

wholmgren commented Sep 10, 2018 • edited Loading

mikofski commented Sep 10, 2018 • edited Loading

wholmgren commented Sep 10, 2018 via email

wholmgren commented Sep 10, 2018

wholmgren commented Sep 11, 2018

mikofski commented Sep 11, 2018

wholmgren commented Sep 7, 2018 •

edited

Loading

mikofski commented Sep 7, 2018 •

edited

Loading

test `extras_require` using `pip install pvlib[complete]`:

mikofski left a comment •

edited

Loading

wholmgren commented Sep 10, 2018 •

edited

Loading

mikofski commented Sep 10, 2018 •

edited

Loading