New command pep517.meta

[jaraco]

As outlined in #26, this command implements a pep517.build-metapep517.meta command, allowing for building metadata for a source directory.

The new build_metameta module also exposes an API for retrieving the metadata as an in-memory zip file allowing consumers to receive metadata in-process without thread safety and other race concerns.

Additionally, this request adds a compat_system function which, given a source directory, produces a "build system", either by loading it from pyproject.toml or falling back to "setuptools (legacy)".

This proposed technique fulfills the needs of importlib_metadata or other consumers that wish to load metadata from PEP 517 projects on disk, or if compat_system() is passed for system, for most any project.

Edit: This technique additionally introduces a load() function for loading an importlib_metadata.Distribution object from a source directory.

One downside I'd like to still investigate:

• When generating the metadata, a lot of output is emitted. It would be nice to be able to suppress that output.

[jaraco]

The code in b35a8dd is currently released to https://m.devpi.net/jaraco/pep517-meta.

[jaraco]

See importlib_metadata!67 for the corresponding importlib_metadata work.

[pfmoore]

If I understand correctly, this implements a proposed extension to PEP 517? If that's true, then I think that we should hold off on merging this PR until that proposed extension is accepted,

[jaraco]

this implements a proposed extension to PEP 517?

No. This PR only implements APIs around the already specified hooks defined in PEP 517, namely prepare_metadata_for_build_wheel.

This new command and API follow the patterns found in pep517.build and pep517.check to present comparable functionality for metadata.

It does go one step further. Based on the discussion and recommendation in pypa/packaging-problems#226, the build_meta_as_zip function builds the metadata in a temporary directory, zips it up in-memory, and then returns that in-memory zipfile.

But as you can see, each of these functions are disentangled from the more fundamental functionality (building the metadata in some target directory).

[pfmoore]

this implements a proposed extension to PEP 517?

No.

OK, cool. My apologies, I got the idea it was an extension from this comment. I must have misread the thread (I was skimming quickly, I don't have time right now for a full review).

[jaraco]

Here's an example of this functionality in action:

~ $ pip-run -i https://m.devpi.net/jaraco/pep517-meta pep517                                                                                                                   
Looking in indexes: https://m.devpi.net/jaraco/pep517-meta
Collecting pep517
  Downloading https://m.devpi.net/jaraco/pep517-meta/%2Bf/1f7/c95a2a4d63948/pep517-0.5.0-py2.py3-none-any.whl (49kB)
     |████████████████████████████████| 51kB 440kB/s 
Collecting pytoml (from pep517)
Installing collected packages: pytoml, pep517
Successfully installed pep517-0.5.0 pytoml-0.1.20
>>> Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import pep517.build_meta
>>> import functools
>>> import os
>>> builder = functools.partial(pep517.build_meta.build_meta, source_dir=os.path.expanduser('~/m/jaraco.functools'))
>>> meta_zip = pep517.build_meta.build_meta_as_zip(builder)
Collecting setuptools>=34.4
  Using cached https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl
Collecting setuptools_scm>=1.15
  Using cached https://files.pythonhosted.org/packages/1d/70/97966deebaeeda0b81d3cd63ba9f8ec929b838871ed17476de9d8159db3e/setuptools_scm-3.3.3-py2.py3-none-any.whl
Installing collected packages: setuptools, wheel, setuptools-scm
Successfully installed setuptools setuptools-scm wheel
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl
Collecting setuptools_scm>=1.15.0
  Using cached https://files.pythonhosted.org/packages/1d/70/97966deebaeeda0b81d3cd63ba9f8ec929b838871ed17476de9d8159db3e/setuptools_scm-3.3.3-py2.py3-none-any.whl
Installing collected packages: wheel, setuptools-scm
Successfully installed setuptools-scm-3.3.3 wheel-0.33.4
running dist_info
creating /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.egg-info
writing /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.egg-info/PKG-INFO
writing dependency_links to /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.egg-info/dependency_links.txt
writing requirements to /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.egg-info/requires.txt
writing top-level names to /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.egg-info/top_level.txt
writing manifest file '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.egg-info/SOURCES.txt'
writing manifest file '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.egg-info/SOURCES.txt'
creating '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp04lgx4m_/jaraco.functools.dist-info'
>>> meta_zip
<zipfile.ZipFile file=<_io.BytesIO object at 0x10f9985c8> mode='w'>

Another thing I'm considering is instead of stopping at generating the zip file, perhaps pep517 could own this functionality. Then, importlib_metadata would facilitate the metadata parsing, but pep517 could own the metadata loading logic. That feels like a better separation of concerns, especially as importlib_metadata enters the stdlib and can't reliably depend on pep517.

• Migrate PathDistribution construction logic to pep517.

[jaraco]

Also, as is demonstrated by failures I discovered above, we should add tests.

• Add tests for this functionality.

[gaborbernat]

Overall looks ok but needs cleanup and tests.

[gaborbernat]

I think this generally looks good and promising, I would say add a few tests to cover the new code and we're good to give it the first go (tox definitely would like to use this e.g.).

[jaraco]

As you can see, tests are failing on Python 2.7 due to flit not being available on Python 2.7 (and as a result, it's not possible to build the metadata for pep517 on Python 2.7). I guess that's an xfail?

[jaraco]

I feel a lot better about this latest implementation. It suppresses the output from the pip_install invocation by redirecting the output to the log.

[gaborbernat]

@pfmoore can we land this?

[pfmoore]

I'm going to leave that decision to @takluyver as owner of this project. My only real interest is in the wrappers, which pip uses.

[takluyver]

Thanks, and sorry it's taken me a while to get to looking at this.

[jaraco]

Thanks for the review. All good points. I'll follow up later.

[jaraco]

I believe I've addressed all of the concerns that @takluyver raised. Any remaining concerns or comments?

[takluyver]

Thanks!

[gaborbernat]

@takluyver want to do a release with this? thanks!

[takluyver]

There you go, 0.6.0 now out.

[takluyver]

PS @jaraco or anyone: if you got a notification about the deploy failure, don't worry. I had forgotten that the releases were automated, and my personal release habit is to upload to PyPI first and then create the tag, so it failed because the packages were already on PyPI.