Change the editable VCS directory location for pip and Pipenv #1753
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a dependency from a version control system (eg Git) is installed in editable mode, the package manager has to clone the repository somewhere long-lived, that is then referenced by the `.pth` file added to `site-packages`. (When installed in normal, non-editable mode, the repo checkouts are instead deleted after the package is installed.) Until now, the buildpack configured pip and Pipenv to store these repos at `/app/.heroku/src/`, then later copied those files into the build directory and build cache. However, this approach isn't needed with the `.pth` rewriting we have now. In addition, the implementation didn't restore the cached `src/` directory, so the repos stored in the cache were never actually re-used on subsequent builds anyway. Now, pip and pipenv are configured to store the repositories at `<BUILD_DIR>/.heroku/python/src/`, which means: - The behaviour now matches that when using Poetry. - The repos get cached/restored/invalidated for free, as part of the existing handling of the `.heroku/python/` directory, and we avoid the additional directory copy from `/app` to `/tmp`, both of which help reduce build times.
Malax
approved these changes
Feb 19, 2025
edmorley
added a commit
that referenced
this pull request
Feb 21, 2025
Heroku builds occur at a different path to which the app will be run at run-time. As such, we have to perform path rewriting for editable dependencies, so that they work after relocation. The existing rewriting is performed at app boot (see code comment for more details), and works fine with pip and Poetry. However, I discovered that Pipenv doesn't correctly reinstall editable VCS dependencies if they use the new PEP660 style editable interface, which I've reported upstream here: pypa/pipenv#6348 This issue has affected apps using editable VCS dependencies with Pipenv for some time, but until now only at build-time for cached builds. However, after #1753 (which thankfully isn't yet released, due to me catching this as part of updating the tests to exercise the new PEP660 style editable interface) would otherwise affect apps at run-time too. As a workaround, we can perform build time rewriting of paths too, but must do so only for VCS dependencies (see code comment for why). Lastly, the Pipenv bug also requires that we perform explicit cache invalidation for Pipenv apps after the src dir move in #1753.
edmorley
added a commit
that referenced
this pull request
Feb 21, 2025
Heroku builds occur at a different path to which the app will be run at run-time. As such, we have to perform path rewriting for editable dependencies, so that they work after relocation. The existing rewriting is performed at app boot (see code comment for more details), and works fine with pip and Poetry. However, I discovered that Pipenv doesn't correctly reinstall editable VCS dependencies if they use the new PEP660 style editable interface, which I've reported upstream here: pypa/pipenv#6348 This issue has affected apps using editable VCS dependencies with Pipenv for some time, but until now only at build-time for cached builds. However, after #1753 (which thankfully isn't yet released, due to me catching this as part of updating the tests to exercise the new PEP660 style editable interface) would otherwise affect apps at run-time too. As a workaround, we can perform build time rewriting of paths too, but must do so only for VCS dependencies (see code comment for why). Lastly, the Pipenv bug also requires that we perform explicit cache invalidation for Pipenv apps after the src dir move in #1753.
edmorley
added a commit
that referenced
this pull request
Feb 21, 2025
Heroku builds occur at a different path to which the app will be run at run-time. As such, we have to perform path rewriting for editable dependencies, so that they work after relocation. The existing rewriting is performed at app boot (see code comment for more details), and works fine with pip and Poetry. However, I discovered that Pipenv doesn't correctly reinstall editable VCS dependencies if they use the new PEP660 style editable interface, which I've reported upstream here: pypa/pipenv#6348 This issue has affected apps using editable VCS dependencies with Pipenv for some time, but until now only at build-time for cached builds. However, after #1753 (which thankfully isn't yet released, due to me catching this as part of updating the tests to exercise the new PEP660 style editable interface) would otherwise affect apps at run-time too. As a workaround, we can perform build time rewriting of paths too, but must do so only for VCS dependencies (see code comment for why). Lastly, the Pipenv bug also requires that we perform explicit cache invalidation for Pipenv apps after the src dir move in #1753. GUS-W-17884520.
edmorley
added a commit
that referenced
this pull request
Feb 22, 2025
Heroku builds occur at a different path to which the app will be run at run-time. As such, we have to perform path rewriting for editable dependencies, so that they work after relocation. The existing rewriting is performed at app boot (see code comment for more details), and works fine with pip and Poetry. However, I discovered that Pipenv doesn't correctly reinstall editable VCS dependencies if they use the new PEP660 style editable interface, which I've reported upstream here: pypa/pipenv#6348 This issue has affected apps using editable VCS dependencies with Pipenv for some time, but until now only at build-time for cached builds. However, after #1753 (which thankfully isn't yet released, due to me catching this as part of updating the tests to exercise the new PEP660 style editable interface) would otherwise affect apps at run-time too. As a workaround, we can perform build time rewriting of paths too, but must do so only for VCS dependencies (see code comment for why). Lastly, the Pipenv bug also requires that we perform explicit cache invalidation for Pipenv apps after the src dir move in #1753. GUS-W-17884520.
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a dependency from a version control system (eg Git) is installed in editable mode, the package manager has to clone the repository somewhere long-lived, that is then referenced by the
.pthfile added tosite-packages. (When installed in normal non-editable mode, the repo checkouts are instead saved to a temporary directory and deleted after the package is installed.)Until now, the buildpack configured pip and Pipenv to store these repos at
/app/.heroku/src/, then later copied those files into the build directory and build cache. However, this approach isn't needed with the.pthrewriting we have now. In addition, the existing implementation didn't actually restore the cachedsrc/directory, so the repos stored in the cache were never re-used on subsequent builds anyway.Now, pip and pipenv are configured to store the repositories at
<BUILD_DIR>/.heroku/python/src/, which means:.heroku/python/directory, and we avoid the additional directory copy from/appto/tmp, both of which help reduce build times.GUS-W-17863838.