Skip to content

Commit 8ef2b3d

Browse files
committed
Merge PR #123 into unstable/v1
This patch implements support for secret-less OIDC-based publishing to PyPI-like package indexes. The OIDC flow is activated when neither username, nor password action inputs are set. The OIDC "token exchange," is an authentication technique that PyPI (and TestPyPI, and hopefully some future others) supports as an alternative to long-lived username/password combinations or API tokens. OIDC token exchange boils down to the following set of steps: 1. A user (currently only someone in the OIDC beta on PyPI) configured a particular GitHub Actions workflow in their repository as a trusted OIDC publisher; 2. That workflow uses this action to mint an OIDC token; 3. That OIDC token is sent to PyPI (or another index), which exchanges it for a temporary API token; 4. That API token is used as normal. For the seamless configuration-free upload to work, the end-users are expected to explicitly assign the `id-token: write` privilege to the auto-injected `GITHUB_TOKEN` secret on the job level. They should also set up GHA workflow trust on the PyPI side. PyPI's documentation: https://pypi.org/help/#openid-connect Beta test enrollment: pypi/warehouse#12965
2 parents 22b4d1f + 2b46bad commit 8ef2b3d

8 files changed

+234
-2
lines changed

.pre-commit-config.yaml

+4
Original file line numberDiff line numberDiff line change
@@ -105,11 +105,15 @@ repos:
105105
name: flake8 WPS-only
106106
args:
107107
- --ignore
108+
# NOTE: WPS326: Found implicit string concatenation
109+
# NOTE: WPS332: Found walrus operator
108110
- >-
109111
WPS102,
110112
WPS110,
111113
WPS111,
112114
WPS305,
115+
WPS326,
116+
WPS332,
113117
WPS347,
114118
WPS360,
115119
WPS421,

Dockerfile

+1
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ WORKDIR /app
2525
COPY LICENSE.md .
2626
COPY twine-upload.sh .
2727
COPY print-hash.py .
28+
COPY oidc-exchange.py .
2829

2930
RUN chmod +x twine-upload.sh
3031
ENTRYPOINT ["/app/twine-upload.sh"]

README.md

+48
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,51 @@ The secret used in `${{ secrets.PYPI_API_TOKEN }}` needs to be created on the
6262
settings page of your project on GitHub. See [Creating & using secrets].
6363

6464

65+
### Publishing with OpenID Connect
66+
67+
> **IMPORTANT**: This functionality is in beta, and will not work for you
68+
> unless you're a member of the PyPI OIDC beta testers' group. For more
69+
> information, see [warehouse#12965].
70+
71+
This action supports PyPI's [OpenID Connect publishing]
72+
implementation, which allows authentication to PyPI without a manually
73+
configured API token or username/password combination. To perform
74+
[OIDC publishing][OpenID Connect Publishing] with this action, your project's
75+
OIDC publisher must already be configured on PyPI.
76+
77+
To enter the OIDC flow, configure this action's job with the `id-token: write`
78+
permission and **without** an explicit username or password:
79+
80+
```yaml
81+
jobs:
82+
pypi-publish:
83+
name: Upload release to PyPI
84+
runs-on: ubuntu-latest
85+
permissions:
86+
id-token: write # IMPORTANT: this permission is mandatory for OIDC publishing
87+
steps:
88+
# retrieve your distributions here
89+
90+
- name: Publish package distributions to PyPI
91+
uses: pypa/gh-action-pypi-publish@release/v1
92+
```
93+
94+
Other indices that support OIDC publishing can also be used, like TestPyPI:
95+
96+
```yaml
97+
- name: Publish package distributions to TestPyPI
98+
uses: pypa/gh-action-pypi-publish@release/v1
99+
with:
100+
repository-url: https://test.pypi.org/legacy/
101+
```
102+
103+
> **Pro tip**: only set the `id-token: write` permission in the job that does
104+
> publishing, not globally. Also, try to separate building from publishing
105+
> — this makes sure that any scripts maliciously injected into the build
106+
> or test environment won't be able to elevate privileges while flying under
107+
> the radar.
108+
109+
65110
## Non-goals
66111

67112
This GitHub Action [has nothing to do with _building package
@@ -221,3 +266,6 @@ https://packaging.python.org/glossary/#term-Distribution-Package
221266
https://raw.githubusercontent.com/vshymanskyy/StandWithUkraine/main/banner-direct-single.svg
222267
[SWUdocs]:
223268
https://github.com/vshymanskyy/StandWithUkraine/blob/main/docs/README.md
269+
270+
[warehouse#12965]: https://github.com/pypi/warehouse/issues/12965
271+
[OpenID Connect Publishing]: https://pypi.org/help/#openid-connect

action.yml

+1
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ inputs:
2020
The inputs have been normalized to use kebab-case.
2121
Use `repository-url` instead.
2222
required: false
23+
default: https://pypi.org/legacy/
2324
packages-dir: # Canonical alias for `packages_dir`
2425
description: The target directory for distribution
2526
required: false

oidc-exchange.py

+156
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
import os
2+
import sys
3+
from http import HTTPStatus
4+
from pathlib import Path
5+
from typing import NoReturn
6+
from urllib.parse import urlparse
7+
8+
import id # pylint: disable=redefined-builtin
9+
import requests
10+
11+
_GITHUB_STEP_SUMMARY = Path(os.getenv("GITHUB_STEP_SUMMARY"))
12+
13+
# Rendered if OIDC identity token retrieval fails for any reason.
14+
_TOKEN_RETRIEVAL_FAILED_MESSAGE = """
15+
OIDC token retrieval failed: {identity_error}
16+
17+
This generally indicates a workflow configuration error, such as insufficient
18+
permissions. Make sure that your workflow has `id-token: write` configured
19+
at the job level, e.g.:
20+
21+
```yaml
22+
permissions:
23+
id-token: write
24+
```
25+
26+
Learn more at https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings.
27+
"""
28+
29+
# Rendered if the package index refuses the given OIDC token.
30+
_SERVER_REFUSED_TOKEN_EXCHANGE_MESSAGE = """
31+
Token request failed: the server refused the request for the following reasons:
32+
33+
{reasons}
34+
"""
35+
36+
# Rendered if the package index's token response isn't valid JSON.
37+
_SERVER_TOKEN_RESPONSE_MALFORMED_JSON = """
38+
Token request failed: the index produced an unexpected
39+
{status_code} response.
40+
41+
This strongly suggests a server configuration or downtime issue; wait
42+
a few minutes and try again.
43+
"""
44+
45+
# Rendered if the package index's token response isn't a valid API token payload.
46+
_SERVER_TOKEN_RESPONSE_MALFORMED_MESSAGE = """
47+
Token response error: the index gave us an invalid response.
48+
49+
This strongly suggests a server configuration or downtime issue; wait
50+
a few minutes and try again.
51+
"""
52+
53+
54+
def die(msg: str) -> NoReturn:
55+
with _GITHUB_STEP_SUMMARY.open("a", encoding="utf-8") as io:
56+
print(msg, file=io)
57+
58+
# NOTE: `msg` is Markdown formatted, so we emit only the header line to
59+
# avoid clogging the console log with a full Markdown formatted document.
60+
header = msg.splitlines()[0]
61+
print(f"::error::OIDC exchange failure: {header}", file=sys.stderr)
62+
sys.exit(1)
63+
64+
65+
def debug(msg: str):
66+
print(f"::debug::{msg.title()}", file=sys.stderr)
67+
68+
69+
def get_normalized_input(name: str) -> str | None:
70+
name = f"INPUT_{name.upper()}"
71+
if val := os.getenv(name):
72+
return val
73+
return os.getenv(name.replace("-", "_"))
74+
75+
76+
def assert_successful_audience_call(resp: requests.Response, domain: str):
77+
if resp.ok:
78+
return
79+
80+
match resp.status_code:
81+
case HTTPStatus.FORBIDDEN:
82+
# This index supports OIDC, but forbids the client from using
83+
# it (either because it's disabled, limited to a beta group, etc.)
84+
die(f"audience retrieval failed: repository at {domain} has OIDC disabled")
85+
case HTTPStatus.NOT_FOUND:
86+
# This index does not support OIDC.
87+
die(
88+
"audience retrieval failed: repository at "
89+
f"{domain} does not indicate OIDC support",
90+
)
91+
case other:
92+
status = HTTPStatus(other)
93+
# Unknown: the index may or may not support OIDC, but didn't respond with
94+
# something we expect. This can happen if the index is broken, in maintenance mode,
95+
# misconfigured, etc.
96+
die(
97+
"audience retrieval failed: repository at "
98+
f"{domain} responded with unexpected {other}: {status.phrase}",
99+
)
100+
101+
102+
repository_url = get_normalized_input("repository-url")
103+
repository_domain = urlparse(repository_url).netloc
104+
token_exchange_url = f"https://{repository_domain}/_/oidc/github/mint-token"
105+
106+
# Indices are expected to support `https://{domain}/_/oidc/audience`,
107+
# which tells OIDC exchange clients which audience to use.
108+
audience_url = f"https://{repository_domain}/_/oidc/audience"
109+
audience_resp = requests.get(audience_url)
110+
assert_successful_audience_call(audience_resp, repository_domain)
111+
112+
oidc_audience = audience_resp.json()["audience"]
113+
114+
debug(f"selected OIDC token exchange endpoint: {token_exchange_url}")
115+
116+
try:
117+
oidc_token = id.detect_credential(audience=oidc_audience)
118+
except id.IdentityError as identity_error:
119+
die(_TOKEN_RETRIEVAL_FAILED_MESSAGE.format(identity_error=identity_error))
120+
121+
# Now we can do the actual token exchange.
122+
mint_token_resp = requests.post(
123+
token_exchange_url,
124+
json={"token": oidc_token},
125+
)
126+
127+
try:
128+
mint_token_payload = mint_token_resp.json()
129+
except requests.JSONDecodeError:
130+
# Token exchange failure normally produces a JSON error response, but
131+
# we might have hit a server error instead.
132+
die(
133+
_SERVER_TOKEN_RESPONSE_MALFORMED_JSON.format(
134+
status_code=mint_token_resp.status_code,
135+
),
136+
)
137+
138+
# On failure, the JSON response includes the list of errors that
139+
# occurred during minting.
140+
if not mint_token_resp.ok:
141+
reasons = "\n".join(
142+
f"* `{error['code']}`: {error['description']}"
143+
for error in mint_token_payload["errors"]
144+
)
145+
146+
die(_SERVER_REFUSED_TOKEN_EXCHANGE_MESSAGE.format(reasons=reasons))
147+
148+
pypi_token = mint_token_payload.get("token")
149+
if pypi_token is None:
150+
die(_SERVER_TOKEN_RESPONSE_MALFORMED_MESSAGE)
151+
152+
# Mask the newly minted PyPI token, so that we don't accidentally leak it in logs.
153+
print(f"::add-mask::{pypi_token}", file=sys.stderr)
154+
155+
# This final print will be captured by the subshell in `twine-upload.sh`.
156+
print(pypi_token)

requirements/runtime.in

+8
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
twine
22

3+
# NOTE: Used to detect an ambient OIDC credential for OIDC publishing.
4+
id ~= 1.0
5+
6+
# NOTE: This is pulled in transitively through `twine`, but we also declare
7+
# NOTE: it explicitly here because `oidc-exchange.py` uses it.
8+
# Ref: https://github.com/di/id
9+
requests
10+
311
# NOTE: `pkginfo` is a transitive dependency for us that is coming from Twine.
412
# NOTE: It is declared here only to avoid installing a broken combination of
513
# NOTE: the distribution packages. This should be removed once a fixed version

requirements/runtime.txt

+10-2
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ cryptography==39.0.1
1818
# via secretstorage
1919
docutils==0.19
2020
# via readme-renderer
21+
id==1.0.0
22+
# via -r requirements/runtime.in
2123
idna==3.4
2224
# via requests
2325
importlib-metadata==5.1.0
@@ -36,10 +38,12 @@ more-itertools==9.0.0
3638
# via jaraco-classes
3739
pkginfo==1.9.2
3840
# via
39-
# -r runtime.in
41+
# -r requirements/runtime.in
4042
# twine
4143
pycparser==2.21
4244
# via cffi
45+
pydantic==1.10.6
46+
# via id
4347
pygments==2.13.0
4448
# via
4549
# readme-renderer
@@ -48,6 +52,8 @@ readme-renderer==37.3
4852
# via twine
4953
requests==2.28.1
5054
# via
55+
# -r requirements/runtime.in
56+
# id
5157
# requests-toolbelt
5258
# twine
5359
requests-toolbelt==0.10.1
@@ -61,7 +67,9 @@ secretstorage==3.3.3
6167
six==1.16.0
6268
# via bleach
6369
twine==4.0.1
64-
# via -r runtime.in
70+
# via -r requirements/runtime.in
71+
typing-extensions==4.5.0
72+
# via pydantic
6573
urllib3==1.26.13
6674
# via
6775
# requests

twine-upload.sh

+6
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,12 @@ INPUT_VERIFY_METADATA="$(get-normalized-input 'verify-metadata')"
4040
INPUT_SKIP_EXISTING="$(get-normalized-input 'skip-existing')"
4141
INPUT_PRINT_HASH="$(get-normalized-input 'print-hash')"
4242

43+
if [[ "${INPUT_USER}" == "__token__" && -z "${INPUT_PASSWORD}" ]] ; then
44+
# No password supplied by the user implies that we're in the OIDC flow;
45+
# retrieve the OIDC credential and exchange it for a PyPI API token.
46+
echo "::notice::In OIDC flow"
47+
INPUT_PASSWORD="$(python /app/oidc-exchange.py)"
48+
fi
4349

4450
if [[
4551
"$INPUT_USER" == "__token__" &&

0 commit comments

Comments
 (0)