Skip to content

Commit 539cd04

Browse files
alexeaglealexeagle
committed
Add pip_repository documentation
The macros are leaky and otherwise you have to read sources to find out about the kwargs
1 parent f480ebe commit 539cd04

File tree

5 files changed

+169
-11
lines changed

5 files changed

+169
-11
lines changed

‎docs/BUILD

+13
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ licenses(["notice"]) # Apache 2.0
2424
_DOCS = {
2525
"packaging": "//docs:packaging-docs",
2626
"pip": "//docs:pip-docs",
27+
"pip_repository": "//docs:pip-repository",
2728
"python": "//docs:core-docs",
2829
}
2930

@@ -104,6 +105,18 @@ stardoc(
104105
],
105106
)
106107

108+
stardoc(
109+
name = "pip-repository",
110+
out = "pip_repository.md_",
111+
input = "//python/pip_install:pip_repository.bzl",
112+
target_compatible_with = _NOT_WINDOWS,
113+
deps = [
114+
":bazel_repo_tools",
115+
":pip_install_bzl",
116+
"//third_party/github.com/bazelbuild/bazel-skylib/lib:versions",
117+
],
118+
)
119+
107120
stardoc(
108121
name = "packaging-docs",
109122
out = "packaging.md_",

‎docs/pip.md

+7-5
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,8 @@ Accepts a `requirements.txt` file and installs the dependencies listed within.
7474

7575
Those dependencies become available in a generated `requirements.bzl` file.
7676

77-
This macro runs a repository rule that invokes `pip`. In your WORKSPACE file:
77+
This macro wraps the [`pip_repository`](./pip_repository) rule that invokes `pip`.
78+
In your WORKSPACE file:
7879

7980
```python
8081
pip_install(
@@ -142,7 +143,7 @@ alias(
142143
| :-------------: | :-------------: | :-------------: |
143144
| requirements | A 'requirements.txt' pip requirements file. | <code>None</code> |
144145
| name | A unique name for the created external repository (default 'pip'). | <code>"pip"</code> |
145-
| kwargs | Keyword arguments passed directly to the <code>pip_repository</code> repository rule. | none |
146+
| kwargs | Additional arguments to the [<code>pip_repository</code>](./pip_repository) repository rule. | none |
146147

147148

148149
<a name="#pip_parse"></a>
@@ -158,7 +159,8 @@ Accepts a locked/compiled requirements file and installs the dependencies listed
158159
Those dependencies become available in a generated `requirements.bzl` file.
159160
You can instead check this `requirements.bzl` file into your repo, see the "vendoring" section below.
160161

161-
This macro runs a repository rule that invokes `pip`. In your WORKSPACE file:
162+
This macro wraps the [`pip_repository`](./pip_repository) rule that invokes `pip`, with `incremental` set.
163+
In your WORKSPACE file:
162164

163165
```python
164166
load("@rules_python//python:pip.bzl", "pip_parse")
@@ -242,9 +244,9 @@ See the example in rules_python/examples/pip_parse_vendored.
242244

243245
| Name | Description | Default Value |
244246
| :-------------: | :-------------: | :-------------: |
245-
| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. | none |
247+
| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. Note that if your lockfile is platform-dependent, you can use the <code>requirements_[platform]</code> attributes. | none |
246248
| name | The name of the generated repository. The generated repositories containing each requirement will be of the form &lt;name&gt;_&lt;requirement-name&gt;. | <code>"pip_parsed_deps"</code> |
247-
| kwargs | Additional keyword arguments for the underlying <code>pip_repository</code> rule. | none |
249+
| kwargs | Additional arguments to the [<code>pip_repository</code>](./pip_repository) repository rule. | none |
248250

249251

250252
<a name="#pip_repositories"></a>

‎docs/pip_repository.md

+140
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
<!-- Generated with Stardoc: http://skydoc.bazel.build -->
2+
3+
<a name="#pip_repository"></a>
4+
5+
## pip_repository
6+
7+
<pre>
8+
pip_repository(<a href="#pip_repository-name">name</a>, <a href="#pip_repository-annotations">annotations</a>, <a href="#pip_repository-enable_implicit_namespace_pkgs">enable_implicit_namespace_pkgs</a>, <a href="#pip_repository-environment">environment</a>, <a href="#pip_repository-extra_pip_args">extra_pip_args</a>,
9+
<a href="#pip_repository-incremental">incremental</a>, <a href="#pip_repository-isolated">isolated</a>, <a href="#pip_repository-pip_data_exclude">pip_data_exclude</a>, <a href="#pip_repository-python_interpreter">python_interpreter</a>, <a href="#pip_repository-python_interpreter_target">python_interpreter_target</a>,
10+
<a href="#pip_repository-quiet">quiet</a>, <a href="#pip_repository-repo_prefix">repo_prefix</a>, <a href="#pip_repository-requirements">requirements</a>, <a href="#pip_repository-requirements_darwin">requirements_darwin</a>, <a href="#pip_repository-requirements_linux">requirements_linux</a>,
11+
<a href="#pip_repository-requirements_lock">requirements_lock</a>, <a href="#pip_repository-requirements_windows">requirements_windows</a>, <a href="#pip_repository-timeout">timeout</a>)
12+
</pre>
13+
14+
A rule for importing `requirements.txt` dependencies into Bazel.
15+
16+
This rule imports a `requirements.txt` file and generates a new
17+
`requirements.bzl` file. This is used via the `WORKSPACE` pattern:
18+
19+
```python
20+
pip_repository(
21+
name = "foo",
22+
requirements = ":requirements.txt",
23+
)
24+
```
25+
26+
You can then reference imported dependencies from your `BUILD` file with:
27+
28+
```python
29+
load("@foo//:requirements.bzl", "requirement")
30+
py_library(
31+
name = "bar",
32+
...
33+
deps = [
34+
"//my/other:dep",
35+
requirement("requests"),
36+
requirement("numpy"),
37+
],
38+
)
39+
```
40+
41+
Or alternatively:
42+
```python
43+
load("@foo//:requirements.bzl", "all_requirements")
44+
py_binary(
45+
name = "baz",
46+
...
47+
deps = [
48+
":foo",
49+
] + all_requirements,
50+
)
51+
```
52+
53+
54+
**ATTRIBUTES**
55+
56+
57+
| Name | Description | Type | Mandatory | Default |
58+
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
59+
| name | A unique name for this repository. | <a href="https://bazel.build/docs/build-ref.html#name">Name</a> | required | |
60+
| annotations | Optional annotations to apply to packages | <a href="https://bazel.build/docs/skylark/lib/dict.html">Dictionary: String -> String</a> | optional | {} |
61+
| enable_implicit_namespace_pkgs | If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary and py_test targets must specify either <code>legacy_create_init=False</code> or the global Bazel option <code>--incompatible_default_to_explicit_init_py</code> to prevent <code>__init__.py</code> being automatically generated in every directory.<br><br>This option is required to support some packages which cannot handle the conversion to pkg-util style. | Boolean | optional | False |
62+
| environment | Environment variables to set in the pip subprocess. Can be used to set common variables such as <code>http_proxy</code>, <code>https_proxy</code> and <code>no_proxy</code> Note that pip is run with "--isolated" on the CLI so PIP_&lt;VAR&gt;_&lt;NAME&gt; style env vars are ignored, but env vars that control requests and urllib3 can be passed. | <a href="https://bazel.build/docs/skylark/lib/dict.html">Dictionary: String -> String</a> | optional | {} |
63+
| extra_pip_args | Extra arguments to pass on to pip. Must not contain spaces. | List of strings | optional | [] |
64+
| incremental | Create the repository in incremental mode. | Boolean | optional | False |
65+
| isolated | Whether or not to pass the [--isolated](https://pip.pypa.io/en/stable/cli/pip/#cmdoption-isolated) flag to the underlying pip command. Alternatively, the <code>RULES_PYTHON_PIP_ISOLATED</code> enviornment varaible can be used to control this flag. | Boolean | optional | True |
66+
| pip_data_exclude | Additional data exclusion parameters to add to the pip packages BUILD file. | List of strings | optional | [] |
67+
| python_interpreter | The python interpreter to use. This can either be an absolute path or the name of a binary found on the host's <code>PATH</code> environment variable. If no value is set <code>python3</code> is defaulted for Unix systems and <code>python.exe</code> for Windows. | String | optional | "" |
68+
| python_interpreter_target | If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
69+
| quiet | If True, suppress printing stdout and stderr output to the terminal. | Boolean | optional | True |
70+
| repo_prefix | Prefix for the generated packages. For non-incremental mode the packages will be of the form<br><br>@&lt;name&gt;//&lt;prefix&gt;&lt;sanitized-package-name&gt;/...<br><br>For incremental mode the packages will be of the form<br><br>@&lt;prefix&gt;&lt;sanitized-package-name&gt;//... | String | optional | "" |
71+
| requirements | A 'requirements.txt' pip requirements file. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
72+
| requirements_darwin | Override the requirements_lock attribute when the host platform is Mac OS | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
73+
| requirements_linux | Override the requirements_lock attribute when the host platform is Linux | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
74+
| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
75+
| requirements_windows | Override the requirements_lock attribute when the host platform is Windows | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
76+
| timeout | Timeout (in seconds) on the rule's execution duration. | Integer | optional | 600 |
77+
78+
79+
<a name="#whl_library"></a>
80+
81+
## whl_library
82+
83+
<pre>
84+
whl_library(<a href="#whl_library-name">name</a>, <a href="#whl_library-annotation">annotation</a>, <a href="#whl_library-enable_implicit_namespace_pkgs">enable_implicit_namespace_pkgs</a>, <a href="#whl_library-environment">environment</a>, <a href="#whl_library-extra_pip_args">extra_pip_args</a>, <a href="#whl_library-isolated">isolated</a>,
85+
<a href="#whl_library-pip_data_exclude">pip_data_exclude</a>, <a href="#whl_library-python_interpreter">python_interpreter</a>, <a href="#whl_library-python_interpreter_target">python_interpreter_target</a>, <a href="#whl_library-quiet">quiet</a>, <a href="#whl_library-repo">repo</a>, <a href="#whl_library-repo_prefix">repo_prefix</a>,
86+
<a href="#whl_library-requirement">requirement</a>, <a href="#whl_library-timeout">timeout</a>)
87+
</pre>
88+
89+
90+
Download and extracts a single wheel based into a bazel repo based on the requirement string passed in.
91+
Instantiated from pip_repository and inherits config options from there.
92+
93+
**ATTRIBUTES**
94+
95+
96+
| Name | Description | Type | Mandatory | Default |
97+
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
98+
| name | A unique name for this repository. | <a href="https://bazel.build/docs/build-ref.html#name">Name</a> | required | |
99+
| annotation | Optional json encoded file containing annotation to apply to the extracted wheel. See <code>package_annotation</code> | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
100+
| enable_implicit_namespace_pkgs | If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary and py_test targets must specify either <code>legacy_create_init=False</code> or the global Bazel option <code>--incompatible_default_to_explicit_init_py</code> to prevent <code>__init__.py</code> being automatically generated in every directory.<br><br>This option is required to support some packages which cannot handle the conversion to pkg-util style. | Boolean | optional | False |
101+
| environment | Environment variables to set in the pip subprocess. Can be used to set common variables such as <code>http_proxy</code>, <code>https_proxy</code> and <code>no_proxy</code> Note that pip is run with "--isolated" on the CLI so PIP_&lt;VAR&gt;_&lt;NAME&gt; style env vars are ignored, but env vars that control requests and urllib3 can be passed. | <a href="https://bazel.build/docs/skylark/lib/dict.html">Dictionary: String -> String</a> | optional | {} |
102+
| extra_pip_args | Extra arguments to pass on to pip. Must not contain spaces. | List of strings | optional | [] |
103+
| isolated | Whether or not to pass the [--isolated](https://pip.pypa.io/en/stable/cli/pip/#cmdoption-isolated) flag to the underlying pip command. Alternatively, the <code>RULES_PYTHON_PIP_ISOLATED</code> enviornment varaible can be used to control this flag. | Boolean | optional | True |
104+
| pip_data_exclude | Additional data exclusion parameters to add to the pip packages BUILD file. | List of strings | optional | [] |
105+
| python_interpreter | The python interpreter to use. This can either be an absolute path or the name of a binary found on the host's <code>PATH</code> environment variable. If no value is set <code>python3</code> is defaulted for Unix systems and <code>python.exe</code> for Windows. | String | optional | "" |
106+
| python_interpreter_target | If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
107+
| quiet | If True, suppress printing stdout and stderr output to the terminal. | Boolean | optional | True |
108+
| repo | Pointer to parent repo name. Used to make these rules rerun if the parent repo changes. | String | required | |
109+
| repo_prefix | Prefix for the generated packages. For non-incremental mode the packages will be of the form<br><br>@&lt;name&gt;//&lt;prefix&gt;&lt;sanitized-package-name&gt;/...<br><br>For incremental mode the packages will be of the form<br><br>@&lt;prefix&gt;&lt;sanitized-package-name&gt;//... | String | optional | "" |
110+
| requirement | Python requirement string describing the package to make available | String | required | |
111+
| timeout | Timeout (in seconds) on the rule's execution duration. | Integer | optional | 600 |
112+
113+
114+
<a name="#package_annotation"></a>
115+
116+
## package_annotation
117+
118+
<pre>
119+
package_annotation(<a href="#package_annotation-additive_build_content">additive_build_content</a>, <a href="#package_annotation-copy_files">copy_files</a>, <a href="#package_annotation-copy_executables">copy_executables</a>, <a href="#package_annotation-data">data</a>, <a href="#package_annotation-data_exclude_glob">data_exclude_glob</a>,
120+
<a href="#package_annotation-srcs_exclude_glob">srcs_exclude_glob</a>)
121+
</pre>
122+
123+
Annotations to apply to the BUILD file content from package generated from a `pip_repository` rule.
124+
125+
[cf]: https://github.com/bazelbuild/bazel-skylib/blob/main/docs/copy_file_doc.md
126+
127+
128+
**PARAMETERS**
129+
130+
131+
| Name | Description | Default Value |
132+
| :-------------: | :-------------: | :-------------: |
133+
| additive_build_content | Raw text to add to the generated <code>BUILD</code> file of a package. | <code>None</code> |
134+
| copy_files | A mapping of <code>src</code> and <code>out</code> files for [@bazel_skylib//rules:copy_file.bzl][cf] | <code>{}</code> |
135+
| copy_executables | A mapping of <code>src</code> and <code>out</code> files for [@bazel_skylib//rules:copy_file.bzl][cf]. Targets generated here will also be flagged as executable. | <code>{}</code> |
136+
| data | A list of labels to add as <code>data</code> dependencies to the generated <code>py_library</code> target. | <code>[]</code> |
137+
| data_exclude_glob | A list of exclude glob patterns to add as <code>data</code> to the generated <code>py_library</code> target. | <code>[]</code> |
138+
| srcs_exclude_glob | A list of labels to add as <code>srcs</code> to the generated <code>py_library</code> target. | <code>[]</code> |
139+
140+

‎python/pip.bzl

+8-5
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,8 @@ def pip_install(requirements = None, name = "pip", **kwargs):
2525
2626
Those dependencies become available in a generated `requirements.bzl` file.
2727
28-
This macro runs a repository rule that invokes `pip`. In your WORKSPACE file:
28+
This macro wraps the [`pip_repository`](./pip_repository) rule that invokes `pip`.
29+
In your WORKSPACE file:
2930
3031
```python
3132
pip_install(
@@ -88,7 +89,7 @@ def pip_install(requirements = None, name = "pip", **kwargs):
8889
Args:
8990
requirements (Label): A 'requirements.txt' pip requirements file.
9091
name (str, optional): A unique name for the created external repository (default 'pip').
91-
**kwargs (dict): Keyword arguments passed directly to the `pip_repository` repository rule.
92+
**kwargs (dict): Additional arguments to the [`pip_repository`](./pip_repository) repository rule.
9293
"""
9394

9495
# Just in case our dependencies weren't already fetched
@@ -107,7 +108,8 @@ def pip_parse(requirements_lock, name = "pip_parsed_deps", **kwargs):
107108
Those dependencies become available in a generated `requirements.bzl` file.
108109
You can instead check this `requirements.bzl` file into your repo, see the "vendoring" section below.
109110
110-
This macro runs a repository rule that invokes `pip`. In your WORKSPACE file:
111+
This macro wraps the [`pip_repository`](./pip_repository) rule that invokes `pip`, with `incremental` set.
112+
In your WORKSPACE file:
111113
112114
```python
113115
load("@rules_python//python:pip.bzl", "pip_parse")
@@ -191,10 +193,11 @@ def pip_parse(requirements_lock, name = "pip_parsed_deps", **kwargs):
191193
of 'requirements' no resolve will take place and pip_repository will create
192194
individual repositories for each of your dependencies so that wheels are
193195
fetched/built only for the targets specified by 'build/run/test'.
196+
Note that if your lockfile is platform-dependent, you can use the `requirements_[platform]`
197+
attributes.
194198
name (str, optional): The name of the generated repository. The generated repositories
195199
containing each requirement will be of the form <name>_<requirement-name>.
196-
**kwargs (dict): Additional keyword arguments for the underlying
197-
`pip_repository` rule.
200+
**kwargs (dict): Additional arguments to the [`pip_repository`](./pip_repository) repository rule.
198201
"""
199202

200203
# Just in case our dependencies weren't already fetched

‎python/pip_install/pip_repository.bzl

+1-1
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ def _locked_requirements(rctx):
126126
requirements_txt = rctx.attr.requirements_darwin
127127
elif os.startswith("linux") and rctx.attr.requirements_linux != None:
128128
requirements_txt = rctx.attr.requirements_linux
129-
elif "win" in os:
129+
elif "win" in os and rctx.attr.requirements_windows != None:
130130
requirements_txt = rctx.attr.requirements_windows
131131
if not requirements_txt:
132132
fail("""\

0 commit comments

Comments
 (0)