|
| 1 | +# Azure SDK for Python Conda Distributions |
| 2 | + |
| 3 | +## Local Environment Setup |
| 4 | + |
| 5 | +Follow the instructions [here](https://docs.conda.io/projects/conda-build/en/latest/install-conda-build.html) to install `conda` and `conda-build`. |
| 6 | + |
| 7 | +## CI Build Process |
| 8 | + |
| 9 | +There will be a `CondaArtifact` defined in the `ci.yml` of each service directory. (`sdk/<service>`) |
| 10 | + |
| 11 | +A Conda Artifact defines: |
| 12 | +- The location of the `meta.yml` |
| 13 | +- Which packages will be pulled into the combined artifact |
| 14 | +- The name of the combined artifact |
| 15 | +- Any other necessary details. |
| 16 | + |
| 17 | +## How to Build an Azure SDK Conda Package Locally |
| 18 | + |
| 19 | +### Set up your conda environment |
| 20 | + |
| 21 | +You will notice that all the azure-sdk conda distributions have the **same** version number and requirement set. This is due to the fact that the azure-sdk team pushes our conda packages out in waves. To support this, all versions are set via a common environment variable `AZURESDK_CONDA_VERSION`. |
| 22 | + |
| 23 | +We keep this environment variable set properly across all our builds by using a common `conda_env.yml` when creating our build environment. This environment definition ensures that: |
| 24 | + |
| 25 | +1. Our channel `https://azuresdkconda.blob.core.windows.net/channel1/` is added to the set to download packages |
| 26 | +2. The environment variable `AZURESDK_CONDA_VERSION` will be set exactly once. |
| 27 | + |
| 28 | + |
| 29 | +Reference the `conda_env.yml` in your local build by pass `-f <path to conda_env.yml>` when you create your conda environment. |
| 30 | + |
| 31 | +``` |
| 32 | +conda env create --yes --quiet --name ${{ artifact.name }} -f $(Build.SourcesDirectory)/eng/conda_env.yml |
| 33 | +``` |
| 34 | + |
| 35 | +### Create Your Build Directory |
| 36 | +Given how Conda packages are comprised of multiple source distributions _combined_, the buildable source does not exist directly within the azure-sdk-for-python repo. Currently, there is _some_ manual work that needs to be done. |
| 37 | + |
| 38 | +To begin, check your `ci.yml` for a `CondaArtifact`. Each these artifacts will become a single conda package. Let's use `storage/ci.yml` as an example. |
| 39 | + |
| 40 | +``` |
| 41 | + - name: azure-storage |
| 42 | + meta_source: meta.yml |
| 43 | + common_root: azure/storage |
| 44 | + checkout: |
| 45 | + - package: azure-storage-blob |
| 46 | + checkout_path: sdk/storage |
| 47 | + version: 12.8.0 |
| 48 | + - package: azure-storage-queue |
| 49 | + checkout_path: sdk/storage |
| 50 | + version: 12.1.5 |
| 51 | + - package: azure-storage-file-share |
| 52 | + checkout_path: sdk/storage |
| 53 | + version: 12.4.1 |
| 54 | + - package: azure-storage-file-datalake |
| 55 | + checkout_path: sdk/storage |
| 56 | + version: 12.3.0 |
| 57 | +``` |
| 58 | + |
| 59 | +- `name: azure-storage`: will be the name of the "combined" sdist package that we generate. |
| 60 | +- `meta_source: meta.yml`: this is the path (relative to the service directory) to the target conda package meta.yml. |
| 61 | +- `common_root: azure/storage`: when generating the combined package, where will we begin combining? This is tightly bound to folder structure within the generated sdist. |
| 62 | +- `checkout`: the `checkout` setting is a list of target packages that will go into the combined artifact. These targets will be individually sparse cloned, and copied into the conda build directory. Currently, this is a **manual step** in your local build. Reference `eng/pipelines/templates/get-tagged-code.yml` for exact details on how CI does it. |
| 63 | + |
| 64 | +Before we continue, you should be aware of two primary locations that are necessary, but not referenced directly in the `ci.yml`. |
| 65 | + |
| 66 | +The `build` folder and the `output` folder. The `build` folder (`$(Conda.Build)` variable in CI) is where we will... |
| 67 | + |
| 68 | +- store the cloned package code |
| 69 | +- generate the combined sdist |
| 70 | + |
| 71 | +To locally repro without magic given a specific `checkout` artifact: |
| 72 | + |
| 73 | +``` |
| 74 | +<cd sdk-for-python> |
| 75 | +git checkout `<package>_<version>` |
| 76 | +grab the entire <package> directory under the <checkout_path>. place into your `build` folder. |
| 77 | +``` |
| 78 | + |
| 79 | +Given the `storage` example. This is what your `build` folder should look like prior to invoking `build_conda_artifacts.py`. |
| 80 | + |
| 81 | +``` |
| 82 | +<build directory>/ |
| 83 | + azure-storage-blob/ <-- the package directly ripped from specified tag |
| 84 | + azure-storage-file-datalake/ |
| 85 | + azure-storage-file-share/ |
| 86 | + azure-storage-queue/ |
| 87 | +``` |
| 88 | + |
| 89 | +### Create the Combined SDist |
| 90 | + |
| 91 | +Once you have a directory assembled, invoke the script to build. The below command is formatted for visibility, recombine the lines however necessary for your chosen shell environment. |
| 92 | + |
| 93 | + |
| 94 | +``` |
| 95 | +python `build_conda_artifacts.py` |
| 96 | + -d "<output folder>" |
| 97 | + -b "<build folder>" |
| 98 | + -m "<resolvable path to sdk/storage/meta.yml>" |
| 99 | + -r "azure/storage" |
| 100 | + -n "azure-storage" |
| 101 | + -s "storage" |
| 102 | + -e "<resolvable path to repo root>/eng/conda_env.yml" |
| 103 | + -c "<resolvable path to sdk/storage/ci.yml>" |
| 104 | +``` |
| 105 | + |
| 106 | +### Generate the Conda Package |
| 107 | + |
| 108 | +Locally, from the anaconda prompt, set the environment variable `STORAGE_SOURCE_DISTRIBUTION` to the location of the generated sdist. After that: |
| 109 | + |
| 110 | +```bash |
| 111 | +export STORAGE_SOURCE_DISTRIBUTION=<path/to/generated/sdist> |
| 112 | +cd <meta.yml directory> |
| 113 | +conda-build . --output-folder <conda.output> |
| 114 | +``` |
0 commit comments