feat(build): Add support for Python scripts via pythonExtension (#1686)

* add PythonExtension

* remove potential shell injection risk

* Filter out blank lines or comment lines

* fix spelling

* add pythonExtension's `runInline`

* changes to requirements don’t invalidate the entire install layer

* copy script files on-demand

* improve PythonExtension types and logging

* add changeset

* fix broken imports

* Improve security of inline script execution

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Add file existence check for requirementsFile

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* update lock file

* Enhance error handling with detailed error information

* Add portable type annotation

* fix error TS18046: 'e' is of type 'unknown'

* export the python extension

* add `pythonExtension` to the catalog

* fix `Cannot find module '@trigger.dev/build/extensions/core' (TS2307)

* replace execa by tinyexec

* Update pnpm-lock.yaml

* add custom traces instead of logging

* The cleanup in the finally block does not fail silently anymore

* move python runtime/extension to independent package

* fix build package readme

* update lock file

* add documentation to python's package

* add missing dependency

* Update little-trains-begin.md

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Matt Aitken <[email protected]>
Co-authored-by: Eric Allam <[email protected]>

Author: 4 people

.changeset/little-trains-begin.md

@@ -0,0 +1,5 @@
+---
+"@trigger.dev/python": patch
+---
+
+Introduced a new Python extension to enhance the build process. It now allows users to execute Python scripts with improved support and error handling.

packages/build/README.md

@@ -1,3 +1,3 @@
-# Official TypeScript SDK for Trigger.dev
+# Official Build Package of Trigger.dev
 
-View the full documentation for the [here](https://trigger.dev/docs)
+View the full documentation [here](https://trigger.dev/docs)

packages/python/CHANGELOG.md

@@ -0,0 +1 @@
+# @trigger.dev/python

packages/python/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Trigger.dev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

packages/python/README.md

@@ -0,0 +1,102 @@
+# Python Extension for Trigger.dev
+
+The Python extension enhances Trigger.dev's build process by enabling limited support for executing Python scripts within your tasks.
+
+## Overview
+
+This extension introduces the <code>pythonExtension</code> build extension, which offers several key capabilities:
+
+- **Install Python Dependencies (Except in Dev):** Automatically installs Python and specified dependencies using <code>pip</code>.
+- **Requirements File Support:** You can specify dependencies in a <code>requirements.txt</code> file.
+- **Inline Requirements:** Define dependencies directly within your <code>trigger.config.ts</code> file using the <code>requirements</code> option.
+- **Virtual Environment:** Creates a virtual environment (<code>/opt/venv</code>) inside containers to isolate Python dependencies.
+- **Helper Functions:** Provides a variety of functions for executing Python code:
+  - <code>run</code>: Executes Python commands with proper environment setup.
+  - <code>runInline</code>: Executes inline Python code directly from Node.
+  - <code>runScript</code>: Executes standalone <code>.py</code> script files.
+- **Custom Python Path:** In development, you can configure <code>pythonBinaryPath</code> to point to a custom Python installation.
+
+## Usage
+
+1. Add the extension to your <code>trigger.config.ts</code> file:
+
+```typescript
+import { defineConfig } from "@trigger.dev/sdk/v3";
+import pythonExtension from "@trigger.dev/python/extension";
+
+export default defineConfig({
+  project: "<project ref>",
+  build: {
+    extensions: [
+      pythonExtension({
+        requirementsFile: "./requirements.txt", // Optional: Path to your requirements file
+        pythonBinaryPath: path.join(rootDir, `.venv/bin/python`), // Optional: Custom Python binary path
+        scripts: ["my_script.py"], // List of Python scripts to include
+      }),
+    ],
+  },
+});
+```
+
+2. (Optional) Create a <code>requirements.txt</code> file in your project root with the necessary Python dependencies.
+
+3. Execute Python scripts within your tasks using one of the provided functions:
+
+### Running a Python Script
+
+```typescript
+import { task } from "@trigger.dev/sdk/v3";
+import python from "@trigger.dev/python";
+
+export const myScript = task({
+  id: "my-python-script",
+  run: async () => {
+    const result = await python.runScript("my_script.py", ["hello", "world"]);
+    return result.stdout;
+  },
+});
+```
+
+### Running Inline Python Code
+
+```typescript
+import { task } from "@trigger.dev/sdk/v3";
+import python from "@trigger.dev/python";
+
+export const myTask = task({
+  id: "to_datetime-task",
+  run: async () => {
+    const result = await python.runInline(`
+import pandas as pd
+
+pandas.to_datetime("${+new Date() / 1000}")
+`);
+    return result.stdout;
+  },
+});
+```
+
+### Running Lower-Level Commands
+
+```typescript
+import { task } from "@trigger.dev/sdk/v3";
+import python from "@trigger.dev/python";
+
+export const pythonVersionTask = task({
+  id: "python-version-task",
+  run: async () => {
+    const result = await python.run(["--version"]);
+    return result.stdout; // Expected output: Python 3.12.8
+  },
+});
+```
+
+## Limitations
+
+- This is a **partial implementation** and does not provide full Python support as an execution runtime for tasks.
+- Only basic Python script execution is supported; scripts are not automatically copied to staging/production containers.
+- Manual intervention may be required for installing and configuring binary dependencies in development environments.
+
+## Additional Information
+
+For more detailed documentation, visit the official docs at [Trigger.dev Documentation](https://trigger.dev/docs).

packages/python/package.json

@@ -0,0 +1,93 @@
+{
+  "name": "@trigger.dev/python",
+  "version": "3.3.16",
+  "description": "Python runtime and build extension for Trigger.dev",
+  "license": "MIT",
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/triggerdotdev/trigger.dev",
+    "directory": "packages/python"
+  },
+  "type": "module",
+  "files": [
+    "dist"
+  ],
+  "tshy": {
+    "selfLink": false,
+    "main": true,
+    "module": true,
+    "project": "./tsconfig.src.json",
+    "exports": {
+      "./package.json": "./package.json",
+      ".": "./src/index.ts",
+      "./extension": "./src/extension.ts"
+    },
+    "sourceDialects": [
+      "@triggerdotdev/source"
+    ]
+  },
+  "typesVersions": {
+    "*": {
+      "extension": [
+        "dist/commonjs/extension.d.ts"
+      ]
+    }
+  },
+  "scripts": {
+    "clean": "rimraf dist",
+    "build": "tshy && pnpm run update-version",
+    "dev": "tshy --watch",
+    "typecheck": "tsc --noEmit -p tsconfig.src.json",
+    "update-version": "tsx ../../scripts/updateVersion.ts",
+    "check-exports": "attw --pack ."
+  },
+  "dependencies": {
+    "@trigger.dev/build": "workspace:3.3.16",
+    "@trigger.dev/core": "workspace:3.3.16",
+    "@trigger.dev/sdk": "workspace:3.3.16",
+    "tinyexec": "^0.3.2"
+  },
+  "devDependencies": {
+    "@types/node": "20.14.14",
+    "rimraf": "6.0.1",
+    "tshy": "^3.0.2",
+    "typescript": "^5.5.4",
+    "tsx": "4.17.0",
+    "esbuild": "^0.23.0",
+    "@arethetypeswrong/cli": "^0.15.4"
+  },
+  "engines": {
+    "node": ">=18.20.0"
+  },
+  "exports": {
+    "./package.json": "./package.json",
+    ".": {
+      "import": {
+        "@triggerdotdev/source": "./src/index.ts",
+        "types": "./dist/esm/index.d.ts",
+        "default": "./dist/esm/index.js"
+      },
+      "require": {
+        "types": "./dist/commonjs/index.d.ts",
+        "default": "./dist/commonjs/index.js"
+      }
+    },
+    "./extension": {
+      "import": {
+        "@triggerdotdev/source": "./src/extension.ts",
+        "types": "./dist/esm/extension.d.ts",
+        "default": "./dist/esm/extension.js"
+      },
+      "require": {
+        "types": "./dist/commonjs/extension.d.ts",
+        "default": "./dist/commonjs/extension.js"
+      }
+    }
+  },
+  "main": "./dist/commonjs/index.js",
+  "types": "./dist/commonjs/index.d.ts",
+  "module": "./dist/esm/index.js"
+}

packages/python/src/extension.ts

@@ -0,0 +1,119 @@
+import fs from "node:fs";
+import assert from "node:assert";
+import { additionalFiles } from "@trigger.dev/build/extensions/core";
+import { BuildManifest } from "@trigger.dev/core/v3";
+import { BuildContext, BuildExtension } from "@trigger.dev/core/v3/build";
+
+export type PythonOptions = {
+  requirements?: string[];
+  requirementsFile?: string;
+  /**
+   * [Dev-only] The path to the python binary.
+   *
+   * @remarks
+   * This option is typically used during local development or in specific testing environments
+   * where a particular Python installation needs to be targeted.  It should point to the full path of the python executable.
+   *
+   * Example: `/usr/bin/python3` or `C:\\Python39\\python.exe`
+   */
+  pythonBinaryPath?: string;
+  /**
+   * An array of glob patterns that specify which Python scripts are allowed to be executed.
+   *
+   * @remarks
+   * These scripts will be copied to the container during the build process.
+   */
+  scripts?: string[];
+};
+
+const splitAndCleanComments = (str: string) =>
+  str
+    .split("\n")
+    .map((line) => line.trim())
+    .filter((line) => line && !line.startsWith("#"));
+
+export function pythonExtension(options: PythonOptions = {}): BuildExtension {
+  return new PythonExtension(options);
+}
+
+class PythonExtension implements BuildExtension {
+  public readonly name = "PythonExtension";
+
+  constructor(private options: PythonOptions = {}) {
+    assert(
+      !(this.options.requirements && this.options.requirementsFile),
+      "Cannot specify both requirements and requirementsFile"
+    );
+
+    if (this.options.requirementsFile) {
+      assert(
+        fs.existsSync(this.options.requirementsFile),
+        `Requirements file not found: ${this.options.requirementsFile}`
+      );
+      this.options.requirements = splitAndCleanComments(
+        fs.readFileSync(this.options.requirementsFile, "utf-8")
+      );
+    }
+  }
+
+  async onBuildComplete(context: BuildContext, manifest: BuildManifest) {
+    await additionalFiles({
+      files: this.options.scripts ?? [],
+    }).onBuildComplete!(context, manifest);
+
+    if (context.target === "dev") {
+      if (this.options.pythonBinaryPath) {
+        process.env.PYTHON_BIN_PATH = this.options.pythonBinaryPath;
+      }
+
+      return;
+    }
+
+    context.logger.debug(`Adding ${this.name} to the build`);
+
+    context.addLayer({
+      id: "python-installation",
+      image: {
+        instructions: splitAndCleanComments(`
+          # Install Python
+          RUN apt-get update && apt-get install -y --no-install-recommends \
+              python3 python3-pip python3-venv && \
+              apt-get clean && rm -rf /var/lib/apt/lists/*
+
+          # Set up Python environment
+          RUN python3 -m venv /opt/venv
+          ENV PATH="/opt/venv/bin:$PATH"
+        `),
+      },
+      deploy: {
+        env: {
+          PYTHON_BIN_PATH: `/opt/venv/bin/python`,
+        },
+        override: true,
+      },
+    });
+
+    context.addLayer({
+      id: "python-dependencies",
+      build: {
+        env: {
+          REQUIREMENTS_CONTENT: this.options.requirements?.join("\n") || "",
+        },
+      },
+      image: {
+        instructions: splitAndCleanComments(`
+          ARG REQUIREMENTS_CONTENT
+          RUN echo "$REQUIREMENTS_CONTENT" > requirements.txt
+
+          # Install dependencies
+          RUN pip install --no-cache-dir -r requirements.txt
+        `),
+      },
+      deploy: {
+        override: true,
+      },
+    });
+  }
+}
+
+export default pythonExtension;