examples/operator-postgres-tls-demo/02-Install-feast.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Install Feast on Kubernetes with the Feast Operator\n",
    "## Objective\n",
    "\n",
    "Provide a reference implementation of a runbook to deploy a Feast environment on a Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/docs/user/quick-start) and the [Feast Operator](../../infra/feast-operator/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "* Kubernetes Cluster\n",
    "* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) Kubernetes CLI tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the Feast Operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/feast-operator-system created\n",
      "customresourcedefinition.apiextensions.k8s.io/featurestores.feast.dev created\n",
      "serviceaccount/feast-operator-controller-manager created\n",
      "role.rbac.authorization.k8s.io/feast-operator-leader-election-role created\n",
      "clusterrole.rbac.authorization.k8s.io/feast-operator-featurestore-editor-role created\n",
      "clusterrole.rbac.authorization.k8s.io/feast-operator-featurestore-viewer-role created\n",
      "clusterrole.rbac.authorization.k8s.io/feast-operator-manager-role created\n",
      "clusterrole.rbac.authorization.k8s.io/feast-operator-metrics-auth-role created\n",
      "clusterrole.rbac.authorization.k8s.io/feast-operator-metrics-reader created\n",
      "rolebinding.rbac.authorization.k8s.io/feast-operator-leader-election-rolebinding created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/feast-operator-manager-rolebinding created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/feast-operator-metrics-auth-rolebinding created\n",
      "service/feast-operator-controller-manager-metrics-service created\n",
      "deployment.apps/feast-operator-controller-manager created\n",
      "deployment.apps/feast-operator-controller-manager condition met\n"
     ]
    }
   ],
   "source": [
    "## Use this install command from a release branch (e.g. 'v0.46-branch')\n",
    "!kubectl apply -f ../../infra/feast-operator/dist/install.yaml\n",
    "\n",
    "## OR, for the latest code/builds, use one the following commands from the 'master' branch\n",
    "# !make -C ../../infra/feast-operator install deploy IMG=quay.io/feastdev-ci/feast-operator:develop FS_IMG=quay.io/feastdev-ci/feature-server:develop\n",
    "# !make -C ../../infra/feast-operator install deploy IMG=quay.io/feastdev-ci/feast-operator:$(git rev-parse HEAD) FS_IMG=quay.io/feastdev-ci/feature-server:$(git rev-parse HEAD)\n",
    "\n",
    "!kubectl wait --for=condition=available --timeout=5m deployment/feast-operator-controller-manager -n feast-operator-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the Feast services via FeatureStore CR\n",
    "Next, we'll use the running Feast Operator to install the feast services. Before doing that it is important to understand basic understanding of operator support of Volumes and volumeMounts and how to mount TLS certificates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mounting TLS Certificates with Volumes in Feast Operator  \n",
    "\n",
    "The Feast operator supports **volumes** and **volumeMounts**, allowing you to mount TLS certificates onto a pod. This approach provides flexibility in how you mount these files, supporting different Kubernetes resources such as **Secrets, ConfigMaps,** and **Persistent Volumes (PVs).**  \n",
    "\n",
    "#### Example: Mounting Certificates Using Kubernetes Secrets  \n",
    "\n",
    "In this example, we demonstrate how to mount TLS certificates using **Kubernetes Secrets** that were created in a previous notebook.  \n",
    "\n",
    "#### PostgreSQL Connection Parameters  \n",
    "\n",
    "When connecting to PostgreSQL with TLS, some important parameters in the connection URL are:  \n",
    "\n",
    "- **`sslrootcert`** – Specifies the path to the **CA certificate** file used to validate trusted certificates.  \n",
    "- **`sslcert`** – Provides the client certificate for **mutual TLS (mTLS) encryption**.  \n",
    "- **`sslkey`** – Specifies the private key for the client certificate.  \n",
    "\n",
    "If mutual TLS authentication is not required, you can **omit** the `sslcert` and `sslkey` parameters. However, the `sslrootcert` parameter is still necessary for validating server certificates.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b><font color=red> Note: Please deploy either option 1 or 2 only. Don't deploy both of them at the same time to avoid conflicts in the lateral steps. </font></b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 1: Directly Setting the CA Certificate Path**  \n",
    "\n",
    "In this approach, we specify the CA certificate path directly in the Feast PostgreSQL URL using the `sslrootcert` parameter.  \n",
    "\n",
    "You can refer to the `v1alpha1_featurestore_postgres_db_volumes_tls.yaml` file for the complete configuration details.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "secret/postgres-secret created\n",
      "secret/feast-data-stores created\n",
      "featurestore.feast.dev/sample-db-ssl created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ../../infra/feast-operator/config/samples/v1alpha1_featurestore_postgres_db_volumes_tls.yaml --namespace=feast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Option 2: Using an Environment Variable for the CA Certificate**  \n",
    "\n",
    "In this approach, you define the CA certificate path as an environment variable. You can refer to the `v1alpha1_featurestore_postgres_tls_volumes_ca_env.yaml` file for the complete configuration details.   \n",
    "\n",
    "```bash\n",
    "FEAST_CA_CERT_FILE_PATH=<path-to-ca-cert>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "secret/postgres-secret created\n",
      "secret/feast-data-stores created\n",
      "featurestore.feast.dev/sample-db-ssl created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ../../infra/feast-operator/config/samples/v1alpha1_featurestore_postgres_tls_volumes_ca_env.yaml --namespace=feast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate the running FeatureStore deployment\n",
    "Validate the deployment status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment.apps/feast-sample-db-ssl condition met\n",
      "NAME                                     READY   STATUS    RESTARTS   AGE\n",
      "pod/feast-sample-db-ssl-86b47d54-hclb9   1/1     Running   0          27s\n",
      "pod/postgresql-0                         1/1     Running   0          13h\n",
      "\n",
      "NAME                                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE\n",
      "service/feast-sample-db-ssl-online   ClusterIP   10.96.61.65   <none>        80/TCP     27s\n",
      "service/postgresql                   ClusterIP   10.96.228.3   <none>        5432/TCP   13h\n",
      "service/postgresql-hl                ClusterIP   None          <none>        5432/TCP   13h\n",
      "\n",
      "NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE\n",
      "deployment.apps/feast-sample-db-ssl   1/1     1            1           27s\n",
      "\n",
      "NAME                                           DESIRED   CURRENT   READY   AGE\n",
      "replicaset.apps/feast-sample-db-ssl-86b47d54   1         1         1       27s\n",
      "\n",
      "NAME                          READY   AGE\n",
      "statefulset.apps/postgresql   1/1     13h\n"
     ]
    }
   ],
   "source": [
    "!kubectl wait --for=condition=available --timeout=8m deployment/feast-sample-db-ssl -n feast\n",
    "!kubectl get all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validate that the FeatureStore CR is in a `Ready` state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME            STATUS   AGE\n",
      "sample-db-ssl   Ready    33s\n"
     ]
    }
   ],
   "source": [
    "!kubectl get feast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that the DB includes the expected tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulted container \"postgresql\" out of: postgresql, init-chmod-data (init)\n",
      "                               List of relations\n",
      " Schema |                         Name                         | Type  | Owner \n",
      "--------+------------------------------------------------------+-------+-------\n",
      " public | data_sources                                         | table | admin\n",
      " public | entities                                             | table | admin\n",
      " public | feast_metadata                                       | table | admin\n",
      " public | feature_services                                     | table | admin\n",
      " public | feature_views                                        | table | admin\n",
      " public | managed_infra                                        | table | admin\n",
      " public | on_demand_feature_views                              | table | admin\n",
      " public | permissions                                          | table | admin\n",
      " public | postgres_tls_sample_env_ca_driver_hourly_stats       | table | admin\n",
      " public | postgres_tls_sample_env_ca_driver_hourly_stats_fresh | table | admin\n",
      " public | projects                                             | table | admin\n",
      " public | saved_datasets                                       | table | admin\n",
      " public | stream_feature_views                                 | table | admin\n",
      " public | validation_references                                | table | admin\n",
      "(14 rows)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!kubectl exec postgresql-0 -- env PGPASSWORD=password psql -U admin -d feast -c '\\dt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify the client `feature_store.yaml` and create the sample feature store definitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "project: postgres_tls_sample_env_ca\n",
      "provider: local\n",
      "offline_store:\n",
      "    host: ${POSTGRES_HOST}\n",
      "    type: postgres\n",
      "    port: 5432\n",
      "    database: ${POSTGRES_DB}\n",
      "    db_schema: public\n",
      "    password: ${POSTGRES_PASSWORD}\n",
      "    sslcert_path: /var/lib/postgresql/certs/tls.crt\n",
      "    sslkey_path: /var/lib/postgresql/certs/tls.key\n",
      "    sslmode: verify-full\n",
      "    sslrootcert_path: system\n",
      "    user: ${POSTGRES_USER}\n",
      "online_store:\n",
      "    type: postgres\n",
      "    database: ${POSTGRES_DB}\n",
      "    db_schema: public\n",
      "    host: ${POSTGRES_HOST}\n",
      "    password: ${POSTGRES_PASSWORD}\n",
      "    port: 5432\n",
      "    sslcert_path: /var/lib/postgresql/certs/tls.crt\n",
      "    sslkey_path: /var/lib/postgresql/certs/tls.key\n",
      "    sslmode: verify-full\n",
      "    sslrootcert_path: system\n",
      "    user: ${POSTGRES_USER}\n",
      "registry:\n",
      "    path: postgresql+psycopg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/${POSTGRES_DB}?sslmode=verify-full&sslrootcert=system&sslcert=/var/lib/postgresql/certs/tls.crt&sslkey=/var/lib/postgresql/certs/tls.key\n",
      "    registry_type: sql\n",
      "    cache_ttl_seconds: 60\n",
      "    sqlalchemy_config_kwargs:\n",
      "        echo: false\n",
      "        pool_pre_ping: true\n",
      "auth:\n",
      "    type: no_auth\n",
      "entity_key_serialization_version: 3\n",
      "<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)\n",
      "<jemalloc>: (This is the expected behaviour if you are running under QEMU)\n",
      "/opt/app-root/src/sdk/python/feast/feature_view.py:48: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  DUMMY_ENTITY = Entity(\n",
      "/feast-data/postgres_tls_sample_env_ca/feature_repo/example_repo.py:27: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity 'driver'.\n",
      "  driver = Entity(name=\"driver\", join_keys=[\"driver_id\"])\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity 'driver'.\n",
      "  entity = cls(\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  entity = cls(\n",
      "Applying changes for project postgres_tls_sample_env_ca\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity 'driver'.\n",
      "  entity = cls(\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  entity = cls(\n",
      "/opt/app-root/src/sdk/python/feast/feature_store.py:579: RuntimeWarning: On demand feature view is an experimental feature. This API is stable, but the functionality does not scale well for offline retrieval\n",
      "  warnings.warn(\n",
      "Deploying infrastructure for driver_hourly_stats\n",
      "Deploying infrastructure for driver_hourly_stats_fresh\n",
      " Feast apply is completed. You can go to next step.\n"
     ]
    }
   ],
   "source": [
    "!kubectl exec deploy/feast-sample-db-ssl -c online -- cat feature_store.yaml\n",
    "!kubectl exec deploy/feast-sample-db-ssl -c online -- feast apply\n",
    "print(\" Feast apply is completed. You can go to next step.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List the registered feast projects & feature views."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)\n",
      "<jemalloc>: (This is the expected behaviour if you are running under QEMU)\n",
      "/opt/app-root/src/sdk/python/feast/feature_view.py:48: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  DUMMY_ENTITY = Entity(\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity 'driver'.\n",
      "  entity = cls(\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  entity = cls(\n",
      "NAME                        DESCRIPTION                      TAGS    OWNER\n",
      "postgres_tls_sample                                          {}\n",
      "postgres_tls_sample_env_ca  A project for driver statistics  {}\n",
      "<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)\n",
      "<jemalloc>: (This is the expected behaviour if you are running under QEMU)\n",
      "/opt/app-root/src/sdk/python/feast/feature_view.py:48: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  DUMMY_ENTITY = Entity(\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity 'driver'.\n",
      "  entity = cls(\n",
      "/opt/app-root/src/sdk/python/feast/entity.py:173: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  entity = cls(\n",
      "NAME                         ENTITIES    TYPE\n",
      "driver_hourly_stats_fresh    {'driver'}  FeatureView\n",
      "driver_hourly_stats          {'driver'}  FeatureView\n",
      "transformed_conv_rate        {'driver'}  OnDemandFeatureView\n",
      "transformed_conv_rate_fresh  {'driver'}  OnDemandFeatureView\n"
     ]
    }
   ],
   "source": [
    "!kubectl exec deploy/feast-sample-db-ssl -c online -- feast projects list\n",
    "!kubectl exec deploy/feast-sample-db-ssl -c online -- feast feature-views list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's verify the feast version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)\n",
      "<jemalloc>: (This is the expected behaviour if you are running under QEMU)\n",
      "/opt/app-root/src/sdk/python/feast/feature_view.py:48: DeprecationWarning: Entity value_type will be mandatory in the next release. Please specify a value_type for entity '__dummy'.\n",
      "  DUMMY_ENTITY = Entity(\n",
      "Feast SDK Version: \"0.1.dev1+g6c92447.d20250213\"\n"
     ]
    }
   ],
   "source": [
    "!kubectl exec deployment/feast-sample-db-ssl -c online -- feast version"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}