demo-notebooks/additional-demos/hf_interactive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3260669-c7ec-4d06-a655-590c5e7ab152",
   "metadata": {},
   "source": [
    "# Transfer learning with Huggingface using CodeFlare"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4acfb10-1aa1-445d-947e-396ea5ebed1a",
   "metadata": {},
   "source": [
    "In this notebook you will learn how to leverage the **[huggingface](https://huggingface.co/)** support in ray ecosystem to carry out a text classification task using transfer learning. We will be referencing the example **[here](https://huggingface.co/docs/transformers/tasks/sequence_classification)**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70b77929-e96c-434e-ada3-8b14795bfbb1",
   "metadata": {},
   "source": [
    "The example carries out a text classification task on **[imdb dataset](https://huggingface.co/datasets/imdb)** and tries to classify the movie reviews as positive or negative. Huggingface library provides an easy way to build a model and the dataset to carry out this classification task. In this case we will be using **distilbert-base-uncased** model which is a **BERT** based model.\n",
    "\n",
    "Huggingface has a **[built in support for ray ecosystem](https://docs.ray.io/en/releases-1.13.0/_modules/ray/ml/train/integrations/huggingface/huggingface_trainer.html)** which allows the huggingface trainer to scale on CodeFlare and can scale the training as we add additional gpus and can run distributed training across multiple GPUs that will help scale out the training.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02593d04-40b9-4a07-a32e-40b649444ab5",
   "metadata": {},
   "source": [
    "### Getting all the requirements in place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c737a768-6e31-4767-a301-60ae932b4ed9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pieces from codeflare-sdk\n",
    "from codeflare_sdk import Cluster, ClusterConfiguration, TokenAuthentication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0734734b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create authentication object for user permissions\n",
    "# IF unused, SDK will automatically check for default kubeconfig, then in-cluster config\n",
    "# KubeConfigFileAuthentication can also be used to specify kubeconfig path manually\n",
    "auth = TokenAuthentication(\n",
    "    token = \"XXXX\",\n",
    "    server = \"XXXX\",\n",
    "    skip_tls = False\n",
    ")\n",
    "auth.login()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc27f84c",
   "metadata": {},
   "source": [
    "Here, we want to define our cluster by specifying the resources we require for our batch workload. Below, we define our cluster object (which generates a corresponding Ray Cluster).\n",
    "\n",
    "NOTE: We must specify the `image` which will be used in our RayCluster, we recommend you bring your own image which suits your purposes. \n",
    "The example here is a community image.\n",
    "\n",
    "NOTE: By default the SDK uses Kueue as it's scheduling solution to use MCAD set the `mcad=True` option in `ClusterConfiguration`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "220b9d85-3a3c-4c0c-aaf2-0d866823dcd8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Written to: hfgputest.yaml\n"
     ]
    }
   ],
   "source": [
    "# Create our cluster and submit\n",
    "# The SDK will try to find the name of your default local queue based on the annotation \"kueue.x-k8s.io/default-queue\": \"true\"\n",
    "cluster = Cluster(ClusterConfiguration(name='hfgputest', \n",
    "                                       namespace=\"default\",\n",
    "                                       num_workers=1,\n",
    "                                       min_cpus=8, \n",
    "                                       max_cpus=8, \n",
    "                                       min_memory=16, \n",
    "                                       max_memory=16, \n",
    "                                       num_gpus=4,\n",
    "                                       image=\"quay.io/project-codeflare/ray:latest-py39-cu118\",\n",
    "                                       # local_queue=\"local-queue-name\" # Specify the local queue manually\n",
    "                                       ))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12eef53c",
   "metadata": {},
   "source": [
    "Next, we want to bring our cluster up, so we call the `up()` function below to submit our Ray Cluster onto the queue, and begin the process of obtaining our resource cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ae1d861-b743-4c05-903b-5799072b942b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.up()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "657ebdfb",
   "metadata": {},
   "source": [
    "Now, we want to check on the initial status of our resource cluster, then wait until it is finally ready for use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4d0db5f5-22f1-4806-ae7e-a0ee865625c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">╭─────────────────────────╮\n",
       "│ <span style=\"font-style: italic\"> </span><span style=\"font-weight: bold; font-style: italic\"> 🚀 List of CodeFlare</span><span style=\"font-style: italic\"> </span> │\n",
       "│ <span style=\"font-style: italic\">  </span><span style=\"font-weight: bold; font-style: italic\">clusters in queue🚀</span><span style=\"font-style: italic\">  </span> │\n",
       "│ +-----------+---------+ │\n",
       "│ |<span style=\"font-weight: bold\"> Name      </span>|<span style=\"font-weight: bold\"> Status  </span>| │\n",
       "│ +===========+=========+ │\n",
       "│ |<span style=\"color: #008080; text-decoration-color: #008080\"> hfgputest </span>|<span style=\"color: #800080; text-decoration-color: #800080\"> pending </span>| │\n",
       "│ |<span style=\"color: #008080; text-decoration-color: #008080\">           </span>|<span style=\"color: #800080; text-decoration-color: #800080\">         </span>| │\n",
       "│ +-----------+---------+ │\n",
       "╰─────────────────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "╭─────────────────────────╮\n",
       "│ \u001b[3m \u001b[0m\u001b[1;3m 🚀 List of CodeFlare\u001b[0m\u001b[3m \u001b[0m │\n",
       "│ \u001b[3m  \u001b[0m\u001b[1;3mclusters in queue🚀\u001b[0m\u001b[3m  \u001b[0m │\n",
       "│ +-----------+---------+ │\n",
       "│ |\u001b[1m \u001b[0m\u001b[1mName     \u001b[0m\u001b[1m \u001b[0m|\u001b[1m \u001b[0m\u001b[1mStatus \u001b[0m\u001b[1m \u001b[0m| │\n",
       "│ +===========+=========+ │\n",
       "│ |\u001b[36m \u001b[0m\u001b[36mhfgputest\u001b[0m\u001b[36m \u001b[0m|\u001b[35m \u001b[0m\u001b[35mpending\u001b[0m\u001b[35m \u001b[0m| │\n",
       "│ |\u001b[36m \u001b[0m\u001b[36m         \u001b[0m\u001b[36m \u001b[0m|\u001b[35m \u001b[0m\u001b[35m       \u001b[0m\u001b[35m \u001b[0m| │\n",
       "│ +-----------+---------+ │\n",
       "╰─────────────────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(False, <CodeFlareClusterStatus.QUEUED: 2>)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster.status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d26275e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.wait_ready()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2969a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "477ac246",
   "metadata": {},
   "source": [
    "Let's quickly verify that the specs of the cluster are as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "06a54428-f186-4c27-948e-4eaf9c0e34b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                 </span><span style=\"font-weight: bold; font-style: italic\"> 🚀 List of CodeFlare clusters 🚀</span><span style=\"font-style: italic\">                  </span>\n",
       "<span style=\"font-weight: bold\">                                                                    </span>\n",
       " ╭────────────────────────────────────────────────────────────────╮ \n",
       " │   <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; background-color: #008000; font-weight: bold\">Owner</span>                                                        │ \n",
       " │   <span style=\"font-weight: bold; text-decoration: underline\">hfgputest</span>                                        Active ✅   │ \n",
       " │                                                                │ \n",
       " │   <span style=\"font-weight: bold\">URI:</span> ray://hfgputest-head-svc.default.svc:10001              │ \n",
       " │                                                                │ \n",
       " │   <a href=\"ray-dashboard-hfgputest-default.apps.prepfullinstall.psap.aws.rhperfscale.org\" target=\"_blank\"><span style=\"color: #000080; text-decoration-color: #000080; text-decoration: underline\">Dashboard🔗</span></a>                                                  │ \n",
       " │                                                                │ \n",
       " │  <span style=\"font-style: italic\">                    Cluster Resources                     </span>    │ \n",
       " │   ╭─ Workers ──╮  ╭───────── Worker specs(each) ─────────╮     │ \n",
       " │   │ <span style=\"font-weight: bold\"> Min  Max </span> │  │ <span style=\"font-weight: bold\"> Memory      CPU         GPU        </span> │     │ \n",
       " │   │ <span style=\"color: #008080; text-decoration-color: #008080\">     </span><span style=\"color: #800080; text-decoration-color: #800080\">     </span> │  │ <span style=\"color: #008080; text-decoration-color: #008080\">            </span><span style=\"color: #800080; text-decoration-color: #800080\">                        </span> │     │ \n",
       " │   │ <span style=\"color: #008080; text-decoration-color: #008080\"> 1   </span><span style=\"color: #800080; text-decoration-color: #800080\"> 1   </span> │  │ <span style=\"color: #008080; text-decoration-color: #008080\"> 16G~16G    </span><span style=\"color: #800080; text-decoration-color: #800080\"> 8           4          </span> │     │ \n",
       " │   │ <span style=\"color: #008080; text-decoration-color: #008080\">     </span><span style=\"color: #800080; text-decoration-color: #800080\">     </span> │  │ <span style=\"color: #008080; text-decoration-color: #008080\">            </span><span style=\"color: #800080; text-decoration-color: #800080\">                        </span> │     │ \n",
       " │   ╰────────────╯  ╰──────────────────────────────────────╯     │ \n",
       " ╰────────────────────────────────────────────────────────────────╯ \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                 \u001b[0m\u001b[1;3m 🚀 List of CodeFlare clusters 🚀\u001b[0m\u001b[3m                  \u001b[0m\n",
       "\u001b[1m \u001b[0m\u001b[1m                                                                  \u001b[0m\u001b[1m \u001b[0m\n",
       " ╭────────────────────────────────────────────────────────────────╮ \n",
       " │   \u001b[1;37;42mOwner\u001b[0m                                                        │ \n",
       " │   \u001b[1;4mhfgputest\u001b[0m                                        Active ✅   │ \n",
       " │                                                                │ \n",
       " │   \u001b[1mURI:\u001b[0m ray://hfgputest-head-svc.default.svc:10001              │ \n",
       " │                                                                │ \n",
       " │   \u001b]8;id=552692;ray-dashboard-hfgputest-default.apps.prepfullinstall.psap.aws.rhperfscale.org\u001b\\\u001b[4;34mDashboard🔗\u001b[0m\u001b]8;;\u001b\\                                                  │ \n",
       " │                                                                │ \n",
       " │  \u001b[3m                    Cluster Resources                     \u001b[0m    │ \n",
       " │   ╭─ Workers ──╮  ╭───────── Worker specs(each) ─────────╮     │ \n",
       " │   │ \u001b[1m \u001b[0m\u001b[1mMin\u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mMax\u001b[0m\u001b[1m \u001b[0m │  │ \u001b[1m \u001b[0m\u001b[1mMemory    \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mCPU       \u001b[0m\u001b[1m \u001b[0m\u001b[1m \u001b[0m\u001b[1mGPU       \u001b[0m\u001b[1m \u001b[0m │     │ \n",
       " │   │ \u001b[36m \u001b[0m\u001b[36m   \u001b[0m\u001b[36m \u001b[0m\u001b[35m \u001b[0m\u001b[35m   \u001b[0m\u001b[35m \u001b[0m │  │ \u001b[36m \u001b[0m\u001b[36m          \u001b[0m\u001b[36m \u001b[0m\u001b[35m \u001b[0m\u001b[35m          \u001b[0m\u001b[35m \u001b[0m\u001b[35m \u001b[0m\u001b[35m          \u001b[0m\u001b[35m \u001b[0m │     │ \n",
       " │   │ \u001b[36m \u001b[0m\u001b[36m1  \u001b[0m\u001b[36m \u001b[0m\u001b[35m \u001b[0m\u001b[35m1  \u001b[0m\u001b[35m \u001b[0m │  │ \u001b[36m \u001b[0m\u001b[36m16G~16G   \u001b[0m\u001b[36m \u001b[0m\u001b[35m \u001b[0m\u001b[35m8         \u001b[0m\u001b[35m \u001b[0m\u001b[35m \u001b[0m\u001b[35m4         \u001b[0m\u001b[35m \u001b[0m │     │ \n",
       " │   │ \u001b[36m \u001b[0m\u001b[36m   \u001b[0m\u001b[36m \u001b[0m\u001b[35m \u001b[0m\u001b[35m   \u001b[0m\u001b[35m \u001b[0m │  │ \u001b[36m \u001b[0m\u001b[36m          \u001b[0m\u001b[36m \u001b[0m\u001b[35m \u001b[0m\u001b[35m          \u001b[0m\u001b[35m \u001b[0m\u001b[35m \u001b[0m\u001b[35m          \u001b[0m\u001b[35m \u001b[0m │     │ \n",
       " │   ╰────────────╯  ╰──────────────────────────────────────╯     │ \n",
       " ╰────────────────────────────────────────────────────────────────╯ \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<RayClusterStatus.READY: 'ready'>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster.details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8ac46c87-70f1-4c70-9648-881151665355",
   "metadata": {},
   "outputs": [],
   "source": [
    "ray_cluster_uri = cluster.cluster_uri()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44dba6a0-8275-4726-8911-6b6ec467b6a3",
   "metadata": {},
   "source": [
    "**NOTE**: Now we have our resource cluster with the desired GPUs, so we can interact with it to train the HuggingFace model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4c458589-5a17-47c6-a8db-625427ae4fe7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ray cluster is up and running:  True\n"
     ]
    }
   ],
   "source": [
    "#before proceeding make sure the cluster exists and the uri is not empty\n",
    "assert ray_cluster_uri, \"Ray cluster needs to be started and set before proceeding\"\n",
    "\n",
    "import ray\n",
    "from ray.air.config import ScalingConfig\n",
    "\n",
    "# reset the ray context in case there's already one. \n",
    "ray.shutdown()\n",
    "# establish connection to ray cluster\n",
    "\n",
    "#install additional libraries that will be required for this training\n",
    "runtime_env = {\"pip\": [\"transformers\", \"datasets\", \"evaluate\", \"pyarrow<7.0.0\", \"accelerate\"]}\n",
    "\n",
    "# NOTE: This will work for in-cluster notebook servers (RHODS/ODH), but not for local machines\n",
    "# To see how to connect from your laptop, go to demo-notebooks/additional-demos/local_interactive.ipynb\n",
    "ray.init(address=ray_cluster_uri, runtime_env=runtime_env)\n",
    "\n",
    "print(\"Ray cluster is up and running: \", ray.is_initialized())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a38146-1321-4b7b-9152-9ebca4eb9444",
   "metadata": {},
   "source": [
    "**NOTE** : in this case since we are running a task for which we need additional pip packages. we can install those by passing them in the `runtime_env` variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76a1945b-d6c8-49b8-9a4c-b82724cffba9",
   "metadata": {},
   "source": [
    "### Transfer learning code from huggingface"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bdbe888-4f38-4e9a-ae43-67ce89ff9d42",
   "metadata": {},
   "source": [
    "We are using the code based on the example **[here](https://huggingface.co/docs/transformers/tasks/sequence_classification)** . "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e69994b4-1a13-43fe-b698-2a5374cb941b",
   "metadata": {},
   "outputs": [],
   "source": [
    "@ray.remote\n",
    "def train_fn():\n",
    "    from datasets import load_dataset\n",
    "    import transformers\n",
    "    from transformers import AutoTokenizer, TrainingArguments\n",
    "    from transformers import AutoModelForSequenceClassification\n",
    "    import numpy as np\n",
    "    from datasets import load_metric\n",
    "    import ray\n",
    "    from ray import tune\n",
    "    from ray.train.huggingface import HuggingFaceTrainer\n",
    "\n",
    "    dataset = load_dataset(\"imdb\")\n",
    "    tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n",
    "\n",
    "    def tokenize_function(examples):\n",
    "        return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n",
    "\n",
    "    tokenized_datasets = dataset.map(tokenize_function, batched=True)\n",
    "\n",
    "    #using a fraction of dataset but you can run with the full dataset\n",
    "    small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(100))\n",
    "    small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(100))\n",
    "\n",
    "    print(f\"len of train {small_train_dataset} and test {small_eval_dataset}\")\n",
    "\n",
    "    ray_train_ds = ray.data.from_huggingface(small_train_dataset)\n",
    "    ray_evaluation_ds = ray.data.from_huggingface(small_eval_dataset)\n",
    "\n",
    "    def compute_metrics(eval_pred):\n",
    "        metric = load_metric(\"accuracy\")\n",
    "        logits, labels = eval_pred\n",
    "        predictions = np.argmax(logits, axis=-1)\n",
    "        return metric.compute(predictions=predictions, references=labels)\n",
    "\n",
    "    def trainer_init_per_worker(train_dataset, eval_dataset, **config):\n",
    "        model = AutoModelForSequenceClassification.from_pretrained(\"distilbert-base-uncased\", num_labels=2)\n",
    "\n",
    "        training_args = TrainingArguments(\"/tmp/hf_imdb/test\", eval_steps=1, disable_tqdm=True, \n",
    "                                          num_train_epochs=1, skip_memory_metrics=True,\n",
    "                                          learning_rate=2e-5,\n",
    "                                          per_device_train_batch_size=16,\n",
    "                                          per_device_eval_batch_size=16,                                \n",
    "                                          weight_decay=0.01,)\n",
    "        return transformers.Trainer(\n",
    "            model=model,\n",
    "            args=training_args,\n",
    "            train_dataset=train_dataset,\n",
    "            eval_dataset=eval_dataset,\n",
    "            compute_metrics=compute_metrics\n",
    "        )\n",
    "\n",
    "    scaling_config = ScalingConfig(num_workers=4, use_gpu=True) #num workers is the number of gpus\n",
    "\n",
    "    # we are using the ray native HuggingFaceTrainer, but you can swap out to use non ray Huggingface Trainer. Both have the same method signature. \n",
    "    # the ray native HFTrainer has built in support for scaling to multiple GPUs\n",
    "    trainer = HuggingFaceTrainer(\n",
    "        trainer_init_per_worker=trainer_init_per_worker,\n",
    "        scaling_config=scaling_config,\n",
    "        datasets={\"train\": ray_train_ds, \"evaluation\": ray_evaluation_ds},\n",
    "    )\n",
    "    result = trainer.fit()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9593fee-2b2b-415f-8902-bceec014385f",
   "metadata": {},
   "source": [
    "**NOTE:** This code will produce a lot of output and will run for **approximately 2 minutes.** As a part of execution it will download the `imdb` dataset, `distilbert-base-uncased` model and then will start transfer learning task for training the model with this dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7f0985e9-5e88-4d36-ab38-c3001c13f97c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading builder script: 100%|██████████| 4.31k/4.31k [00:00<00:00, 5.60MB/s]\n",
      "Downloading metadata: 100%|██████████| 2.17k/2.17k [00:00<00:00, 3.13MB/s]\n",
      "Downloading readme: 100%|██████████| 7.59k/7.59k [00:00<00:00, 9.75MB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Downloading and preparing dataset imdb/plain_text to /home/ray/.cache/huggingface/datasets/imdb/plain_text/1.0.0/2fdd8b9bcadd6e7055e742a706876ba43f19faee861df134affd7a3f60fc38a1...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading data:   0%|          | 0.00/84.1M [00:00<?, ?B/s]\n",
      "Downloading data:   0%|          | 30.7k/84.1M [00:00<05:22, 261kB/s]\n",
      "Downloading data:   0%|          | 89.1k/84.1M [00:00<03:31, 397kB/s]\n",
      "Downloading data:   0%|          | 184k/84.1M [00:00<02:24, 582kB/s] \n",
      "Downloading data:   0%|          | 373k/84.1M [00:00<01:25, 981kB/s]\n",
      "Downloading data:   1%|          | 778k/84.1M [00:00<00:44, 1.86MB/s]\n",
      "Downloading data:   2%|▏         | 1.34M/84.1M [00:00<00:29, 2.83MB/s]\n",
      "Downloading data:   2%|▏         | 2.02M/84.1M [00:00<00:21, 3.79MB/s]\n",
      "Downloading data:   3%|▎         | 2.86M/84.1M [00:00<00:16, 4.85MB/s]\n",
      "Downloading data:   5%|▍         | 3.98M/84.1M [00:01<00:12, 6.27MB/s]\n",
      "Downloading data:   6%|▋         | 5.39M/84.1M [00:01<00:09, 8.02MB/s]\n",
      "Downloading data:   9%|▉         | 7.69M/84.1M [00:01<00:06, 11.8MB/s]\n",
      "Downloading data:  13%|█▎        | 11.2M/84.1M [00:01<00:04, 17.4MB/s]\n",
      "Downloading data:  18%|█▊        | 15.3M/84.1M [00:01<00:03, 22.5MB/s]\n",
      "Downloading data:  23%|██▎       | 19.7M/84.1M [00:01<00:02, 28.5MB/s]\n",
      "Downloading data:  27%|██▋       | 23.1M/84.1M [00:01<00:02, 29.9MB/s]\n",
      "Downloading data:  31%|███▏      | 26.4M/84.1M [00:01<00:01, 30.7MB/s]\n",
      "Downloading data:  37%|███▋      | 30.7M/84.1M [00:01<00:01, 34.5MB/s]\n",
      "Downloading data:  42%|████▏     | 35.5M/84.1M [00:02<00:01, 38.4MB/s]\n",
      "Downloading data:  47%|████▋     | 39.4M/84.1M [00:02<00:01, 38.6MB/s]\n",
      "Downloading data:  52%|█████▏    | 43.6M/84.1M [00:02<00:01, 39.6MB/s]\n",
      "Downloading data:  58%|█████▊    | 48.7M/84.1M [00:02<00:00, 42.8MB/s]\n",
      "Downloading data:  63%|██████▎   | 53.0M/84.1M [00:02<00:00, 42.7MB/s]\n",
      "Downloading data:  68%|██████▊   | 57.3M/84.1M [00:02<00:00, 42.9MB/s]\n",
      "Downloading data:  74%|███████▎  | 62.0M/84.1M [00:02<00:00, 43.2MB/s]\n",
      "Downloading data:  80%|███████▉  | 67.3M/84.1M [00:02<00:00, 46.0MB/s]\n",
      "Downloading data:  85%|████████▌ | 71.9M/84.1M [00:02<00:00, 45.5MB/s]\n",
      "Downloading data:  91%|█████████ | 76.5M/84.1M [00:02<00:00, 45.3MB/s]\n",
      "Downloading data: 100%|██████████| 84.1M/84.1M [00:03<00:00, 27.2MB/s]\n",
      "Generating train split:   0%|          | 0/25000 [00:00<?, ? examples/s]\n",
      "Generating train split:   0%|          | 1/25000 [00:02<15:40:26,  2.26s/ examples]\n",
      "Generating train split:   3%|▎         | 749/25000 [00:02<00:54, 445.61 examples/s]\n",
      "Generating train split:   6%|▌         | 1514/25000 [00:02<00:23, 1010.13 examples/s]\n",
      "Generating train split:   9%|▉         | 2285/25000 [00:02<00:13, 1684.51 examples/s]\n",
      "Generating train split:  12%|█▏        | 3061/25000 [00:02<00:08, 2447.86 examples/s]\n",
      "Generating train split:  15%|█▌        | 3836/25000 [00:02<00:06, 3254.59 examples/s]\n",
      "Generating train split:  18%|█▊        | 4596/25000 [00:02<00:05, 4030.18 examples/s]\n",
      "Generating train split:  21%|██▏       | 5368/25000 [00:02<00:04, 4780.15 examples/s]\n",
      "Generating train split:  25%|██▍       | 6132/25000 [00:03<00:03, 5423.09 examples/s]\n",
      "Generating train split:  28%|██▊       | 6913/25000 [00:03<00:03, 6002.61 examples/s]\n",
      "Generating train split:  31%|███       | 7683/25000 [00:03<00:02, 6439.96 examples/s]\n",
      "Generating train split:  34%|███▍      | 8448/25000 [00:03<00:02, 6757.88 examples/s]\n",
      "Generating train split:  37%|███▋      | 9219/25000 [00:03<00:02, 7020.62 examples/s]\n",
      "Generating train split:  40%|████      | 10000/25000 [00:03<00:02, 6475.62 examples/s]\n",
      "Generating train split:  43%|████▎     | 10774/25000 [00:03<00:02, 6811.79 examples/s]\n",
      "Generating train split:  46%|████▌     | 11561/25000 [00:03<00:01, 7102.16 examples/s]\n",
      "Generating train split:  49%|████▉     | 12348/25000 [00:03<00:01, 7317.96 examples/s]\n",
      "Generating train split:  53%|█████▎    | 13143/25000 [00:04<00:01, 7498.06 examples/s]\n",
      "Generating train split:  56%|█████▌    | 13919/25000 [00:04<00:01, 7573.16 examples/s]\n",
      "Generating train split:  59%|█████▉    | 14707/25000 [00:04<00:01, 7659.25 examples/s]\n",
      "Generating train split:  62%|██████▏   | 15494/25000 [00:04<00:01, 7721.19 examples/s]\n",
      "Generating train split:  65%|██████▌   | 16273/25000 [00:04<00:01, 7739.71 examples/s]\n",
      "Generating train split:  68%|██████▊   | 17056/25000 [00:04<00:01, 7765.06 examples/s]\n",
      "Generating train split:  71%|███████▏  | 17839/25000 [00:04<00:00, 7783.64 examples/s]\n",
      "Generating train split:  75%|███████▍  | 18628/25000 [00:04<00:00, 7813.80 examples/s]\n",
      "Generating train split:  78%|███████▊  | 19411/25000 [00:04<00:00, 7804.00 examples/s]\n",
      "Generating train split:  81%|████████  | 20193/25000 [00:04<00:00, 7072.48 examples/s]\n",
      "Generating train split:  84%|████████▍ | 20967/25000 [00:05<00:00, 7257.29 examples/s]\n",
      "Generating train split:  87%|████████▋ | 21749/25000 [00:05<00:00, 7416.22 examples/s]\n",
      "Generating train split:  90%|█████████ | 22540/25000 [00:05<00:00, 7556.79 examples/s]\n",
      "Generating train split:  93%|█████████▎| 23327/25000 [00:05<00:00, 7646.65 examples/s]\n",
      "Generating train split:  96%|█████████▋| 24108/25000 [00:05<00:00, 7694.25 examples/s]\n",
      "Generating train split: 100%|█████████▉| 24881/25000 [00:05<00:00, 7674.07 examples/s]\n",
      "                                                                                      \n",
      "Generating test split:   0%|          | 0/25000 [00:00<?, ? examples/s]\n",
      "Generating test split:   0%|          | 1/25000 [00:00<3:10:01,  2.19 examples/s]\n",
      "Generating test split:   3%|▎         | 766/25000 [00:00<00:13, 1826.18 examples/s]\n",
      "Generating test split:   6%|▌         | 1544/25000 [00:00<00:07, 3338.57 examples/s]\n",
      "Generating test split:   9%|▉         | 2332/25000 [00:00<00:04, 4546.72 examples/s]\n",
      "Generating test split:  12%|█▏        | 3116/25000 [00:00<00:04, 5453.13 examples/s]\n",
      "Generating test split:  16%|█▌        | 3905/25000 [00:00<00:03, 6140.31 examples/s]\n",
      "Generating test split:  19%|█▉        | 4688/25000 [00:01<00:03, 6624.84 examples/s]\n",
      "Generating test split:  22%|██▏       | 5474/25000 [00:01<00:02, 6982.92 examples/s]\n",
      "Generating test split:  25%|██▌       | 6257/25000 [00:01<00:02, 7230.53 examples/s]\n",
      "Generating test split:  28%|██▊       | 7046/25000 [00:01<00:02, 7423.93 examples/s]\n",
      "Generating test split:  31%|███▏      | 7829/25000 [00:01<00:02, 7543.45 examples/s]\n",
      "Generating test split:  34%|███▍      | 8614/25000 [00:01<00:02, 7633.64 examples/s]\n",
      "Generating test split:  38%|███▊      | 9394/25000 [00:01<00:02, 7680.82 examples/s]\n",
      "Generating test split:  41%|████      | 10174/25000 [00:01<00:02, 7005.77 examples/s]\n",
      "Generating test split:  44%|████▍     | 10949/25000 [00:01<00:01, 7213.25 examples/s]\n",
      "Generating test split:  47%|████▋     | 11730/25000 [00:01<00:01, 7381.64 examples/s]\n",
      "Generating test split:  50%|█████     | 12515/25000 [00:02<00:01, 7516.21 examples/s]\n",
      "Generating test split:  53%|█████▎    | 13301/25000 [00:02<00:01, 7615.13 examples/s]\n",
      "Generating test split:  56%|█████▋    | 14087/25000 [00:02<00:01, 7684.18 examples/s]\n",
      "Generating test split:  60%|█████▉    | 14876/25000 [00:02<00:01, 7744.99 examples/s]\n",
      "Generating test split:  63%|██████▎   | 15664/25000 [00:02<00:01, 7779.58 examples/s]\n",
      "Generating test split:  66%|██████▌   | 16456/25000 [00:02<00:01, 7817.82 examples/s]\n",
      "Generating test split:  69%|██████▉   | 17240/25000 [00:02<00:00, 7819.40 examples/s]\n",
      "Generating test split:  72%|███████▏  | 18024/25000 [00:02<00:00, 7816.77 examples/s]\n",
      "Generating test split:  75%|███████▌  | 18808/25000 [00:02<00:00, 7823.24 examples/s]\n",
      "Generating test split:  78%|███████▊  | 19593/25000 [00:02<00:00, 7829.38 examples/s]\n",
      "Generating test split:  82%|████████▏ | 20377/25000 [00:03<00:00, 7091.67 examples/s]\n",
      "Generating test split:  85%|████████▍ | 21155/25000 [00:03<00:00, 7283.38 examples/s]\n",
      "Generating test split:  88%|████████▊ | 21937/25000 [00:03<00:00, 7434.19 examples/s]\n",
      "Generating test split:  91%|█████████ | 22724/25000 [00:03<00:00, 7560.18 examples/s]\n",
      "Generating test split:  94%|█████████▍| 23514/25000 [00:03<00:00, 7658.45 examples/s]\n",
      "Generating test split:  97%|█████████▋| 24285/25000 [00:03<00:00, 7624.86 examples/s]\n",
      "Generating unsupervised split:   0%|          | 0/50000 [00:00<?, ? examples/s]      \n",
      "Generating unsupervised split:   0%|          | 1/50000 [00:04<56:42:02,  4.08s/ examples]\n",
      "Generating unsupervised split:   2%|▏         | 771/50000 [00:04<03:08, 260.56 examples/s]\n",
      "Generating unsupervised split:   3%|▎         | 1525/50000 [00:04<01:21, 595.91 examples/s]\n",
      "Generating unsupervised split:   5%|▍         | 2297/50000 [00:04<00:46, 1035.82 examples/s]\n",
      "Generating unsupervised split:   6%|▌         | 3065/50000 [00:04<00:29, 1574.39 examples/s]\n",
      "Generating unsupervised split:   8%|▊         | 3795/50000 [00:04<00:21, 2168.20 examples/s]\n",
      "Generating unsupervised split:   9%|▉         | 4564/50000 [00:04<00:15, 2880.92 examples/s]\n",
      "Generating unsupervised split:  11%|█         | 5351/50000 [00:04<00:12, 3657.19 examples/s]\n",
      "Generating unsupervised split:  12%|█▏        | 6099/50000 [00:04<00:11, 3947.54 examples/s]\n",
      "Generating unsupervised split:  14%|█▎        | 6874/50000 [00:05<00:09, 4672.28 examples/s]\n",
      "Generating unsupervised split:  15%|█▌        | 7653/50000 [00:05<00:07, 5336.59 examples/s]\n",
      "Generating unsupervised split:  17%|█▋        | 8430/50000 [00:05<00:07, 5905.86 examples/s]\n",
      "Generating unsupervised split:  18%|█▊        | 9206/50000 [00:05<00:06, 6366.96 examples/s]\n",
      "Generating unsupervised split:  20%|█▉        | 9971/50000 [00:05<00:05, 6703.01 examples/s]\n",
      "Generating unsupervised split:  21%|██▏       | 10725/50000 [00:05<00:06, 6205.86 examples/s]\n",
      "Generating unsupervised split:  23%|██▎       | 11505/50000 [00:05<00:05, 6617.62 examples/s]\n",
      "Generating unsupervised split:  25%|██▍       | 12291/50000 [00:05<00:05, 6952.09 examples/s]\n",
      "Generating unsupervised split:  26%|██▌       | 13080/50000 [00:05<00:05, 7212.23 examples/s]\n",
      "Generating unsupervised split:  28%|██▊       | 13852/50000 [00:05<00:04, 7356.03 examples/s]\n",
      "Generating unsupervised split:  29%|██▉       | 14628/50000 [00:06<00:04, 7471.67 examples/s]\n",
      "Generating unsupervised split:  31%|███       | 15399/50000 [00:06<00:04, 7539.92 examples/s]\n",
      "Generating unsupervised split:  32%|███▏      | 16181/50000 [00:06<00:04, 7619.82 examples/s]\n",
      "Generating unsupervised split:  34%|███▍      | 16967/50000 [00:06<00:04, 7690.56 examples/s]\n",
      "Generating unsupervised split:  36%|███▌      | 17753/50000 [00:06<00:04, 7738.38 examples/s]\n",
      "Generating unsupervised split:  37%|███▋      | 18531/50000 [00:06<00:04, 7742.28 examples/s]\n",
      "Generating unsupervised split:  39%|███▊      | 19311/50000 [00:06<00:03, 7754.75 examples/s]\n",
      "Generating unsupervised split:  40%|████      | 20089/50000 [00:06<00:04, 7009.65 examples/s]\n",
      "Generating unsupervised split:  42%|████▏     | 20862/50000 [00:06<00:04, 7208.36 examples/s]\n",
      "Generating unsupervised split:  43%|████▎     | 21633/50000 [00:07<00:03, 7348.02 examples/s]\n",
      "Generating unsupervised split:  45%|████▍     | 22408/50000 [00:07<00:03, 7463.78 examples/s]\n",
      "Generating unsupervised split:  46%|████▋     | 23192/50000 [00:07<00:03, 7573.20 examples/s]\n",
      "Generating unsupervised split:  48%|████▊     | 23971/50000 [00:07<00:03, 7636.86 examples/s]\n",
      "Generating unsupervised split:  49%|████▉     | 24739/50000 [00:07<00:03, 7621.64 examples/s]\n",
      "Generating unsupervised split:  51%|█████     | 25517/50000 [00:07<00:03, 7667.30 examples/s]\n",
      "Generating unsupervised split:  53%|█████▎    | 26286/50000 [00:07<00:03, 7661.28 examples/s]\n",
      "Generating unsupervised split:  54%|█████▍    | 27058/50000 [00:07<00:02, 7675.99 examples/s]\n",
      "Generating unsupervised split:  56%|█████▌    | 27848/50000 [00:07<00:02, 7740.61 examples/s]\n",
      "Generating unsupervised split:  57%|█████▋    | 28629/50000 [00:07<00:02, 7756.47 examples/s]\n",
      "Generating unsupervised split:  59%|█████▉    | 29411/50000 [00:08<00:02, 7774.99 examples/s]\n",
      "Generating unsupervised split:  60%|██████    | 30189/50000 [00:08<00:02, 6960.63 examples/s]\n",
      "Generating unsupervised split:  62%|██████▏   | 30922/50000 [00:08<00:02, 7061.87 examples/s]\n",
      "Generating unsupervised split:  63%|██████▎   | 31660/50000 [00:08<00:02, 7149.84 examples/s]\n",
      "Generating unsupervised split:  65%|██████▍   | 32440/50000 [00:08<00:02, 7335.25 examples/s]\n",
      "Generating unsupervised split:  66%|██████▋   | 33224/50000 [00:08<00:02, 7481.89 examples/s]\n",
      "Generating unsupervised split:  68%|██████▊   | 34007/50000 [00:08<00:02, 7582.15 examples/s]\n",
      "Generating unsupervised split:  70%|██████▉   | 34795/50000 [00:08<00:01, 7669.23 examples/s]\n",
      "Generating unsupervised split:  71%|███████   | 35582/50000 [00:08<00:01, 7728.71 examples/s]\n",
      "Generating unsupervised split:  73%|███████▎  | 36366/50000 [00:08<00:01, 7759.63 examples/s]\n",
      "Generating unsupervised split:  74%|███████▍  | 37151/50000 [00:09<00:01, 7784.00 examples/s]\n",
      "Generating unsupervised split:  76%|███████▌  | 37935/50000 [00:09<00:01, 7798.60 examples/s]\n",
      "Generating unsupervised split:  77%|███████▋  | 38719/50000 [00:09<00:01, 7808.77 examples/s]\n",
      "Generating unsupervised split:  79%|███████▉  | 39501/50000 [00:09<00:01, 7745.30 examples/s]\n",
      "Generating unsupervised split:  81%|████████  | 40277/50000 [00:09<00:01, 6887.20 examples/s]\n",
      "Generating unsupervised split:  82%|████████▏ | 41043/50000 [00:09<00:01, 7098.58 examples/s]\n",
      "Generating unsupervised split:  84%|████████▎ | 41820/50000 [00:09<00:01, 7285.74 examples/s]\n",
      "Generating unsupervised split:  85%|████████▌ | 42600/50000 [00:09<00:00, 7433.09 examples/s]\n",
      "Generating unsupervised split:  87%|████████▋ | 43379/50000 [00:09<00:00, 7533.84 examples/s]\n",
      "Generating unsupervised split:  88%|████████▊ | 44161/50000 [00:10<00:00, 7616.56 examples/s]\n",
      "Generating unsupervised split:  90%|████████▉ | 44939/50000 [00:10<00:00, 7663.82 examples/s]\n",
      "Generating unsupervised split:  91%|█████████▏| 45719/50000 [00:10<00:00, 7703.42 examples/s]\n",
      "Generating unsupervised split:  93%|█████████▎| 46507/50000 [00:10<00:00, 7753.52 examples/s]\n",
      "Generating unsupervised split:  95%|█████████▍| 47285/50000 [00:10<00:00, 7758.62 examples/s]\n",
      "Generating unsupervised split:  96%|█████████▌| 48063/50000 [00:10<00:00, 7751.50 examples/s]\n",
      "Generating unsupervised split:  98%|█████████▊| 48854/50000 [00:10<00:00, 7796.96 examples/s]\n",
      "Generating unsupervised split:  99%|█████████▉| 49641/50000 [00:10<00:00, 7818.59 examples/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Dataset imdb downloaded and prepared to /home/ray/.cache/huggingface/datasets/imdb/plain_text/1.0.0/2fdd8b9bcadd6e7055e742a706876ba43f19faee861df134affd7a3f60fc38a1. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 696.30it/s]                                               \n",
      "Downloading: 100%|██████████| 28.0/28.0 [00:00<00:00, 32.1kB/s]\n",
      "Downloading: 100%|██████████| 483/483 [00:00<00:00, 600kB/s]\n",
      "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]\n",
      "Downloading: 100%|██████████| 232k/232k [00:00<00:00, 4.80MB/s]\n",
      "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]\n",
      "Downloading: 100%|██████████| 466k/466k [00:00<00:00, 7.88MB/s]\n",
      "  0%|          | 0/25 [00:00<?, ?ba/s]\n",
      "  4%|▍         | 1/25 [00:00<00:15,  1.52ba/s]\n",
      "  8%|▊         | 2/25 [00:01<00:14,  1.57ba/s]\n",
      " 12%|█▏        | 3/25 [00:01<00:13,  1.59ba/s]\n",
      " 16%|█▌        | 4/25 [00:02<00:13,  1.59ba/s]\n",
      " 20%|██        | 5/25 [00:03<00:13,  1.52ba/s]\n",
      " 24%|██▍       | 6/25 [00:03<00:12,  1.54ba/s]\n",
      " 28%|██▊       | 7/25 [00:04<00:11,  1.55ba/s]\n",
      " 32%|███▏      | 8/25 [00:05<00:11,  1.53ba/s]\n",
      " 36%|███▌      | 9/25 [00:05<00:10,  1.54ba/s]\n",
      " 40%|████      | 10/25 [00:06<00:09,  1.54ba/s]\n",
      " 44%|████▍     | 11/25 [00:07<00:08,  1.58ba/s]\n",
      " 48%|████▊     | 12/25 [00:07<00:08,  1.58ba/s]\n",
      " 52%|█████▏    | 13/25 [00:08<00:07,  1.60ba/s]\n",
      " 56%|█████▌    | 14/25 [00:08<00:07,  1.57ba/s]\n",
      " 60%|██████    | 15/25 [00:09<00:06,  1.57ba/s]\n",
      " 64%|██████▍   | 16/25 [00:10<00:05,  1.55ba/s]\n",
      " 68%|██████▊   | 17/25 [00:10<00:05,  1.55ba/s]\n",
      " 72%|███████▏  | 18/25 [00:11<00:04,  1.56ba/s]\n",
      " 76%|███████▌  | 19/25 [00:12<00:03,  1.56ba/s]\n",
      " 80%|████████  | 20/25 [00:12<00:03,  1.55ba/s]\n",
      " 84%|████████▍ | 21/25 [00:13<00:02,  1.50ba/s]\n",
      " 88%|████████▊ | 22/25 [00:14<00:01,  1.51ba/s]\n",
      " 92%|█████████▏| 23/25 [00:14<00:01,  1.55ba/s]\n",
      " 96%|█████████▌| 24/25 [00:15<00:00,  1.55ba/s]\n",
      " 96%|█████████▌| 24/25 [00:16<00:00,  1.49ba/s]\n",
      "  0%|          | 0/25 [00:00<?, ?ba/s]\n",
      "  4%|▍         | 1/25 [00:00<00:14,  1.67ba/s]\n",
      "  8%|▊         | 2/25 [00:01<00:13,  1.67ba/s]\n",
      " 12%|█▏        | 3/25 [00:01<00:13,  1.64ba/s]\n",
      " 16%|█▌        | 4/25 [00:02<00:12,  1.64ba/s]\n",
      " 20%|██        | 5/25 [00:03<00:12,  1.63ba/s]\n",
      " 24%|██▍       | 6/25 [00:03<00:11,  1.61ba/s]\n",
      " 28%|██▊       | 7/25 [00:04<00:11,  1.62ba/s]\n",
      " 32%|███▏      | 8/25 [00:04<00:10,  1.61ba/s]\n",
      " 36%|███▌      | 9/25 [00:05<00:10,  1.58ba/s]\n",
      " 40%|████      | 10/25 [00:06<00:09,  1.58ba/s]\n",
      " 44%|████▍     | 11/25 [00:06<00:08,  1.60ba/s]\n",
      " 48%|████▊     | 12/25 [00:07<00:08,  1.59ba/s]\n",
      " 52%|█████▏    | 13/25 [00:08<00:07,  1.55ba/s]\n",
      " 56%|█████▌    | 14/25 [00:08<00:06,  1.58ba/s]\n",
      " 60%|██████    | 15/25 [00:09<00:06,  1.58ba/s]\n",
      " 64%|██████▍   | 16/25 [00:10<00:05,  1.57ba/s]\n",
      " 68%|██████▊   | 17/25 [00:10<00:05,  1.56ba/s]\n",
      " 72%|███████▏  | 18/25 [00:11<00:04,  1.56ba/s]\n",
      " 76%|███████▌  | 19/25 [00:11<00:03,  1.57ba/s]\n",
      " 80%|████████  | 20/25 [00:12<00:03,  1.60ba/s]\n",
      " 84%|████████▍ | 21/25 [00:13<00:02,  1.61ba/s]\n",
      " 88%|████████▊ | 22/25 [00:13<00:01,  1.58ba/s]\n",
      " 92%|█████████▏| 23/25 [00:14<00:01,  1.58ba/s]\n",
      " 96%|█████████▌| 24/25 [00:15<00:00,  1.58ba/s]\n",
      " 96%|█████████▌| 24/25 [00:15<00:00,  1.53ba/s]\n",
      "  0%|          | 0/50 [00:00<?, ?ba/s]\n",
      "  2%|▏         | 1/50 [00:00<00:29,  1.68ba/s]\n",
      "  4%|▍         | 2/50 [00:01<00:29,  1.63ba/s]\n",
      "  6%|▌         | 3/50 [00:01<00:29,  1.58ba/s]\n",
      "  8%|▊         | 4/50 [00:02<00:30,  1.51ba/s]\n",
      " 10%|█         | 5/50 [00:03<00:29,  1.51ba/s]\n",
      " 12%|█▏        | 6/50 [00:03<00:28,  1.56ba/s]\n",
      " 14%|█▍        | 7/50 [00:04<00:27,  1.54ba/s]\n",
      " 16%|█▌        | 8/50 [00:05<00:27,  1.53ba/s]\n",
      " 18%|█▊        | 9/50 [00:05<00:26,  1.56ba/s]\n",
      " 20%|██        | 10/50 [00:06<00:25,  1.55ba/s]\n",
      " 22%|██▏       | 11/50 [00:07<00:25,  1.55ba/s]\n",
      " 24%|██▍       | 12/50 [00:07<00:24,  1.56ba/s]\n",
      " 26%|██▌       | 13/50 [00:08<00:23,  1.57ba/s]\n",
      " 28%|██▊       | 14/50 [00:08<00:22,  1.57ba/s]\n",
      " 30%|███       | 15/50 [00:09<00:22,  1.55ba/s]\n",
      " 32%|███▏      | 16/50 [00:10<00:21,  1.55ba/s]\n",
      " 34%|███▍      | 17/50 [00:10<00:21,  1.56ba/s]\n",
      " 36%|███▌      | 18/50 [00:11<00:20,  1.56ba/s]\n",
      " 38%|███▊      | 19/50 [00:12<00:19,  1.56ba/s]\n",
      " 40%|████      | 20/50 [00:12<00:19,  1.56ba/s]\n",
      " 42%|████▏     | 21/50 [00:13<00:18,  1.53ba/s]\n",
      " 44%|████▍     | 22/50 [00:14<00:18,  1.55ba/s]\n",
      " 46%|████▌     | 23/50 [00:14<00:17,  1.54ba/s]\n",
      " 48%|████▊     | 24/50 [00:15<00:16,  1.56ba/s]\n",
      " 50%|█████     | 25/50 [00:16<00:15,  1.56ba/s]\n",
      " 52%|█████▏    | 26/50 [00:16<00:15,  1.57ba/s]\n",
      " 54%|█████▍    | 27/50 [00:17<00:14,  1.55ba/s]\n",
      " 56%|█████▌    | 28/50 [00:17<00:13,  1.60ba/s]\n",
      " 58%|█████▊    | 29/50 [00:18<00:13,  1.57ba/s]\n",
      " 60%|██████    | 30/50 [00:19<00:12,  1.59ba/s]\n",
      " 62%|██████▏   | 31/50 [00:19<00:12,  1.55ba/s]\n",
      " 64%|██████▍   | 32/50 [00:20<00:11,  1.55ba/s]\n",
      " 66%|██████▌   | 33/50 [00:21<00:10,  1.56ba/s]\n",
      " 68%|██████▊   | 34/50 [00:21<00:10,  1.58ba/s]\n",
      " 70%|███████   | 35/50 [00:22<00:09,  1.61ba/s]\n",
      " 72%|███████▏  | 36/50 [00:23<00:08,  1.60ba/s]\n",
      " 74%|███████▍  | 37/50 [00:23<00:08,  1.54ba/s]\n",
      " 76%|███████▌  | 38/50 [00:24<00:07,  1.56ba/s]\n",
      " 78%|███████▊  | 39/50 [00:24<00:07,  1.56ba/s]\n",
      " 80%|████████  | 40/50 [00:25<00:06,  1.57ba/s]\n",
      " 82%|████████▏ | 41/50 [00:26<00:05,  1.54ba/s]\n",
      " 84%|████████▍ | 42/50 [00:26<00:05,  1.55ba/s]\n",
      " 86%|████████▌ | 43/50 [00:27<00:04,  1.55ba/s]\n",
      " 88%|████████▊ | 44/50 [00:28<00:03,  1.54ba/s]\n",
      " 90%|█████████ | 45/50 [00:28<00:03,  1.56ba/s]\n",
      " 92%|█████████▏| 46/50 [00:29<00:02,  1.56ba/s]\n",
      " 94%|█████████▍| 47/50 [00:30<00:01,  1.54ba/s]\n",
      " 96%|█████████▌| 48/50 [00:30<00:01,  1.54ba/s]\n",
      " 98%|█████████▊| 49/50 [00:31<00:00,  1.54ba/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m len of train Dataset({\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m     features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m     num_rows: 100\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m }) and test Dataset({\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m     features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m     num_rows: 100\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m })\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 49/50 [00:32<00:00,  1.53ba/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m To disable this warning, you can either:\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \t- Avoid using `tokenizers` before the fork if possible\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:55:58 (running for 00:00:05.07)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 6.4/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m 2022-11-04 07:56:02,047\tINFO torch.py:346 -- Setting up process group for: env:// [rank=0, world_size=4]\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m 2022-11-04 07:56:02,045\tINFO torch.py:346 -- Setting up process group for: env:// [rank=2, world_size=4]\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m 2022-11-04 07:56:02,047\tINFO torch.py:346 -- Setting up process group for: env:// [rank=1, world_size=4]\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m 2022-11-04 07:56:02,048\tINFO torch.py:346 -- Setting up process group for: env:// [rank=3, world_size=4]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:03 (running for 00:00:10.07)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 7.2/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: 100%|██████████| 483/483 [00:00<00:00, 588kB/s]\n",
      "Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s] \n",
      "Downloading:   0%|          | 893k/268M [00:00<00:29, 8.93MB/s]\n",
      "Downloading:   3%|▎         | 6.70M/268M [00:00<00:06, 37.8MB/s]\n",
      "Downloading:   5%|▍         | 12.9M/268M [00:00<00:05, 48.7MB/s]\n",
      "Downloading:   7%|▋         | 19.2M/268M [00:00<00:04, 54.4MB/s]\n",
      "Downloading:  10%|▉         | 25.7M/268M [00:00<00:04, 58.3MB/s]\n",
      "Downloading:  12%|█▏        | 32.3M/268M [00:00<00:03, 60.8MB/s]\n",
      "Downloading:  14%|█▍        | 38.8M/268M [00:00<00:03, 62.1MB/s]\n",
      "Downloading:  17%|█▋        | 45.3M/268M [00:00<00:03, 63.3MB/s]\n",
      "Downloading:  19%|█▉        | 51.8M/268M [00:00<00:03, 63.7MB/s]\n",
      "Downloading:  22%|██▏       | 58.4M/268M [00:01<00:03, 64.3MB/s]\n",
      "Downloading:  24%|██▍       | 64.9M/268M [00:01<00:03, 64.7MB/s]\n",
      "Downloading:  27%|██▋       | 71.5M/268M [00:01<00:03, 65.2MB/s]\n",
      "Downloading:  29%|██▉       | 78.1M/268M [00:01<00:02, 65.1MB/s]\n",
      "Downloading:  32%|███▏      | 84.6M/268M [00:01<00:02, 65.1MB/s]\n",
      "Downloading:  34%|███▍      | 91.2M/268M [00:01<00:02, 65.3MB/s]\n",
      "Downloading:  36%|███▋      | 97.7M/268M [00:01<00:02, 65.3MB/s]\n",
      "Downloading:  39%|███▉      | 104M/268M [00:01<00:02, 65.4MB/s] \n",
      "Downloading:  41%|████▏     | 111M/268M [00:01<00:02, 65.5MB/s]\n",
      "Downloading:  44%|████▍     | 117M/268M [00:01<00:02, 65.5MB/s]\n",
      "Downloading:  46%|████▋     | 124M/268M [00:02<00:02, 65.4MB/s]\n",
      "Downloading:  49%|████▊     | 130M/268M [00:02<00:02, 65.4MB/s]\n",
      "Downloading:  51%|█████     | 137M/268M [00:02<00:01, 65.5MB/s]\n",
      "Downloading:  54%|█████▎    | 144M/268M [00:02<00:01, 65.5MB/s]\n",
      "Downloading:  56%|█████▌    | 150M/268M [00:02<00:01, 65.4MB/s]\n",
      "Downloading:  58%|█████▊    | 157M/268M [00:02<00:01, 65.4MB/s]\n",
      "Downloading:  61%|██████    | 163M/268M [00:02<00:01, 65.6MB/s]\n",
      "Downloading:  63%|██████▎   | 170M/268M [00:02<00:01, 65.4MB/s]\n",
      "Downloading:  66%|██████▌   | 176M/268M [00:02<00:01, 65.4MB/s]\n",
      "Downloading:  68%|██████▊   | 183M/268M [00:02<00:01, 65.4MB/s]\n",
      "Downloading:  71%|███████   | 190M/268M [00:03<00:01, 65.6MB/s]\n",
      "Downloading:  73%|███████▎  | 196M/268M [00:03<00:01, 65.6MB/s]\n",
      "Downloading:  76%|███████▌  | 203M/268M [00:03<00:00, 65.7MB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:08 (running for 00:00:15.07)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 7.5/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading:  78%|███████▊  | 209M/268M [00:03<00:00, 65.7MB/s]\n",
      "Downloading:  81%|████████  | 216M/268M [00:03<00:00, 65.7MB/s]\n",
      "Downloading:  83%|████████▎ | 223M/268M [00:03<00:00, 66.0MB/s]\n",
      "Downloading:  86%|████████▌ | 229M/268M [00:03<00:00, 66.0MB/s]\n",
      "Downloading:  88%|████████▊ | 236M/268M [00:03<00:00, 65.8MB/s]\n",
      "Downloading:  90%|█████████ | 242M/268M [00:03<00:00, 65.8MB/s]\n",
      "Downloading:  93%|█████████▎| 249M/268M [00:03<00:00, 65.7MB/s]\n",
      "Downloading:  95%|█████████▌| 255M/268M [00:04<00:00, 65.7MB/s]\n",
      "Downloading:  98%|█████████▊| 262M/268M [00:04<00:00, 65.8MB/s]\n",
      "Downloading: 100%|██████████| 268M/268M [00:04<00:00, 63.9MB/s]\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_projector.bias', 'vocab_layer_norm.bias', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_transform.weight', 'vocab_projector.weight']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m - This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m - This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.bias', 'classifier.bias', 'classifier.weight', 'pre_classifier.weight']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.bias', 'vocab_projector.bias', 'vocab_projector.weight', 'vocab_layer_norm.weight']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m - This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m - This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.bias', 'pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_projector.weight', 'vocab_transform.weight', 'vocab_projector.bias', 'vocab_transform.bias', 'vocab_layer_norm.bias', 'vocab_layer_norm.weight']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m - This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m - This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.bias', 'pre_classifier.bias', 'pre_classifier.weight', 'classifier.weight']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_layer_norm.weight', 'vocab_projector.weight', 'vocab_transform.weight', 'vocab_projector.bias', 'vocab_layer_norm.bias', 'vocab_transform.bias']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m - This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m - This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m /tmp/ray/session_2022-11-04_07-51-23_507232_7/runtime_resources/pip/4a36d7bd0bbff8fccea52f9c0d942dd63707933f/virtualenv/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   warnings.warn(\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m /tmp/ray/session_2022-11-04_07-51-23_507232_7/runtime_resources/pip/4a36d7bd0bbff8fccea52f9c0d942dd63707933f/virtualenv/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m   warnings.warn(\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m /tmp/ray/session_2022-11-04_07-51-23_507232_7/runtime_resources/pip/4a36d7bd0bbff8fccea52f9c0d942dd63707933f/virtualenv/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m   warnings.warn(\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m /tmp/ray/session_2022-11-04_07-51-23_507232_7/runtime_resources/pip/4a36d7bd0bbff8fccea52f9c0d942dd63707933f/virtualenv/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m   warnings.warn(\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m ***** Running training *****\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   Num examples = 6250\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   Num Epochs = 1\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   Instantaneous batch size per device = 16\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   Total train batch size (w. parallel, distributed & accumulation) = 64\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   Gradient Accumulation steps = 1\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   Total optimization steps = 391\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m   Number of trainable parameters = 66955010\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:13 (running for 00:00:20.08)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 12.3/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m [W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=184, ip=10.129.66.16)\u001b[0m [W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=183, ip=10.129.66.16)\u001b[0m [W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=185, ip=10.129.66.16)\u001b[0m [W reducer.cpp:1251] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:18 (running for 00:00:25.08)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:23 (running for 00:00:30.08)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:28 (running for 00:00:35.09)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:33 (running for 00:00:40.09)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:38 (running for 00:00:45.10)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:43 (running for 00:00:50.10)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:48 (running for 00:00:55.10)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:53 (running for 00:01:00.10)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:56:59 (running for 00:01:05.11)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:04 (running for 00:01:10.11)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:09 (running for 00:01:15.11)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:14 (running for 00:01:20.12)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:19 (running for 00:01:25.12)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:24 (running for 00:01:30.12)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:29 (running for 00:01:35.13)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:34 (running for 00:01:40.13)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:39 (running for 00:01:45.13)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:44 (running for 00:01:50.13)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:49 (running for 00:01:55.14)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:54 (running for 00:02:00.14)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:57:59 (running for 00:02:05.15)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 13.7/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m Saving model checkpoint to /tmp/hf_imdb/test/checkpoint-391\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m Configuration saved in /tmp/hf_imdb/test/checkpoint-391/config.json\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m Model weights saved in /tmp/hf_imdb/test/checkpoint-391/pytorch_model.bin\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result for HuggingFaceTrainer_c7d60_00000:\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   _time_this_iter_s: 118.07144260406494\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   _timestamp: 1667573883\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   _training_iteration: 1\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   date: 2022-11-04_07-58-03\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   done: false\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   epoch: 1.0\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   experiment_id: 7bc6ab25d0414fcbb589bcb5d0f29b99\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   hostname: hfgputest-worker-small-group-hfgputest-q4758\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   iterations_since_restore: 1\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   node_ip: 10.129.66.16\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   pid: 146\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   should_checkpoint: true\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   step: 391\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   time_since_restore: 124.55581378936768\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   time_this_iter_s: 124.55581378936768\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   time_total_s: 124.55581378936768\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   timestamp: 1667573883\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   timesteps_since_restore: 0\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_loss: 0.2760564701636429\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_runtime: 109.7668\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_samples_per_second: 56.939\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_steps_per_second: 3.562\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   training_iteration: 1\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   trial_id: c7d60_00000\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   warmup_time: 0.003995656967163086\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   \n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m {'train_runtime': 109.7668, 'train_samples_per_second': 56.939, 'train_steps_per_second': 3.562, 'train_loss': 0.2760564701636429, 'epoch': 1.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m \n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m \n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m \n",
      "\u001b[2m\u001b[36m(BaseWorkerMixin pid=182, ip=10.129.66.16)\u001b[0m \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:58:13 (running for 00:02:19.36)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 16.0/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 5.0/10 CPUs, 4.0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 RUNNING)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+--------+------------------+-----------------+----------------------------+--------------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status   | loc              |   iter |   total time (s) |   train_runtime |   train_samples_per_second |   train_steps_per_second |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+----------+------------------+--------+------------------+-----------------+----------------------------+--------------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | RUNNING  | 10.129.66.16:146 |      1 |          124.556 |         109.767 |                     56.939 |                    3.562 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+----------+------------------+--------+------------------+-----------------+----------------------------+--------------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m 2022-11-04 07:58:13,248\tWARNING util.py:214 -- The `process_trial_save` operation took 9.709 s, which may be a performance bottleneck.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m 2022-11-04 07:58:13,248\tWARNING trial_runner.py:856 -- Consider turning off forced head-worker trial checkpoint syncs by setting sync_on_checkpoint=False. Note that this may result in faulty trial restoration if a failure occurs while the checkpoint is being synced from the worker to the head node.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result for HuggingFaceTrainer_c7d60_00000:\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   _time_this_iter_s: 118.07144260406494\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   _timestamp: 1667573883\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   _training_iteration: 1\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   date: 2022-11-04_07-58-03\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   done: true\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   epoch: 1.0\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   experiment_id: 7bc6ab25d0414fcbb589bcb5d0f29b99\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   experiment_tag: '0'\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   hostname: hfgputest-worker-small-group-hfgputest-q4758\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   iterations_since_restore: 1\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   node_ip: 10.129.66.16\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   pid: 146\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   should_checkpoint: true\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   step: 391\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   time_since_restore: 124.55581378936768\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   time_this_iter_s: 124.55581378936768\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   time_total_s: 124.55581378936768\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   timestamp: 1667573883\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   timesteps_since_restore: 0\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_loss: 0.2760564701636429\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_runtime: 109.7668\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_samples_per_second: 56.939\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   train_steps_per_second: 3.562\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   training_iteration: 1\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   trial_id: c7d60_00000\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   warmup_time: 0.003995656967163086\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m   \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m == Status ==\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Current time: 2022-11-04 07:58:16 (running for 00:02:22.40)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Memory usage on this node: 9.1/240.1 GiB\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Using FIFO scheduling algorithm.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Resources requested: 0/10 CPUs, 0/4 GPUs, 0.0/22.35 GiB heap, 0.0/6.59 GiB objects\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Result logdir: /home/ray/ray_results/HuggingFaceTrainer_2022-11-04_07-55-53\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m Number of trials: 1/1 (1 TERMINATED)\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+------------+------------------+--------+------------------+-----------------+----------------------------+--------------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | Trial name                     | status     | loc              |   iter |   total time (s) |   train_runtime |   train_samples_per_second |   train_steps_per_second |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m |--------------------------------+------------+------------------+--------+------------------+-----------------+----------------------------+--------------------------|\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m | HuggingFaceTrainer_c7d60_00000 | TERMINATED | 10.129.66.16:146 |      1 |          124.556 |         109.767 |                     56.939 |                    3.562 |\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m +--------------------------------+------------+------------------+--------+------------------+-----------------+----------------------------+--------------------------+\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m 2022-11-04 07:58:16,286\tWARNING util.py:214 -- The `process_trial_save` operation took 2.161 s, which may be a performance bottleneck.\n",
      "\u001b[2m\u001b[36m(train_fn pid=250)\u001b[0m 2022-11-04 07:58:16,398\tINFO tune.py:747 -- Total run time: 142.70 seconds (142.40 seconds for the tuning loop).\n"
     ]
    }
   ],
   "source": [
    "#call the above cell as a remote ray function\n",
    "ray.get(train_fn.remote())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5af8cd32",
   "metadata": {},
   "source": [
    "Finally, we bring our resource cluster down and release/terminate the associated resources, bringing everything back to the way it was before our cluster was brought up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec421113-0e49-4043-a3b5-66efa5021cdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.down()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61bf4946",
   "metadata": {},
   "outputs": [],
   "source": [
    "auth.logout()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b7a183b-5e8e-4adb-b9a6-a349e13512a0",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "As shown in the above example, you can easily run your Huggingface transfer learning tasks easily and natively on CodeFlare. You can scale them from 1 to n GPUs without requiring you to make any significant code changes and leveraging the native Huggingface trainer. \n",
    "\n",
    "Also refer to additional notebooks that showcase other use cases\n",
    "In our next notebook [./02_codeflare_workflows_encoding.ipynb ] shows an sklearn example and how you can leverage workflows to run experiment pipelines and explore multiple pipelines in parallel on CodeFlare cluster. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2677c868-a052-4893-9493-6f1dacd8fa27",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "f9f85f796d01129d0dd105a088854619f454435301f6ffec2fea96ecbd9be4ac"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}