Dendra_Query_Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dendra Query Examples\n",
    "Author: Collin Bode   \n",
    "Email: collin@berkeley.edu\n",
    "\n",
    "<u>Purpose</u>: Example code to pull all datastreams from one station.         \n",
    "<u>Requires</u>: dendra_api_client.py file defined in python path. Please download from:       \n",
    "https://github.com/DendraScience/dendra-api-client-python    \n",
    "and place in your working directory.\n",
    "\n",
    "Please note the following functions:   \n",
    "\n",
    ">df = <b>dendra.get_datapoints</b>(<em>datastream_id,begins_at,ends_before=time_format(),time_type='local',name='default’</em>):   returns one datastream as dataframe.\n",
    "\n",
    ">df = <b>dendra.get_datapoints_from_id_list</b>(<em>datastream_id_list,begins_at,ends_before=time_format(),time_type='local’</em>):  returns one dataframe of all datastreams.  Input is an array of datastream_ids, e.g.  Permittivity_Avg = [\"5d488fe302e4cd88409c2bde\", \"5d488fda02e4cd135e9c2bc0\",“5d488fda02e4cd5ecf9c2bc2\"]   \n",
    "\n",
    ">df = <b>dendra.list_datastreams_by_measurement</b>(<em>measurement,optional:aggregate,station_id,orgslug</em>): returns a list of datastream names and ids for all datastreams that fit your query. This list can then be used in get_datapoints_from_id_list.  Measurements are a controlled vocabulary in Dendra found here https://dendra.science/vocabulary.  Select \"Dendra Query Vocabularies (dq)\" and use the \"label\" version of the measurement. It should have no spaces in the label.  \n",
    "\n",
    "<u>Arguments</u> common to all three functions: \n",
    "\n",
    "<i>datastream_id</i> and <i>station_id</i>:  these are Mongo database ID’s.  They must be in quotes to be processed. 5d488fe302e4cd88409c2bde throws an error. \"5d488fe302e4cd88409c2bde\" works. Use dendra.list_datastreams   \n",
    "\n",
    "<i>begins_at</i> is an ISO compliant timestamp. ’T’ is placed between date and time.  Time is hours:minutes:seconds in two digits, e.g. '2020-02-20T00:00:00’.   The first timestamp is included in the query (>=).   \n",
    "\n",
    "<i>ends_before</i> is optional.  It will default to today if left empty. ends_before is NOT included in the query (<).    \n",
    "\n",
    "<i>time_type</i> is optional.  It will default to ‘local’ if left empty.  This mean Pacific Standard Time (UTC-8hours).  The only other option is ‘utc’ which requires your input time parameters be in UTC.   \n",
    "\n",
    "<u>Functions to list</u> Organizations, Stations, and Datastreams:   \n",
    "- list_organizations(orgslug='all')   \n",
    "- get_organization_id(orgslug)   \n",
    "- list_stations(orgslug='all',query_add='none')   \n",
    "- list_datastreams_by_station_id(station_id,query_add = '')   \n",
    "- list_datastreams_by_query(query_add = '',station_id = '')   \n",
    "- list_datastreams_by_medium_variable(medium = '',variable = '',aggregate = '', station_id = '', orgslug = '', query_add = '')   \n",
    "- list_datastreams_by_measurement(measurement = '',aggregate = '', station_id = [], orgslug = '', query_add = '')   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import json\n",
    "import os\n",
    "import sys\n",
    "path_to_git = '../dendra-api-client-python/'  # <-- Please change this to match the location you have pulled github\n",
    "sys.path.append(path_to_git)\n",
    "import dendra_api_client as dendra"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " ········\n"
     ]
    }
   ],
   "source": [
    "# Authentication\n",
    "# If you have a login and the data is not public, you must authenticatte using your Dendra login\n",
    "dendra.authenticate('collin@berkeley.edu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters: start and end dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters: start and end time\n",
    "# Note the queries default to local time. Add a captial 'Z' to the end of the timestamp to indicate UTC and many functions have a parameter for local vs utc\n",
    "begins_at = '2024-05-01T00:00:00'  \n",
    "ends_before = dendra.time_format() # time_format without argument gives current datetime. #'2020-03-01T00:00:00'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List stations for one organization, e.g. UC Natural Reserve System"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UCNRS Weather Station List\n",
      "{'_id': '58e68cabdf5ce600012602b3', 'name': 'Angelo South Meadow', 'slug': 'angelo-south-meadow'}\n",
      "{'_id': '58e68cabdf5ce600012602b4', 'name': 'Anza Borrego HQ', 'slug': 'anza-borrego'}\n",
      "{'_id': '5ed6f71d5ceeff3f92a505b0', 'name': 'Anza Borrego Sentenac', 'slug': 'anza-borrego-sentenac'}\n",
      "{'_id': '630ffc7fbcd7f8d98fe03a37', 'name': 'Año Nuevo', 'slug': 'ano-nuevo'}\n",
      "{'_id': '58e68cabdf5ce600012602b5', 'name': 'BigCreek Gatehouse', 'slug': 'bigcreek-gatehouse'}\n",
      "{'_id': '58e68cabdf5ce600012602b6', 'name': 'BigCreek Highlands', 'slug': 'bigcreek-highlands'}\n",
      "{'_id': '58e68cabdf5ce600012602b7', 'name': 'BigCreek Whale', 'slug': 'bigcreek-whale'}\n",
      "{'_id': '58e68cabdf5ce600012602b8', 'name': 'Blue Oak Ranch', 'slug': 'blue-oak-ranch'}\n",
      "{'_id': '58e68cabdf5ce600012602b9', 'name': 'Bodega', 'slug': 'bodega'}\n",
      "{'_id': '58e68cabdf5ce600012602ba', 'name': 'Burns', 'slug': 'burns'}\n",
      "{'_id': '5e5d4f9257a6e41c0adeb739', 'name': 'Carpinteria', 'slug': 'carpinteria'}\n",
      "{'_id': '58e68cabdf5ce600012602bc', 'name': 'Chickering', 'slug': 'chickering'}\n",
      "{'_id': '5a9cd13b18af1700013b110a', 'name': 'Dawson', 'slug': 'dawson'}\n",
      "{'_id': '58e68cabdf5ce600012602bd', 'name': 'Deep Canyon', 'slug': 'deep-canyon'}\n",
      "{'_id': '58e68cabdf5ce600012602be', 'name': 'Elliott', 'slug': 'elliott'}\n",
      "{'_id': '58e68cabdf5ce600012602bf', 'name': 'Fort Ord', 'slug': 'fort-ord'}\n",
      "{'_id': '58e68cabdf5ce600012602c0', 'name': 'Granites', 'slug': 'granites'}\n",
      "{'_id': '61e74b0ea824b2bad7ad646c', 'name': 'Granites Dripping  Spring', 'slug': 'granitedrip'}\n",
      "{'_id': '58e68cabdf5ce600012602c1', 'name': 'Hastings', 'slug': 'hastings'}\n",
      "{'_id': '58e68cacdf5ce600012602c3', 'name': 'James', 'slug': 'james'}\n",
      "{'_id': '58e68cacdf5ce600012602c4', 'name': 'Jepson', 'slug': 'jepson'}\n",
      "{'_id': '58e68cacdf5ce600012602c5', 'name': 'McLaughlin', 'slug': 'mclaughlin'}\n",
      "{'_id': '58e68cacdf5ce600012602c6', 'name': 'Motte', 'slug': 'motte'}\n",
      "{'_id': '621d0b127cbce863e92e6fd1', 'name': 'Oasis de los Osos', 'slug': 'oasis-de-los-osos'}\n",
      "{'_id': '5e1fc5eaa73b5d4389152376', 'name': 'Point Reyes', 'slug': 'point-reyes'}\n",
      "{'_id': '5a37e71ba546db0001547681', 'name': 'Quail Ridge', 'slug': 'quail-ridge'}\n",
      "{'_id': '58e68cacdf5ce600012602c7', 'name': 'Rancho Marino', 'slug': 'rancho-marino'}\n",
      "{'_id': '58e68cacdf5ce600012602d7', 'name': 'SNARL', 'slug': 'snarl'}\n",
      "{'_id': '58e68cacdf5ce600012602d1', 'name': 'Sagehen Creek', 'slug': 'sagehen-creek'}\n",
      "{'_id': '58e68cacdf5ce600012602d3', 'name': 'Santa Cruz Island', 'slug': 'santa-cruz-island'}\n",
      "{'_id': '58e68cacdf5ce600012602d6', 'name': 'Sedgwick', 'slug': 'sedgwick'}\n",
      "{'_id': '58e68cacdf5ce600012602d9', 'name': 'Stunt Ranch', 'slug': 'stunt-ranch'}\n",
      "{'_id': '5fdb6dca2589ba0773655262', 'name': 'Valentine', 'slug': 'valentine'}\n",
      "{'_id': '58e68caddf5ce600012602de', 'name': 'White Mt Summit', 'slug': 'whitemt-summit'}\n",
      "{'_id': '58e68cacdf5ce600012602dc', 'name': 'WhiteMt Barcroft', 'slug': 'whitemt-barcroft'}\n",
      "{'_id': '58e68cacdf5ce600012602dd', 'name': 'WhiteMt Crooked', 'slug': 'whitemt-crooked'}\n",
      "{'_id': '6185c316480e75133e10af77', 'name': 'Yosemite Mariposa Grove', 'slug': 'yosemite-mariposa-grove'}\n",
      "{'_id': '58e68caddf5ce600012602e0', 'name': 'Younger', 'slug': 'younger'}\n"
     ]
    }
   ],
   "source": [
    "# Output is a JSON list with id, name, and web slug\n",
    "# easiest to query stations by id\n",
    "print('UCNRS Weather Station List')\n",
    "stations = dendra.list_stations('ucnrs')\n",
    "for station in stations:\n",
    "    print(station)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download all datastream metadata for one weather station"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 StuntRanch_Air_Temp_Max NEW dataframe created!\n",
      "1 StuntRanch_Air_Temp_Delta_10m2m_Min added.\n",
      "2 StuntRanch_Air_Temp_2_m_Max added.\n",
      "3 StuntRanch_Air_Temp_10_m_Avg added.\n",
      "4 StuntRanch_Air_Temp_Min added.\n",
      "5 StuntRanch_Air_Temp_Delta_10m2m_Avg added.\n",
      "6 StuntRanch_Battery_Voltage_Avg added.\n",
      "7 StuntRanch_Air_Temp_Avg added.\n",
      "8 StuntRanch_Barometric_Pressure_Avg added.\n",
      "9 StuntRanch_Air_Temp_2_m_Avg added.\n",
      "10 StuntRanch_Air_Temp_Delta_10m2m_Max added.\n",
      "11 StuntRanch_Air_Temp_10_m_Max added.\n",
      "12 StuntRanch_Air_Temp_10_m_Min added.\n",
      "13 StuntRanch_Air_Temp_2_m_Min added.\n",
      "14 StuntRanch_Battery_Voltage_Min added.\n",
      "15 StuntRanch_Logger_Panel_Temp_Avg added.\n",
      "16 StuntRanch_Rainfall_Cumulative added.\n",
      "17 StuntRanch_Battery_Voltage_Max added.\n",
      "18 StuntRanch_Photosynthetically_Active_Radiation_Max added.\n",
      "19 StuntRanch_Rainfall added.\n",
      "20 StuntRanch_Photosynthetically_Active_Radiation_Min added.\n",
      "21 StuntRanch_Rain_Gauge_Temp_Avg added.\n",
      "22 StuntRanch_Photosynthetically_Active_Radiation_Avg added.\n",
      "23 StuntRanch_Relative_Humidity_Max added.\n",
      "24 StuntRanch_Relative_Humidity_Avg added.\n",
      "25 StuntRanch_Soil_Temp_100_mm_Avg added.\n",
      "26 StuntRanch_Relative_Humidity_Min added.\n",
      "27 StuntRanch_Rainfall_Cumulative_(logger) added.\n",
      "28 StuntRanch_Soil_Temp_100_mm_Max added.\n",
      "29 StuntRanch_Soil_Temp_200_mm_Avg added.\n",
      "30 StuntRanch_Soil_Temp_200_mm_Min added.\n",
      "31 StuntRanch_Soil_Temp_200_mm_Max added.\n",
      "32 StuntRanch_Soil_Temp_100_mm_Min added.\n",
      "33 StuntRanch_Soil_Temp_50_mm_Max added.\n",
      "34 StuntRanch_Soil_Temp_500_mm_Max added.\n",
      "35 StuntRanch_Soil_Temp_500_mm_Avg added.\n",
      "36 StuntRanch_Soil_Temp_50_mm_Min added.\n",
      "37 StuntRanch_Soil_Temp_500_mm_Min added.\n",
      "38 StuntRanch_Soil_Temp_50_mm_Avg added.\n",
      "39 StuntRanch_Total_Solar_Radiation_Avg added.\n",
      "40 StuntRanch_TX_Reflected_Power added.\n",
      "41 StuntRanch_TX_Forward_Power added.\n",
      "42 StuntRanch_Total_Solar_Radiation_Max added.\n",
      "43 StuntRanch_Total_Solar_Radiation_Min added.\n",
      "44 StuntRanch_Total_Solar_Radiation_Std added.\n",
      "45 StuntRanch_Volumetric_Water_Content_Period_Vertical_Avg added.\n",
      "46 StuntRanch_Volumetric_Water_Content_Period_Horizontal_Avg added.\n",
      "47 StuntRanch_Volumetric_Water_Content_Horizontal_Avg added.\n",
      "48 StuntRanch_Wind_Direction_Avg added.\n",
      "49 StuntRanch_Wind_Speed_Avg added.\n",
      "50 StuntRanch_Volumetric_Water_Content_Vertical_Avg added.\n",
      "51 StuntRanch_Wind_Direction_Std added.\n",
      "52 StuntRanch_Wind_Speed_Max added.\n",
      "53 StuntRanch_Wind_Speed_Std added.\n",
      "54 StuntRanch_Wind_Speed_Min added.\n",
      "55 StuntRanch_Wind_Speed_Vector_Magnitude added.\n"
     ]
    }
   ],
   "source": [
    "station_id = '58e68cacdf5ce600012602d9'  # 'Stunt Ranch'\n",
    "# def get_datapoints_from_station_id(station_id,begins_at,ends_before=time_format(),time_type='local'):\n",
    "# Returns a dataframe with ALL datastreams associated with a particular station for the time period \n",
    "df = dendra.get_datapoints_from_station_id(station_id,begins_at,ends_before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp_utc</th>\n",
       "      <th>StuntRanch_Air_Temp_Max</th>\n",
       "      <th>StuntRanch_Air_Temp_Delta_10m2m_Min</th>\n",
       "      <th>StuntRanch_Air_Temp_2_m_Max</th>\n",
       "      <th>StuntRanch_Air_Temp_10_m_Avg</th>\n",
       "      <th>StuntRanch_Air_Temp_Min</th>\n",
       "      <th>StuntRanch_Air_Temp_Delta_10m2m_Avg</th>\n",
       "      <th>StuntRanch_Battery_Voltage_Avg</th>\n",
       "      <th>StuntRanch_Air_Temp_Avg</th>\n",
       "      <th>StuntRanch_Barometric_Pressure_Avg</th>\n",
       "      <th>...</th>\n",
       "      <th>StuntRanch_Volumetric_Water_Content_Period_Horizontal_Avg</th>\n",
       "      <th>StuntRanch_Volumetric_Water_Content_Horizontal_Avg</th>\n",
       "      <th>StuntRanch_Wind_Direction_Avg</th>\n",
       "      <th>StuntRanch_Wind_Speed_Avg</th>\n",
       "      <th>StuntRanch_Volumetric_Water_Content_Vertical_Avg</th>\n",
       "      <th>StuntRanch_Wind_Direction_Std</th>\n",
       "      <th>StuntRanch_Wind_Speed_Max</th>\n",
       "      <th>StuntRanch_Wind_Speed_Std</th>\n",
       "      <th>StuntRanch_Wind_Speed_Min</th>\n",
       "      <th>StuntRanch_Wind_Speed_Vector_Magnitude</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp_local</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2024-05-01 00:00:00</th>\n",
       "      <td>2024-05-01 08:00:00+00:00</td>\n",
       "      <td>11.82</td>\n",
       "      <td>0.390</td>\n",
       "      <td>11.82</td>\n",
       "      <td>12.23</td>\n",
       "      <td>11.62</td>\n",
       "      <td>0.656</td>\n",
       "      <td>12.70</td>\n",
       "      <td>11.73</td>\n",
       "      <td>965.779661</td>\n",
       "      <td>...</td>\n",
       "      <td>27.89</td>\n",
       "      <td>0.302</td>\n",
       "      <td>91.20</td>\n",
       "      <td>0.432</td>\n",
       "      <td>0.477</td>\n",
       "      <td>17.92</td>\n",
       "      <td>1.143</td>\n",
       "      <td>0.322</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-01 00:10:00</th>\n",
       "      <td>2024-05-01 08:10:00+00:00</td>\n",
       "      <td>11.65</td>\n",
       "      <td>0.136</td>\n",
       "      <td>11.47</td>\n",
       "      <td>11.75</td>\n",
       "      <td>11.22</td>\n",
       "      <td>0.584</td>\n",
       "      <td>12.69</td>\n",
       "      <td>11.36</td>\n",
       "      <td>965.762712</td>\n",
       "      <td>...</td>\n",
       "      <td>27.88</td>\n",
       "      <td>0.302</td>\n",
       "      <td>267.00</td>\n",
       "      <td>0.435</td>\n",
       "      <td>0.477</td>\n",
       "      <td>27.31</td>\n",
       "      <td>1.143</td>\n",
       "      <td>0.397</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-01 00:20:00</th>\n",
       "      <td>2024-05-01 08:20:00+00:00</td>\n",
       "      <td>11.29</td>\n",
       "      <td>0.373</td>\n",
       "      <td>11.26</td>\n",
       "      <td>11.61</td>\n",
       "      <td>11.02</td>\n",
       "      <td>0.618</td>\n",
       "      <td>12.65</td>\n",
       "      <td>11.17</td>\n",
       "      <td>965.728814</td>\n",
       "      <td>...</td>\n",
       "      <td>27.87</td>\n",
       "      <td>0.302</td>\n",
       "      <td>66.71</td>\n",
       "      <td>0.071</td>\n",
       "      <td>0.477</td>\n",
       "      <td>24.55</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-01 00:30:00</th>\n",
       "      <td>2024-05-01 08:30:00+00:00</td>\n",
       "      <td>11.06</td>\n",
       "      <td>0.322</td>\n",
       "      <td>10.95</td>\n",
       "      <td>11.42</td>\n",
       "      <td>10.75</td>\n",
       "      <td>0.680</td>\n",
       "      <td>12.64</td>\n",
       "      <td>10.90</td>\n",
       "      <td>965.745763</td>\n",
       "      <td>...</td>\n",
       "      <td>27.86</td>\n",
       "      <td>0.301</td>\n",
       "      <td>284.60</td>\n",
       "      <td>0.501</td>\n",
       "      <td>0.476</td>\n",
       "      <td>13.26</td>\n",
       "      <td>1.143</td>\n",
       "      <td>0.331</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-05-01 00:40:00</th>\n",
       "      <td>2024-05-01 08:40:00+00:00</td>\n",
       "      <td>10.82</td>\n",
       "      <td>0.356</td>\n",
       "      <td>10.84</td>\n",
       "      <td>11.26</td>\n",
       "      <td>10.69</td>\n",
       "      <td>0.619</td>\n",
       "      <td>12.67</td>\n",
       "      <td>10.75</td>\n",
       "      <td>965.813559</td>\n",
       "      <td>...</td>\n",
       "      <td>27.85</td>\n",
       "      <td>0.301</td>\n",
       "      <td>241.40</td>\n",
       "      <td>0.509</td>\n",
       "      <td>0.476</td>\n",
       "      <td>29.85</td>\n",
       "      <td>1.111</td>\n",
       "      <td>0.316</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-11 16:20:00</th>\n",
       "      <td>2024-07-12 00:20:00+00:00</td>\n",
       "      <td>36.18</td>\n",
       "      <td>-1.869</td>\n",
       "      <td>36.32</td>\n",
       "      <td>34.35</td>\n",
       "      <td>35.68</td>\n",
       "      <td>-1.365</td>\n",
       "      <td>12.99</td>\n",
       "      <td>35.95</td>\n",
       "      <td>965.169492</td>\n",
       "      <td>...</td>\n",
       "      <td>21.72</td>\n",
       "      <td>0.127</td>\n",
       "      <td>259.20</td>\n",
       "      <td>2.401</td>\n",
       "      <td>0.289</td>\n",
       "      <td>22.81</td>\n",
       "      <td>4.312</td>\n",
       "      <td>1.007</td>\n",
       "      <td>0.000</td>\n",
       "      <td>2.210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-11 16:30:00</th>\n",
       "      <td>2024-07-12 00:30:00+00:00</td>\n",
       "      <td>36.36</td>\n",
       "      <td>-2.128</td>\n",
       "      <td>36.46</td>\n",
       "      <td>34.36</td>\n",
       "      <td>35.79</td>\n",
       "      <td>-1.560</td>\n",
       "      <td>13.00</td>\n",
       "      <td>36.14</td>\n",
       "      <td>965.169492</td>\n",
       "      <td>...</td>\n",
       "      <td>21.73</td>\n",
       "      <td>0.127</td>\n",
       "      <td>265.00</td>\n",
       "      <td>2.150</td>\n",
       "      <td>0.289</td>\n",
       "      <td>24.19</td>\n",
       "      <td>4.377</td>\n",
       "      <td>1.042</td>\n",
       "      <td>0.163</td>\n",
       "      <td>1.958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-11 16:40:00</th>\n",
       "      <td>2024-07-12 00:40:00+00:00</td>\n",
       "      <td>36.57</td>\n",
       "      <td>-1.950</td>\n",
       "      <td>36.49</td>\n",
       "      <td>34.65</td>\n",
       "      <td>36.13</td>\n",
       "      <td>-1.329</td>\n",
       "      <td>13.00</td>\n",
       "      <td>36.37</td>\n",
       "      <td>965.237288</td>\n",
       "      <td>...</td>\n",
       "      <td>21.74</td>\n",
       "      <td>0.127</td>\n",
       "      <td>283.10</td>\n",
       "      <td>1.219</td>\n",
       "      <td>0.289</td>\n",
       "      <td>28.31</td>\n",
       "      <td>3.920</td>\n",
       "      <td>0.882</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-11 16:50:00</th>\n",
       "      <td>2024-07-12 00:50:00+00:00</td>\n",
       "      <td>36.70</td>\n",
       "      <td>-2.320</td>\n",
       "      <td>36.64</td>\n",
       "      <td>34.40</td>\n",
       "      <td>35.97</td>\n",
       "      <td>-1.349</td>\n",
       "      <td>13.01</td>\n",
       "      <td>36.36</td>\n",
       "      <td>965.288136</td>\n",
       "      <td>...</td>\n",
       "      <td>21.74</td>\n",
       "      <td>0.128</td>\n",
       "      <td>285.20</td>\n",
       "      <td>1.246</td>\n",
       "      <td>0.289</td>\n",
       "      <td>30.05</td>\n",
       "      <td>2.940</td>\n",
       "      <td>0.647</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2024-07-11 17:00:00</th>\n",
       "      <td>2024-07-12 01:00:00+00:00</td>\n",
       "      <td>36.10</td>\n",
       "      <td>-1.903</td>\n",
       "      <td>36.09</td>\n",
       "      <td>34.24</td>\n",
       "      <td>35.70</td>\n",
       "      <td>-1.094</td>\n",
       "      <td>13.01</td>\n",
       "      <td>35.89</td>\n",
       "      <td>965.271186</td>\n",
       "      <td>...</td>\n",
       "      <td>21.74</td>\n",
       "      <td>0.128</td>\n",
       "      <td>270.80</td>\n",
       "      <td>1.295</td>\n",
       "      <td>0.289</td>\n",
       "      <td>29.06</td>\n",
       "      <td>3.267</td>\n",
       "      <td>0.824</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.124</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10315 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                timestamp_utc  StuntRanch_Air_Temp_Max  \\\n",
       "timestamp_local                                                          \n",
       "2024-05-01 00:00:00 2024-05-01 08:00:00+00:00                    11.82   \n",
       "2024-05-01 00:10:00 2024-05-01 08:10:00+00:00                    11.65   \n",
       "2024-05-01 00:20:00 2024-05-01 08:20:00+00:00                    11.29   \n",
       "2024-05-01 00:30:00 2024-05-01 08:30:00+00:00                    11.06   \n",
       "2024-05-01 00:40:00 2024-05-01 08:40:00+00:00                    10.82   \n",
       "...                                       ...                      ...   \n",
       "2024-07-11 16:20:00 2024-07-12 00:20:00+00:00                    36.18   \n",
       "2024-07-11 16:30:00 2024-07-12 00:30:00+00:00                    36.36   \n",
       "2024-07-11 16:40:00 2024-07-12 00:40:00+00:00                    36.57   \n",
       "2024-07-11 16:50:00 2024-07-12 00:50:00+00:00                    36.70   \n",
       "2024-07-11 17:00:00 2024-07-12 01:00:00+00:00                    36.10   \n",
       "\n",
       "                     StuntRanch_Air_Temp_Delta_10m2m_Min  \\\n",
       "timestamp_local                                            \n",
       "2024-05-01 00:00:00                                0.390   \n",
       "2024-05-01 00:10:00                                0.136   \n",
       "2024-05-01 00:20:00                                0.373   \n",
       "2024-05-01 00:30:00                                0.322   \n",
       "2024-05-01 00:40:00                                0.356   \n",
       "...                                                  ...   \n",
       "2024-07-11 16:20:00                               -1.869   \n",
       "2024-07-11 16:30:00                               -2.128   \n",
       "2024-07-11 16:40:00                               -1.950   \n",
       "2024-07-11 16:50:00                               -2.320   \n",
       "2024-07-11 17:00:00                               -1.903   \n",
       "\n",
       "                     StuntRanch_Air_Temp_2_m_Max  \\\n",
       "timestamp_local                                    \n",
       "2024-05-01 00:00:00                        11.82   \n",
       "2024-05-01 00:10:00                        11.47   \n",
       "2024-05-01 00:20:00                        11.26   \n",
       "2024-05-01 00:30:00                        10.95   \n",
       "2024-05-01 00:40:00                        10.84   \n",
       "...                                          ...   \n",
       "2024-07-11 16:20:00                        36.32   \n",
       "2024-07-11 16:30:00                        36.46   \n",
       "2024-07-11 16:40:00                        36.49   \n",
       "2024-07-11 16:50:00                        36.64   \n",
       "2024-07-11 17:00:00                        36.09   \n",
       "\n",
       "                     StuntRanch_Air_Temp_10_m_Avg  StuntRanch_Air_Temp_Min  \\\n",
       "timestamp_local                                                              \n",
       "2024-05-01 00:00:00                         12.23                    11.62   \n",
       "2024-05-01 00:10:00                         11.75                    11.22   \n",
       "2024-05-01 00:20:00                         11.61                    11.02   \n",
       "2024-05-01 00:30:00                         11.42                    10.75   \n",
       "2024-05-01 00:40:00                         11.26                    10.69   \n",
       "...                                           ...                      ...   \n",
       "2024-07-11 16:20:00                         34.35                    35.68   \n",
       "2024-07-11 16:30:00                         34.36                    35.79   \n",
       "2024-07-11 16:40:00                         34.65                    36.13   \n",
       "2024-07-11 16:50:00                         34.40                    35.97   \n",
       "2024-07-11 17:00:00                         34.24                    35.70   \n",
       "\n",
       "                     StuntRanch_Air_Temp_Delta_10m2m_Avg  \\\n",
       "timestamp_local                                            \n",
       "2024-05-01 00:00:00                                0.656   \n",
       "2024-05-01 00:10:00                                0.584   \n",
       "2024-05-01 00:20:00                                0.618   \n",
       "2024-05-01 00:30:00                                0.680   \n",
       "2024-05-01 00:40:00                                0.619   \n",
       "...                                                  ...   \n",
       "2024-07-11 16:20:00                               -1.365   \n",
       "2024-07-11 16:30:00                               -1.560   \n",
       "2024-07-11 16:40:00                               -1.329   \n",
       "2024-07-11 16:50:00                               -1.349   \n",
       "2024-07-11 17:00:00                               -1.094   \n",
       "\n",
       "                     StuntRanch_Battery_Voltage_Avg  StuntRanch_Air_Temp_Avg  \\\n",
       "timestamp_local                                                                \n",
       "2024-05-01 00:00:00                           12.70                    11.73   \n",
       "2024-05-01 00:10:00                           12.69                    11.36   \n",
       "2024-05-01 00:20:00                           12.65                    11.17   \n",
       "2024-05-01 00:30:00                           12.64                    10.90   \n",
       "2024-05-01 00:40:00                           12.67                    10.75   \n",
       "...                                             ...                      ...   \n",
       "2024-07-11 16:20:00                           12.99                    35.95   \n",
       "2024-07-11 16:30:00                           13.00                    36.14   \n",
       "2024-07-11 16:40:00                           13.00                    36.37   \n",
       "2024-07-11 16:50:00                           13.01                    36.36   \n",
       "2024-07-11 17:00:00                           13.01                    35.89   \n",
       "\n",
       "                     StuntRanch_Barometric_Pressure_Avg  ...  \\\n",
       "timestamp_local                                          ...   \n",
       "2024-05-01 00:00:00                          965.779661  ...   \n",
       "2024-05-01 00:10:00                          965.762712  ...   \n",
       "2024-05-01 00:20:00                          965.728814  ...   \n",
       "2024-05-01 00:30:00                          965.745763  ...   \n",
       "2024-05-01 00:40:00                          965.813559  ...   \n",
       "...                                                 ...  ...   \n",
       "2024-07-11 16:20:00                          965.169492  ...   \n",
       "2024-07-11 16:30:00                          965.169492  ...   \n",
       "2024-07-11 16:40:00                          965.237288  ...   \n",
       "2024-07-11 16:50:00                          965.288136  ...   \n",
       "2024-07-11 17:00:00                          965.271186  ...   \n",
       "\n",
       "                     StuntRanch_Volumetric_Water_Content_Period_Horizontal_Avg  \\\n",
       "timestamp_local                                                                  \n",
       "2024-05-01 00:00:00                                              27.89           \n",
       "2024-05-01 00:10:00                                              27.88           \n",
       "2024-05-01 00:20:00                                              27.87           \n",
       "2024-05-01 00:30:00                                              27.86           \n",
       "2024-05-01 00:40:00                                              27.85           \n",
       "...                                                                ...           \n",
       "2024-07-11 16:20:00                                              21.72           \n",
       "2024-07-11 16:30:00                                              21.73           \n",
       "2024-07-11 16:40:00                                              21.74           \n",
       "2024-07-11 16:50:00                                              21.74           \n",
       "2024-07-11 17:00:00                                              21.74           \n",
       "\n",
       "                     StuntRanch_Volumetric_Water_Content_Horizontal_Avg  \\\n",
       "timestamp_local                                                           \n",
       "2024-05-01 00:00:00                                              0.302    \n",
       "2024-05-01 00:10:00                                              0.302    \n",
       "2024-05-01 00:20:00                                              0.302    \n",
       "2024-05-01 00:30:00                                              0.301    \n",
       "2024-05-01 00:40:00                                              0.301    \n",
       "...                                                                ...    \n",
       "2024-07-11 16:20:00                                              0.127    \n",
       "2024-07-11 16:30:00                                              0.127    \n",
       "2024-07-11 16:40:00                                              0.127    \n",
       "2024-07-11 16:50:00                                              0.128    \n",
       "2024-07-11 17:00:00                                              0.128    \n",
       "\n",
       "                     StuntRanch_Wind_Direction_Avg  StuntRanch_Wind_Speed_Avg  \\\n",
       "timestamp_local                                                                 \n",
       "2024-05-01 00:00:00                          91.20                      0.432   \n",
       "2024-05-01 00:10:00                         267.00                      0.435   \n",
       "2024-05-01 00:20:00                          66.71                      0.071   \n",
       "2024-05-01 00:30:00                         284.60                      0.501   \n",
       "2024-05-01 00:40:00                         241.40                      0.509   \n",
       "...                                            ...                        ...   \n",
       "2024-07-11 16:20:00                         259.20                      2.401   \n",
       "2024-07-11 16:30:00                         265.00                      2.150   \n",
       "2024-07-11 16:40:00                         283.10                      1.219   \n",
       "2024-07-11 16:50:00                         285.20                      1.246   \n",
       "2024-07-11 17:00:00                         270.80                      1.295   \n",
       "\n",
       "                     StuntRanch_Volumetric_Water_Content_Vertical_Avg  \\\n",
       "timestamp_local                                                         \n",
       "2024-05-01 00:00:00                                             0.477   \n",
       "2024-05-01 00:10:00                                             0.477   \n",
       "2024-05-01 00:20:00                                             0.477   \n",
       "2024-05-01 00:30:00                                             0.476   \n",
       "2024-05-01 00:40:00                                             0.476   \n",
       "...                                                               ...   \n",
       "2024-07-11 16:20:00                                             0.289   \n",
       "2024-07-11 16:30:00                                             0.289   \n",
       "2024-07-11 16:40:00                                             0.289   \n",
       "2024-07-11 16:50:00                                             0.289   \n",
       "2024-07-11 17:00:00                                             0.289   \n",
       "\n",
       "                     StuntRanch_Wind_Direction_Std  StuntRanch_Wind_Speed_Max  \\\n",
       "timestamp_local                                                                 \n",
       "2024-05-01 00:00:00                          17.92                      1.143   \n",
       "2024-05-01 00:10:00                          27.31                      1.143   \n",
       "2024-05-01 00:20:00                          24.55                      0.588   \n",
       "2024-05-01 00:30:00                          13.26                      1.143   \n",
       "2024-05-01 00:40:00                          29.85                      1.111   \n",
       "...                                            ...                        ...   \n",
       "2024-07-11 16:20:00                          22.81                      4.312   \n",
       "2024-07-11 16:30:00                          24.19                      4.377   \n",
       "2024-07-11 16:40:00                          28.31                      3.920   \n",
       "2024-07-11 16:50:00                          30.05                      2.940   \n",
       "2024-07-11 17:00:00                          29.06                      3.267   \n",
       "\n",
       "                     StuntRanch_Wind_Speed_Std  StuntRanch_Wind_Speed_Min  \\\n",
       "timestamp_local                                                             \n",
       "2024-05-01 00:00:00                      0.322                      0.000   \n",
       "2024-05-01 00:10:00                      0.397                      0.000   \n",
       "2024-05-01 00:20:00                      0.150                      0.000   \n",
       "2024-05-01 00:30:00                      0.331                      0.000   \n",
       "2024-05-01 00:40:00                      0.316                      0.000   \n",
       "...                                        ...                        ...   \n",
       "2024-07-11 16:20:00                      1.007                      0.000   \n",
       "2024-07-11 16:30:00                      1.042                      0.163   \n",
       "2024-07-11 16:40:00                      0.882                      0.000   \n",
       "2024-07-11 16:50:00                      0.647                      0.000   \n",
       "2024-07-11 17:00:00                      0.824                      0.000   \n",
       "\n",
       "                     StuntRanch_Wind_Speed_Vector_Magnitude  \n",
       "timestamp_local                                              \n",
       "2024-05-01 00:00:00                                   0.405  \n",
       "2024-05-01 00:10:00                                   0.360  \n",
       "2024-05-01 00:20:00                                   0.042  \n",
       "2024-05-01 00:30:00                                   0.484  \n",
       "2024-05-01 00:40:00                                   0.423  \n",
       "...                                                     ...  \n",
       "2024-07-11 16:20:00                                   2.210  \n",
       "2024-07-11 16:30:00                                   1.958  \n",
       "2024-07-11 16:40:00                                   1.059  \n",
       "2024-07-11 16:50:00                                   1.070  \n",
       "2024-07-11 17:00:00                                   1.124  \n",
       "\n",
       "[10315 rows x 57 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Take a look at the dataframe\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('stuntranch_export_wy2019.csv')  # export to disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List Datastreams by Measurement\n",
    "Optional.  If you wish to pull only one kind of measurement, say 'RainfallCumulative' from many locations, you can list all datastreams which perform that measurement. To see what measurements exist, check our vocabulary under 'DQ' or Dendra Queries:  https://dendra.science/vocabulary    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "measurement = 'RainfallCumulative'  \n",
    "query_refinement = { 'is_hidden': False } \n",
    "measurement_list = []   # list of only datastreams that you wish to download data from\n",
    "ds_list = dendra.list_datastreams_by_measurement(measurement,'',[],'ucnrs',query_refinement)\n",
    "for ds in ds_list:\n",
    "    dsm = dendra.get_meta_datastream_by_id(ds['_id'])  # This will pull full datastream metadata in JSON format\n",
    "    station_name = dsm['station_lookup']['name']\n",
    "    print(station_name,ds['name'],ds['_id'])\n",
    "    measurement_list.append(ds['_id'])\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download data for RainfallCumulative\n",
    "The list of datastreams will be fed to 'get_datapoints' which will pull all data for the date range given earlier.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See parameters above for date ranges\n",
    "df = dendra.get_datapoints_from_id_list(measurement_list,begins_at,ends_before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check columns\n",
    "for col in df.columns:\n",
    "    print(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a look at the full DataFrame\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export to disk\n",
    "df.to_csv('rainfallcumulative_measurement_export_wy2019.csv') "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}