Merge pull request #1 from AlSaeed/tests-conflict-resolution

Tests conflict resolution

Author: melange396

Diff for: README.md

@@ -5,15 +5,15 @@
 # About
 
 This is the home of [Delphi](https://delphi.cmu.edu/)'s epidemiological data
-API. See our [API documentation](docs/api/README.md) for details on the
-available data sets, APIs, and clients.
+API. See our [API documentation](https://cmu-delphi.github.io/delphi-epidata/)
+for details on the available data sets, APIs, and clients.
 
 # COVID-19 Notice
 
 We are working on collecting several new data sources that may be useful for
 nowcasting and forecasting ILI during the COVID-19 pandemic.  Each of these will
 make these available as soon as individually possible, through our [covidcast
-endpoint](docs/api/covidcast.md).
+endpoint](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html).
 
 For a list of many other data sources relevant to COVID-19, publicly available
 through external sites, we have compiled a simple

Diff for: docs/api/covidcast.md

@@ -22,13 +22,15 @@ terms; we encourage academic users to [cite](README.md#citing) the data if they
 use it in any publications. Further documentation on Delphi's APIs is available
 in the [API overview](README.md).
 
-**For users:** Delphi operates a [mailing
-list](https://lists.andrew.cmu.edu/mailman/listinfo/delphi-covidcast-api) for
-users of the COVIDcast API. We will use the list to announce API changes,
-corrections to data, and new features; API users may also use the mailing list
-to ask general questions about its use. If you use the API, we strongly
-encourage you to
-[subscribe](https://lists.andrew.cmu.edu/mailman/listinfo/delphi-covidcast-api).
+<div style="background-color:#FCC; padding: 10px 30px;"><strong>For
+users:</strong> Delphi operates a <a
+href="https://lists.andrew.cmu.edu/mailman/listinfo/delphi-covidcast-api">mailing
+list</a> for users of the COVIDcast API. We will use the list to announce API
+changes, corrections to data, and new features; API users may also use the
+mailing list to ask general questions about its use. If you use the API, we
+strongly encourage you to <a
+href="https://lists.andrew.cmu.edu/mailman/listinfo/delphi-covidcast-api">subscribe</a>.
+</div>
 
 ## Accessing the API
 
@@ -38,7 +40,7 @@ the appropriate client for your programming language, accessing data is as easy
 as (in [R](https://www.r-project.org/)):
 
 ```r
-library(covidcastR)
+library(covidcast)
 
 data <- covidcast_signal("fb-survey", "smoothed_cli", start_day = "20200501",
                          end_day = "20200507")
@@ -82,6 +84,8 @@ and lists.
 
 ### Parameters
 
+#### Required
+
 | Parameter | Description | Type |
 | --- | --- | --- |
 | `data_source` | name of upstream data source (e.g., `doctor-visits` or `fb-survey`; [see full list](covidcast_signals.md)) | string |
@@ -96,9 +100,16 @@ The current set of signals available for each data source is returned by the
 
 #### Optional
 
-The default API behavior is to return the most recently issued value for each `time_value` selected.
+Estimates for a specific `time_value` and `geo_value` are sometimes updated
+after they are first published. Many of our data sources issue corrections or
+backfill estimates as data arrives; see the [documentation for each
+source](covidcast_signals.md) for details.
+
+The default API behavior is to return the most recently issued value for each
+`time_value` selected.
 
-We also provide access to previous versions of data using the optional parameters below.
+We also provide access to previous versions of data using the optional query
+parameters below.
 
 | Parameter | Description | Type |
 | --- | --- | --- |
@@ -109,7 +120,7 @@ We also provide access to previous versions of data using the optional parameter
 Use cases:
 
 * To pretend like you queried the API on June 1, such that the returned results
-  do not include any updates which became available after June 1, use
+  do not include any updates that became available after June 1, use
   `as_of=20200601`.
 * To retrieve only data that was published or updated on June 1, and exclude
   records whose most recent update occured earlier than June 1, use
@@ -121,10 +132,12 @@ Use cases:
 * To retrieve only data that was published or updated exactly 3 days after the
   underlying events occurred, use `lag=3`.
 
-NB: Each issue in the versioning system contains only the records that were
-added or updated during that time unit; we exclude records whose values remain
-the same as a previous issue. If you have a research problem that would require
-knowing when an unchanged value was last confirmed, please get in touch.
+You should specify only one of these three parameters in any given query.
+
+**Note:** Each issue in the versioning system contains only the records that
+were added or updated during that time unit; we exclude records whose values
+remain the same as a previous issue. If you have a research problem that would
+require knowing when an unchanged value was last confirmed, please get in touch.
 
 ### Response
 

Diff for: docs/api/covidcast_clients.md

@@ -8,9 +8,8 @@ nav_order: 1
 
 Dedicated COVIDcast clients are available for several languages:
 
-* [covidcast-py](https://cmu-delphi.github.io/covidcast/covidcast-py/html/) for
-  Python users
-* [covidcastR](https://cmu-delphi.github.io/covidcast/covidcastR/) for R users
+* R: [covidcast](https://cmu-delphi.github.io/covidcast/covidcastR/)
+* Python: [covidcast](https://cmu-delphi.github.io/covidcast/covidcast-py/html/)
 
 These packages provide a convenient way to obtain COVIDcast data as a data frame
 ready to be used in further analyses. For installation instructions and
@@ -81,7 +80,7 @@ print(res['result'], res['message'], len(res['epidata']))
 
 ### R
 
-**Note:** For COVIDcast usage, R users should prefer the [covidcastR
+**Note:** For COVIDcast usage, R users should prefer the [covidcast
 package](https://cmu-delphi.github.io/covidcast/covidcastR/); these instructions
 are for advanced users who want access to the entire Epidata API, including data
 on influenza, dengue, and norovirus.

Diff for: docs/api/covidcast_meta.md

@@ -33,13 +33,16 @@ See [this documentation](README.md) for details on specifying epiweeks, dates, a
 | `epidata[].signal` | signal name | string |
 | `epidata[].time_type` | temporal resolution of the signal (e.g., `day`, `week`) | string |
 | `epidata[].geo_type` | geographic resolution (e.g. `county`, `hrr`, `msa`, `dma`, `state`) | string |
-| `epidata[].min_time` | minimum time (e.g., 20200406) | integer |
-| `epidata[].max_time` | maximum time (e.g., 20200413) | integer |
+| `epidata[].min_time` | minimum observation time (e.g., 20200406) | integer |
+| `epidata[].max_time` | maximum observation time (e.g., 20200413) | integer |
 | `epidata[].num_locations` | number of distinct geographic locations with data | integer |
 | `epidata[].min_value` | minimum value | float |
 | `epidata[].max_value` | maximum value | float |
 | `epidata[].mean_value` | mean of value | float |
 | `epidata[].stdev_value` | standard deviation of value | float |
+| `epidata[].max_issue` | most recent date data was issued (e.g., 20200710) | integer |
+| `epidata[].min_lag` | smallest lag from observation to issue, in `time_type` units | integer |
+| `epidata[].max_lag` | largest lag from observation to issue, in `time_type` units | integer |
 | `message` | `success` or error message | string |
 
 ## Example URLs

Diff for: docs/symptom-survey/index.md

@@ -13,9 +13,8 @@ economic and health impacts they have experienced as a result of the pandemic. A
 high-level overview of the survey is posted [on the COVIDcast
 website](https://covidcast.cmu.edu/survey.html).
 
-Aggregate data from this survey is available through the [COVIDcast
-API](../api/covidcast.md) as the [`fb-survey` data
-source](../api/covidcast-signals/fb-survey.md).
+Aggregate data from this survey is available through the [COVIDcast API](../api/covidcast.md)
+as the [`fb-survey` data source](../api/covidcast-signals/fb-survey.md).
 
 This documentation is for users who have a signed Data Use Agreement to receive
 individual response data from the survey. It describes the survey items, data

Diff for: integrations/acquisition/covidcast/test_covidcast_meta_caching.py

@@ -66,14 +66,14 @@ def test_caching(self):
     self.cur.execute('''
       insert into covidcast values
         (0, 'src', 'sig', 'day', 'state', 20200422, 'pa',
-          123, 1, 2, 3, 456, 1, 20200422, 0, False),
+          123, 1, 2, 3, 456, 1, 20200422, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200422, 'wa',
-          789, 1, 2, 3, 456, 1, 20200423, 1, False)
+          789, 1, 2, 3, 456, 1, 20200423, 1, 1, False)
     ''')
     self.cur.execute('''
       insert into covidcast values
         (100, 'src', 'wip_sig', 'day', 'state', 20200422, 'pa',
-          456, 4, 5, 6, 789, -1, 20200422, 0, True)
+          456, 4, 5, 6, 789, -1, 20200422, 0, 1, True)
     ''')
 
     self.cnx.commit()

Diff for: integrations/acquisition/covidcast/test_direction_updating.py

@@ -51,6 +51,8 @@ def test_uploading(self):
     """Update rows having a stale `direction` field and serve the results."""
 
     # insert some sample data
+    # src, sig1, 1111:
+    #   direction should be updated to None as there are no historical data for (src, sig1, state).
     # CA 20200301:
     #   timeline should be x=[-2, -1, 0], y=[2, 6, 5] with direction=1
     # FL 20200517:
@@ -60,31 +62,77 @@ def test_uploading(self):
     #   wrong) is fresh
     self.cur.execute('''
       insert into covidcast values
+        (0, 'src', 'sig1', 'day', 'state', 20201028, '1111',
+          123, 2, 0, 0, 0, -1, 20201028, 0, 1, False),
+        (0, 'src', 'sig1', 'day', 'state', 20201029, '1111',
+          123, 6, 0, 0, 0, 0, 20201029, 0, 1, False),
+        (0, 'src', 'sig1', 'day', 'state', 20201030, '1111',
+          123, 5, 0, 0, 0, 1, 20201030, 0, 1, False),  
         (0, 'src', 'sig', 'day', 'state', 20200228, 'ca',
-          123, 2, 0, 0, 0, NULL, 20200228, 0, False),
+          123, 2, 0, 0, 0, NULL, 20200228, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200229, 'ca',
-          123, 6, 0, 0, 0, NULL, 20200229, 0, False),
+          123, 6, 0, 0, 0, NULL, 20200229, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200301, 'ca',
-          123, 5, 0, 0, 0, NULL, 20200301, 0, False),
+          123, 5, 0, 0, 0, NULL, 20200301, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200511, 'fl',
-          123, 1, 0, 0, 0, NULL, 20200511, 0, False),
+          123, 1, 0, 0, 0, NULL, 20200511, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200512, 'fl',
-          123, 2, 0, 0, 0, NULL, 20200512, 0, False),
+          123, 2, 0, 0, 0, NULL, 20200512, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200517, 'fl',
-          123, 2, 0, 0, 0, NULL, 20200517, 0, False),
+          123, 2, 0, 0, 0, NULL, 20200517, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200615, 'tx',
-          123, 9, 0, 0, 456, NULL, 20200615, 0, False),
+          123, 9, 0, 0, 456, NULL, 20200615, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200616, 'tx',
-          123, 5, 0, 0, 456, NULL, 20200616, 0, False),
+          123, 5, 0, 0, 456, NULL, 20200616, 0, 1, False),
         (0, 'src', 'sig', 'day', 'state', 20200617, 'tx',
-          123, 1, 0, 0, 456, 1, 20200617, 0, False)
+          123, 1, 0, 0, 456, 1, 20200617, 0, 1, False)
     ''')
     self.cnx.commit()
 
     # update direction (only 20200417 has enough history)
-    args = None
+    args = get_argument_parser().parse_args('')
     main(args)
 
+    # The Quick-Fix is working
+    response = Epidata.covidcast(
+      'src', 'sig1', 'day', 'state', '20200101-20201231', '*')
+
+    self.assertEqual(response, {
+        'result': 1,
+        'epidata': [{
+        'time_value': 20201028,
+        'geo_value': '1111',
+        'value': 2,
+        'stderr': 0,
+        'sample_size': 0,
+        'direction': None,
+        'issue': 20201028,
+        'lag': 0
+      },
+      {
+        'time_value': 20201029,
+        'geo_value': '1111',
+        'value': 6,
+        'stderr': 0,
+        'sample_size': 0,
+        'direction': None,
+        'issue': 20201029,
+        'lag': 0
+      },
+      {
+        'time_value': 20201030,
+        'geo_value': '1111',
+        'value': 5,
+        'stderr': 0,
+        'sample_size': 0,
+        'direction': None,
+        'issue': 20201030,
+        'lag': 0
+      },
+    ],
+      'message': 'success',
+    })
+
     # request data from the API
     response = Epidata.covidcast(
         'src', 'sig', 'day', 'state', '20200101-20201231', '*')
@@ -190,9 +238,9 @@ def test_uploading(self):
     # verify secondary timestamps were updated
     self.cur.execute('select direction_updated_timestamp from covidcast order by id asc')
     timestamps = [t for (t,) in self.cur]
-    for t in timestamps[:6]:
-      # first 6 rows had `direction` updated
+    for t in timestamps[:9]:
+      # first 9 rows had `direction` updated
       self.assertGreater(t, 0)
-    for t in timestamps[6:]:
+    for t in timestamps[9:]:
       # last 3 rows were not updated
       self.assertEqual(t, 456)

Diff for: integrations/acquisition/covidcast/test_fill_is_latest_issue.py

@@ -0,0 +1,94 @@
+"""Integration tests for covidcast's direction updating."""
+
+# standard library
+import unittest
+
+# third party
+import mysql.connector
+
+# first party
+from delphi.epidata.client.delphi_epidata import Epidata
+import delphi.operations.secrets as secrets
+
+# py3tester coverage target (equivalent to `import *`)
+__test_target__ = 'delphi.epidata.acquisition.covidcast.fill_is_latest_issue'
+
+
+class FillIsLatestIssueTests(unittest.TestCase):
+  """Tests filling is_latest_issue column"""
+
+  def setUp(self):
+    """Perform per-test setup."""
+
+    # connect to the `epidata` database and clear the `covidcast` table
+    cnx = mysql.connector.connect(
+        user='user',
+        password='pass',
+        host='delphi_database_epidata',
+        database='epidata')
+    cur = cnx.cursor()
+    cur.execute('truncate table covidcast')
+    cnx.commit()
+    cur.close()
+
+    # make connection and cursor available to test cases
+    self.cnx = cnx
+    self.cur = cnx.cursor()
+
+    # use the local instance of the epidata database
+    secrets.db.host = 'delphi_database_epidata'
+    secrets.db.epi = ('user', 'pass')
+
+    # use the local instance of the Epidata API
+    Epidata.BASE_URL = 'http://delphi_web_epidata/epidata/api.php'
+
+  def tearDown(self):
+    """Perform per-test teardown."""
+    self.cur.close()
+    self.cnx.close()
+
+  def test_fill_is_latest_issue(self):
+    """Update rows having a stale `direction` field and serve the results."""
+
+    self.cur.execute('''
+      insert into covidcast values
+        (0, 'src', 'sig', 'day', 'state', 20200228, 'ca',
+          123, 2, 5, 5, 5, NULL, 20200228, 0, 1, False),
+        (0, 'src', 'sig', 'day', 'state', 20200228, 'ca',
+          123, 2, 0, 0, 0, NULL, 20200229, 1, 1, False),
+        (0, 'src', 'sig', 'day', 'state', 20200229, 'ca',
+          123, 6, 0, 0, 0, NULL, 20200301, 1, 1, False),
+        (0, 'src', 'sig', 'day', 'state', 20200229, 'ca',
+          123, 6, 9, 9, 9, NULL, 20200229, 0, 1, False),
+        (0, 'src', 'sig', 'day', 'state', 20200301, 'ca',
+          123, 5, 0, 0, 0, NULL, 20200303, 2, 1, False),
+        (0, 'src', 'sig', 'day', 'state', 20200301, 'ca',
+          123, 5, 5, 5, 5, NULL, 20200302, 1, 1, False),
+        (0, 'src', 'sig', 'day', 'state', 20200301, 'ca',
+          123, 5, 9, 8, 7, NULL, 20200301, 0, 1, False)
+    ''')
+    self.cnx.commit()
+
+    # fill is_latest_issue
+    main()
+
+    self.cur.execute('''select * from covidcast''')
+    result = list(self.cur)
+    expected = [
+        (1, 'src', 'sig', 'day', 'state', 20200228, 'ca',
+          123, 2, 5, 5, 5, None, 20200228, 0, bytearray(b'0'), bytearray(b'0')),
+        (2, 'src', 'sig', 'day', 'state', 20200228, 'ca',
+          123, 2, 0, 0, 0, None, 20200229, 1, bytearray(b'1'), bytearray(b'0')),
+        (3, 'src', 'sig', 'day', 'state', 20200229, 'ca',
+          123, 6, 0, 0, 0, None, 20200301, 1, bytearray(b'1'), bytearray(b'0')),
+        (4, 'src', 'sig', 'day', 'state', 20200229, 'ca',
+          123, 6, 9, 9, 9, None, 20200229, 0, bytearray(b'0'), bytearray(b'0')),
+        (5, 'src', 'sig', 'day', 'state', 20200301, 'ca',
+          123, 5, 0, 0, 0, None, 20200303, 2, bytearray(b'1'), bytearray(b'0')),
+        (6, 'src', 'sig', 'day', 'state', 20200301, 'ca',
+          123, 5, 5, 5, 5, None, 20200302, 1, bytearray(b'0'), bytearray(b'0')),
+        (7, 'src', 'sig', 'day', 'state', 20200301, 'ca',
+          123, 5, 9, 8, 7, None, 20200301, 0, bytearray(b'0'), bytearray(b'0'))
+    ]
+
+    self.assertEqual(result, expected)