Skip to content

Commit c6bafc0

Browse files
authored
Finalize production setup for deploying Safegraph (#264)
* Add updated encrypted credentials * Add wip_signal to production params template * Add newline * Run on 12 cores * Add shell script to run the indicator * Update run script - Use production ingestion dir - Remove hard fail to work around an issue with `cp`. - `cp` fails when files don't exist. I thought we could squash the error by sending stderr to /dev/null, but `set -eo` still catches and fails the script. Even worse, it happens silently. There should be a better way to handle this so will add it to a future task.
1 parent 3988a3b commit c6bafc0

File tree

1 file changed

+4
-5
lines changed

1 file changed

+4
-5
lines changed

safegraph/run-safegraph.sh

+4-5
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Run the Safegraph indicator
44
#
55

6-
set -eo pipefail
6+
#set -eo pipefail
77

88
# Purge the receiving directory.
99
echo "Purging ./receiving..."
@@ -14,8 +14,7 @@ echo "Running the indicator..."
1414
env/bin/python -m delphi_safegraph
1515

1616
# Copy the files to the ingestion directory.
17-
#scp $(date +"receiving/%Y%m*") delphi.midas.cs.cmu.edu:/common/covidcast/receiving/safegraph/
18-
#scp $(date --date='-1 month' +"receiving/%Y%m*") delphi.midas.cs.cmu.edu:/common/covidcast/receiving/safegraph/
1917
echo "Copying files to the ingestion directory..."
20-
cp $(date +"receiving/%Y%m*") ./test-output/safegraph 2>/dev/null # Hack to make cp care less about missing files.
21-
cp $(date --date='-1 month' +"receiving/%Y%m*") ./test-output/safegraph
18+
# Hack to make cp care less about missing recent files since we don't always have them.
19+
cp $(date +"receiving/%Y%m*") /common/covidcast/receiving/safegraph 2>/dev/null
20+
cp $(date --date='-1 month' +"receiving/%Y%m*") /common/covidcast/receiving/safegraph

0 commit comments

Comments
 (0)