CHC SFTP Downloads

[rumackaaron]

Download CHC files through SFTP to allow for automation of CHC signal

[rumackaaron]

@krivard @chinandrew The script now downloads files from the CHC ftp server and generates the CHC signal from those files. I'm not sure what we want to do about testing since I don't want to push the credentials to this repo.

[krivard]

You are right not to upload them to the repo in plaintext; we'll add them to the Ansible vault instead. @korlaxxalrok can get you set up.

[krivard]

Why 1:?

[rumackaaron]

This is my super hi-tech encryption method. I store the password in plaintext as "dpasswor" and then the client connects using "password." It's almost as foolproof as RSA, just don't tell anyone!

[chinandrew]

Do we need to do this if credentials are encrypted at rest in ansible?

[krivard]

Can this be mocked out for testing?

[chinandrew]

+1

[chinandrew]

This looks good, just the open question on testing and the password obfuscation

[chinandrew]

callback is cool, didn't know of this feature in paramiko

[rumackaaron]

Sorry for the delay in creating the tests, I've never worked with mocks before. Let me know what else you'd like me to do

[chinandrew]

I'm getting the following test errors

ERROR test_load_data.py - KeyError: 'input_covid_file'
ERROR test_sensor.py - KeyError: 'input_covid_file'
ERROR test_update_sensor.py - KeyError: 'input_covid_file'

[rumackaaron]

Can you copy the current params.json.template to params.json and see if the errors go away?

[chinandrew]

Can you copy the current params.json.template to params.json and see if the errors go away?

Thanks, that did the trick. Any reason we can't just keep it in the params.json (the one in the tests folder)?

[rumackaaron]

Can you copy the current params.json.template to params.json and see if the errors go away?

Thanks, that did the trick. Any reason we can't just keep it in the params.json (the one in the tests folder)?

Theoretically no, but I assume there's some reason that params.json is included in .gitignore

[capnrefsmmat]

Mostly to prevent production params from getting into the repo. I think test params are fine to force-commit.

[chinandrew]

So I had an old params.json present which I think was causing the issue. Normal behavior will just have it read the template if a params isn't present, which I think is fine. Just fix linting and should be good to go.

[chinandrew]

Linting still raising some issues on things like line length, but I think they snuck in the previous PR and are not due to this one, so I'm fine having those fixed in a separate PR. Tests are passing.