Skip to content
This repository was archived by the owner on Sep 11, 2023. It is now read-only.

Add Optical Flow Data Source #314

Merged
merged 213 commits into from
Dec 6, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
213 commits
Select commit Hold shift + click to select a range
e4d61c7
Start on Optical Flow data source
jacobbieker Nov 1, 2021
a294eb9
Add more to optical flow
jacobbieker Nov 1, 2021
572e247
Get previous timestep flow
jacobbieker Nov 1, 2021
b88ac47
Reorder inputs
jacobbieker Nov 1, 2021
dc6765e
Add to fake Batch
jacobbieker Nov 1, 2021
0d47d0a
Add to init
jacobbieker Nov 1, 2021
def12f5
Add to configuration
jacobbieker Nov 1, 2021
443124b
Add padding Optical Flow
jacobbieker Nov 1, 2021
2a4d9e1
Add crop center
jacobbieker Nov 2, 2021
ab30f36
Add getting number of future timesteps
jacobbieker Nov 2, 2021
c1a5f97
Update to newer format
jacobbieker Nov 3, 2021
c76f869
Remove get_batch
jacobbieker Nov 3, 2021
3bd99a1
Misc update
jacobbieker Nov 3, 2021
ee604db
Add opencv
jacobbieker Nov 3, 2021
1df138c
Fix requirements
jacobbieker Nov 3, 2021
6097286
Change to headless OpenCV
jacobbieker Nov 3, 2021
7c438b5
Fix linter errors
jacobbieker Nov 3, 2021
add230a
Add unit tests
jacobbieker Nov 3, 2021
9fd2201
Update for OpticalFlowDataSource
jacobbieker Nov 3, 2021
2293905
Make new dataarray with the predictions
jacobbieker Nov 3, 2021
c05b8f3
Return the correct DataArray
jacobbieker Nov 3, 2021
ce45ced
Update test
jacobbieker Nov 3, 2021
a3742fe
Fix tests
jacobbieker Nov 3, 2021
6340093
Fix test path
jacobbieker Nov 3, 2021
919d4d2
Minor docstring fixes
jacobbieker Nov 3, 2021
ed42e8a
Address PR comments
jacobbieker Nov 3, 2021
fa2330c
Fix from rebase
jacobbieker Nov 3, 2021
c3d1119
Add docstring
jacobbieker Nov 8, 2021
00af8b2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 9, 2021
35968b7
Readd OpticalFlowDataSource
jacobbieker Nov 9, 2021
c5e2b1b
Start adding DerivedDataSource
jacobbieker Nov 9, 2021
322897e
Simplify Optical Flow data source a bit
jacobbieker Nov 10, 2021
41dcc64
Remove making example dim in derived sources
jacobbieker Nov 10, 2021
54b4d4d
Remove tests
jacobbieker Nov 10, 2021
cbfebbc
Merge branch 'main' into jacob/optical-flow-datasource
jacobbieker Nov 10, 2021
539ea33
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
b884496
Update docstrings
jacobbieker Nov 10, 2021
ee8a5ad
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
d188a9c
Add getting which batches are needed for derived data sources
jacobbieker Nov 10, 2021
944322e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
d49781f
Auto stash before merge of "jacob/optical-flow-datasource" and "origi…
jacobbieker Nov 10, 2021
e8aea84
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
221ce7e
Auto stash before merge of "jacob/optical-flow-datasource" and "origi…
jacobbieker Nov 10, 2021
ef1eee3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
9e0d0e3
Add adding to derived data source dict
jacobbieker Nov 10, 2021
877cf80
Try to get around circular import
jacobbieker Nov 10, 2021
8a3934e
Simplify OF model
jacobbieker Nov 10, 2021
5a6953e
Remove init config netcdf
jacobbieker Nov 10, 2021
2c6f814
Change name
jacobbieker Nov 10, 2021
a45e386
Fix linting error
jacobbieker Nov 10, 2021
6ea1a48
Add in tests for OpticalFlow Data Source
jacobbieker Nov 10, 2021
e4b2f7c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
dc41d10
Fix lint
jacobbieker Nov 10, 2021
3a6f7a3
Fix names
jacobbieker Nov 10, 2021
8c03f10
Fix outside temp directory
jacobbieker Nov 10, 2021
58ee6ef
Fix numpy
jacobbieker Nov 10, 2021
d17dcfb
Change to use x_index
jacobbieker Nov 10, 2021
94f5d91
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
a80d24c
Update index name
jacobbieker Nov 10, 2021
0cb7efd
Add more history to minutes
jacobbieker Nov 10, 2021
cb20097
Fix time index
jacobbieker Nov 10, 2021
e7f3694
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
8aa0835
Reshape
jacobbieker Nov 10, 2021
955ed74
Fix shapes
jacobbieker Nov 10, 2021
4daf634
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
17bd0fb
Add to on premise config
jacobbieker Nov 10, 2021
f2203a3
Fix name
jacobbieker Nov 10, 2021
01e6376
Add assert
jacobbieker Nov 10, 2021
658f4e2
Add prints
jacobbieker Nov 10, 2021
69c2ad1
More debug
jacobbieker Nov 10, 2021
76b046c
More debug
jacobbieker Nov 10, 2021
d61633d
More debug
jacobbieker Nov 10, 2021
ee6ba3c
More debug
jacobbieker Nov 10, 2021
98930ad
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
9b00926
More debug
jacobbieker Nov 10, 2021
aba4be7
More debug
jacobbieker Nov 10, 2021
0341ae1
More debug
jacobbieker Nov 10, 2021
2e2c2b5
More debug
jacobbieker Nov 10, 2021
094d215
Remove deubg statements
jacobbieker Nov 10, 2021
65b655a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 10, 2021
5074fe6
Merge remote-tracking branch 'origin/main' into jacob/optical-flow-da…
jacobbieker Nov 10, 2021
e3c742a
Update testing of OpticalFlowDataSource in Manager
jacobbieker Nov 11, 2021
9adaf4b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
3a8875f
Do Optical Flow per channel
jacobbieker Nov 11, 2021
5b7bbeb
Update nowcasting_dataset/data_sources/data_source.py
jacobbieker Nov 11, 2021
87a5127
Switch to ProcessPoolExecuter
jacobbieker Nov 11, 2021
609736b
Address some PR comments
jacobbieker Nov 11, 2021
c5233e5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
81d5b40
Address more PR comments
jacobbieker Nov 11, 2021
b11e064
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
e286034
Address more PR comments
jacobbieker Nov 11, 2021
36f2bf5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
f8465a8
Add multi-timestep fixes
jacobbieker Nov 11, 2021
24fb0cf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
0cc39c8
Update docstring
jacobbieker Nov 11, 2021
9f9d8b0
Merge remote-tracking branch 'origin/jacob/optical-flow-datasource' i…
jacobbieker Nov 11, 2021
147c5ee
Update model
jacobbieker Nov 11, 2021
3eb38a7
Update nowcasting_dataset/data_sources/data_source.py
jacobbieker Nov 11, 2021
3980f01
Make docstring more descriptive
jacobbieker Nov 11, 2021
f2e6591
Add links to #367
jacobbieker Nov 11, 2021
77169ea
Fix tests
jacobbieker Nov 11, 2021
0768c40
Add assert, fix error
jacobbieker Nov 11, 2021
f5165ca
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
5e800b7
Fix assert
jacobbieker Nov 11, 2021
df661f3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
39aacd9
Add test for assert
jacobbieker Nov 11, 2021
dae02db
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
d523d27
Add giving different data sources
jacobbieker Nov 11, 2021
2f44b8c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 11, 2021
83ebe15
Fix error
jacobbieker Nov 11, 2021
44dfc69
Update test configs
jacobbieker Nov 12, 2021
a2e3014
Fix name
jacobbieker Nov 12, 2021
e25462a
Fix name
jacobbieker Nov 12, 2021
d50a73d
Add assert
jacobbieker Nov 12, 2021
a24fc54
Try different way of getting metadata file
jacobbieker Nov 12, 2021
7a6c1eb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
0397912
Remove NWP for now
jacobbieker Nov 12, 2021
008853a
Try metadata more
jacobbieker Nov 12, 2021
2f232e0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
7fdfb97
Use CSV instead of MetadataDataSource
jacobbieker Nov 12, 2021
1c9504e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
e8cabf1
Update tests
jacobbieker Nov 12, 2021
854c281
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
e14126a
Readd trying metadata
jacobbieker Nov 12, 2021
35447eb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
4317c34
Change dim name in metadat
jacobbieker Nov 12, 2021
8a3c287
Add metadata test
jacobbieker Nov 12, 2021
792267f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
ab8693d
Fix linter error
jacobbieker Nov 12, 2021
c3ff1cc
Rearrange metadata time
jacobbieker Nov 12, 2021
a350da2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
fa1197e
Merge remote-tracking branch 'origin/main' into jacob/optical-flow-da…
jacobbieker Nov 12, 2021
f091295
Remove adding dim index a second time
jacobbieker Nov 12, 2021
82d8cc9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2021
1115d38
Merge branch 'main' into jacob/optical-flow-datasource
jacobbieker Nov 17, 2021
6bd6de6
Refactor 'create_batches' between 'DataSource' and 'DerivedDataSource…
lenassero Nov 23, 2021
1a84aa7
add fastai::opencv-python-headless to environment.yml
JackKelly Nov 25, 2021
ea38641
merge with main
JackKelly Nov 25, 2021
aa89b88
fix circular import of Batch
JackKelly Nov 25, 2021
80dfc96
fixed all but one test failure in test_batch
JackKelly Nov 25, 2021
778053f
all tests in test_batch pass now
JackKelly Nov 25, 2021
ec92942
metadata.t0_datetime_utc should be pd.Timestamp
JackKelly Nov 25, 2021
986b6cf
fix number of channels in test. The number of timesteps is still wrong
JackKelly Nov 29, 2021
795ac36
all tests pass!
JackKelly Nov 29, 2021
8ce5495
Fix duplicate entries in log output. Fixes #446
JackKelly Nov 30, 2021
506b514
Fixed bug where Manager thought all DerivedDataSources were complete.…
JackKelly Nov 30, 2021
0303b3e
It is now creating OpticalFlow batches!
JackKelly Nov 30, 2021
ca614dc
a number of small fixes. But the opt flow predictions are still not …
JackKelly Nov 30, 2021
01a4be5
optical flow now moves image forwards. But now need much larger inpu…
JackKelly Nov 30, 2021
aa542db
making a start on removing DerivedDataSource
JackKelly Nov 30, 2021
d95fdde
finish removing DerivedDataSource
JackKelly Nov 30, 2021
8a4c213
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 30, 2021
86ec620
Tests run again. But they do not pass!
JackKelly Nov 30, 2021
bae8aaa
Merge branch 'jacob/optical-flow-datasource' of github.com:openclimat…
JackKelly Nov 30, 2021
76c1cb9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 30, 2021
148fe1e
test_optical_flow_get_example passes!
JackKelly Nov 30, 2021
7fa47ad
Merge branch 'jacob/optical-flow-datasource' of github.com:openclimat…
JackKelly Nov 30, 2021
9545dbb
tiny update
JackKelly Nov 30, 2021
1339424
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 30, 2021
f43dd1c
Merge branch 'jacob/optical-flow-datasource' of github.com:openclimat…
JackKelly Nov 30, 2021
bdaa060
Slight redesign: OpticalFlowDataSource now inherits from DataSource (…
JackKelly Dec 1, 2021
c1c5423
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 1, 2021
83c5118
manager tests pass
JackKelly Dec 1, 2021
42841c3
merge
JackKelly Dec 1, 2021
76bff4e
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 1, 2021
e468dea
prepare_ml_data.py runs again
JackKelly Dec 1, 2021
08fd7ce
Merge branch 'jacob/optical-flow-datasource' of github.com:openclimat…
JackKelly Dec 1, 2021
83e1e16
fix test_optical_flow_data_source
JackKelly Dec 1, 2021
39a15f0
border values should be -1
JackKelly Dec 1, 2021
fd66db5
improve logging when exception occurs
JackKelly Dec 2, 2021
4ba7d63
improve logging when exception occurs
JackKelly Dec 2, 2021
78ea315
More informative logging when requested region of interest steps outs…
JackKelly Dec 2, 2021
5e6a09a
prepare_ml_data.py runs! Now using input_image_size_pixels=106 and b…
JackKelly Dec 2, 2021
742d389
fix compression and dtype
JackKelly Dec 2, 2021
445705c
update docstring and tidy crop_center
JackKelly Dec 3, 2021
58a86a2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 3, 2021
e6e1dd4
fix passing crop_center too many args
JackKelly Dec 3, 2021
7af6726
Merge branch 'jacob/optical-flow-datasource' of github.com:openclimat…
JackKelly Dec 3, 2021
92f8b60
refactor
JackKelly Dec 3, 2021
ae6aa75
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 3, 2021
e772d3b
fix bug with _convert_array_to_uint8
JackKelly Dec 3, 2021
83d6aa5
Merge branch 'jacob/optical-flow-datasource' of github.com:openclimat…
JackKelly Dec 3, 2021
6175d3f
fix bug with _put_predictions_into_data_array
JackKelly Dec 3, 2021
34d5c22
update docs
JackKelly Dec 3, 2021
8523aa2
fix linter errors in optical_flow_data_source.py
JackKelly Dec 3, 2021
322bb72
tests/test_manager.py passes
JackKelly Dec 3, 2021
d961377
log the correct data_source_name if an exception occurs!
JackKelly Dec 3, 2021
6ecaa3d
All tests pass!
JackKelly Dec 3, 2021
3cedd23
add notebook to plot optical flow batches
JackKelly Dec 3, 2021
8121dab
Merge branch 'main' into jacob/optical-flow-datasource
JackKelly Dec 3, 2021
36c2c95
fix linter error with local_temp_path_to_path_object_expanduser
JackKelly Dec 3, 2021
268ed89
remove total_number_batches from DataSource.create_batches
JackKelly Dec 3, 2021
906d6ac
remove total_number_batches from assertion message
JackKelly Dec 3, 2021
cbdaf46
improve comments
JackKelly Dec 3, 2021
31a3f3c
improve comments
JackKelly Dec 3, 2021
1b36c8d
more comments!
JackKelly Dec 3, 2021
1baa27c
update nwp_size_test.yaml
JackKelly Dec 3, 2021
0de7c00
update gcp.yaml
JackKelly Dec 3, 2021
ee62079
clip fake PV data after smoothing
JackKelly Dec 3, 2021
5d9efc0
I the PV test is fixed. Not entirely sure
JackKelly Dec 3, 2021
b0ee333
revert back to using np.random.randn
JackKelly Dec 3, 2021
84055b7
no need for Manager to convert local_temp_path to Path
JackKelly Dec 3, 2021
cade179
use Path as default for local_temp_path
JackKelly Dec 3, 2021
1d9f02d
update description for local_temp_path
JackKelly Dec 3, 2021
cf4b18e
avoid divide by zero
JackKelly Dec 3, 2021
765ae1f
raise numpy errors for division
JackKelly Dec 3, 2021
36eee38
fix test_load_yaml_configuration
JackKelly Dec 3, 2021
7dd2e5e
add error log when file is not there
peterdudfield Dec 6, 2021
11ca14e
Merge branch 'main' into jacob/optical-flow-datasource
peterdudfield Dec 6, 2021
3302ed6
add doc strings
peterdudfield Dec 6, 2021
c7c0660
refactor functions - just move code to separate files
peterdudfield Dec 6, 2021
f5933ec
reduce pydantic opticalflow model - DataSourceMixin
peterdudfield Dec 6, 2021
bd25021
update
peterdudfield Dec 6, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: nowcasting_dataset
channels:
- pvlib
- conda-forge
- fastai
dependencies:
- python>=3.9
- pip
Expand All @@ -16,6 +17,7 @@ dependencies:
- xarray
- ipykernel
- h5netcdf # For opening NetCDF files from cloud buckets.
- fastai::opencv-python-headless

# Cloud & distributed compute
- gcsfs
Expand Down
Loading