Skip to content

Allow to periodically save with any function #362

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Oct 5, 2022

Conversation

basnijholt
Copy link
Member

@basnijholt basnijholt commented Sep 19, 2022

Description

This allows saving with e.g., pandas, like:

def save_dataframe(
    fname: str | list[str] | tuple[str, ...], **to_dataframe_kwargs: Any
) -> Callable[[BaseLearner], None]:
    def save(learner):
        df = learner.to_dataframe(**to_dataframe_kwargs)
        fname_df = fname_to_dataframe(fname)
        df.to_parquet(fname_df)

    return save

save_task = runner.start_periodic_saving(interval=5, method=save_dataframe("test.parquet"))

Checklist

  • Fixed style issues using pre-commit run --all (first install using pip install pre-commit)
  • pytest passed

Type of change

Check relevant option(s).

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • (Code) style fix or documentation update
  • This change requires a documentation update

@basnijholt basnijholt changed the title Allow to periodically save with pandas WIP: Allow to periodically save with pandas Sep 19, 2022
@basnijholt basnijholt changed the title WIP: Allow to periodically save with pandas Allow to periodically save with any function Oct 5, 2022
@basnijholt basnijholt merged commit 3148552 into master Oct 5, 2022
@basnijholt basnijholt deleted the periodic_saving_with_pandas branch October 5, 2022 20:04
basnijholt added a commit that referenced this pull request Oct 9, 2022
* Allow to periodically save with pandas

* Generalize saving

* add future annotations

* Fix defaults

* remove typing_extensions again

* rename
basnijholt added a commit that referenced this pull request Apr 7, 2023
* Allow to periodically save with pandas

* Generalize saving

* add future annotations

* Fix defaults

* remove typing_extensions again

* rename
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant