Improve default VCF compression #937
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Issue warning for VCF FORMAT float fields
tomwhite
force-pushed
the
improve-default-vcf-compression
branch
from
109c194 to
6260a9b
Compare
benjeffery
approved these changes
Oct 18, 2022
There was a problem hiding this comment.
Nice! One question about testing.
A thought - you could add a generic asv benchmark for compression ratio of some example VCFs:
https://asv.readthedocs.io/en/stable/writing_benchmarks.html#tracking-generic
| or "filters" not in merged_encoding[var] | ||
| ) | ||
| ): | ||
| warnings.warn( |
There was a problem hiding this comment.
Might be nice to have an explicit test for the warning with pytest.warns?
There was a problem hiding this comment.
Good idea - added
I opened #938 for this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements some of the findings in #925
I considered always rounding FORMAT float fields to 2dp, but there are problems with NaN (can't use FixedScaleOffset since it doesn't preserve NaNs) and precision (which integer type to use for FixedScaleOffset depends on the range of the values).
Quantize and Bitround do better here (both support NaN), but they are not as compact as FixedScaleOffset. It's probably worth doing more investigation into them (another PR?), but so far in this PR I have just added a warning that suggests setting filters, which is a good start.