[Text Generation] Support for causal masks, internal KV cache, and initial testing framework #1172
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dbogunowicz
changed the title
causal mask support in the pipeline.
dbogunowicz
commented
Aug 8, 2023
bfineran
reviewed
Aug 8, 2023
Co-authored-by: Benjamin Fineran <[email protected]>
Co-authored-by: Benjamin Fineran <[email protected]>
…ttps://github.com/neuralmagic/deepsparse into feature/damian/backward_comp_no_causal_mask_models
bfineran
reviewed
Aug 9, 2023
bfineran
reviewed
Aug 9, 2023
bfineran
reviewed
Aug 9, 2023
bfineran
reviewed
Aug 9, 2023
bfineran
previously approved these changes
Aug 9, 2023
Satrat
previously approved these changes
Aug 10, 2023
Satrat
previously approved these changes
Aug 10, 2023
bfineran
previously approved these changes
Aug 10, 2023
…rk (#1163) * Create test_nl_decoder_engine.py * [Text Generation][Tests] DecoderKVCache (#1154) * [Text Generation][Tests] NLDecoderEngine (#1155) * initial commit * initial commit * [Text Generation][Tests] Text Generation Pipeline (#1162) * initial implementation * problems with multitoken prefill * almost there... * finally all tests pass * just need to change to stub * fix bad merge * Make tests work with stub (as much as possible), cleanup test names, disable heavy tests, include patch for running without causal mask * use patch from unittest library - remove additional dependency * Update tests/deepsparse/transformers/pipelines/test_text_generation.py * clarify todo comment * [Text Generation] KV Cache internal Deepsparse support (#1135) * fix kv cache * refactor * add validation pathway * avx2 support * initial commit * initial commit * initial implementation * problems with multitoken prefill * its working * almost there... * finally all tests pass * just need to change to stub * fix bad merge * added some tests * ready for review * full support --------- Co-authored-by: dbogunowicz <[email protected]> Co-authored-by: Damian <[email protected]> * incomplete string in parametrize * few nits before the merge --------- Co-authored-by: Benjamin Fineran <[email protected]> Co-authored-by: Sage Moore <[email protected]>
bfineran
changed the title
causal mask support in the pipeline.
bfineran
deleted the
feature/damian/backward_comp_no_causal_mask_models
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
INCLUDED PRs:
Includes PR #1163 which adds testing and internal KV cache support, made up of: