Skip to content

[Sampler] Vectorized sampling (simplified) #1048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Sep 23, 2023
Merged

[Sampler] Vectorized sampling (simplified) #1048

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
526ef1f
Modify _prune_hidden_states
zhuohan123 Sep 13, 2023
a810b71
Merge branch 'main' into simpler-batched-sampling
zhuohan123 Sep 14, 2023
d3a2f5e
[WIP] finish argmax and random sampling
zhuohan123 Sep 14, 2023
0f2b02c
First version
zhuohan123 Sep 14, 2023
53127ae
further simplification
zhuohan123 Sep 15, 2023
55be676
fix
zhuohan123 Sep 15, 2023
33e4ceb
fix
zhuohan123 Sep 15, 2023
e68ba9f
format
zhuohan123 Sep 15, 2023
78b27dd
Merge branch 'main' into simpler-batched-sampling
zhuohan123 Sep 15, 2023
8be02fc
add correctness test
zhuohan123 Sep 15, 2023
657ce5b
speedup with batched query of logprobs
zhuohan123 Sep 16, 2023
1b3e4ec
bug fix
zhuohan123 Sep 16, 2023
e97dcee
Fix bugs
zhuohan123 Sep 17, 2023
33d2730
format
zhuohan123 Sep 17, 2023
cfdb268
Fix
zhuohan123 Sep 17, 2023
05a13aa
Optimize the logic of max_best_of
zhuohan123 Sep 17, 2023
d8123d6
fix
zhuohan123 Sep 17, 2023
8fca640
Further optimize performance
zhuohan123 Sep 17, 2023
d3e950b
Merge branch 'main' into simpler-batched-sampling
zhuohan123 Sep 17, 2023
cfb2155
Fix small review comments
zhuohan123 Sep 21, 2023
84ac156
reduce one concat overhead
zhuohan123 Sep 21, 2023
924ff08
Fix tests
zhuohan123 Sep 21, 2023
20dfb39
fix
zhuohan123 Sep 21, 2023
e1e2540
batched apply_penalty
zhuohan123 Sep 21, 2023
1775b2d
format
zhuohan123 Sep 21, 2023
264a813
fix cached_property
zhuohan123 Sep 21, 2023
35e4928
Fix review comments
zhuohan123 Sep 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading