Add pytorch-labs/tokenizers into ET submodules #9074
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
larryliu0820
requested review from
jackzhxng,
iseeyuan,
cccclai and
lucylq
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9074
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 1 Cancelled JobAs of commit fba9914 with merge base 94a8bc7 ( NEW FAILURES - The following jobs have failed:
CANCELLED JOB - The following job was cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
force-pushed
the
export-D70862880
branch
from
6c4644d to
5a8d7a3
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 9, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
5a8d7a3 to
ff667fe
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 9, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
ff667fe to
42b1b48
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 9, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
42b1b48 to
2dcee58
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 9, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
2dcee58 to
ba117d2
Compare
iseeyuan
approved these changes
Mar 10, 2025
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 10, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Reviewed By: iseeyuan Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
ba117d2 to
ff88673
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 10, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Reviewed By: iseeyuan Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
ff88673 to
431cf41
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 10, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Reviewed By: iseeyuan Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
431cf41 to
9dbeb4a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 10, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Reviewed By: iseeyuan Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
9dbeb4a to
0a55c33
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
added a commit
that referenced
this pull request
Mar 10, 2025
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Reviewed By: iseeyuan Differential Revision: D70862880
larryliu0820
force-pushed
the
export-D70862880
branch
from
0a55c33 to
3791a2d
Compare
Summary: Pull Request resolved: #9074 This stack is aiming to replace `extension/llm/tokenizer` with `pytorch-labs/tokenizers` repo. This PR is adding `pytorch-labs/tokenizers` into ET submodules (`extension/llm/tokenizers`) and remove `extension/llm/third-party/` and use `extension/llm/tokenizers/third-party/` instead. Reviewed By: iseeyuan Differential Revision: D70862880
|
This pull request was exported from Phabricator. Differential Revision: D70862880 |
larryliu0820
force-pushed
the
export-D70862880
branch
from
3791a2d to
fba9914
Compare
larryliu0820
requested review from
GregoryComer and
kirklandsign
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This stack is aiming to replace
extension/llm/tokenizerwithpytorch-labs/tokenizersrepo.This PR is adding
pytorch-labs/tokenizersinto ET submodules(
extension/llm/tokenizers) and removeextension/llm/third-party/and useextension/llm/tokenizers/third-party/instead.
Differential Revision: D70862880