-
-
Notifications
You must be signed in to change notification settings - Fork 7.7k
[V1] Move usage stats to worker and start logging TPU hardware #16211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 13 commits
Commits
Show all changes
58 commits
Select commit
Hold shift + click to select a range
f9d82ea
Track TPU usages in vLLM's data dashboards
dyli-google 731b68a
Merge branch 'vllm-project:main' into main
dyli-google d2d9b9e
Make the code more robust
dyli-google f168647
Merge branch 'main' of https://github.com/dyli-google/vllm
dyli-google ee00cf7
Merge branch 'vllm-project:main' into main
dyli-google 39d610f
Your descriptive message about the changes you made
dyli-google 558c60f
format
dyli-google 639f77b
use new API
dyli-google d5e7533
Merge branch 'vllm-project:main' into main
dyli-google d9b9d61
Merge branch 'vllm-project:main' into main
dyli-google 8f055c9
address Simon's comments
dyli-google 63bea36
Silence ImportError
dyli-google 25fa30b
Merge branch 'vllm-project:main' into main
dyli-google 8124c99
Merge branch 'vllm-project:main' into main
dyli-google 6a4eea4
Use torch_xla.tpu.get_tpu_type() to get TPU version
dyli-google ae2f5a6
Merge branch 'vllm-project:main' into main
dyli-google 5d2f2b6
Merge branch 'vllm-project:main' into main
dyli-google 9b3a67c
Merge branch 'vllm-project:main' into main
dyli-google 35fb26b
Merge branch 'vllm-project:main' into main
dyli-google b0912f0
Merge branch 'vllm-project:main' into main
dyli-google 88dd6c6
Merge branch 'vllm-project:main' into main
dyli-google 727bed5
Add usage to more engines
dyli-google 4f94631
Merge branch 'vllm-project:main' into main
dyli-google 619e496
fix error
dyli-google a1ae7ff
format
dyli-google 1667fab
Merge branch 'vllm-project:main' into main
dyli-google 9f725f6
Revert "format"
dyli-google b17dbc9
format
dyli-google 5286466
Merge branch 'vllm-project:main' into main
dyli-google 3bd0c9b
Use import torch_xla
dyli-google 625d21c
Merge branch 'main' of https://github.com/dyli-google/vllm
dyli-google 718729a
format
dyli-google 6e61fba
format
dyli-google 737646d
format
dyli-google 0e093cc
Merge branch 'vllm-project:main' into main
dyli-google 9940dad
Merge branch 'vllm-project:main' into main
dyli-google f825349
Try Qiliang's idea
dyli-google 7798bde
Merge branch 'vllm-project:main' into main
dyli-google bbd7f5a
Use Yarong's 2nd idea
dyli-google 5bf9f34
Merge branch 'main' into main
dyli-google 4e38e67
revert vllm/engine/async_llm_engine.py
dyli-google fc18a7a
simplify code
dyli-google cf7997a
simplify
dyli-google 3bd5730
fix typo
dyli-google 4374c3c
format
dyli-google 6829371
simplify
dyli-google 3c55fc7
silence error
dyli-google bbee546
Suppress all exceptions
dyli-google 429b6aa
format
dyli-google 8939235
remove comment
dyli-google bc284db
Merge branch 'vllm-project:main' into main
dyli-google bac067a
report usage of TPU and GPU during worker init time
dyli-google 3ad33a2
remove useless import
dyli-google 5b0ab6d
format
dyli-google 1f592e4
Merge branch 'vllm-project:main' into main
dyli-google 98e7ae0
Merge branch 'vllm-project:main' into main
dyli-google 689d343
Merge branch 'vllm-project:main' into main
dyli-google 4eea0a9
Merge branch 'vllm-project:main' into main
dyli-google File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
xm.xla_device_hw(xm.xla_device())
returnTPU
as result.Or do we want something like
v6e
,v5e
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yarongmu-google @simon-mo What do you think? I believe TPU should be OK?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Version number will be useful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Simon.
@yaochengji Do we have ways to get the version number?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can use
torch_xla.tpu.get_tpu_type()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, thanks. I just updated the code to use
torch_xla.tpu.get_tpu_type()