Skip to content

Update nvidia-physical-ai.md #2750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 18, 2025
Merged

Update nvidia-physical-ai.md #2750

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 1 addition & 31 deletions nvidia-physical-ai.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,23 +43,6 @@ The model is built by training individual ControlNets separately for each sensor
</div>
<p style="text-align: center; font-style: italic;">Input types include 3D bounding box map, Trajectory map, Depth map, Segmentation map.</p>

| ![Cosmos Transfer input types](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/physical-ai/3D_bounding_box_map.gif) |
| :--: |
| *Input types include 3D bounding box map, Trajectory map, Depth map, Segmentation map.* |

| ![Cosmos Transfer input types](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/physical-ai/trajectory_map.gif) |
| :--: |
| *Input types include 3D bounding box map, Trajectory map, Depth map, Segmentation map.* |

| ![Cosmos Transfer input types](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/physical-ai/depth_map.gif) |
| :--: |
| *Input types include 3D bounding box map, Trajectory map, Depth map, Segmentation map.* |

| ![Cosmos Transfer input types](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/physical-ai/segmentation_map.gif) |
| :--: |
| *Input types include 3D bounding box map, Trajectory map, Depth map, Segmentation map.* |


- At inference time, developers can use various input types, including structured visual or geometric data such as segmentation maps, depth maps, edge maps, human motion keypoints, LiDAR scans, trajectories, HD maps, and 3D bounding boxes to guide the output.
- The control signals from each control branch are multiplied by their corresponding adaptive spatiotemporal control maps and then summed before being added to the transformer blocks of the base model.
- The generated output is photorealistic video sequences with controlled layout, object placement, and motion. Developers can control the output in multiple ways, such as preserving structure and appearance or allowing appearance variations while maintaining structure.
Expand All @@ -71,19 +54,6 @@ The model is built by training individual ControlNets separately for each sensor
</div>
<p style="text-align: center; font-style: italic;">Outputs from Cosmos Transfer varying environments and weather conditions.</p>

| ![Cosmos Transfer output](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/physical-ai/output_1.gif) |
| :--: |
| *Outputs from Cosmos Transfer varying environments and weather conditions.* |

| ![Cosmos Transfer output](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/physical-ai/output_2.gif) |
| :--: |
| *Outputs from Cosmos Transfer varying environments and weather conditions.* |

| ![Cosmos Transfer output](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/physical-ai/output_3.gif) |
| :--: |
| *Outputs from Cosmos Transfer varying environments and weather conditions.* |


Cosmos Transfer coupled with the NVIDIA Omniverse platform is driving controllable synthetic data generation for robotics and autonomous vehicle development at scale. Find more Cosmos Transfer [Examples](https://github.com/nvidia-cosmos/cosmos-transfer1#examples) on GitHub.

Cosmos Transfer [samples](https://huggingface.co/nvidia/Cosmos-Transfer1-7B-Sample-AV) built using post-training base model are also available for autonomous vehicles.
Expand Down Expand Up @@ -115,4 +85,4 @@ Check out GitHub for [Cosmos Predict](https://github.com/nvidia-cosmos/cosmos-pr

The NVIDIA Isaac GR00T-N1-2B model is available on [Hugging Face](https://huggingface.co/nvidia/GR00T-N1-2B). Sample datasets and PyTorch scripts for post-training using custom user datasets, which is compatible with the Hugging Face LeRobot format are available on [GitHub](http://github.com/NVIDIA/Isaac-GR00T). For more information about the Isaac GR00T N1 model, see the [research paper](https://research.nvidia.com/publication/2025-03_nvidia-isaac-gr00t-n1-open-foundation-model-humanoid-robots).

Follow [NVIDIA on Hugging Face](https://huggingface.co/nvidia) for more updates.
Follow [NVIDIA on Hugging Face](https://huggingface.co/nvidia) for more updates.