The world of robotics is undergoing a significant transformation, driven by rapid advancements in physical AI. This evolution is accelerating the time to market for new robotic solutions, enhancing confidence in their safety capabilities, and contributing to the powering of physical AI in factories and warehouses.
Announced at GTC, Newton is an open-source, extensible physics engine developed by NVIDIA, Google DeepMind, and Disney Research to advance robot learning and development.
NVIDIA Cosmos launched as a world foundation model (WFM) platform under an open model license to accelerate physical AI development of autonomous machines such as autonomous vehicles and robots.
The latest NVIDIA Omniverse Kit SDK 107 release is another major milestone for robotics applications. It upgrades to OpenUSD version 24.05, along with updates in Python, C++, and Linux ABI. NVIDIA Isaac Sim 5.0 will be developed on top of Kit 107. These enhancements provide significant advances in sensor simulation, language support, and binary compatibility, which can greatly enhance the development of robotics applications.
Systems and tools in robot development
Whether they are industrial machines or humanoid assistants, AI-driven robots require extensive training, testing, and validation before they can be deployed in the real world. Aggregating heterogeneous data originating from specialized toolsets is the cornerstone of this process, and there are numerous methods available to facilitate the development of robots:
CAD software
Creating detailed 3D models of robots.
Digital content creation tools
Creating content such as animation and simulation.
Simulators
Testing and validating robot behaviors in response to a wide range of scenarios perceived by real sensor models in virtual environments.
Fleet management software
Coordinating and managing multiple robots in a single environment.
At the heart of every robot is its schema—its data models that define its behavior for completing tasks. Schemas serve as a common interface for storing high-level data and metadata, including the following:
- Kinematic attributes: Information about the robot’s movement and structure.
- Robot semantics: Data that defines the robot’s purpose and capabilities.
- Bodies hierarchy: The organization of the robot’s components.
- Configuration parameters: Settings that control the robot’s operation.
Data management challenges
Currently, the robotic ecosystem is disjointed, as developers have to manage multiple data sources for various areas of the robot. Different subsystems of a robot often rely on multiple fragmented data sources, which can be a time-consuming process to connect. Other challenges include the following:
Different robot formats
Robots can be designed and represented in various formats that currently lack a common bridge.
Different robot data
Robotic data types can range from CAD designs and robotic rigging to electrical and mechanical information, sensor data, and IoT metrics.
Various simulation solvers
There are multiple solvers to specialize targeted components of a robot.
To address these challenges, it is essential to consolidate all data in a unified manner, replicating real-world perception for training, optimizing, and operating robots.
The value of OpenUSD
Universal Scene Description (OpenUSD) is key for unifying content from disparate data sources and empowering large-scale simulations. Here are some of the key benefits:
- Interoperability: OpenUSD is designed to be extensible, enabling it to support existing robot data sources from CAD and URDF, imbue the simulation environment with physically accurate materials and kinematics, and transport live operational data from sensors on robots or in industrial facilities.
- Aggregation: OpenUSD uses file format plugins and references to enable cohesive datasets while maintaining a canonical source of truth.
- Collaboration: OpenUSD’s composition and non-destructive editing features enable new collaborative workflows between humans, AI, and simulators.
Introducing the OpenUSD Asset Structure Pipeline for Robotics
At GTC 2025, NVIDIA introduced the OpenUSD Asset Structure Pipeline for Robotics with Disney Research and Intrinsic AI. This new structure and data pipeline uses best practices within OpenUSD to unify robotic workflows and provide a common language for all data sources.
In Figure 2, left-to-right is workflow and right-to left is component dependencies.
The OpenUSD Asset Structure Pipeline is designed to streamline the preparation and enhancement of robotic assets for simulation and development. This workflow ensures efficient management, optimization, and enhancement while preserving the original integrity and hierarchy of the assets.
OpenUSD Asset Structure Pipeline stages
By following the OpenUSD Asset Structure Pipeline based on extract, transform, load (ETL), you can seamlessly integrate robotic assets into simulation environments, making it easier to test, refine, and deploy the assets in various applications.
Asset source
- Import assets directly into USD (converted with Omniverse CAD converter or open-source file format plug-in).
- Structure assets either as a USD layer per robot component, or as a single layer for the entire robot.
- Preserve the original organization and hierarchy of the assets to facilitate management and referencing.
Transformation
- Reorganize the asset to optimize its structure or enhance its performance.
- Ensure the asset is fully simulation-ready while maintaining its integrity.
Features
- Add simulation features to each asset.
- Compose all simulation features through payloads that can be configured via variants.
- Use a modular approach to enable various workflows and promote efficient simulation.
Key benefits
The pipeline supports transformative reshaping of the content structure from source to simulation, enabling large-scale training and optimizations for real-time simulation within runtimes such as NVIDIA Isaac Sim, a reference application for robotic simulation.
It is designed with reusability and modularity in mind, supporting current workflows and future innovations. A single interface layer configured through variant sets unifies the robot asset structure pipeline while providing the flexibility to tailor the asset to the task at hand.
The pipeline uses references to maintain a link to the CAD data source, ensuring that any updates or changes can be pushed or pulled downstream.
It enables scalable synthetic data generation to train physical AI models by transforming structured simulations into photoreal video with new Cosmos world foundation models.
Considerations
OpenUSD is extremely flexible because its composition engine is domain-agnostic. At its most foundational levels, OpenUSD composes key-value pairs without having to know what those key-value pairs mean in 3D terms. However, this also means that domain-specific considerations must be taken into account when aggregating data in OpenUSD, making OpenUSD standardization critical for ensuring interoperability and data interchangeability
One of the most important of these considerations are the scene units. When working with USD for robots in runtimes such as Isaac Sim, we recommend working with a root layer that has metersPerUnit set to meters, referencing in robots that are already in meters, and using Metrics Assembler to ensure that any environments referenced into the simulation are properly scaled if needed to be in meters. We do not recommend sublayering in any content that is not already in meters.
The Alliance for OpenUSD (AOUSD) will also be exploring more comprehensive approaches for dealing with mixed units and other stage metrics in USD layer stacks. However, such approaches must be considered carefully to meet the following requirements:
- Maintain performance
- Mitigate code and content incompatibilities
- Centralize computations of final transforms and other values affected by units
Upcoming in OpenUSD
The roadmap to evolve OpenUSD for Robotics includes the following efforts:
- Robotics format conceptual data mappings: Mapping data models from robotics formats such as URDF, MJCF, and SDFormat to OpenUSD, and posting schema proposals to fill conceptual gaps as needed.
- AOUSD announces OpenUSD expansion of Working Groups in new domains: Driving industry-specific innovation and expanded OpenUSD applications across emerging sectors including Industrial and Engineering Digital Twin (IEDT) Interest Group and Web Interest Group.
- AOUSD proposals for deformables and B-reps:
- The AOUSD Physics Working Group is proposing to add deformables (soft body dynamics) to UsdPhysics for training robot manipulators and grippers.
- The AOUSD Geometry Working Group is developing a preliminary schema for B-reps. This could be used to derive different tessellations of CAD geometry on demand for various physics solvers in the same industrial environment.
- USD + FMI proposal: Co-standardization and conceptual data mapping between OpenUSD and Functional Mock-up Interface (FMI) for co-simulation (such as multiple solvers simulating different components of the same robot or its surrounding environment).
- Dynamic Behaviors proposal: Ability to implement dynamic behaviors needed to simulate robot actions, such as those encapsulated in ROS bridges, or those needed to simulate safety zones. The proposal will identify the functional requirements of OmniGraph that would be needed to realize such dynamic behaviors on top of the forthcoming OpenExec initiative from Pixar.
