Best practices to develop XR for industry applications

Executive summary

Over the last few decades, developers have brought industry into the digital realm. They’ve coded online platforms where people can buy or sell anything. They’ve created open libraries teaching everything from 3D CAD to raw ore smelting. They’ve produced networking tools to collaborate on design optimizations and 3D geometry, and much more. As a result, industry experts can share and learn anything with just a few clicks. But to do so, they still must connect to that digital realm.

This has led to the next stage of industry’s digital transformation. Instead of bridging industry’s physical realm to the digital one, developers intend to blend them. In essence, the digital realm is now coming directly onto the production floor via virtual reality (VR), augmented reality (AR) and mixed reality (MR) — collectively referred to as extended reality (XR). Current industry applications of these technologies include:

• VR training on digital production lines to reduce the risks to people, product and property.
• AR assisted maintenance, enabling people on the line to see how designers intend a problem to be fixed.
• MR enabled shops where customers virtually tour or try on products before purchasing.
• Industry applications of XR technologies are still in their infancy. So, industry developers may question what applications are available.

However, the recency of adding XR to industry has developers asking another question: What are the best practices to produce these experiences? That will be discussed here.

This e-book was commissioned from engineering.com.

What is XR, and how will it change industry?

Though most developers instinctively know the difference between XR technologies, it’s best to define them here to establish a baseline:

• Virtual reality (VR) is a completely digital experience; the only connections to the real world are a user’s input and display devices.
• Augmented reality (AR) is a soft-blend between the physical world and virtual realm; digital assets, information and data are overlaid on-top of real-world assets.
• Mixed reality (MR) fully blends the digital realm and physical world; digital assets interact with real world assets and vice versa.
• Extended reality (XR) encompasses all of the above.

“I would broadly define [XR] as creating immersive experiences that blur the line between the physical and digital realities,” agrees Seth Glaze, senior technical product manager on the industry team at Unity. “That includes specifically VR, or virtual reality, AR, augmented reality and then mixed reality.”

An industrial example of VR could involve a digital meeting or workshop. Here, employees from around the world simultaneously interact with a digital landscape and digital twins of real-world assets. In reality, each employee is wearing a VR headset and using an input device to interact with the virtual environment and digital twins. But within the digital realm, the employees could scale the twins to any size (to inspect or test the tiniest details) or simulate a product’s performance within any environment.

Another good application of VR is “immersive training [as it] is often the only way to train at scale,” says Jerome Maurey-Delaunay, senior solution architect on the industry team at Unity. “You can't take [trainees] into a manufacturing plant willy-nilly. [VR] can give [facility] access to [these] employees … getting people used to the hazards [and best practices].”

The aforementioned maintenance example, however, is a good descriptor of AR. Imagine a maintenance engineer is wearing a hardhat with a Google Lens like attachment. The display can notify them that a machine on the production line needs maintenance, direct them to where the machine is located, show them the part on the machine that is broken, and then overlay a video explaining how to fix that part.

Mixed reality takes the concept of AR’s blended reality a step further and, in so doing, often combines it with VR elements. Imagine the AR maintenance example above. Only now, the maintenance engineer is working from home with VR input devices and a headset. From the engineer’s perspective, they are traversing a digital landscape made up of the production line’s digital twin. Meanwhile, at the physical production line, a robot controlled by that engineer is physically fixing a machine.

Challenges making industrial XR experiences

With XR terms fully defined and their potential for industrial applications clear, developers will naturally start to wonder about the challenges they will face when creating these immersive experiences. Jonathan de Belle, software development manager on the industry team at Unity, explains that his team is often called to help customers adapt their data and designs for XR applications. As a result, he knows the common XR challenges developers face.

He explains that the first challenge tends to be that the customer’s 3D models are too big to be easily integrated into a digital experience. De Belle says his team often works with “very large, very detailed models … The part count is huge and each of those parts are highly detailed because [we are working with] manufacturing data. This is not traditional video game data.”

A related but different challenge comes from the large volume of metadata related to 3D models and other digital assets. Examples of this non-geometric data might include the material properties of an asset. In an XR experience, those values may be pulled and added to the digital display. Though one asset’s yield strength isn’t a lot of data, it is likely pulled from materials property data that spans the whole facility — which can be a lot of data.

“You’ve got the non-visual data as well,” agrees de Belle. “You want that data to be accessible within the application in a lot of cases. And getting that … relevant information through is also a challenge because there is a lot of it.”

This volume of data brings another challenge: XR hardware compatibility. There is no telling what device a user might have when accessing an XR experience in the wild. Thus, this vast amount of data, must be transferable and compatible with numerous software and hardware devices, including legacy CPUs, GPUs and headsets.

By extension this also means that developers face the challenge of future proofing their XR applications. Since many mobile devices, like phones, tend to have a yearly turnover between models, it’s just as important to ensure the XR experience works on future devices as well as legacy ones.

Another challenge designers face when building XR models for industry applications is that they typically need access to digital twins, proprietary data and information locked up within various company data silos. This problem is exacerbated when dealing with real-time data. Since it is important that the digital realm mirror the physical one, version control of 3D assets and various other data sources is critical.

Best practices when making XR experiences

Though XR experiences will be different, Unity has shown that many of the challenges developers face remain the same. As a result, the company has developed best practices to address many of these challenges.

Ask yourself, “What does this XR experience solve?”

Glaze notes that the first and often most important best practice when creating XR applications is to understand and define the use case and benefits at the start of development. Adding XR to industrial workflows shouldn’t be about jumping onto the latest tech trend. For example, there is no benefit of bringing every meeting into an XR environment when a video call can suffice. There is no need to force immersive experiences onto users unless it solves a tangible problem.

By defining the use case and benefit of an XR application first, developers can use this information to reign in its scope, budget and development time. It also helps them limit many of the aforementioned challenges. Once that initial XR experience is completed, then developers can look to expand its scope or build another.

The best XR experiences are tailored to the problem being solved

Once the developers understand the problem their XR experience is solving, they can better tailor it to that need. For instance, if the application is supposed to:

• Help customers try products out, then the ultimate show room is the customers’ ‘backyard.’
• Train maintenance workers, then the ultimate training room is your own equipment.
• Test product designs, then the ultimate testing grounds is your product’s natural environment.

For example, Glaze references an Australian developer that focuses on immersive VR training experiences for industrial applications. One of these experiences involved training workers to fix offshore oil rigs. The equipment runs 24/7 and costs millions of dollars to stop. Meanwhile, simulating emergency scenarios on these assets could be dangerous. Therefore, it’s difficult to train workers on the equipment in the real world.

To solve this problem, the developer created customized digital twins of real-world equipment and added them into XR applications. During training, employees now use this tool to practice solving potential issues that can arise in reality.

The developer discovered that it isn’t enough for these XR simulations to model any oil rig; it needs to model the oil rig that employee is working on. Otherwise, they may stumble during a pivotal moment while trying to translate their training knowledge to the current equipment setup.

Use the models you have

Industry is flush with data, models and 3D geometry that can be utilized within XR applications. Glaze used an example of a company developing rides and attractions for a theme park. 3D CAD models of those rides and attractions already exist. So, they can be used to speed up the development of XR applications.

Theme parks are sought after locations, but not everyone has the means or ability to visit them. This can be considered lost revenue for the park and lost opportunity for would-be visitors. Instead, the park could build immersive experiences using its 3D models to reach these fans. Now more people can access these attractions, from afar, regardless of their physical location or condition. Meanwhile, the park creates a new revenue stream improving the ROI of those attractions.

“You can leverage those assets a lot of times again,” says Glaze. “They’re defeatured at that point from a security [standpoint]. You’re not going to throw in your entire CAD model. But you can reuse those [defeatured models] to power your sales team. … These [experiences] could be things like … virtual try-ons, or you’re doing a walk-through of facilities you’ve created using these [models].”

Clean and streamline your data to work in real-time 3D

Much of the inhouse 3D data industry developers have access to comes in the form of CAD files. This makes sense, as CAD tools are designed for industrial workflows such as product design, development and manufacturing. These workflows require more information than is needed, or compatible, with XR environments. But as the previous best practice claims, this CAD data should be put to use. De Belle and Maurey-Delaunay explain that this contradiction feeds into another XR development best practice: streamline and simplify data.

“Dissociate your application from your data, externalize the data,” says Maurey-Delaunay. “That's really important because the next hardware that's going to come out, you might need to rework the data to fit within the hardware requirements of that platform. If your data is embedded into your application, it's going to create a whole [lot of work] to reopen the project [to ingest the data].”

The first way to do this is to create hierarchical level of detail (or LODing) loading systems, where the features of a 3D object are variable. Essentially, when the user sees an object from afar it’s a simple outline of the 3D model — even if that object is made of thousands of parts. As users get closer, more details are drawn, and more models are added to the experience. Eventually, enough data is brought into the digital reality that users can zoom into individual parts. De Belle refers to this as a polygon streaming method to 3D geometry.

He also explains that tools can be used to extract data from CAD models so they can be used in a more ‘gamified’ environment. This has an additional benefit of protecting intellectual property (IP). These tools can vary how much of a 3D model is abstracted based on the XR use case.

“This links back to the first [best practice],” says de Belle. “Ask yourself, ‘what are you trying to do with this data and then what parts of the data can I defeature. This is going to remove security problems, [reducing] data leaks, but on the flip side, it’s going to reduce the total weight of the data and make it much easier to process [it] to actually get it into a real-time experience.”

Future proof for new XR equipment, data sources and integrations

To optimize the ROI of an XR experience, developers need to ensure they are future proofed. This not only means that a digital twin within these applications must reflect a real-world asset in real-time, but it also means that the XR application itself is compatible with current XR hardware. To do this, developers should ensure that their data adheres to open standards. This can be difficult as standards are always evolving and not well defined for XR applications.

To combat this, Maurey-Delaunay suggests that automated systems are vital. “Having intelligence built into our tooling makes a huge difference,” he says. “It’s not just a conversion [of 3D models], its dynamic conversions that adapts to your use case … Complex assembly can see 60 updates a day, right? Because there are 30 people working on it on various parts. So having those automations is critical.”

Start small, then go wide

An XR experience doesn’t have to encompass a whole organization from its release. It can, and often should, start small to simplify development, reduce challenges and improve workflow integrations. It is much better to make an XR application that shows a maintenance engineer how to fix one part than to never release it due to feature creep. Once the experience has proven its worth, it can then be expanded to other situations.

Maurey-Delaunay notes that this once again relates back to asking what the XR experience is going to solve, as this helps define its initial scope. He says, “It's finding those small problems, solving them and then you build on that once you have proven the return on investment for that solution. You build on that to tackle more and more problems within the organization.”

How Unity can help make XR experiences

With the help of Unity, developers can build industrial XR experiences while adhering to best practices and limiting common challenges.

For instance, Unity Pixyz can be a great asset for developers looking to bring their organization’s 3D data into XR applications. The tool is able to import and convert CAD, BIM, mesh, point cloud and many 3D data formats, optimize them and manage the metadata inside to make them suitable for use in XR experiences. Unity also offers asset management tools, that make it easier for developers to find and use that data so they can build immersive experiences.

As proof, Glaze points to a Unity customer example from a Japanese construction company . The company digitized all of its assets and designs. Any machine it uses or builds can be used in any XR experience the company wishes to create. And because the company uses the Unity ecosystem, these applications can be deployed on both web browsers and native applications. The Unity ecosystem also offered an end-to-end platform that streamlines the ingestion, processing, preparation and management of the construction company’s 3D assets.

“The power of Unity is that we can help you create XR experiences that perform on a tablet, or the latest headset,” says Glaze, “but we also help you support your legacy devices, extending your budgets and the need to train employees on new equipment.”

As for other ways Unity can help developers produce XR experiences, Glaze says, “I think it comes down to, do you want to do that yourself, or do you want to have a whole team at Unity helping make sure that you're compatible with the latest device that’s about to come out. In some cases, we even get early access to hardware which helps our users be the first to deploy on the next big thing … We're also staying up to date on the latest open standards. Whether it's Meta’s latest open source project, or file formats like universal scene description (USD). We have the resources to be able to follow those and make sure that our users are staying as up to date as possible.”

Unity also offers educational resources to help developers produce industrial XR applications. These resources include trainings, consulting and community discussion boards.