Ethereal Matter: Connecting full-body haptics with VR experiences

Industrial Designer Scott Summit has designed products for Apple, Silicon Graphics and Nike, taught design at Stanford, Singularity and Carnegie Mellon, and holds 46 patents related to design + technology. He founded Ethereal Matter, Inc to address fitness and health opportunities afforded by emerging VR and gaming trends.

The world-changing potential for VR has been soundly demonstrated. It transports us out of our skin and into a reality of our choosing like no other technology has yet shown. This capability alone has driven high adoption for military and medical training, especially where effective, alternative IRL simulations prove costly or impossible. Even at the consumer level, VR has seen success in addressing challenges like the fear of heights and public speaking.

Despite the rich technological evolutions behind VR in recent years, to many it demonstrates unfinished business. The potential is palpable, but today’s offerings often qualify as a solid first step. The market has confirmed this with anticlimactic adoption rates. VR has yet to demonstrate across-the-board relevance in our daily lives or show us how life would be lacking without it. In short, it promises full immersion with real-world outcomes but mostly delivers an eyes-and-ears illusion of what we hoped would engage our entirety.

The physical world is bidirectional

The world is, by definition, a bidirectional haptic environment. That’s the best part. You interact, and it interacts back. You pedal a bike, it moves you forward. You steer and lean, and it turns under your command, while the scenery changes around you. On the water, you find the ‘sweet spot’ in your sail, lean in, and you’re windsurfing.

In VR, some of the engaging, low hanging haptics applications - stationary cycling and rowing - have succeeded in showing those bold enough to go there how the monotony of their stationary counterparts can be transformed into magical escapism, enticing us into more energetic output. Their success lies in the ability to generate a media rich experience from an otherwise anaesthetic activity. The potential is solidly demonstrated, though limited to few activities.

Current haptic solutions: The Hexapod approach

Mostly, however, the industry go-to for full-body haptic immersion, remains the ‘Hexapod’. When coupled with a visual or immersive experience, this mechanically synchronizes the 6 Degree of Freedom (6DoF) motion that confirms to your inner ear exactly what your eyes & ears expect. Add a controllable fan, and the resulting devices range from fully immersive theme park experiences to the military’s tank and helicopter trainers.

This harmony among our sensory inputs actually diminishes ‘sim sickness’ resulting from dissonance between the senses, simply by connecting the sensory dots. But the Hexapod platform was designed as a multipurpose, universal motion control device, only recently shoehorned into service for VR in order to deceive our inner ear. It leaves the remainder of our physicality mostly disengaged.

Enter full-body interactive platforms

Ethereal Matter’s MEK is an interactive motion control platform that engages the arms, legs, and balance in interactive virtual experiences, just as one would in the real world. Like the Hexapod approach, MEK marries what you expect to feel with the sensation you physically experience. Most importantly, MEK was fundamentally designed to embrace the opportunities enabled by VR, inserting the user into a rich, imaginative world by generating full-body interaction with the objects within it.

To this end, it comprises:

- Two robotic armatures that actively cover the human range of motion.

- Active foot platforms that produce haptic conditions for the legs.

- Gimbal grips on the motorized armatures, allowing the wrists to move freely and safely, while sensors detect the slightest user input

- LiDAR Motion Capture scanning of the user, enabling a richer level of teleportation into the VR scene as their first person avatar to enhance deep immersion.

How MEK achieves bidirectional haptics

Bidirectional haptics results from a combination of sensors in the end effectors (gimbal grip assembly), and code that dovetails with Unity, allowing a synergy between visual and physical experience.

For example, a user can grip what virtually appears as throttle controls on a Star Wars-style Pod Racer, and thrust them forward to accelerate. The sensors detect forward intent, the motion control guides the motion into a specified sliding throttle track on the virtual Speeder, and the user sees their own avatar hands gunning the motors. The motors rotate into position, the sound increases, they feel vibration from the startup, and the scene around them starts into motion. They turn using differential steering - pushing one throttle more than the other, like on a tractor - while banking into the turn with their legs, which perceive a buoyant and reactive “Speeder” beneath them, resembling the IRL motion dynamics of a jetski. To the user, the net effect is an entirely natural experience in a surreal environment.

The Speeder is just one example from a nearly infinite range of simulation, recreation, and training opportunities. What takes shape from here is entirely driven by game design on the robot-enabled Unity software. Entirely new sports, interactions, recreation and activity - all straight from imagination - can now become a real experience.

Powered by Unity: The core of immersive interaction

Beneath the sensors, motors, and VR lies the Unity core. It concatenates all incoming data, blends it with the gaming experience, and drives the motor controllers. Unity integrates these ingredients, with a layer of C++ as needed, and includes:

- Incoming sensor data collection

- Integration between sensor data and motion controllers

- The Unity Physics package to enhance haptic realism

- User Interface, allowing simple, graphical user interaction

- Vast, high-quality content from the Unity Asset Store

- Spatial audio to enhance the sense of depth and immersion

- VR output, enabling MEK’s headset-agnostic platform

- Universal Render Pipeline for high-quality rendering and photorealism, enhancing the sense of realism

- Multiplayer functionality for training, competition, or social interaction.

- First-person avatar integration

The built-in physics allows precise control over all motion, convincing the brain that the virtual entity has mass and inertia, as well as resistance, wind, buoyancy, or any other effect required to reify the experience. The rendering style can be realistic to fulfill training and simulation needs, or stylized like Manga, watercolor, wood block, film noir, or any other flavor that contributes to a sense of escapism. And the vast Unity Asset Store offers boundless, quality materials, effects, environments and audio to make new content creation rapid and richly imaginative. This is vital to MEK, as a constant flow of new worlds drives return engagements.

Unlocking new possibilities across industries

Fitness & Training

Ethereal allows enhancement of traditional experiences, including typically sedentary gaming. By applying bidirectional motion control, you may become Tom Cruise climbing the Burj Khalifa, flap mechanical wings to fly your steam-punk ornithopter over the Great Wall of China, or navigate a Man-of-War in the Battle of Trafalgar. You can even don a 300-foot Pacific Rim robot avatar and box your college roommate from across the world, crushing Tokyo beneath your feet.

The net effect is a truly captivating engagement simultaneously more compelling than a video game, more immersive than VR alone, and more inviting than the gym. And with the resistance turned up, those motions do become legitimate fitness - bench presses and rows, flies, squats, curls, or nearly anything one might find in a well-equipped gym. Only instead of miring you in boredom, this amps your sense of escapism and adventure, blending fitness with gaming, based entirely on the user’s choice for haptic resistance settings. Add physicality to gaming, and eSport suddenly becomes sport, recreation and fitness, with IRL results.

Since the armatures cover the human range of motion, nearly any fitness activity can be recreated and morphed into gaming experiences - or vice versa - tailored to specific demographics. Unity’s inverse kinematics determines the user’s proportions and self-scales the scene or assets, ensuring that all motion is biomechanically optimal, and adjusted for the user’s age. Add complete control over force feedback parameters, and subtle haptic interactions become an art, contributing to the magic of the overall experience. That giant robot punch starts with a resistive wind-up, followed by the full body force required to drive the upper cut home. You feel the shudder of impact, see the opponent’s armor erupt in sparks and smoke, and hear the cacophony as your adversary flies backward. All haptic cues ultimately derive from Unity’s visual, kinematic, auditory and physics-enabled code, resulting in a richly immersive experience.

Physical Therapy

Unity’s multiplayer functionality transforms the platform into a complete sports, fitness & training solution, allowing cohorts to compete or collaborate regardless of physical location. This results in team sports with authentic fitness, and training where the entire body takes part in the activity. Immersive eSports becomes a social, physically demanding, full-body experience. Colleges may compete in international, intramural competition, regardless of their equipment, location, weather, travel budget, and time available. With MEK and Unity, a catch-up call with Mom can become an hour of paddleboarding in the canals of Venice.

With MEK and Unity, physical therapy (PT) transforms from tedium to enjoyment, increasing the compliance while optimizing outcomes. Here, Unity’s inverse kinematic models - combined with the LiDAR-driven onboard scanning - enable close motion monitoring, as if a vigilant Physical Therapist were present. The virtual motion paths are defined by specific vectors, scaled to the patient’s proportions, and designed with the exact resistance for optimized results. A layer of gamification addresses the most common shortcoming of PT: lack of compliance due to its inherent monotony. Suddenly, rowing a magic carpet with friends down the ancient Nile becomes the best way to fix your shoulder impingement.

All of this is powered by the marriage of Unity and robotics through the platforms software controlled armatures and bi-directional motion.

Space training & simulation

In Space, where failure is notoriously not an option, accurate simulation is vital to mission success. Currently, Ethereal is generating environments that target the Space training & simulation market, which presents its own set of nuanced challenges that are impossible to replicate on Earth.

Specifically, by using Unity’s Physics Engine package, MEK generates complex motions and interactions in a photorealistically virtual setting, resulting in a highly accurate training environment on all levels. Unity’s high-quality, stereoscopic rendering simulates the intense glare and deep shadows in Space, while recreating the exact orbital dynamics of Earth, Moon, Sun and stars, traveling at high speed, seemingly around the trainee. The resulting environment familiarizes the astronaut trainee with the visual experience of space before going aloft, thereby minimizing the overall cognitive load in situ.

Physically, the Unity blends an astronaut trainee’s motion with the inherent resistance of a space suit, and with the mass-without-gravity that they will encounter while moving heavy objects in the actual zero-gravity environment. The same motion-control code also allows the astronaut to move themselves weightlessly about the exterior of the space station, hand over hand, carefully balancing the eccentric inertia resulting from the astronaut and 250lb space suit in motion.

Unity's built-in physics enables accurate simulation of astronaut motion, rotation, and multi body collisions in zero G. Unity's simple and performant implementation of rigid body physics and colliders, and even inertia tensors fundamentally enables modeling EVA motion. The inertia tensors specifically allows MEK to simulate the difference in spin between axial and sagittal spin in space. And on a more nuanced level, MEK uses a manual center of mass assignment to simulate the eccentric mass of a spacesuit’s back-heavy life support system. Specifically, MEK’s code uses the ‘add force at position’ to avoid calculating the resultant torque, thus simplifying the movement code.

Such fidelity in immersive training leads to vastly improved familiarity and muscle memory around the complex motion dynamics to be encountered in space, long before leaving the Earth.

The future of immersive technology

MEK stands to expand the physical dimension and depth to the sensory illusion initiated by today’s VR headsets. Driven by Unity’s integrated technologies, MEK coalesces a wide range of data - AI, motion control, sensor fusion, gaming - into unique and deeply immersive experiences ranging from fitness to social engagement, to complex training. At MEK’s core, Unity generates the experience by harmonizing the sensory inputs and outputs, resulting in the magic of deep immersion and a true escape to another world.