Mercedes-Benz mixed-reality HoloLens connects techs with remote experts

Oct. 30, 2020
In September, Mercedes-Benz announced the nationwide rollout to its 383 U.S. dealerships of the industry’s first mixed reality automotive maintenance system, Mercedes-Benz Virtual Remote Support.

When the Iron Man movie came out in 2008, Tony Stark’s holographic computer seemed to be far in the future. With a wave of his hands, our protagonist could rapidly grab and scroll through holographic data screens seemingly floating at arm’s length. It’s not just the stuff of fiction anymore.

In September, Mercedes-Benz announced the nationwide rollout to its 383 U.S. dealerships of the industry’s first mixed reality automotive maintenance system, Mercedes-Benz Virtual Remote Support. Developed in partnership with Microsoft and piloted in 13 dealerships, it uses the tech company’s HoloLens 2 headset and Dynamics 365 Remote Assist software.

The system allows onsite dealership technicians to work hands-free, sharing real-time views and sounds of the vehicle while talking with Mercedes-Benz technical specialists in Jacksonville, Fla. Mercedes-Benz describes the technology as an “immersive experience where they both can view intricate 3D images and holograms, see where changes need to be made, annotate the visual information, add documents, insert arrows, circles and more, all to highlight areas on which to focus.”

Prior to the system’s introduction, after exhausting onsite resources, a dealership technician would call a remote specialist, but if further assistance was required for a particularly complex issue, it would sometimes require an onsite visit, and of course a wait.

Those in the pilot program reported a “dramatic reduction in wait times for technical assistance,” according to the company. Additionally, the launch of the technology comes at a time when social distancing is encouraged and travel is minimized.

“This is a massive shift in the way we do business – helping us to serve our customers more quickly – and is especially timely with the new realities of COVID-19 and our desire to keep employees safe,” said Christian Treiber, Vice President of Customer Services at MBUSA. “Today’s vehicles feature more than 100 million lines of software code. Through our partnership with Microsoft, we have a new paradigm for technology support and communication that helps our dealers and technicians master the complexity of these vehicles while eliminating travel time and onsite visits. It’s like having an expert on your shoulder.”

A spokesperson for Microsoft Azure Edge Communications – IoT & Mixed Reality, said the HoloLens 2 was designed to “provide an experience as innate to humans as possible. Interaction models for HoloLens 2 include voice control, direct hand manipulation, and gaze.”

Microsoft’s HoloLens 2 and Dynamics 365 have been used together in a number of industries to allow an onsite technician to consult a remote expert. The quality of the images displayed on the technician’s headset depend on the brightness of ambient lighting and even the user’s eye calibration, and the content displayed.

As used by Mercedes-Benz Virtual Remote Support, dealership technicians can also download images, schematics, and repair instructions to the HoloLens 2, which are then projected in 2D onto the headset while they are not on a live call with a Mercedes-Benz USA field technical specialist.

Technology has applications for collision repair
The most common application of the technology is for mechanical problems, but there are cases for its use in collision repair, too.

“The HoloLens 2 headsets used for Mercedes-Benz Virtual Remote Support allow our field specialists to annotate the live image in front of a technician’s eyes,” said a spokesperson for the Mercedes-Benz USA Engineering Services Team. “This makes the HoloLens 2 particularly useful in the identification of hard-to-find sensors or wiring, along with any visible trouble areas in the repair area.

 “For example, if there are two similarly designed sensors located in close proximity, our field specialists can circle the correct sensor to address. The HoloLens 2’s live imaging can also allow our field specialists to perform up-close visual inspections of worn parts such as wiring harnesses to determine if a replacement is warranted. For a disassembled transmission, it can be used to determine what may be causing a malfunction or even to forensically diagnose the cause of a bearing failure.”

For collision repair, the primary use of the technology is for field technical specialists to visually diagnose any repair issues arising from a collision, including collaborating on complex diagnostics.

 “However, there have also been instances where our specialists have used the HoloLens 2 headsets to guide a technician through an uncommon body repair. For example, one of our field technical specialists in Jacksonville provided live, step-by-step, guidance to help a dealership technician through the tricky process of removing and replacing a damaged ground stud, which involved carefully cutting the damaged stud from the car and welding in a new part.”

Although the company spokesperson would not comment on whether the technology would in the future be offered to independent repairers, such as Mercedes-Benz Certified Collision Centers, the company does foresee other uses for it.

“In the short term, Mercedes-Benz Virtual Remote Support will be used to provide remote support and training to Mercedes-Benz service technicians working at dealerships. Looking ahead, we will continue to seek opportunities to integrate mixed reality into our business landscape, potentially leveraging it for training and peer-to-peer collaboration.”

About the Author

Jay Sicht | Editor-in-Chief, FenderBender and ABRN

Jay Sicht is editor-in-chief of FenderBender and ABRN. He has worked in the automotive aftermarket for more than 29 years, including in a number of sales and technical support roles in paint/parts distribution and service/repair. He has a bachelor's degree in journalism from the University of Central Missouri with a minor in aviation, and as a writer and editor, he has covered all segments of the automotive aftermarket for more than 20 of those years, including formerly serving as editor-in-chief of Motor Age and Aftermarket Business World. Connect with him on LinkedIn.

Sponsored Recommendations

ADAS Applications: What They Are & What They Do

Learn how ADAS utilizes sensors such as radar, sonar, lidar and cameras to perceive the world around the vehicle, and either provide critical information to the driver or take...

Banking on Bigger Profits with a Heavy-Duty Truck Paint Booth

The addition of a heavy-duty paint booth for oversized trucks & vehicles can open the door to new or expanded service opportunities.

The Autel IA700: Advanced Modular ADAS is Here

The Autel IA700 is a state-of-the-art and versatile wheel alignment pre-check and ADAS calibration system engineered for both in-shop and mobile applications...

Boosting Your Shop's Bottom Line with an Extended Height Paint Booths

Discover how the investment in an extended-height paint booth is a game-changer for most collision shops with this Free Guide.