How Mixed Reality Can Place Knowledge Exactly Where You Need It


                                                                                        Image credit:  Microsoft

In my work we're seeing a lot of evidence that spatial mapping and placement is the next big breakthrough in how we experience information.  Culturally, we're starting to become used to accessing and receiving information exactly when we need it (with a connected phone always at our fingertips, and with AI on that device providing more suggestions the better it knows us), but with the spatial location component added, we'll soon be able to access information exactly where we need it.

For the consumer space, Bill Kilday of Niantic said it well:

"…there’s a second wave of innovation coming in mapping, and it has to do with the hot new tech trend of augmented reality. Pokemon GO is sort of a true north for that whole industry as the first use case. But you can imagine all these other geographic data sets, like a Yelp or a HotelTonight. Imagine holding your phone up and getting a Yelp review of the restaurant that you’re looking at through your phone, or a hotel availability with room rates over a hotel, or a bus schedule over a bus stop."

Consumers will see rapid innovation in this space as companies realize they're no longer tethered to screens and physical signs and billboards to serve us information - they can now surround us with the information they want us to receive, in the virtual world.  And as with most innovations for the consumer space, the enterprise space will benefit as well.

Recently I was doing a demo for a customer in the Manufacturing industry - showing them the Remote Assist application on the HoloLens.  (The app allows an authenticated user to place a call to another user on their network, and share the world through their eyes.  The typical scenario is a worker in a factory who needs to troubleshoot an issue - they place the HoloLens on their head (or use the app through a mobile phone), initiate the call, and can then visually collaborate with another user to solve the problem.  Either user can mark up the virtual space (e.g. with circles or arrows), capture images and video, and place documents and other files in the space.)

One of the attendees at my demo asked - "Could it know what part of the plant I'm in, and show me just the documents I need for the equipment that's there?"

The answer - it's not a feature of that particular application today, however it is absolutely possible.

The challenge of finding information on the manufacturing floor is real and well-documented. Dave Jones at KMWorld summed up the stark current state:

"The software used in manufacturing... is now starting to look seriously dated, which isn’t surprising given that many of the applications people continue to use were designed and built in an era before mobile phones and cloud technology even existed. Consequently, the tools still in use deliver confusing interfaces that make storing and searching for files a nightmare, where maintenance is more and more complicated every year and the likelihood of a critical system failure is now more of a certainty than risk."

My demo attendee's simple question lit up a whole series of new ideas for me.  If digital files can be associated to a precise physical location, and viewed in a virtual space, why should anyone have to spend time searching for the right manual, policy, or SOP for a particular piece of equipment or process?  What if all the documentation, specs, training, issue history, and support contact info could be virtually attached to every piece of equipment in a facility?

The components for achieving this are:

  • For each location where you want to show the documents (images, videos, data, etc.), a spatial anchor or spatial locator (e.g. a QR code) that will act as the key to the documentation repository.
  • Connectivity for that location - in other words, wifi access throughout the facility or at least at the points where the documentation is needed / displayed.
  • An AR-enabled device on which to view the documents
  • An application on the device that will serve the documentation (ideally with consideration given to a way for the user to interact if there are many documents or options).  At the time of this writing it would likely be a custom application.
  • The document repository (and/or other information sources e.g. the database of the issue history)

Now that we've got the mechanism in place, how do we make sure the right documents appear in the right spaces?  What determines the most relevant document to show?  How do we security-trim the documents based on the user's level of training or privileges?

For details about the that side of the equation, let me turn this over to Mike Gilronan.  Mike discusses the knowledge management considerations for serving spatially-based files, and lays out some next steps toward making this kind of KM a reality.

The year 2020 is just a month away - it's definitely time for factories and manufacturing plants to make breakthroughs in their knowledge management.  The technology is here - are you ready to catapult your systems right over the "mobile" era to an entirely new form of interaction?


Further Reading:

KMworld:  Tech is Powering the Factory of the Future - But Also Hindering It

Forbes:  Augmented Reality And Geographic Information Show You What You Can't See In The World

Wharton:  Why Augmented Reality Is the Next Big Innovation in Mapping


How Mixed Reality Can Place Knowledge Exactly Where You Need It