Written by Junyu Sun

What is MR?

MR, or Mixed Reality, is appearing across the arts world, but its actual meaning is not well understood.  A strong beginning includes a foundational understanding of the concepts of reality and virtuality. The concept of the reality-virtuality continuum was introduced as an anthropological concept by Paul Milgram, referring to the scale of continuity between fully virtual and fully real with an application in displays.

A more intuitive presentation of this concept is shown in the figure below, starting from the left indicating "real environment," followed by "Augmented Reality (AR)," and "Augmented Virtual (AV)," ending on the right indicating "virtual environment." In this model, "Mixed Reality" includes "Augmented Reality" and "Augmented Virtual".

Most individuals have heard of virtual reality (VR).   VR uses computing devices to simulate a three-dimensional virtual world, providing the user with the simulation of visual, auditory, and other senses, creating an experience of full immersion and real presence in the space. Yet, no matter how real it feels, everything seen and heard is computer-generated and virtual – there is no reality in it.  MR may include elements of VR but is distinctly different.

MR also differs from AR.  Many smartphone apps are categorized as AR, but are often not referred to as such.  For example, some LBS (location-based services) applications are AR.  When opening an app and pointing the phone camera at a building, information about the building will appear on the phone screen, such as the name, number of floors, and other similar information. Another example of AR in apps the Memoji on the iPhone.  Memoji captures the user's head in real time, generating a cartoon-like avatar or sticker of the user over their real face on the screen. This type of application is a simple overlay the virtual content on the real content, creating the distinction from MR – there is no actual "mixed" content that is produced.

Thus, mixed reality is a combination of the real world and the virtual world through produced through a new visual environment. Virtual information is added in the MR space, and then rendered onto the world’s environment in real-time, which allows users to interact with this new, MR-enhanced world.  This allows the preservation of virtual and real components that can be freely switched, thus enriching the environment. In some ways, MR better utilizes the components of AR.  This is why AR is sometimes seen as a form of MR, even though the  technical differences are significant, especially when the development of the hardware for each is concerned.

MR Technologies

Since the immersive experience is intrinsic to MR, it is impossible to achieve true MR solely through mobile devices. Many technology companies are currently working on MR devices, such as Microsoft's HoloLens, Magic Leap's Magic Leap 1, and Meta's products. While Google Glasses and Holokit are also notable products, they are better categorized as AR devices.

HoloLens

Priced at $3,500, HoloLens is an MR head-mounted display developed by Microsoft and was first released on January 22, 2015. The second generation, HoloLens 2, was released on November 8, 2019. It achieves a mixed reality effect by superimposing certain computer-generated effects on top of the real world. Unlike VR glasses, users wearing HoloLens are able to walk freely and talk to people without worrying about crashing into walls. While wearing the MR glasses, they will track your movement and vision, thus generating the appropriate virtual objects which are projected into the user’s eyes through light. Additionally, since the device knows the user’s orientation, virtual 3D objects can be interacted with through gestures alone. However, for HoloLens to achieve its lifelike effect, the use of a vast amount of hardware is required. Various sensors track the user’s movement around the room and then create objects that can be interacted with from different angles through layers of colored lenses. The glasses look through the camera at objects in the room, so the device can learn the orientation of tables, chairs, and other objects, giving it the ability to project 3D images on the surface or interior of those objects.  For example, by projecting virtual explosives on a tabletop, the user can observe the explosives’ internal action after pressing the trigger.

Magic Leap 1

First recognized in 2016 after presenting the video below of a whale swimming through a school gym, created through MR, Magic Leap came onto the MR scene. However, Magic Leap 1 MR glasses, priced at $2,295, did not begin gaining traction and start selling until 2019.

Magic Leap uses fiber optics to project the entire Digital Light Field directly onto the retina to produce Cinematic Reality, which is how people view and register objects in the real world. In contrast, HoloLens uses a translucent glass combined with a side DLP projection for display to create these effects. This causes the image of the virtual object to always appear solid, lacking the perception of it as a real object with light effects.  Theoretically, in using this technique there is no loss of light information. However, there are downsides to this device.  It has light leakage when used, hindering the immersion of the experience.  Additionally, since the calculations to execute the Magic Leap glasses are so large, it must be connected to a small computer at all times, unlike the portable, more accessible HoloLens.

Google Glass

Google's AR glasses were released much earlier than HoloLens and Magic Leap.  Google Project Glass entered the market as early as 2012, officially open to public consumers in 2014 at the price of $1,500. This was short-lived, as the production of these glasses ceased in 2015. At the time of the product’s release, the main features advertised were video calling, map navigation, and email handling. Its intention is to work as an assistant for busy professionals. For example, the glasses can display pick-up information for DHL employees and medical records for doctors. In 2017 and 2019, Google released two iterations of the glasses marketed for enterprise use. However, after the latest software update on February 25, 2020, Google Glass only had the ability to take photos, video recordings and answer calls.   Regardless of its success, these glasses are best categorized as AR rather than MR because its functions only show virtual content on a transparent display and do not mix the virtual and reality.

 Holokit

Most MR and AR devices are exceedingly expensive. Holokit, however, is an affordablen AR experience device, only costing $36. Holokit's construction is like Google Cardboard, which is basically a cardboard headset that requires the user’s cell phone to project the screen. Unlike most products, where its software designed by the company, the code of the Holokit8 is open source, which allows other users to develop it. However, its display effect is relatively poor and does not achieve the mixture of real and virtual content, similar to the AR technology experience.

MR Application in the Arts Industry

Since the effects of MR technology depend not only on the virtual image, but also on real-world content, its use is ideal in art spaces for the enhancement of audiences’ artistic experiences. With Microsoft's aggressive promotion of HoloLens, MR has made its way into museums, music performances, art education, and more.

Museums

At the Intrepid Sea, Air & Space Museum, visitors can see the first woman of color who entered space, Dr. Mae Jemison, by putting on the HoloLens. Her life-size hologram will take visitors through the history of women in the U.S. space program while sharing her own story. To create the hologram, 106 cameras simultaneously filmed Dr. Jamison from all angles to complete the lighting information for the virtual image, which was produced by Microsoft's Mixed Reality Capture Studio for the hologram.

In addition to the Space Museum, the Egyptian Museum of Cairo invested in the Microsoft HoloLens to create a different type of interactive MR experience. The experience is more akin to being thrown into Night at the Museum, causing visitors to dodge warring charioteers and dust off ancient relics. When HoloLens is worn in the museum and the software "Museum Eye" opened, holograms of King Tutankhamun and his consort, Queen Ankhesenamun, guide visitors through the museum and speak about their lives, demonstrating how they used different kinds of tools and artifacts. Visitors can make gestures sensed by HoloLens to "take objects from the museum's collections on the shelves" and examine them in detail. In some parts of the museum, Microsoft used animation to recreate the wars that once took place in ancient Egypt, with soldiers and chariots flying around the galleries. With the HoloLens, visitors have the opportunity to become archaeologists, discovering treasures among the museum and earning credit when successful. Not only is the experience fun and exciting, but the treasure hunt improves the overall education of museum visitors. At the museum, 171 participants evaluated their experience with this technology, and more than 80% thought that the MR experience had significantly improved their visit in terms of interactivity, entertainment, and educational value.

Live Music

Octosense is mixed reality company that presented a Rihanna concert through MR by capturing all the angles of Rihanna while performing in order to create her hologram.  Using the HoloLens, users can view the hologram appear on stage and perform at what feels like a live concert experience. However, while MR provides an immersive experience, this particular use is impractical, as almost all of these features can be achieved through VR technology.

ArtiShock used MR to a fuller extent during an orchestral performance that featured Gustav Holst’s “The Planets.”  When the HoloLens is donned, the audience experiences holographic images of planets and heavenly bodies circling around the concert hall, simulating a sense of watching from the middle of the universe. The well-integrated MR technology in this context enriches the overall symphonic experience, providing immersive visuals that express the composition’s meaning. Using MR in the orchestral field also presents the opportunity to attract audiences who would otherwise be less interested in the traditional concert experience.

Arts Education

MR experiences have more inherent advantages in arts education than those presented through VR, especially in learning an instrument. Music Everywhere is a program that uses holograms to provide students with music lessons through visual instruction. HoloLens draws a virtual overlay on the keyboard, signaling to the user what to play next. After the user plays as instructed, HoloLens will gather data on the user’s performance, determining if the notes played were correct. In addition, Music Everywhere projects a virtual band playing along with the user, adding to the fun of learning.

The Difficulties and Future of MR

The future application of MR technology is unlimited, increasing the possibility of creating experiences that reflect what is seen in sci-fi films. While many practitioners believe that MR is the future, the technology lacks popularity in the consumer market.  This is because MR's display production is very complicated. The lighting conditions needed to restore or create a hologram of an object is difficult to achieve, even more so since it must be presented on a transparent screen.  In comparison, VR glasses require less advanced technology and only need to display images on two monitors. Since the application of MR scenarios must be combined with realistic environments, the content development of that specific environment is expensive and limits its use to one place. While this is feasible for industry use, the technology is not conducive for home use. Additionally, the complexities of MR makes the devices and their content exceedingly expensive. Compared to the Oculus Quest 2, which can be purchased for $300, MR devices that cost thousands of dollars are inaccessible and unnecessary to individual consumers. However, with the continuous development of MR technology, more cost-effective solutions will be found, and the future of a truly immersive experience will come.