Written by Katelyn Clement
What is augmented reality?
Augmented reality (AR) places digital objects and sounds into the physical world. Computer vision tracks what is in the real world through a camera feed, and digital content is then displayed realistically through a screen. This differs from virtual reality, which transports users into a digital world. AR is used in a variety of contexts, such as public safety, gaming, education, tourism, marketing, and equipment maintenance. In the past few years, both Pokémon GO and Snapchat brought AR into the limelight, but there are many more applications of the technology.
In the arts realm, AR has been used in both the classical music and popular music realms to cultivate audience engagement with visual components to the listening experience. This article provides an overview of AR as a technology and its usage in the music industry.
History of augmented reality in everyday life
1968: Sword of Damocles
The first head-mounted display was called the "Sword of Damocles.” With it, users could see computer-generated graphics through a pair of googles.
1974: Videoplace
The University of Connecticut and Myron Krueger built an AR lab called "Videoplace" that projected silhouettes that surrounded and interacted with each other. Two people in separate rooms each with a projection and screen and video camera were able to communicate through their projected images.
1990: Boeing
Boeing researchers Tom Caudell and David Mizell coined the term "augmented reality.”
1994: Dancing in Cyberspace
Julie Martin brought AR to the entertainment industry with a production called "Dancing in Cyberspace” where acrobats danced alongside virtual objects.
1998: National Football League
The NFL added a yellow line marker to broadcasted games so that fans could see where and how fast players moved across the field. It is still used today, although the technology has upgraded. Most people are so used to it that they don't realize it's augmented reality.
2009: Esquire Magazine
Esquire Magazine created the first AR magazine cover where users could scan the cover using their computer scanners and watch a feature by Robert Downy Jr. who "popped out" of the cover.
2013: Volkswagen
Volkswagen released the MARTA app, which overlaid car repair instructions for technicians.
2017: Ikea
Ikea released "IKEA Place,” which allowed consumers to preview IKEA decor in their own homes before purchasing items.
How does augmented reality work?
Augmented reality seemingly places virtual elements into the real world through a variety of elements, including computer vision and registration tools. A depth-sensing camera records visual information to add to an existing object or place. For example, Ikea first records an image of a couch before placing it into a consumer's room. Computer vision can take one image in the user's room and reference it with similar-looking objects so that the objects appear to be three-dimensional. Registration tools, such as accelerometers and motion sensors, help the computer define the space to superimpose the objects that should be around the AR user. Users then view the augmented reality content, such as a video or image, on a phone, tablet, goggles, or computer monitor.
Figure 1: The process involved in augmented reality. Source: MLSDev.
Types of augmented reality
MARKER-BASED AR
Marker-based augmented reality needs a distinct marker or image, such as a QR code or custom artwork, in order for the computer to generate information. This type of AR is inexpensive and easy to implement. It is commonly used in album covers, but users need to download specific apps to recognize the image.
MARKERLESS AR
Markerless augmented reality does not require anything specific in the real world. The camera can identify any physical space and augment it with an image. An example would be trying on virtual glasses.
LOCATION-BASED AR
Location-based augmented reality takes data from GPS and places objects in the real world based on users’ locations. A great example of this is Pokémon GO.
PROJECTOR-BASED AR
Projector-based augmented reality uses advanced projection technology to project onto specific objects. It is most commonly used to provide manufacturing instructions.
AR in the music industry
In 2016, streaming apps such as Spotify and Apple Music transformed the music industry and brought millions of songs to users’ fingertips in a legal and convenient way. Even though the music itself is much more accessible, artists struggle to personalize the listening experience. Streaming has taken away the need for fans to buy physical albums. While people used to connect with artists by buying CDs and records, they are now missing that step between streaming a song and attending a live performance. Artists are beginning to use augmented reality to increase audience engagement and make listening experiences more personal. Augmented reality can provide the bridge between the physical and digital world.
Album artwork
Streaming has taken away the need for fans to buy physical albums, so artists need to find new ways to promote sales and engage with their fans. The band Haim’s album "Women in Music Pt. III" created a sonic augmented reality experience when fans pointed their Instagram camera at the album cover. The AR experience altered both the album artwork and music at the same time. By using Instagram, Haim eliminated the need to download an external app and place more work on the listener.
Music videos
The Lego Vidiyo app and products allow children to explore legos and augmented reality by creating, directing, and sharing their own music videos. They can buy Lego Vidiyo products, build figurines and buildings, and then animate their creations via the Vidiyo app.
Startup company Wavy helps musicians enhance their music videos with augmented reality. Its goal is to enhance music listening experiences by adding three-dimensional animation and storytelling.
Music marketing
Artists can use augmented reality to make their posters more interactive.
In October 2020, Sam Smith collaborated with Spotify and created a 3D augmented reality music video to their single "Diamonds," viewable when users scanned the album art with another device.
In 2017, Michael Jackson's album "Scream" came with a bundle that featured a scannable poster.
Live shows
Adding augmented reality to live performances can personalize audience experiences as well.
The 2019 Coachella music festival transformed one of its stages into an AR experience. People could see completely new visuals by pointing their phones at the screen.
Figure 2: Interactive AR stage at the Coachella music festival. Source: VR Scout.
AR in classical music
Using augmented reality in classical music can help attract new audiences and keep retaining audiences more engaged. It can be used in a variety of arenas, including education, composition, acoustics, and performances.
Figure 3: Image of AR technology in action at Oakland Symphony. Source: Ryland Webb on Medium.
Acoustics
Augmented reality concert halls have microphones that can adapt to the medium being performed. Stage managers can dial in acoustics for a solo performance or large orchestra, and the room will adapt to the ensemble. These halls are typically dry-sounding without the microphones on, but with the microphones, they can use acoustic samples from halls from around the world. "Living Pianos" brought the concept to light and discussed a similar feature in the Indianapolis Hilbert Circle Theater.
Opera
Opera companies can use augmented reality to help audience members connect with storylines and characters. The Welsh National Opera developed an AR experience, "A Vixen's Tale," which combines a physical tunnel book stage with an augmented reality virtual fox that takes audience members down the tale of "The Cunning Little Vixen.”
The English National Opera suggests that augmented reality apps can be used to help opera singers and theatrical makeup artists using virtual makeup try-on apps. The apps can help vocalists transition into their roles and help makeup artists develop a look for each character. Audience members could also test out the apps to see how they would look if they were also opera characters.
Notation
In 2013, Google Glass wanted to replace music stands and sheet music with scrolling notes through the glasses. They discussed changing traditional music notation to one with colors and shapes instead of a staff, which could be extremely problematic for musicians. The app would also need to adjust to the conductor. They decided that this technology may take a long time to develop.
In 2019, the National Theater in London introduced "smart caption glasses" that displayed dialogue as the actors spoke. The glasses allowed audience members to read the captions and watch the stage at the same time instead of switching between reading captions above or below the stage or on a caption reader and the stage. I could see this technology adapting to operas in the near future.
Education
Augmented Reality can be used to teach a variety of instruments. At the moment, most research has been done using piano and stringed instruments.
A Pittsburgh-based start-up Music Everywhere is currently developing a program in which students can learn to play the piano by putting on a Microsoft HoloLens. The app then overlays icons onto piano keys that tells the students where to place their fingers and for how long to hold the notes.
An app called Instant Musician allows users to point a phone camera onto a piano. Virtual moving blocks, such as the ones used in Guitar Hero, Rock Band, and Dance Dance Revolution, show up on the phone's screen and tell users which notes to play and when.
Performances
Augmented reality can put classical music concerts in audience members’ living rooms. Spacial Orchestra lets users walk through their living room and hear and see how an orchestra would be placed in real life.
The Piano Guys partnered with Massive Technologies to bring Jon Schmidt to users’ home pianos or virtual ones. Users can buy an app, AR Pianist, and pay $2.99 per piece or $59.99 annually to put The Piano Guys in any location.
Strings Pavilion developed an app in which users can purchase a pass to view 360-degree concerts on their mobile devices in a variety of venues.
+ Resources
Blend Media. “5 Ways the Music Industry Is Using Augmented Reality,” December 1, 2020. https://blend.media/blog/5-ways-the-music-industry-is-using-ar.
3rockAR. “Augmented Reality Could Transform the Music Industry,” n.d. https://www.3rockar.com/augmented-reality-could-transform-music-industry/.
Berry, R. “Augmented Reality for Music.” In International Computer Music Association, 100–104, n.d.
Berry, Rodney, Mao Makino, Naoto Hikawa, and Masumi Suzuki. “The Augmented Composer Project: The Music Table.” IEEE Computer Society, 2003.
Brodeur, Michael Andor. “Beethoven’s Birthday, Virtual Holiday Parties and More Classical Streams for December.” The Washington Post (blog), December 2, 2020. https://www.washingtonpost.com/goingoutguide/music/classical-music-recommendations-december/2020/11/30/f8ca9e84-2aa7-11eb-9b14-ad872157ebc9_story.html.
Carlton, Bobby. “This App Brings Music to Life in Augmented Reality.” VR Scout (blog), April 16, 2018. https://vrscout.com/news/app-music-in-augmented-reality/.
Carson, Nick. “How Augmented Reality Gives New Life to Physical Music.” Marketing Week (blog), November 25, 2020. https://www.marketingweek.com/augmented-reality-gives-new-life-physical-music/.
Chen, Reuben. “Your 3-Minute Guide t Augmented Reality (AR): How Does It Work?” Construct Digital (blog), n.d. https://www.constructdigital.com/insight/how-does-augmented-reality-ar-work.
Chow, Jonathon, Haoyang Feng, and Robert Amor. “Music Education Using Augmented Reality with a Head Mounted Display.” In Proceedings of the Fourteenth Australasian User Interface Conference, 73–80. Adelaide, Australia, 2013. https://crpit.scem.westernsydney.edu.au/confpapers/CRPITV139Chow.pdf.
Davie, Owen. “Augmented Reality: The Music Release Format of the Future?” Hypebot (blog), December 9, 2019. https://www.hypebot.com/hypebot/2019/12/augmented-reality-the-music-release-format-of-the-future.html.
Estrin, Robert. “Augmented Reality in Concert Halls.” Living Pianos (blog), October 24, 2019. https://livingpianos.com/augmented-reality-in-concert-halls/.
Garland, Kay, Larry Parker, and Diana Perterson. “AR-Assisted Piano Improvisation.” Music Everywhere: Augmented Reality Music Learning, n.d. https://www.etc.cmu.edu/projects/music-everywhere/.
Javornik, Ana. “The Mainstreaming of Augmented Reality: A Brief History.” Harvard Business Review (blog), October 4, 2016. https://hbr.org/2016/10/the-mainstreaming-of-augmented-reality-a-brief-history.
Joehnig, Jon. “Tech Goes to the Opera with New AR Experience.” AR Post (blog), October 4, 2019. https://arpost.co/2019/10/04/tech-goes-to-the-opera-with-new-ar-experience/.
Kastrenakes, Jacob. “Analog to Digital: Cornell Conductor Tries to Modernize the Orchestra with Google Glass.” The Verge (blog), October 27, 2013. https://www.theverge.com/2013/10/27/5034818/cornell-professor-cynthia-turner-mixes-google-glass-and-classical-music.
Katz, Anthony. “Four Ways Augmented Reality Could Change the Music Industry.” Forbes (blog), March 30, 2017. https://www.forbes.com/sites/forbesagencycouncil/2017/03/30/four-ways-augmented-reality-could-change-the-music-industry/?sh=647c1b5f70da.
VRTIFY. “Key Features,” n.d. https://vrtify.com/#keyfeatures.
Liu, Zeruo, Matt Adcock, and Henry Gardner. “An Evaluation of Augmented Reality Music Notation.” In Association for Computing Machinery, 38:1–2. New York, NY, 2019. https://doi.org/10.1145/3359997.3365748.
Luca. “What Is Augmented Reality (AR) and How Does It Work?” BlippAR (blog), August 21, 2018. https://www.blippar.com/blog/2018/08/21/what-is-augmented-reality-and-how-does-augmented-reality-work.
Melnick, Kyle. “Coachella Debuts Its First Interactive AR Stage.” VR Scout (blog), April 19, 2019. https://vrscout.com/news/coachella-ar-interactive-stage/.
Mileva, Gergana. “Using Augmented Reality to Elevate the Concert Experience.” AR Post (blog), February 4, 2020. https://arpost.co/2020/02/04/augmented-reality-elevate-concert-experience/.
Millman, Ethan. “The Weeknd Is Holding an Augmented-Reality Concert Only on TikTok.” Rolling Stone (blog), June 30, 2020. https://www.rollingstone.com/pro/news/the-weekend-is-airing-an-augmented-reality-show-tiktok-1035977/.
Nojima, Takuya, and Dairoku Sekiguchi. “The SmartTool: A System for Augmented Reality of Haptics.” In Proceedings of the IEEE Virtual Reality 2002. The University of Tokyo, 2002. http://www.nojilab.org/wp-content/uploads/2016/08/IEEEVR2002SmartTool.pdf.
Poetker, Bridget. “A Brief History of Augmented Reality (+ Future Trends & Impact).” G2 (blog), August 22, 2019. https://www.g2.com/articles/history-of-augmented-reality.
Powell, Lisa. “Augmented Reality Will Transport Music Lovers to Italy in UD ArtsLIVE Concert Series.” Dayton Daily News (blog), February 3, 2021. https://www.daytondailynews.com/what-to-do/augmented-reality-will-transport-music-lovers-to-italy-in-ud-artslive-concert-series/QUEUR5QT6RARXPFJ2IH2DOCLQ4/.
Ridden, Paul. “AR App Puts a Virtual Classical Piano Jam in Your Living Room.” New Atlas (blog), October 12, 2020. https://newatlas.com/mobile-technology/ar-pianist-app-piano-guys-fur-elise/.
Serafin, Stefania, and Ali Adjorlu. “Considerations on the Use of Virtual and Augmented Reality Technologies in Music Education.” In IEEE Computer Society. Los Angeles, CA, USA, 2017. https://doi.org/10.1109/KELVAR.2017.7961562.
Soudoplatoff, Dimitri, and Pras Amandine. “Augmented Reality to Improve Orchestra Conductors’ Headphone Monitoring.” In Audio Engineering Society, 142, 2017. http://www.aes.org/e-lib/browse.cfm?elib=18598.
http://tcwav.com. “Spatial Orchestra,” n.d.
Stassen, Murray. “Universal and Lego Launch Augmented Reality Music Video Maping App, Vidiyo.” Music Business Worldwide (blog), January 26, 2021. https://www.musicbusinessworldwide.com/universal-and-lego-launch-augmented-reality-music-video-making-app-vidiyo/.
Strings Music Festival. “Strings Pavilion Augmented Reality,” n.d. https://stringsmusicfestival.com/virtual/.
Taufik, Shadine. “An Augmented-Reality Music Experience with The Changcuters.” The Jakarta Post (blog), September 17, 2020. https://www.thejakartapost.com/life/2020/09/17/an-augmented-reality-music-experience-with-the-changcuters.html.
TCW, AV. Spatial Orchestra: Walk the Orchestra in Augmented Reality. Youtube, n.d. https://www.youtube.com/watch?v=vKuTPWuj68I.
Techslang. “How Does Augmented Reality Work?” Techslang (blog), October 18, 2020. https://www.techslang.com/how-does-augmented-reality-work/.
Thielen, Elisabeth, and Julien Letellier. “Bringing a Virtual String Quartet to Life.” AfriCHI’18, December 2018, 1–4. https://doi.org/10.1145/3283458.3283477.
Vasudevan, Keshav. “Augmented Reality Is Now Accessible to Everyone. Is the Music Industry Ready to Take Advantage of It?” Medium (blog), May 4, 2019. https://medium.com/product-dev-stories/augmented-reality-is-now-accessible-to-everyone-5e74a364e81f.
Wang, Wen. “The Role of Augmented Reality Audio.” Jabil (blog), 2021. https://www.jabil.com/blog/sound-in-augmented-and-virtual-reality.html.
West Claremont Center for Music and the Arts. “WCCMA Virtual Concert Series,” n.d. https://www.wcc-ma.org/virtualconcerts/#.YJWA7S2caCM.
Webb, Ryland. “Augmented Reality and the Oakland Symphony.” Ryland Webb (blog), October 5, 2017.
The Franklin Institute. “What Is Augmented Reality?,” n.d. https://www.fi.edu/what-is-augmented-reality.
Assemblr Blog. “When AR Meets Music, They Can Create These Magic,” November 9, 2020. https://blog.assemblrworld.com/augmented-reality-in-music/.
Wise, Brian. “Symphonies in Space: Orchestras Embrace Virtual Reality.” The Guardian (blog), August 8, 2016. https://www.theguardian.com/music/2016/aug/08/symphonies-in-space-orchestra-virtual-reality.
Younes, Georges, Rany Kahil, Mayssa Jallad, Daniel Asmar, Imad Elhajj, George Turkiyyah, and Howayda Al-Harithy. “Virtual and Augmented Reality for Rich Interaction with Cultural Heritage Sites: A Case Study from the Roman Theater at Byblos.” Digital Applications in Archaeology and Cultural Heritage 5, no. June 2017 (June 2017): 1–9.