Augmented Reality

With the coming of the social web, mobile web and Big Data, information has become plenty and easily accessible. As late as the mid-2010s, much of this information remained within the digital world of servers, computers and smartphones. Augmented Reality (AR) attempts to remedy this by bringing digital information into the real world. In essence, AR re-imagines human-computer interaction.

Augmented Reality is a term that requires our presence and interaction in the real world. At the same time, our experience of the real world is augmented by adding virtual elements created by computers. These virtual elements are typically visuals and sounds. The idea is to give us extra information to do things more efficiently and enjoyably.

Discussion

  • What's the definition of Augmented Reality?

    One possible definition of AR has three characteristics: it combines real and virtual; it's interactive in real time; and it's registered in 3D. This definition is regardless of the display hardware. It applies equally well to visuals that are optical see-throughs or rendered on monitors. A movie such as Jurassic Park is not AR since it's not interactive for viewers.

    Since this definition requires 3D integration into a real environment, 2D overlays on live video are also excluded from AR. Some writers consider this as AR as well, calling it TV AR. Augmenting live video with digital information for a sports game or a weather report would be considered TV AR.

    Others define AR in terms of the interfaces/hardware used. But conceptually, the definition is still includes merging the real and the virtual while retaining presence and interaction with the real world.

  • How is AR different from VR and MR?
    Comparing AR, VR and MR. Source: Bryksin 2018.
    Comparing AR, VR and MR. Source: Bryksin 2018.

    The acronyms AR, VR and MR refer to Augmented Reality, Virtual Reality and Mixed Reality.

    While AR experiences are rooted in the real world, VR experiences are rooted in the virtual world. In VR, the user is immersed in a synthetic world created by computer graphics. Such a virtual world may be inspired by the real world but it's also free to break the laws of physics that govern real-world interactions.

    AR and VR can also be differentiated by user presence. A person sitting in New York, visualizing the Eiffel Tower in Paris surrounds himself in VR, a world that represents the Tower and its surroundings. VR enables him to experience the Tower without going to Paris. A tourist who is already near the Tower, could use an AR app to get directions. When he is at the Tower, the AR app could display or read out relevant information while the user points his camera at the Tower.

    MR is an environment where users can interact with both real and virtual objects.

  • What are some possible applications of AR?
    Pepsi Max uses AR for advertising in an entertaining way. Source: BBC News 2014.

    Early ideas for AR were mostly in the industrial space rather than the consumer space: medical, engineering, military, robotics, etc. Due to smartphone adoption, AR in the consumer segment is gaining interest with education, gaming and advertising showing good potential.

    In education, live 3D models can help medical students visualize veins in patients. Pokémon Go and Real Strike are example AR games. GPS-based AR apps can help walkers or drivers navigate. AR can add value in the treatment of Post-Traumatic Stress Disorder. In real estate, AR can help you visualize your living room for its interior decor.

    Advertisers can give more information about a product via an interactive AR app, such as the BMW mini ad of 2008. A shopper can try out a dress or a makeup in AR before the actual purchase. AR can be used to create public awareness such as the WWF-Coca-Cola Arctic Home Campaign of 2013. Tourist spots or operators can offer self-guided AR-enabled tours. Actors can prepare for their roles in front of AR mirrors.

  • What are the different types of AR?
    Different types of AR. Source: Adapted from Reality Technologies 2018
    Different types of AR. Source: Adapted from Reality Technologies 2018 and Digit 2018.

    AR can be categorized based on the technique used:

    • Marker-based AR: This involves a camera and image/pattern recognition. Markers in the real world are recognized. They are then overlaid with virtual objects or information. This is also called Recognition-based AR.
    • Location-based AR: The use of GPS plus sensor data (in smartphones) enables apps that are location-centric such as finding relevant shops nearby or showing directions. Information is overlaid based on present location. This is also called Markerless AR.
    • Projection-based AR: This works by projecting light onto real-world surfaces. Your empty desk can have a virtual keyboard. Your palm can be lit up with a phone dialler. User interactions are detected based on changes in the projected light. This type of AR includes 3D interactive holograms.
    • Superimposition-based AR: Using object recognition, real-world objects are replaced or augmented with digital equivalents. Virtual objects are thus superimposed on the real world.
    • Outlining AR: Using object recognition, relevant features are outlined virtually. This may be seen as a specialization of superimposition-based AR.

    It's possible to combine many of these within a single app.

  • I've heard of SLAM. What is it?
    Google's Tango uses SLAM to map its environment. Source: Tabatabaie 2017.
    Google's Tango uses SLAM to map its environment. Source: Tabatabaie 2017.

    Simultaneous Localization And Mapping (SLAM) is a well-known problem in mobile robotics. A robot in an unknown environment has the twin tasks of mapping its surroundings while also determining its own location within the map.

    SLAM is being increasingly used in AR and is seen as a successor to marker-based AR. The good thing is SLAM doesn't need a marker. Multiple cameras with depth sensing technology are used in SLAM to create a map of the surroundings. Unlike marker-based AR, SLAM lacks context. GPS can be used to give context. For indoor navigation apps, SLAM maps (databases) can be used to create context. The idea is to obtain context with nothing more than visual input.

    Solutions to the SLAM problem can be combined with other techniques in AR to effectively combine real and virtual worlds. Apple's ARKit uses SLAM and so does Google's Project Tango. The latter's effort is also towards building context.

  • Isn't a QR Code the same thing as AR?

    A Quick Response (QR) code is basically a 2D barcode that's machine readable. Once scanned, say with a smartphone, it can give more information about the item to which it's attached. A marker-based AR can do this and more. QR codes have to conform to a specific layout while in AR the marker can be any object or pattern.

    QR codes were invented in the 1990s when image recognition had to be simple and fast. In comparison, AR requires more processing but it's also more flexible because any object can be a marker. Moreover, contextual information can be shown at different parts of the same object. There's no need to reduce the object to a code like in QR code. Finally, using AR apps, tracking and analytics can be applied. This is rarely done with QR codes.

    With QR codes, information is typically displayed by redirecting the user to a video, URL, registration form, etc. With AR, information is overlaid on top of real-world objects. Immersion of 3D objects in the real world is possible in AR but not so with QR codes. This is a fundamental difference.

  • What are the components of an AR system?
    AR involves a feedback loop between the user and the computer system. Source: Hollerer and Schmalstieg 2016, fig. 1.1.
    AR involves a feedback loop between the user and the computer system. Source: Hollerer and Schmalstieg 2016, fig. 1.1.

    An AR system can be said to contain the following components:

    • Tracking: Via sensors and camera, the system tracks the user's viewpoint.
    • Registration: Virtual objects must be spatially registered or anchored in the real world.
    • Visualization: Based on current location and viewpoint, the visualization of virtual objects has to be adjusted.
    • Spatial Model: This consists of both the real world and the virtual world. Both must be registered in the same coordinate system.
  • Could you mention the technical foundations of AR?

    For AR to work, virtual objects must be placed accurately in the real world. We can identify the following essentials:

    • Visual Keypoint Matching: Also referred to as Marker Detection, this requires image processing, feature extraction and marker detection. The marker's surface is determined so that virtual objects can be placed on the surface.
    • Spatial Mapping: The idea is to map the real world to a virtual model. Depth sensing is involved. The virtual model can be used to detect surfaces (walls, floors, tabletops). When virtual objects are placed, occlusion becomes important.
    • Sensing: Viewer becomes the anchor of the virtual space and content. Viewpoints are adjusted based on inputs coming from sensors: GPS, accelerometer, gyroscope, etc. Since sensing accuracy may be limited, this can be combined with visual tracking.
  • What sort of hardware is needed to realize AR?

    AR typically requires some sort of a display; camera and other sensors to enable detection and interaction; computer processing to blend the real and the virtual. Smartphones have all of these, thus lowering the cost of adoption for end users. Today, AR can be delivered as smartphone apps and easily reach a worldwide audience. Wearables including head-mounted display (HMD) or eyeglasses such as Google Glass are AR-specific hardware. HMDs can be see-through or screen based.

    To bridge real and virtual worlds, cameras and sensors are used. This information is processed to create a virtual model of the real world. For projection-based AR, miniature projectors are needed and these may be part of a headset wearable.

    A typical AR wearable would need sufficient processing power and memory, wireless connectivity and GPS. Sensors may include accelerometer, gyroscope and magnetometer to detect movements and thereby adjust the views of virtual objects. Some devices use mirrors to assist in aligning images to the viewer's eye.

    Explicit user control of AR could be via a touchpad or voice commands.

Milestones

1965

Ivan Sutherland, often called "Father of Computer Graphics", publishes an essay titled "The Ultimate Display". Here he describes ideas that are today part of AR/VR systems.

1968
World's first head-mounted display. Source: AnimaGalaxy 2018.
World's first head-mounted display. Source: AnimaGalaxy 2018.

Ivan Sutherland creates an optical see-through head-mounted display (HMD). The system uses computer-generated graphics. Sutherland calls it the The Sword of Damocles.

1990

The term Augmented Reality is coined by Boeing researcher Tom Caudell. The idea was display schematics of wire bundle assembly on a see-through HMD to assist workers in an airplane factory. By 1994, the term is increasingly used in literature.

1992
An early AR system at the U.S. Air Force. Source: Wikipedia 2018b.
An early AR system at the U.S. Air Force. Source: Wikipedia 2018b.

Louis Rosenberg develops an AR system called Virtual Fixtures at the U.S. Air Force Research Laboratory (AFRL). Because 3D graphics in the early 1990s were too slow for realistic experiences, Virtual Fixtures employed physical robots controlled by an exoskeleton worn by the user.

1994
AR could be seen as a subset of MR. Source: Milgram and Kishino 1994, fig. 1.
AR could be seen as a subset of MR. Source: Milgram and Kishino 1994, fig. 1.

Researchers Milgram and Kishino attempt to create a taxonomy for Mixed Reality (MR). They see AR as a subset of MR. AR consists of real environment augmented with virtual objects. They also define Augmented Virtuality (AV) as a virtual environment to which real objects are added.

1999

Hirokazu Kato creates ARToolkit. It's open source. It overlays computer graphics on real video. In 2009, this is ported to Adobe Flash, thus bringing AR to the web.

2003

Using ARToolkit, researchers at the Vienna University of Technology create an AR system with a Personal Digital Assistant (PDA). This system includes on-device real-time tracking that can at times be offloaded to the network via a wireless connection.

2013

Volkswagen MARTA (Mobile Augmented Reality Technical Assistance) app provides technicians assistance on a repair process.

May
2014

Google releases Google Glass to the public after about a year of beta testing. It's a head-mounted display designed as eyeglasses. It features Bluetooth connectivity to the Internet via user's smartphone, voice control, touchpad, camera and display.

2015

Microsoft announces HoloLens in January. A live demo of the same is done in April.

Jul
2016

Niantic releases a location-based smartphone AR app named Pokémon Go. It quickly becomes one of the most popular games, breaking multiple records (downloads and revenue). The game involves players visiting real-world locations to capture virtual creatures.

Sep
2017
Google Trends showing a peak in September 2017 due to Apple's WWDC event. The earlier peak of July 2016 is due to Pokémon Go. Source: Google Trends 2018.
Google Trends showing a peak in September 2017 due to Apple's WWDC event. The earlier peak of July 2016 is due to Pokémon Go. Source: Google Trends 2018.

Apple shows off capabilities of its ARKit along with its AR-capable iPhone 8 and iPhone X. Just two weeks earlier, Google launched its ARCore.

References

  1. AnimaGalaxy. 2018. "Augmented Reality - History." Accessed 2018-05-22.
  2. Azuma, Ronald T. 1997. "A Survey of Augmented Reality." Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385, August. Accessed 2018-05-21.
  3. BBC News. 2014. "Pepsi Max's Bus Shelter commercial explained." BBC News, YouTube, April 5. Accessed 2021-03-28.
  4. Bordvik, Aleksander. 2016. "Augmented Reality: The Basics." Techstars Blog, November 02. Accessed 2018-05-21.
  5. Bryksin, Gleb. 2018. "VR vs AR vs MR: Differences and Real-Life Applications." Upwork Global Inc, Accessed 2018-05-21.
  6. Burke, Dave. 2017. "ARCore: Augmented reality at Android scale." Android Developers Blog, August 29. Accessed 2018-05-24.
  7. Busel, Matthew. 2017. "The 4 Types of AR (and why you should know them)." Medium, February 07. Accessed 2018-05-21.
  8. Digit. 2018. "Different types of augmented reality." Chapter 3, FastTrack To Augmented Reality. Accessed 2018-05-22.
  9. Durrant-Whyte, Hugh and Tim Bailey. 2006. "Simultaneous Localisation and Mapping (SLAM): Part I The Essential Algorithms."
  10. Hollerer, Tobias and Dieter Schmalstieg. 2016. "Introduction to Augmented Reality." InformIT, Pearson, June 10. Accessed 2018-05-21.
  11. Intel. 2018. "Virtual Reality Vs. Augmented Reality Vs. Mixed Reality." Intel. Accessed 2018-05-21.
  12. Javornik, Ana. 2016. "The Mainstreaming of Augmented Reality: A Brief History." Harvard Business Review, October 04. Accessed 2018-05-21.
  13. Jerome, Nick. 2014. "Top Tips for Using QR Codes vs. Augmented Reality." FASTSIGNS, December 16. Accessed 2018-05-23.
  14. Kahney, Leander. 2018. "Your smartphone is ready to take augmented reality mainstream." Wired, January 4. Accessed 2018-05-23.
  15. Kovacs, Norbert. 2018. "The basics of Augmented Reality -- Interview with an AR expert." INDE, March 22, None. Accessed 2018-05-21.
  16. Kyoto Prize. 2012. "Dr. Ivan Sutherland, 'Father of Computer Graphics,' Selected as 2012 Kyoto Prize Laureate in Advanced Technology." June 22. Accessed 2018-05-22.
  17. Milgram, Paul and Fumio Kishino. 1994. "A Taxonomy of Mixed Reality Visual Displays." IEICE Trans. Inf. & Syst., vol. E77-D, no. 12, pp. 1321-1329, December. Accessed 2018-05-22.
  18. NewGenApps. 2017. "8 Examples of Augmented Reality Apps and their Successful Uses." NewGenApps, June 22. Accessed 2018-05-21.
  19. NewGenApps. 2018. "8 Unconventional Applications of VR & AR [Insights from Experts]." NewGenApps, January 19. Accessed 2018-05-21.
  20. Padmanabhan, Arvind. 2016. "Understanding AR and VR, for Beginners." The IEDF Blog, November 7. Accessed 2018-05-23.
  21. Part Time Evil. 2018. "What are the Types of Augmented Reality?" Part Time Evil, January 14. Accessed 2018-05-21.
  22. Reality Technologies. 2018. "The Ultimate Guide to Augmented Reality (AR) Technology." Reality Technologies. Accessed 2018-05-21.
  23. Statt, Nick. 2017. "Apple shows off breathtaking new augmented reality demos on iPhone 8." The Verge, September 12. Accessed 2018-05-25.
  24. Sterling, Bruce. 2009. "Augmented Reality: 'The Ultimate Display' by Ivan Sutherland, 1965." Wired, September 20. Accessed 2018-05-22.
  25. Tabatabaie, Mojtaba. 2017. "How SLAM technology is redrawing augmented reality’s battle lines." VentureBeat, July 31. Accessed 2018-05-24.
  26. Tabor, Siobhan. 2015. "AR vs. QR Codes: What's The Difference Between Them?" Printing Images, February 13. Accessed 2018-05-22.
  27. Wagner, Daniel and Dieter Schmalstieg. 2003. "First Steps Towards Handheld Augmented Reality." ISWC '03 Proceedings of the 7th IEEE International Symposium on Wearable Computers, October 21-23, pp. 127-135. Accessed 2018-05-21.
  28. Wikipedia. 2018a. "Augmented reality." Wikipedia, May 20. Accessed 2018-05-21.
  29. Wikipedia. 2018b. "Virtual fixture." Wikipedia, April 18. Accessed 2018-05-22.
  30. Wikipedia. 2018c. "Google Glass." Wikipedia, May 11. Accessed 2018-05-22.
  31. Wikipedia. 2018d. "Pokémon Go." Wikipedia, May 20. Accessed 2018-05-22.
  32. Williams II, Dennis. 2016. "The History of Augmented Reality (Infographic)." Huffington Post, May 13. Updated 2017-12-06. Accessed 2018-05-21.
  33. XinReality Wiki. 2017. "Microsoft HoloLens." December 15. Accessed 2018-05-22.
  34. iGreet. 2018. "The 5 Types of Augmented Reality." iGreet. Accessed 2018-05-21.

Further Reading

  1. Hollerer, Tobias and Dieter Schmalstieg. 2016. "Introduction to Augmented Reality." InformIT, Pearson, June 10. Accessed 2018-05-21.
  2. Javornik, Ana. 2016. "The Mainstreaming of Augmented Reality: A Brief History." Harvard Business Review, October 04. Accessed 2018-05-21.
  3. Azuma, Ronald T. 1997. "A Survey of Augmented Reality." Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385, August. Accessed 2018-05-21.

Article Stats

Author-wise Stats for Article Edits

Author
No. of Edits
No. of Chats
DevCoins
User avatar hemanthSK
4
0
1923
User avatar vikasG
1
0
64
Avatar of user arvindpdmn arvindpdmn
3
1
24
DevCoins due to articles, chats, their likes and article hits are included.
2078
Words
3
Authors
8
Edits
3
Chats
5
Likes
11K
Hits

Cite As

Devopedia. 2021. "Augmented Reality." Version 8, June 28. Accessed 2023-11-13. https://devopedia.org/augmented-reality
Contributed by
3 authors

Last updated on
2021-06-28 16:28:35
mobile visualization user experience optical real-time hardware wearable

See Also