# Augmented Reality: Bridging digital and physical to effect behavioural change
/Hyper Island MA in Digital Management, Digital Technologies • Shirley Lai/
This paper discusses Augmented Reality (AR) and its movers and shakers, and suggests that mobile augmented reality used with gamification can be effective in transforming user behaviour towards hawker centre tray returns in Singapore.
In 1992, a Boeing researcher Tom Caudell and his colleague David Mizell first coined the phrase “augmented reality” (AR), to describe a technology that “allows a computer-produced diagram to be superimposed and stabilized on a specific position on a real-world object” (Caudell & Mizell, 1992). He envisioned this technology to support workers who were assembling wires for airplanes, enabling them to easily thread those wires with visual overlay.
AR is widely described now as “taking digital or computer generated information, whether it be images, audio, video and touch or haptic sensations and overlaying them over in a real-time environment.” (Kipper, Rampolla, & Books, 2012) This technology has been gradually gaining acceptance and visibility, moving from aviation and industrial environments into the lifestyle, tourism, education and healthcare industries, to name a few.
The method of deployment for AR was a huge factor in enabling its current mainstream success: AR was finally brought into the consumer's life when improvements in personal technology like tablets and smartphones arrived. (Metz, 2014) .
## Augmented reality
“Father of Virtual Reality (VR)” Morton Heilig invented an experience theatre called Sensorama in 1957, which “delivered visuals, sounds, vibration and smell to the viewer” (Interaction Design Foundation, 2018). “The Telesphere Mask” (The Franklin Institute) was patented by him in 1960, which was foundation to the first AR head-mounted display, “The Sword of Damocles” developed by computer scientist Ivan E. Sutherland in 1968 (Javornik, 2016).
In his talk /The Ultimate Display/ (Sutherland, 1965) and the subsequent patent (Sutherland, 1968), Sutherland envisioned what we currently know as AR to be able to “sense and interpret eye motion data”, and display objects in the space around user, describing what can be done today for the mass consumer markets with Apple's ARKit 2 and Google's AR Core technologies.
### Movers and shakers
The first fully functioning augmented reality system, Virtual Fixture was developed by Louis Rosenberg in the U.S. Air Force Armstrong Labs in 1992. (Rosenburg, 1993) His research and experiments proved that graphic overlays over the user's real world environment “reduce the mental processing” during tasks. The deployment of AR at this point was limited to hardware mounted on heads, or exoskeletons controlled by the user.
Jun Rekimoto advances the idea of a location-based marker, where a computer would be able to detect the place at which digital information is stored. His AR prototype NaviCam is one of the first systems to allow camera tracking, and is still used in today's AR applications (Kipper, Rampolla, & Books, 2012).
Total Immersion, the first AR solutions provider was founded in 2000. D'Fusion was their first proprietary software for AR. The company continues to research and develop AR solutions till this day.
ARToolKit was developed and released by Hirokazu Kato, via the University of Washington HIT Lab in 2001. It is an open-source AR tracking library that uses video to “calculate the real camera position and orientation relative to physical markers in real time” (ARToolKit, 2018). This development advanced the possibility of AR on mobile and web browsers, and is still used today for any web-based AR (Kipper, Rampolla, & Books, 2012)
Following the invention of ARToolKit, BMW kicked off experiential marketing with their print ads. An AR model of a car “was connected to markers on the physical ad, a user was able to control the car on the screen and move it around to view different angles, simply by manipulating the piece of paper” (Javornik, 2016). This use of AR in marketing sparked experiments by brands like National Geographic and Disney in the next decade and into 2018, pushing immersive marketing beyond the browser and into the streets.
AR games were developed in this decade. The popular game Quake was developed in AR, and incorporated “a tracking system, GPS, a digital compass and vision-based marker tracking” (Kipper, Rampolla, & Books, 2012). These components are still essential for games that are developed for mobile phones, by Apple's AR Kit 2 and Google's AR Core, with the help of sensors in mobile phones. The multi-user AR abilities that AR Kit 2 and AR Core has garnered praise for can find its beginnings in this decade as well, with Schmalstieg, Reitmayr & Hesina (2003) researching and developing mobile collaboration capabilities in a shared AR space. Various mobile and game companies advanced mobile AR even further with their developments. Nokia, Mobilizy and SPRXmobile produced AR browsers and applications that make use of the multitude of sensors in mobile phones. (Kipper, Rampolla, & Books, 2012)
While various hardware and software companies like DAQRI (2010), Niantic (2010) and Magic Leap (2011) pushed the envelope of AR applications early on in the 2010s, AR's biggest breakthrough came when the tech industry's largest players started developing their own hardware patents (Google Glass; Microsoft HoloLens, Intel RealSense)(Miller, 2017), and acquired AR/VR startups. Apple bought PrimeSense, an infra-red depth-sensing engine and Metaio, a AR software developer kit company, while Facebook acquired Oculus and 12 other AR / VR companies, indicating the importance of AR / VR in future consumer products and platforms (Pagan Research, 2017).
Early iterations of AR were “not cost-effective or sufficiently intuitive enough to be launched broadly and to have the potential of being adopted by average consumers” (Javornik, 2016). However the release of Apple's ARKit 2 and Google's ARCore mobile AR platforms has enabled mainstream engagement with AR apps and services directly from the user's mobile phone, without outstanding devices. This has made AR more accessible and user friendly than ever before (Merel, 2017). Improvements in consumer mobile product hardware has also enabled higher adoption of new technology by the masses.
Pokémon Go's release in 2016 was a clear sign that mobile AR has gone mainstream: the game's geo-located AR elements can be accessed via mobile devices, and allows players to catch creatures located in the same environment as the players themselves. It recorded 21 million active daily users by the first month, surpassing Snapchat and Candy Crush Saga's peak active user count. (Kain, 2016)
The AR market size is projected to reach USD$161 billion by 2022 (Anon, 2018), in large apart due to the disruption of healthcare (The Medical Futurist, 2017) and education industries (Ang, 2017). China has also been aggressive in embracing AR innovation. In a mobile-first country like China, buy-in for the technology is been widespread, with the Chinese government and major companies like Tencent, Baidu and Alibaba investing in infrastructure, labs and ventures (Fink, 2017)
## The proposal: Gamifying hawker centre tray returns with Augmented Reality
One of Singapore's visions for hawker centres is for them to become “appealing and vibrant social spaces for the community” (Hawker Centre 3.0 Committee Report, 2017). Improving tray return rates is one way of reaching that vision. Interventions the government has taken, including technological solutions, are mostly punitive and have met with limited success, or reaped unintended consequences.
User research done in hawker centres has revealed motivations that are not addressed by current interventions. One such unaddressed motivation is peer influence. My hypothesis is that the use of augmented reality, applied in a multi-player game setting will introduce a social, delightful user experience to the process, thereby improving and cultivating tray return habits in Singaporeans.
I propose that a Tray Returns Challenge be introduced seasonally, allowing for an engaging way to encourage patrons young and old to nurture the habit of returning their trays. This challenge can be hosted inside an app, with functional features that can serve hawker centre patrons in other ways.
[image:D2D42B0F-6009-44FA-8F45-B5A4ACAE1426-4423-0000F9E156994795/Screenshot 2018-12-01 at 6.53.00 PM.png]
Clickable prototype: https://invis.io/SNPEHD8B6U2
User flow: https://overflow.io/s/UNRH0Q/
/Character illustrations © WhenIWasFour/
### App functionalities
* *Challenge:* This section can be updated yearly for a few months each year. This section can also be used for marketing prior to a challenge starting.
* *Search (Makan):* Allows users to search for hawker centres and stalls for closing and washing times. It also serves as a hawker centre directory, and can be used to notify users when their favourite stalls have completely closed, or moved to another location.
* *Events (MakanTours):* Surface events and tours the Hawker Centre Division has organised
### How the Challenge works
1. The National Tray Return Challenge kicks off for 3 months. Patrons sign up to participate
2. A patron returns their tray and gets “rewarded” with a trinket to scan with each return, which allows them to collect new characters. Each hawker centre carries a different trinket, allowing users to discover new characters throughout Singapore. Characters can be well-loved local foods sold in hawker centre. Characters, on first interaction, can also be used to congratulate the patron on doing their part to keep the Hawker centres clean.
3. Each character can interact with its environment, and other characters in the same location. For example, if /You Tiao/ (dough sticks) high-fives a related character (i.e best foods to consume together), like Soy Bean Milk, both gamers could gain bonus points
4. A player can “fill up” a Kopi /Dabao/ (takeaway) character that they have collected (i.e collect 5 of the same character), and receive a gift by a partnering local company. In this prototype, it is a local lifestyle company called WhenIWasFour
### Augmented Reality and Gamification
Augmented reality (AR) is used to enhance natural environments and can offer enriched experiences in the real world, in real time. Coupled with the mechanism of gamification, which is “the use of game design elements in non-game contexts”, the Tray Return Challenge can transform patrons' experiences in the hawker centre.
In line with the Hawker Centre Divisions' vision of transforming hawker centres into vibrant, community-based spaces, the use of AR technology and gamification strategy can:
*Change attitudes and behaviours*
Mobile AR games can “increase the intrinsic motivation of players to take action or modify their beliefs” (Oleksy & Wnuk, 2017). The emotions evoked by playing can change a player's impression of a hawker centre into one that is vibrant, fun and satisfying.
Due to the hedonic characteristics of play, studies suggest that mobile AR gaming has changed behaviours in healthcare, tourism (Zach & Tussyadiah, 2017), transport and public education (Yen, Mulley & Burke, 2018), through voluntary engagement by users (Yen, Mulley & Burke, 2018). The use of AR is a novel, interactive and immersive experience, that can heighten enjoyment, while gamification is said to trigger emotions like flow, excitement and happiness, emotions that would never be associated with tray returns. The Tray Return Challenge can also tap into human instincts like competitiveness, urgency and scarcity, which will increase engagement and social sharing.
The main goal of participating in the Tray Return Challenge might be to collect characters and prizes initially. By adding a factor of enjoyment into the process however, the predicted outcome is that players would eventually build a tray return habit.
A successful gamification case study in Singapore can be seen in Health Promotion Board's National Steps Challenge (NSC). It has proven that slowly but surely, Singaporeans can cultivate habits via friendly competition, easy-to-reach incentives and social influence. Fitness is now fun and success is easy to measure, as each Singaporean can redeem a fitness tracker. Anticipation is also high for the yearly challenge, as it only runs for a few months a year. Since the start of the challenge in 2014, NSC has progressively attracted more participants, with 700,000 registered participants in 2017. This challenge has influenced 63% of past participants into making a sustained behavioural change, continuing to maintain personal fitness goals beyond the challenge implementation dates. (Smart Nation Singapore, 2018)
*Facilitate social interaction*
The multi-player nature of geo-located AR gaming enables players on differing mobile devices and platforms to engage with their environment and others, fostering “a sense of community due to a connection with other players” (Rauschnabel, Rossmann & Dieck, 2017). In the case of the Tray Return Challenge, players can also use this opportunity to form meet up groups for game play, or enjoy the game as a day out with friends and family.
*Improve emotional, physical and mental health*
Pokemon Go is an example of how mobile AR gaming can translate into physical and emotional well-being unintentionally (Zach & Tussyadiah, 2017). The Tray Return Challenge can induce players to walk and travel to hawker centres they have never frequented, and since players are usually happy to engage in game-driven behaviour, it can lead to an uptick in physical activity. Physical activity is generally acknowledged to be beneficial to emotional and mental health (Penedo & Dahn, 2005).
### Other foreseeable benefits of this proposal
*Increase footfall for hawker centres*
In the Tray Return Challenge, collection of new characters only happens when a player returns their tray, receiving a scannable trinket. Time and location-specific characters can also be part of the game mechanics. By prompting players to visit hawker centres they have never patronised, these spaces can see increase traffic and consumption.
*Promote local tourism and local creatives*
The Tray Return Challenge provides opportunities for players to discover new hawker centres, nearby neighbouring spaces and even other locations in Singapore.
With each season of the challenge, the Hawker Centre Division can collaborate with local designers and stores for collectible items, promoting local talent on the way. Gaming tours can also be used to promote the discovery of new hawker centres.
*Create collaboration between government agencies*
The Hawker Centre Division can collaborate with agencies like the Singapore Tourism Board and Land Transport Authority on food tours. The Health Promotion Board can also be a partner in allowing players to benefit from the steps / movement data that will be collected with this hawker centre app. As well, National Heritage Board can take this opportunity to magnify the history of individual hawker centres, in line with its plan to foster a sense of belonging within Singaporeans.
*No other build-up costs necessary*
Patrons will be playing this game via their mobiles phones, therefore no other physical components would need to be installed. RFID tray return systems, which the government is trying to implement in 23 hawker centres, which would necessitate a large space and more equipment like conveyor belts.
Location data collected can be used to advise wait times, crowd size, and crowd engagement in hawker centres, which can in turn be used to improve the hawker centre app's functionalities.
### Budget and timeline
In a location-based AR game like the Tray Return Challenge, several sensors in the players's phone will be used: GPS, Wifi/mobile data and compass. Such apps will need an estimated timeframe of 400–800 hours to be developed. An additional 300 hours will also need to be factored in if developing the rest of the hawker centre app, which will include a log-in, user engagement features (sharing), map and search service, and the importing of data from Data.gov.sg.
If the hourly rate of a developer is estimated at $40/hour, the price estimate of this app will be $28,000—$44,000.
### Challenges and concerns in implementation
*User privacy and safety*
The Tray Return Challenge requires users to identify their location and consent to the use of cameras in their phones. The company building these apps will then also have access to user behaviours (i.e time spent in a certain location), habits, who they are playing with and so forth. The Hawker Centre Division has to take responsibility for this collected data and make sure that it does not fall into nefarious hands.
On the user side, the app should always get user consent and check in constantly for that, and users should have the ability to modify their privacy settings when necessary to protect themselves. Since game play happens in the physical environment, care must be taken to discourage online gaming behaviour from entering the real world: harassment and aggression are common in gaming. (Basu, 2016)
Mobile AR gaming is a seductive, persuasive method that can successfully nudge users to desired social behaviour, and in the case of this project, is it to encourage tray returns. On the flip side, AR can also be used to deceive or manipulate users without their informed consent, as companies can modify how they use collected data, or sell those data to other companies. (Cross, 2016) The Hawker Centre Division has to ensure that any changes in user terms and conditions should be surfaced to the user at all times.
There is unlimited potential in how innovation with Augmented Reality can take place. The real value of AR is in bridging the gap between digital and physical, enhancing user experiences in work and play. In the case of Singapore's tray return problem, this immersive and interactive technology is predicted to modify user behaviour into creating a sustained behavioural change.
...(download the rest of the essay above)