While cutting-edge virtual reality was hogging the spotlight, augmented reality (AR) quietly started to take hold on our everyday lives. While it might not get the same media attention, it’s arguably done the one thing virtual reality can’t seem to do — become mainstream.
Augmented reality is helping users do everything from dissecting animals (minus the actual animal) to wearing funny hats (and mustaches) during video calls. For many, they don’t even realize the app or program they’re using is AR — they just know it’s cool.
For mobile app designers, these early AR applications are just the beginning. Here are a few of the ways augmented reality is transforming mobile app design.
AR Mobile Games Enable New Social Experiences
Early visions of virtual and augmented reality were often social. Artist Myron Krueger, whose work with camera tracking presaged the creation of the multi-touch interface developed an exhibit as early as 1970 that allowed users to interact with the artist (located in another room) using a video camera and electronic drawing tablet. Using very primitive technology by modern standards, he was able to create a series of new, immersive and social experiences for attendees.
Over the following decades, researchers, and enthusiasts continued to stretch the technological capabilities of the medium — whenever a new breakthrough was made in computing, hobbyists would sit down to figure out exactly what it meant for virtual reality and augmented reality.
Graphics, physics, and on-demand rendering needed to improve in order to convincingly model an immersive world, and prices needed to come way down before virtual experiences could be accessible to everyone.
Augmented reality famously reached that point with the release of Pokémon Go. Overnight, people could use practically the entire world as a playground, traveling wherever they pleased to catch coveted Pokémon characters.
Today, the availability of VR web platforms, improved object detection and mapping, and other augmented reality breakthroughs are poised to enable new kinds of gaming, allowing you to interact with the environment — and the people — around you in new and exciting ways.
Apple’s AR game Swift Shot, created to show off the capabilities of ARkit 2.0, is a great example of what the medium is capable of. Players set up virtual blocks on any surface (in the demo, they use a wooden table), then take turns knocking down blocks with a virtual slingshot. The animations and sound effects are real enough to be immersive, and players can get competitive in a fun physics game without the danger of flying objects.
Facebook is going a different route, using augmented reality games to enhance what is already a social platform: its Messenger app.
Facebook’s games aren’t competitive — instead, they’re just designed to be fun, while also increasing engagement. That means you can dodge asteroids (or cats), engage your friends in a staring contest, or simply play a game to see who laughs first.
Mobile Augmented Reality is Pioneering AR Techniques
When devices like the Oculus Rift and the HTC Vive came onto the market, it seemed like we were on the cusp of a complete change in both how we used technology and what technology we used.
Likewise, when mixed reality devices like Google Glass and the Microsoft HoloLens came out, enthusiasts could be forgiven for believing the days of the mobile phone and tablet were numbered. After all, who wants to glance back and forth between a screen and the world, when you can have that visual data just a glance away, hands free?
Most of us, as it turns out.
We love wearables, but it’s probably going to be awhile before we spend all day wearing a headset. And it’s possible those devices will be more like tablets than phones in the long run: devices that serve an essential niche and have clear use cases, but aren’t quite convenient enough for most of us to bring everywhere.
Even if they do get more popular — it’s likely that mobile app designers will be leading the way.
As Paul Reynolds, a software developer and technology consultant, points out, most current mobile augmented reality apps are fairly simple things like stickers or “single-object, single-scene apps.” But mobile designers are rapidly taking the insights learned in simple projects to create more immersive experiences.
To illustrate the progression, Reynolds contrasts Curate and IKEA Place. Curate enables users to place a single object in a single scene (a room of their house), play with positioning, and then purchase it, if they like it.
IKEA Place takes that idea one step further. Instead of allowing users to only position one object, users of IKEA Place can place multiple items in their virtual rooms — customizing them as they see fit. Once you’re done, much like with Curate, you can purchase the items you’ve furnished your virtual room with.
According to Reynolds, as apps make the jump to multi-scene, the applications for designers suddenly become much more powerful. “Multi-object, multi-scene apps will allow designers to build compelling apps with complex interactions and information architectures. They’re still over the horizon but they will be here soon.”
Luckily, developers who want to push augmented reality forward have a whole suite of tools to choose from. 8th Wall, Google ARCore, and Apple ARKit are all viable platforms to explore.
These platforms enable users to access augmented reality applications without downloading apps, making it much easier to get curious targets to try them out. This is especially important for marketing applications and other use cases where users might be interested enough to try an app out, but not enough to commit device memory to a download.
As we’ve found with mobile app prototyping, browser-based AR has another great advantage: it’s really convenient for user testing. Users don’t have to configure anything or login, and they can seamlessly try out the application on any platform without extra work. As designers experiment with new applications and UX paradigms, that in-browser functionality will be invaluable.
Learning to be Better Designers From AR Failures
It’s hard for a designer who is new to augmented reality to avoid getting a little starry-eyed at the possibilities locked within this new medium.
Being able to create an entire new layer on top of the real world can feel like an almost supernatural power. You’re not constrained by traditional borders or boundaries or even rules of physics, and though AR objects don’t physically exist in space, they have the potential to reshape our experience of the world as if they did.
Of course, whenever you have such a new technology, you can’t expect every experiment to work — nor will every prototype live up to your vision perfectly.
Even now that the tech is there, the medium challenges the workflow, expectation, and approach of traditional designers, according to Dr. Jeffrey Lin, PhD., a Design Director at Magic Leap.
Although Lin is working on the cutting edge of AR devices — specifically, the Magic Leap One, which uses a retinal projector to seamlessly blend simulated graphics with real objects — he’s open about the challenges of creating augmented reality applications. When he works with design teams exploring VR, AR, and MR (which stands for mixed reality — a sort of always-on AR), he often hears the same frustrating stories:
“They would start with their previous design process, whether it was from making software or games; however, after they created the design documents, mocked up the wireframes, and began implementing, nothing worked because VR/AR/MR were just too new for anyone to understand what designs would or would not work. After this point, most teams concluded, ‘Let’s just start prototyping in code as soon as possible and iterate from there.’”
Unfortunately, that approach didn’t work well, either. They’d often end up “asking engineering teams to chase leads as they emerge until they hopefully arrive at a solution.” They’d spend less time designing, but more time actually getting a product to market.
To teach teams to stop wandering aimlessly until they find a solution and build a more methodical and efficient approach to AR design, Lin teaches them to think more like scientists. He has them write down predictions of whether a design will work, what factors might make it fail, and what other techniques could make it work.
They study fields as diverse as social psychology and stage magic for new ways to direct user attention and increase usability. They experiment with new mobile app prototyping techniques, such as acting out user interactions with cardboard wireframes. And perhaps most importantly, they remember that “less is more.”
“Each time [designers] think of a new novel mechanic or feature, they want to squeeze it into the current product design. At every point in a project, always take a second look at your feature list and think, ‘How much value is this really adding for the end user?’ ‘Is this something we only use once in the entire experience?’ In most cases, you are better off going with fewer features and just choosing the ones that maximize user value.”
These techniques aren’t just for AR designers — they’re for designers, period. By taking an experimental approach to AR, designers are learning to think beyond the screen and create design approaches that integrate technology more fluidly into everyday life.
That’s something mobile app designers can benefit from, even if they aren’t building for cutting edge, wearable tech.
Smart Objects Are Giving Mobile App Designers New Ways to Interface With the Real World
While augmented reality apps can overlay new objects, sounds, and information on the world around you, they can’t actually affect that world. But with smart objects, that’s starting to change. Smart objects have already proven themselves incredibly useful, doing everything from keeping your thermostat at the right temperature, to helping companies track inventory, to literally saving lives. Now, they’re beginning to empower mobile app designers to make AR applications that are truly interactive.
One app that’s pushing the limits of an interactive world is the aptly named Reality Editor, from MIT’s Fluid Interfaces Lab. The app is designed to empower users to create complex interactions between digital objects around them in ways that smart objects haven’t traditionally done.
Things like smart speakers and thermostats generally attempt to learn the user’s patterns and anticipate their desires using AI. While this can increase convenience for users, it does so by amassing huge amounts of data on users, creating privacy concerns without necessarily giving them granular control.
Reality Editor takes a different approach, allowing users to program a wide range of connected devices. For example, you could program your kettle to heat water when you switch on your bedside lamp in the morning or teach your thermostat to turn down the temperature half an hour before you get home in the evening. You can even use it to create custom programs to control Lego robotics, drones, or other robotic devices from your phone or tablet.
Reality Editor also lets you directly use big data to inform your interactions with the world. One example MIT cites is a grocery assistant that allows you to avoid specific ingredients, such as gluten or high fructose corn syrup. Reality Editor makes it possible to look at products in the supermarket through your phone, and see which ones exclude the correct ingredients, rather than spending 15 minutes squinting over the labels.
Other smart devices take a more whimsical approach to bridging the divide between the digital and physical world. One fun example is the new WRLDS smart ball, a tennis ball-sized playground object with a twist: it’s equipped with an accelerometer and Bluetooth Low Energy connectivity.
That connectivity is used to create data that can be used in a variety of digital games, based on how players interact with the ball. The developers are releasing four free games on Android and iOS, including a competitive game testing reflexes, and non-competitive fun (the ball can splash virtual colors over the playground, or make fart noises as it bounces). There’s also an SDK, so designers can come up with other novels, interactive ways to play with the smart ball.
Augmented Reality Applications Will Remake the World
We’ve only scratched the surface of what augmented reality can do. New connected devices, better technology and more insight into effective design will enhance our interactions with each other and the world around us in ways we’re only just starting to imagine. And mobile app designers will help lead the way.
Proto.io lets anyone build mobile app prototypes that feel real. No coding or design skills required. Bring your ideas to life quickly! Sign up for a free 15-day trial of Proto.io today and get started on your next mobile app design.
How has augmented reality changed your work as a mobile designer? Let us know by tweeting us @Protoio!
Originally published at blog.proto.io on October 2, 2018.
