Intro to Immersive Experience Design (Part 1)
“Intro to Immersive Experience Design” is a guide for everyone who’s interested to learn more about 3D interface design and augmented reality. It is the result of over 5 years of research and exploration in the field of immersive technologies, and is intended to help you navigate through the vast, uncharted territory of this exciting new industry.
The guide consists of 3 parts:
Part 1: Why 3D interfaces are the future of UX/UI design
Part 2: The past. present, and future of the AR industry
Part 3: How to start your career as an immersive experience designer
Ready? Let’s dive into it.
Part 1: Why 3D Interfaces are the Future of UX/UI Design
In this first article, I want to share with you why I believe that augmented reality (AR) will be the next big frontier for UX/UI designers. I will talk about why the transition from 2D to 3D interfaces is inevitable, and how this change will affect our work as designers and product builders. First and foremost, my goal is to inspire you to take advantage of the unique position we are in at this very moment in time: The window of opportunity to get involved and prepare for this upcoming revolution is still wide open — and designers can play a vital role in shaping this young industry and create long-lasting impact.
How it all started
My motivation to write this series comes from a place of deeply rooted discontent with the current state of interface technology. The internet has become an omnipresent force that touches every aspect of our lives. We can access an endless amount of information from anywhere, at any time. Isn’t it rather inconvenient that our window to this vast digital world is only the size of a dollar bill, and content always stays trapped behind the window glass? Sometimes we don’t even notice it anymore, because we simply have gotten so used to it.
As a trained industrial designer, I learned to design physical objects and user interfaces long before I created my first digital product. Coming from this background, I remember being slightly frustrated by the limitations I had to work with when creating my first apps and websites: Everything needed to be reduced to 2 dimensions, and designed in a way so that it could fit neatly into a rectangle. On top of that, I always found “finger on glass” to be a deeply dissatisfying interaction paradigm. There must be something better, I often thought to myself. Something that feels closer to our nature, and the way we perceive and process information in the real world.
3 years ago, I set out on a journey to research and explore what that something might look like and soon came across immersive technologies. I ended up in a deep rabbit hole and spent hundreds of hours reading papers and articles, and experimenting with different apps and devices. The following text is my summary of all the key learnings and insights I gathered on the way — I hope this can save you some time and serve as a shortcut for everyone who’s interested in understanding this fascinating new paradigm.
Augmented Reality
Augmented reality is a technology that enhances the real world by overlaying computer-generated information on top of it. The idea itself isn’t anything new — immersive tech has been a research topic for decades. Only recently, though, has Moore’s Law made the necessary hardware and software components powerful enough, small enough, and cheap enough for it to slowly make its way out of university labs and into the mass market. Since a few years, we can experience some of the first usable applications of AR technology on our smartphones — mostly in the form of games (e.g. Pokémon Go) and face filters on social media.
While it’s exciting to see this technology gain traction on such a large scale it’s quite obvious that AR still has a long way to go to truly reach its full potential. Despite all the great innovation that is currently happening in the mobile AR space, it seems highly unlikely our phones will be able to deliver the magical fusion of digital and physical world that everyone is looking for.
Those of us who have been lucky enough to get their hands on more advanced versions of immersive tech have come much closer to the “real deal”. But while products like Magic Leap or Microsoft HoloLens can give you a much more realistic glimpse into our immersive future, they can still only be regarded as very early, buggy versions of what’s to come.
So why am I so convinced that AR has a very bright future? Let’s look at this through a designer’s lens:
Cognitive Load
The reason why I believe AR will succeed is simple: It’s all about cognitive load. The term was coined by educational psychologist John Sweller in the late 1980’s and is used to describe the total amount of mental effort that is required to complete a task. Sweller’s cognitive load theory recognizes the fact that our ability to hold information in short-time memory is limited. As a result, any task that imposes a heavy cognitive load can have severe negative effects on our ability to complete the task efficiently, and impede learning.
Our job as designers is often to reduce cognitive load for users. The easier it is for someone to navigate through your app or website and complete a task, the better. The more effortless the experience feels for users, the more likely it is for them to choose your product over that of a competitor.
The history of interfaces has seen constant reduction of cognitive load for users. Graphical user interfaces (GUIs) replaced command line interfaces not only because they looked prettier, but because they freed up a good portion of human memory space to focus on the task at hand. Command line interfaces basically required users to constantly memorize and translate control arguments while trying to get things done. GUIs on the other hand created a visual representation of the system’s underlying structure that was in sync with how people organize things in the real world (documents, drawers, folders), thus requiring much less brain power to operate.
With the dawn of the mobile era cognitive load was reduced even further through the invention of the touch screen. The incredible dexterity of our hands and fingers is quite unique to humans, so being able to directly touch and manipulate digital content through gestures takes away another layer of abstraction that was previously needed when using a mouse.
We are now seeing another powerful interface type gain traction: voice UIs (VUIs). Using speech to communicate with our digital devices feels effortless and natural, because that’s what we’re used to. It’s a 180 degree turn from the command line interface, in a sense: Back then, we had to adapt and learn “machine language” to communicate, now machines are learning to understand and speak human language instead.
The next Revolution in Output Technology
Both touch and voice UI are mostly input methods. What’s still missing is an output system that feels as close to human nature as possible. Humans are visual creatures. Our eyes are our most important sense to understand the world around us. A tiny, 2-dimensional screen doesn’t really do this any justice.
Just as the touch screen has moved closer to our hands (mobile devices) and VUIs have moved closer to our ears and mouth (Airpods), the output part of the UI will most likely move closer to our eyes — in the form of smart glasses.
AR glasses will allow us to project digital information directly onto our real-world environment. It’s a very powerful concept because content can finally break free from the confinement of our screens. This means two things:
First, content can now reach “beyond the rectangle”. It can be displayed right in front of our eyes, at the physical place or location where it belongs, without our phone screen as a kind of inconvenient proxy in between.
Second, content can expand from two into three dimensions. This opens up a whole new world of possibilities for better and more intuitive visual communication. More importantly, though, it will improve the user experience in one important aspect — you guessed it: cognitive load. Let’s look at a few examples to make this more tangible.
Early AR Use Cases
One of the most obvious examples is maps: It actually requires a lot of mental effort to translate two-dimensional information into three dimensions and back. Using AR, you can project the information right onto the street you’re walking on, allowing you to focus on what else is going on around you. This can even save lives, in the context of a car ride: By projecting navigation instructions directly onto their field of view (e.g. on the wind shield), drivers don’t have to take their eyes off the road to look at their smartphone or GPS device for map information anymore.
When we are shopping online for new shoes, we have to invest quite a bit of brain power to translate the 4–6 different product photos on the page into a comprehensive model of what the shoes might look like in real life, or even on our feet. The same concept can be applied to furniture, as Ikea and other companies have long realized. AR allows for faster and more informed decision-making by letting users see exactly what a product might look like from all angles, in a real-world context.
And what about instruction manuals? Browsing through 300 pages of tiny text and confusing 2D illustrations to find out how to change the filters of your new air purifier sure isn’t a pleasant experience. With AR, instructions can be attached directly to the actual filter, using easy-to-understand animated 3D graphics. Apply the same logic to electrical installation or car repair work, and tasks that were previous impossible to carry out by non-experts become a piece of cake to anyone.
These examples are merely meant to be an inspiration to show the potential of AR. They are a small fraction of what is possible with the technology, but many of the future use cases are yet to be designed. Which brings us to the role designers will play in this whole story.
Immersive Experience Design
What implications will AR have for UX/UI design? Here’s a few:
Transitioning from 2D to 3D means there is literally a whole new dimension of possibilities accessible to us as designers. We’ll be able to use form instead of shape to communicate an object’s affordances. Interface elements and content can have depth now, and can be looked at and interacted with from multiple angles.
We will also have a new means of creating hierarchy: In addition to color, contrast, and size we’ll be able to use distance to give certain UI or content elements more importance over others. We won’t have to simulate a button protruding from the screen towards the user using fake shadows anymore. We can design it in a way so that it actually reaches out from its background towards the user, just like we are used to from buttons in the physical world.
Because our designs will not be constrained by rectangles anymore, a lot of the rules and structures we rely on in “traditional” design become obsolete. The neat 12 column grid system of your website design? Gone. Content lives in the physical world now, and is anchored to surfaces and objects instead of the edge of the screen.
Responsive design will refer to the practice of changing the appearance and functionality of digital objects based on the user’s position in relation to them, not based on the size of the device screen or browser window. For example, text attached to a product could dynamically increase in size as the user moves away from it, to ensure good readability even from far away.
Storytelling becomes a completely different beast in AR. Each experience is unique and tied to the individual context in which it is used. We won’t be able to design one consistent path for all users — the number of edge cases and side paths will grow infinitely and we have to make sure to accustom for all of them. Research and testing will become more important than ever before.
I could continue on and on. But one thing should be clear from the above collection of design implications: AR will have a profound impact on how data is structured, managed, and delivered on the internet — and with that comes a radically new and different approach to UX and UI design.
The Industry needs You
So now that we have established a common understanding of why the transition from 2D to 3D is inevitable, and how this might affect us as designers — what are we going to do with this information?
When I realized the potential of AR a few years ago, there really only was two options for me:
1. Cross the bridge when we get there: Wait for the change to happen and then react to it
2. Help build the bridge, and be one of the first to cross onto the other side.
I wish as many designers as possible take the second option, get involved early in the bridge-building process, and don’t leave it up completely to the engineers to figure things out. Here’s why I think it’s important:
As with many other new technologies before, it was developers who got involved first — and they are definitely dominating the industry at this point. This really shows: There is a pretty obvious lack of user-centricity in most AR-related product development. The latest advancements of the technology mostly revolve around the question of what’s possible on a technical side. There is lots of solutions out there that are not solving any problems. How AR can be used to address real user needs remains a somewhat under-explored area. This creates kind of a vicious cycle: The biggest AR success stories so far are easy to dismiss as useless games and silly gimmicks. “This technology doesn’t have any real use cases” designers shrug and turn their backs, missing the chance to get involved and create the change that is desperately needed right now.
The field is still relatively young, so there isn’t many tools and guidelines to rely on at this moment. This is both a challenge and a huge opportunity for us as designers. We have a blank slate in front of us. We can help build this new city from the ground up. We can come up with the rules. We shouldn’t leave this task to the developers, business strategists, or marketing departments of this world — like we did so many times before in the past.
I believe we have a unique opportunity in front of us: The future implications of AR are fore-seeable, but it will take another few years until the tech actually reaches a point at which it’s affordable and user-friendly enough for the mass market. Any designer who understands this will have enough time to make the transition into 3D interface design long before everyone else hops on that train.
As more and more companies take the leap into the third dimension (Ikea, Nike, etc.), “immersive experience designer” will most likely become a much sought-after job title in the near future. Are you prepared to play your part in this new chapter of design history?
In my next article, I will give you a birds-eye view of the AR industry — who the most important players are, what their immersive tech strategies look like, and how the coming years will most likely play out. Most importantly, I will outline how you can take advantage of the current industry landscape, and how you can position yourself to play a vital role in this technological revolution. By the end of it, I hope that you you’ll be as excited as I am about the potential prospects of this space. And maybe you’ll even want to join me on my journey to become an immersive experience designer — I have prepared everything that you’ll need to get started.
