Intro to Immersive Experience Design (Part 3)
“Intro to Immersive Experience Design” is a guide for everyone who’s interested to learn more about 3D interface design and augmented reality. It is the result of over 5 years of research and exploration in the field of immersive technologies, and is intended to help you navigate through the vast, uncharted territory of this exciting new industry.
The guide consists of 3 parts:
Part 1: Why 3D interfaces are the future of UX/UI design
Part 2: The past. present, and future of the AR industry
Part 3: How to start your career as an immersive experience designer
Ready? Let’s dive into it.
How to start your career as an immersive experience designer
In the first part of my series, I talked about why immersive tech will be the next big frontier for UX/UI designers. I explained why 3D interfaces are so much more powerful, and so much closer to human nature than the 2D interfaces we currently interact with on our mobile devices. I also laid out some of the implications that this impending evolution in interface technology might have on our work as a designers.
In part 2, I gave an overview of the AR industry, its main players, and the individual strategies they have put in place to win the “AR arms race”. A lot of things have happened since (the biggest news being Facebook’s re-branding and re-positioning as “Meta”), but so far my predictions are still accurate: We are on a good path to witness the release of a mass market AR glasses product within the next two years.
This gives us a potential 1–2 years head start to front-run the huge wave of people who will want to get involved in this new industry as soon as mass adoption happens. It gives us time to get familiar with 3D, to get involved early, to actively help shape this emerging new paradigm. In this article, I want lay out a potential strategy on how to start your career as an immersive experience designer. I will share the best learning resources and tools I have gathered over the last 5 years so you can get up to speed quick and join me on my mission to position designers at the forefront of this movement.
Why Mobile AR is doomed to fail
Let me start by saying something slightly contradictory: Mobile AR is not the future of AR. But it’s the best way to prepare for the future of AR. What do I mean by that? Let me try to break it down.
As I have explained in length in my first article, augmented reality is a logical next step on our path to making computer interfaces more closely embedded into our surroundings, and more natural to interact with. In many regards, AR is one of the last remaining puzzle pieces to truly merge the physical and the digital world. We are not quite there yet, though. In order to unlock the full potential of the technology the following prerequisites must be met:
- Hardware: The output part of our computer interfaces needs to move closer to our eyes, so that content can free itself from the confinements of our phone screens and instead live anywhere within our natural field of vision. In addition, a wide array of sensors is needed to constantly scan our surroundings in order to achieve a high level of “context awareness”.
- Software: Digital content needs to be transferred into three dimensions, and must be able to integrate seamlessly into the real world. This includes dynamically changing appearance, properties, and position based on input from users and the physical space around it, including other real world objects, lighting conditions, laws of physics, weather conditions, and many other factors.
While mobile AR can already deliver decent results in some of the these areas, it will never be able to deliver the “real deal”. For a truly immersive user experience a new device type like “AR glasses” (or maybe in the future: AR contact lenses) is needed.
Why designers should embrace Mobile AR
So why would I recommend designing for mobile AR in the first place? I believe that mobile AR has its place in “AR history” as a sort of transitionary technology, a stepping stone to help us prepare for a more immersive future. Just as tech companies are using mobile technology to try out and develop many of the technical components needed for AR, we as designers can use mobile AR as a perfect learning ground for experimenting with the usability aspects of this new interface type. I can think of at least 5 reasons why mobile AR is the perfect “gateway drug” to learn all the necessary skills needed as an immersive experience designer:
- Mobile UI is our home turf. We know the ins and outs of mobile interface design. We understand all the rules, possibilities, and limitations of the medium. Although there might be some new things you have to learn for mobile AR, the overall experience is still embedded in the context of your mobile device, following similar rules and interaction patterns as all the other (non AR) apps. You are kind of designing in 3D for mobile AR, but not really — because the input part of the interface still happens in 2 dimensions, via a touch screen.
- The entry barriers are low. Anyone with a mobile phone can create mobile AR experiences — there is no expensive hardware required for prototyping and testing. For a real AR experience (with glasses) one usually needs very expensive specialist hardware and software, as well as extensive expert knowledge to deal with all the technical issues that are still prevalent during these early days. This ties nicely into the next point:
- The design tool stack is mature enough. I have tried many different apps and tools over the years and based on my experience it is almost impossible to create a prototype for an AR glasses experience without deep knowledge in 3D and game design. Good luck trying to find your way around Unity as a beginner. The design tools for mobile AR, though? There is some pretty cool stuff out there that is easy to learn. More on this in the later sections of this article (:
- Mobile AR is easier to integrate into people’s lives. Our phones are with us 24/7. We take them everywhere we go. The social acceptance around mobile phone use in public are high, which is definitely not the case with AR glasses. This gives us opportunity to explore use cases that AR glasses are not yet able to cover (yet)— especially when it comes to experiences that happen outside of our homes. This also mean:
- User testing is easy. You can easily build prototypes and test your designs with anyone who owns a smart phone. This is a huge advantage if you compare it to the massive effort and cost it takes to test your latest HoloLens or Magic Leap designs with potential users.
So with these points outlined above, I hope you can see why mobile AR might be a really great way for us designers to dip our toes into the “immersive experience design” waters. The next question is: “Where do I start?”. I have compiled a “mobile AR starter pack” for you that can help you get the ball rolling. It includes many of the resources I found to be very helpful when I started my very own journey into the “AR rabbit hole”.
The mobile AR starter pack
This starter pack is focussed mainly on mobile AR, although many of the links might also be helpful to learn about related disciplines, such as VR. In order to lower the entry barrier as much as possible, I’ve tried to to focus on resources that don’t cost anything — but not everything can be offered for free. The few of them that need to be paid for are definitely worth the investment.
AR app inspiration
Let’s start with some inspiration first, shall we? One way to get familiar with mobile AR is to try out some of the existing apps out there and have fun with them. The following list is a collection of some “best in class” apps, as well as more unknown, experimental apps that are pushing the boundaries in a more playful way.
Ikea Place was one of the first mobile AR apps with a “real” use case — and a great example of a really well-executed solution in this space.
Wanna Kicks plays in a similar arena to the Ikea app, yet has a very specific use case: trying out and buying sneakers. It sounds kind of lame but the experience is surprisingly delightful
The New York Times app was one of the first digital news outlets that fully embraced AR in the early days. They do regular AR features to create a deeper level of immersion for the readers of their stories.
Architecture of Radio was a really early experimental AR app by Dutch designer Richard Vijgen. It’s not free, but really cool to try out (much more impressive than watching it as a video)
Meisai is the result of Takayuki Watanabe’s ongoing explorations of different AR filters and effects. It’s SO MUCH FUN.
When early videos of Clipdrop came out I just couldn’t believe it was real. It feels absolutely magical when you try it for the first time.
Qlone could also be mentioned in the “3D design and prototyping tools” section. It’s a really clever app that you can use to “3D scan” small real world objects to create a 3D model out of.
Must-reads
A good next step is to build up a solid “AR acumen” by reading some relevant content from thought-leaders in the space.
Google’s AR design guidelines are very well done and contain loads of useful information and best practices. The cute animated illustrations add a lot of helpful context and make it a very enjoyable read.
Apple’s AR design guidelines are a bit underwhelming in comparison to Googles — but they still cover some interesting aspects that Google doesn’t, so it’s worth checking them out.
All articles from Bushra Mahmood on Medium. Especially the 4 part “A Quick Guide to Designing for Augmented Reality on Mobile” has been an incredible source of inspiration for me.
Everything from Alexander Jeremiah on Medium: Sadly, his AR prototyping company “WiARframe” is not existing anymore, but some of his writing about mobile AR is still very relevant.
Must-follows
While we’re in reading mood, here’s some great accounts to follow on social media that will keep you informed and inspired on a daily basis:
Tom Emrich will keep you up to date on the latest AR news on both his Twitter feed, as well as his monthly “AR roundups” on LinkedIn.
Charlie Fink is a writer and consultant and is quite active on Twitter. His articles for Forbes are really great, too.
Kitasenju Design (Takayuki Watanabe) is doing incredibly inspiring work with AR. Most of it is explorative/experimental but that’s exactly what makes it so strong. His (previously mentioned) “Meisai” app is basically a collection of the results of his experiments. I could get lost for hours playing around with the different filters in there…
Omega C is one of the most talented Face filter creators out there. His posts document his experiments with different patterns, effects, and tools. Many of his filters can be tried out on his IG page, too.
Ines Alpha does really cool experiments with face filters and AR fashion. Looking at her posts is like entering a different dimension.
Luke Hurd is another great follow when it comes to fun experiments with face filters. Some of them are very funny — try them out on his IG page!
Don Allen Stevenson III is a really great AR creator who’s doing lots of cool experiments with virtual clothes and body tracking.
Vova Kurbatov is an immersive experience designer and prototyper with an obvious background in UX/UI design. His strength is high fidelity mockups/prototypes of AR interfaces, which are on a whole different level from his peers in terms of visual design and attention to detail.
Inna Sparrow has been delivering a lot of amazing prototypes and experiments over the last months. Excited to see what she comes up with next!
Learning 3D
There is really only two approaches to learning 3D: Either you jump into the cold water and try to teach yourself (Youtube is your friend), or you take a class with a slightly more structured approach. If you wanna go with the former approach, you can skip this section and go straight to the “3D design and prototyping tools” section, pick an app that sounds good, and dive right in (:
3D for Designers: This really awesome online course is the best starting point if you come from a design background and want to learn 3D. Devon Ko (the founder of 3D for Designers) is an incredibly funny and talented instructor. You’ll also be able to join an online community of like-minded 3D designers (via their Slack), where it’s easy to get help, find friends, learn together. Not free, but definitely worth the money!
3D design and prototyping tools
Now it’s time to get your hands dirty. There is no way around it: You’ll need to learn some new tools. It can seem daunting at first but trust me: once you get a hang of it, it’s a lot of fun! The following list covers the best 3D and prototyping tools for beginners.
Spline: This is a great choice for your first experiments with 3D. Spline is a young company on a mission to make 3D design easy for everyone. Their product is free and super easy to use. Spline has just enough functionality to get to some really cool results quickly, while not being overwhelming.
Vectary: Vectary is another amazing company with a similar mission to Spline. They are basically trying to translate the ease of use and collaborative aspects of Figma into 3D. Vectary is perfect for taking your first steps in this new medium, but you can also do some really impressive stuff with it once you’ve gotten a hang of the basics.
Blender: If you’ve reached the limits of what Spline and Vectary can do (which might happen eventually), you can graduate to Blender. I don’t know how this software is for free — it has everything you need for creating really professional 3D models and animations. It can be a bit tough to navigate in the beginning, but their learning resources are good and will bring you up to speed fast.
Spark AR: Spark AR was originally developed by Facebook for their own teams, and then released to the public. It basically started the whole face filter craze on Instagram by allowing anyone to create and publish their own AR masks and filters. Although you can use the software to make some really complex stuff, it’s also fairly easy to create your very own face mask quickly, using their very nice learning content on their website (of course there is also a lot of Youtube tutorials on the subject)
Adobe Aero: Of course Adobe has their eyes on the prize, too. Their “Aero” app makes it very easy to drag and drop pre-made or self-made 3D objects on your desk or living room floor, assign behaviors and interactions to them, and create simple prototypes.
Apple Reality Composer: Reality Composer is Apple’s own AR design tool. similar to Adobe Aero, its the right tool if you just want to play around with some existing 3D content, add interactions, and create some simple AR mockups.
Bonus: Pen and Paper This video is one of the rare behind-the-scenes looks of how Apple designers approach AR prototyping. I was very surprised to find out that they are (or at least they were back in 2018) doing a lot of their prototyping with pen, paper, cardboard, and glue! How nice is that? You can get away from your laptop for a few hours and create physical “prototypes” to showcase your digital experience — I’m sure many would welcome such a change to our usual daily routine.
Bonus: Where to get 3D models
Sometimes it’s easier to just take a nice 3D model that someone else made and build an experience around it. A lot of the 3D design tools in the section above have their own libraries and platforms from which you can get 3D models — but if you can’t find what you’re looking for, try the following platforms:
Thangs is a platform for free 3D models. Not everything on there is good quality, but sometimes you might get lucky and find just the right thing you need on there.
Sketchfab isn’t for free but the overall quality of the assets is (obviously) much higher — so if you need that one really detailed pair of sneakers for your online shopping prototype, this is the place to go.
And that’s it for now— I hope you find it useful. Did I miss anything important? Send me a DM or leave a comment, I’m very always happy to learn and extend the list (:
