Apple iOS 12 + Web-Based AR: Shopify’s Daniel Beauchamp on How Retailers Can Sell More Now

This summer Apple previewed a major announcement planned for their annual event next week: with Apple iOS 12, web-based AR (augmented reality) will be introduced to the world. This will create new, compelling possibilities for online retailers to sell in 3D using augmented reality.

As the Head of Augmented and Virtual Reality at Shopify, Daniel Beauchamp sees the future of retail long before most. We recently had the opportunity to talk with him about Apple iOS 12, web-based AR, and how merchants will use it to share their products with customers. The conversation led to a range of future-focused ideas for ecommerce–and how retailers can prepare today.

For readers new to AR and VR, there are a couple of important definitions to know. Augmented reality is when customers can see your product in their own environment. For a fantastic example of this, check out the Magnolia Market app. Once you open the app, click on any product under the Preview in Your Home heading. Prepare jaw to drop. Part of what makes this so cool is that AR experiences happen through mobile devices. Just think: anyone with a smartphone, anywhere, and at anytime can shop like this.

Virtual reality is a more immersive experience, and it takes place while the shopper wears a VR headset. Daniel shares more on this, but as you can imagine, this kind of experience is limited to those who own and wear the VR headset. It can be amazing, but you need a specific piece of equipment to access it.

One of the key takeaways of our talk with Daniel was that the future of AR/VR is not distant. Apple’s new iOS 12 will be able to host AR experiences directly from Safari. Thus, the main barrier to AR–the fact that shoppers had to download an app–will soon become obsolete. Apple iOS 12, web-based AR is here.

Daniel explains more about this important moment for the industry and other aspects of AR/VR in retail.

Command C: What is your best advice for retailers to get ready to sell with AR/VR on their site?

Daniel: With all these new 3D experiences coming out, the core requirement to all of them is that you have 3D models of your products. Without a 3D model, you can’t view it in AR or VR. To get started, you need to get that 3D model.

Shopify is making it easier for merchants to get 3D models of their products by adding 3D makers to our partner network. These partners are the ones who can actually create 3D models at scale. We announced this at Unite in May, and it’s in beta now. We’re rolling it out throughout the year.

Once a retailer has the 3D models, they can use them in a variety of ways. There are viewers that you can embed directly in your Shopify store where the 3D model can be viewed in an interactive way. So customers can zoom in, spin it around, and see the product from any angle they want. This gives them a lot of confidence in what they’re getting ready to buy.

Also, coming this fall with Apple iOS 12, Apple devices will be able to view AR products directly from within Safari. So, if a merchant has a 3D model of their product, they can put it on their Shopify store. Then any consumer shopping on their iOS 12-enabled device, will see a little icon pop-up showing that a product is available to be viewed in AR. They can tap on it, and an AR experience will happen right in front of them. This is without having to download any separate app or plug-in to make it work. It’s a really smooth experience.

Command C: So, Shopify is bullish on web-based AR?

Daniel: Exactly. You’ve probably had this happen before: you go to a retailer’s store, and a pop-up appears, “download our mobile app.” And you think, “No, I don’t want to download your app. You have a beautiful, responsive website. Let me just use my browser to shop.”

So that same logic applies to AR. It would suck if every retailer you went to had big pop-ups that said, “Download our mobile app, download our mobile app.” No, I’m not going to have a 100 different apps on my phone just to do AR. So, we’ve been bullish on web-AR because we believe these experiences should be accessible directly on the web, without needing to download a separate app.

What Apple is launching is specific to the Apple ecosystem, but it does mean that any shopper on an iOS device has the benefit of doing these AR shopping experiences without downloading a separate app.

As an example, Magnolia Market just got even better. We posted videos of how iOS 12 is going to change how you shop online. [Also, see above.] We reproduced the Magnolia experience, that normally you have to do on their AR app, and we put it on their website. So, if you have iOS 12, you can simply just go to this store and any product that has a 3D model for it, you can preview it right there. Right within Safari and buy it with Apple Pay.

Command C: Are there other channels for this as well?

Daniel: Absolutely. Facebook announced 3D posts, so you can actually put 3D models in a newsfeed or an ad. They can be viewed in AR by using Facebook camera. That same functionality is going to be used in Instagram as well. So you can imagine a scenario where a merchant says, “Hey, check out our new product that just came out, and here’s the 3D model of it.” Then someone seeing that post on Instagram can tap on it, and view that product right in their home. With the buy button right there, too.

And that’s the thing with Shopify: we’re a multi-channel commerce platform. We enable you to sell your products wherever your customers may be, whether that’s Facebook, Instagram, Pinterst, Amazon, eBay, the list goes on. Now we want to add in that 3D component. So that, if you have that 3D model, you should be able to connect that 3D model to any channel that supports AR and VR experiences.

Command C: What about AR/VR in terms of fit; how fabrics hang, stretch, etc?

Daniel: That is definitely the experience everyone is looking for. “Hey, will this shirt fit on me?” It would amazing if an app could tell me that and show me exactly how it will look on my body. We’re seeing a few use cases of this with hats, sunglasses, jewelry, mostly things that can be added on top of your body or face.

It gets exponentially harder when you say, “Will this shirt fit on me?,” and you want the sweater to actually look like it will on your body. It’s flowing the way it should and the fabric is moving as it should. That is something we’re not there with yet. The industry is moving in this direction, and there’s been a few interesting approaches.

Just last week I saw a startup where their solution is you order this suit. You put on this skin-tight suit with all these markers on it, and then you position yourself in front of your phone. You rotate around 360, and it captures your body. It [the software] then tries to make an accurate 3D model of your shape, so that clothes that will fit on you.

Now when we’re talking about the barriers to using technology here, having to order to custom skin-tight AR-marker suit to try on, I don’t know if you’re going to reach mass market. Don’t get me wrong; I’m going to get one. But for the average consumer, that’s a tall order to ask.

But the truth is computer vision and machine learning are only getting better year-to-year. Cameras are getting new types of depth sensors, being able to more accurately see the space and get a sense of depth. I wouldn’t be surprised if, in a few years, you don’t need the suit at all. You can just do it all from your mobile device.

Command C: At this time, should merchants prepare for AR, VR or both?

Daniel: The good news is that if you prepare your 3D models, you’re going to be able to have AR and VR experiences. It’s that 3D model that’s important. So, if someone has an AR device, like their smartphone, they can see your product in front of them and see it in their space.

But if someone has a VR headset, they would be able to see your product in front of them with VR, so they could pick it up with their hand controllers and feel more immersed in that experience. Really it’s getting the 3D model that is important. Once you have that, you unlock 3D on the web, you unlock AR, and you unlock VR.

In terms of what will the consumers of today be using more? Definitely AR. You have 100s of millions of devices around the world that now have AR capabilities, whether that’s through ARKit on the iPhone or whether that’s 3D and AR posts in Facebook camera, that’s also available on Instagram. There are just so many different ways for consumers to use AR right now by using their smartphone.

One thing I will add to that: right now AR and VR are seen as separate technologies, but we see them converging in the future. Eventually you will have one type of wearable that will be able to do both virtual and augmented experiences. And really it’s just up to you to define how much of reality do you want to keep and how much do you want to leave behind?

Command C: And how much reality you want may vary from day-to-day.

Daniel: Exactly. If I’m previewing a couch, I want to see my living room. I want to see the couch in front of me. But if I’m previewing a tent, maybe I don’t want to see it in my living room, I want to feel like I’m in the forest. All around me I see these beautiful stars and trees, and at that point, my whole world is virtual reality. But maybe I still want to see my dog in this reality, so then my world is 90% virtual and 10% real.

One other thing I’ll add: phone-based AR is great right now because it’s so accessible. Everyone has a smartphone. But that’s not going to be the end state of AR. Eventually it will find itself in a wearable where you are hands-free. You feel immersed. You will have a sense of depth. Right now, as you’re doing these AR experiences on your phone, you’re still looking at a flat, 2D screen, and one of your hands is taken up by holding the phone.

Command C: Are there start-ups working on AR wearables?

Daniel: Oh, yeah. There are several, Magic Leap being the biggest one out there. A lot of hype around that product. It’s a wearable that you put on and it is AR glasses. The Hololens is one. The Meta is another. But once again, those headsets suffer the same problem as VR: the sense that you need to buy this expensive, clunky-looking headset for right now. But over time, these headsets will become more affordable, sleeker, and more accessible to wear in public transit.

Command C: If you were a retailer selling apparel, where would you put AR in your business plan today? Would you make this a priority or hold off for a bit?

Daniel: It depends which apparel we’re talking about. If it’s strictly clothing, you’re talking tops, skirts, and that kind of stuff, then I would hold off until the technology catches up [with the ability to see fabrics move/show precise fit with AR]. Unless you really wanted to go at it from a PR angle and say, “Hey, look at us, we’re the first ones to do this,” then go for it.

But if you’re selling hats and accessories, for example, then those have been proven to work well in AR with Facebook and Snapchat filters. You can just place glasses or hats right on your face.

Command C: What are you most excited about it your work right now? 

Daniel: I’m most excited about experiences that can be collaborative and shared with someone else. Right now with a lot of AR experiences, you preview a product in front of you. But if a friend is looking at the same spot through a phone, they don’t see the same product where you have it placed. It’s almost an individual activity doing these AR experiences. That changes once you start having them being shared. I see the product in the exact same spot that you see the product, and well now it feels more like real life.

And there’s interesting ways for VR and AR to play together. We did a proof of concept where you’re at home, trying to shop for the perfect vase for your coffee table, but you need a little bit of help from an interior designer. Well, the designer could be somewhere else in the world, using a VR headset and can appear in your space as a 3D avatar. And actually make suggestions about your space, move products about your room, and test out different lay-outs. Now you’re getting an assisted sale from the comfort of home and feeling like you’re actually speaking to a real human and a real expert.

Command C: And are you speaking to a real expert at that point? Is there a person behind the VR?

Daniel: Yes. That would work really well for a situation where you can make an appointment with a merchant, set a specific time, and they can actually show up in your space and help you out with whatever it is you’re looking to buy.

Command C: What do you see as the rate of adoption among shoppers for these technologies?

Adoption has been slow over the past year, even though some retailers have started providing AR experiences in their own app. And that’s because consumers had to download a separate app in order to have this AR experience. Adoption will speed up as the barriers get lower and lower. And being able to preview in your home, right within Safari, right on the website you’re on and be able to buy it right there, that lowers the barriers significantly. And that is going to get more and more consumers used to shopping that way.