Skip to content

AIUX/UI

Slowing down to speed up

A cross-discipline collaboration to ethically leverage AI and improve in-person shopping experiences

By
Juliette Weiss, Rick Barraza, and Jenny Williams

  –   The estimated reading time is 7 min.

A 3D-rendered image of a clothing store with racks of garments and accessories. Mannequins pose near the clothing displays on futuristic platforms, and shoes are arranged on shelves against the wall. A bench sits in the foreground, bathed in soft lighting.

Think of your last satisfying in-person shopping experience (and if you have to reach back pre-pandemic, that’s ok!). Maybe you were browsing your favorite bookstore, pouring over the staff picks until an intriguing title leaped out. Or you got everything on your grocery list in one go, without having to ask a staff person if they had more in the back.

These experiences — delightful, efficient, fulfilling — are possible because store employees think carefully about what you might need, and then build layouts and displays to help you get what you want. But when it comes to understanding and anticipating shoppers’ needs, in-person retailers have traditionally had less data to leverage than online retailers, who have access to site traffic and other analytics.

Because Microsoft is committed to building AI responsibly, the Connected Spaces product team collaborated with Ethics & Society to uphold our responsible AI principles from the start. As an internal product and design agency, Ethics & Society guides technical and experience innovation towards ethical, responsible, and sustainable outcomes. Together, we were able to:

  • Deep dive into key considerations around privacy and transparency
  • Build a UX that gives store employees more control over the AI system
  • Leverage cross-disciplinary expertise to help us achieve better outcomes for individuals and companies

AI can help. This week, Microsoft announced the preview of Dynamics 365 Connected Spaces, coming in December. Connected Spaces harnesses data from video cameras to help companies understand their physical spaces better: how people move and interact, and when and where staff might be needed to open a new register or restock shelves. These signals can make spaces more efficient and empower store employees to provide exceptional customer service — all of which improves your shopping experience. Connected Spaces can be applied to manufacturing and warehousing, too, providing support up and down the supply chain.

Protecting privacy for shoppers and employees

At Microsoft, we pride ourselves on being faithful stewards of people’s data. We want people to feel respected by our technologies — “optimizing for human dignity,” as we sometimes say. Since Connected Spaces uses cameras to make inferences about the physical environment, we needed to be especially thoughtful in how we would protect the privacy of both shoppers and employees who might be captured in video footage.

First, we looked at the technology itself. Connected Spaces was designed to preserve the anonymity of individuals as much as possible by focusing on movement patterns and locations rather than identifying individual people or analyzing facial characteristics. Additionally, the video stream may be processed automatically on premises or securely on the cloud, based on the customer’s privacy needs and concerns.

Second, we wanted to make sure shoppers and employees can understand in a meaningful way how the system works, what observations are being captured by the cameras, and how these observations might be used. Our user researchers invested significant time and energy in understanding people’s concerns: What questions did people have about the technology? What privacy protections made them feel more comfortable and secure? How could we explain complex technical concepts in ways that people could easily understand?

Since both customers and employees are potentially affected in different ways, we used our research findings to create guidance for store owners to best communicate and disclose these topics to each group. This guidance helps store owners create a disclosure strategy that:

  • Raises awareness about the use of Dynamics 365 in their space.
  • Helps people understand how the technology works and what privacy protections are in place.
  • Adapts to individual people’s communication needs, including considerations of fluency, literacy, and accessibility.
  • Empowers employees to both understand and explain the system to others
A warehouse scene with a figure using augmented reality (AR) to manage inventory. The figure interacts with virtual elements projected around various stacks of boxes. The setup includes shelves filled with boxes and a visually highlighted AR interface.
The illuminated areas in this image shows how Connected Spaces lets you choose the specific areas that you’d like to glean insights about. As Connected Spaces collects data, your employees will be able to receive these insights at their fingertips.

Empowering retailers through human-AI collaboration

On the retail side, we then thought deeply about the Connected Spaces interface: who would use it and how we could empower them. Building robust, highly accurate AI systems has been a core priority for Connected Spaces. But there will always be situations that an AI can’t predict. Maybe traffic patterns in a store changed suddenly because it started to rain, and shoppers flocked to a covered entrance. Maybe a traffic zone was drawn incorrectly during setup, so the system’s predictions are off.

A computer screen displays a tutorial on how to draw and analyze the effectiveness of in-store displays using an overhead camera image of a store aisle. The user is being guided to draw a skill zone on the display image for analysis.
An example of how Connected Spaces guides a user in setting up their display effectiveness skill to ensure the collection of data is accurate and meaningful.

For example, when a user is telling the system what part of their store they want Connected Spaces to pay attention to, they are prompted to draw a “zone” in the UI. In early iterations, many users drew the zone over a promotional display since that’s what they wanted to track. But the system actually looks at where people are standing — requiring the zone to be drawn on the floor.

A supermarket aisle with shelves stocked with various beverages and snacks. Several sections are highlighted with purple labels indicating "False Positive" and "False Negative," suggesting a system analyzing product placement. Green boxes outline the highlighted sections.
Visualizing how the AI can makes mistakes helped teams align and design a more thoughtful solution.

Leveraging cross-disciplinary expertise

A collage of news website articles about AI regulation and oversight, with headlines discussing strict rules, AI's unchecked power, and law firms' increased regulation. A prominent image shows a Western-themed scene with a cowboy in a doorway.
These articles highlight the legal concerns surrounding the use of AI, and show examples of recent legislation to better regulate the industry.

All of this exploration required a wide range of expertise. As Jonah Sterling, Head of Design for Digital Transformation, says, “Design can’t work in a bubble. We need researchers, PMs, data scientists, and legal partners. All disciplines in the product team own the UX and it’s imperative that we work together to get it right.”

To design for Connected Spaces, we started with the model makers and data scientists. As a crew, it was essential to deeply understand how the model works and where it can go wrong. What are the false positives? What are the false negatives? Designers from Ethics & Society worked directly with data scientists to visualize how the different ML models interacted behind the scenes, which surfaced some of the misalignments that could occur.

Misalignments might come from variations in how users set up the system, environmental factors such as a mannequin blocking what the camera can see, or people trying to use the system in ways it wasn’t intended. These learnings have informed improvements across the product, from how we guide users during setup, to which analytics we include in our dashboard reporting, to driving continual improvements to the models themselves.

We also partnered closely with our legal team as legislation around the globe is ever-evolving when it comes to AI. There are myriad examples of AI gone wrong, especially with computer vision and privacy, so we need to take note of these and ensure we are understanding the new challenges and emerging legal developments.

As technology continues to evolve, so will the way we approach building it. We’re always learning, and we hope you’ll share your learnings with us, too.

For more information about Dynamics 365 Connected Spaces, please visit our website and blog.

A dedicated group of people came together to bring this effort to life, and we’d like to thank everyone on the Ethics & Society and Connected Spaces team for your engagement!

Read more

To stay in the know with Microsoft Design, follow us on Twitter and Instagram, or join our Windows or Office Insider program. And if you are interested in working with us at Microsoft, head over to aka.ms/DesignCareers.