OMNI

Voice and Touch-Screen Interfaces for a Smart Home System

Back to My Works

Overview

What is Omni?

Omni is an integrated smart home automation platform that programs itself. Its network of devices works together to recognize household members’ patterns and uses them to create a seamless smart home experience.

The initial platform allowed user control via touchscreen devices or voice commands while integrating machine learning capabilities. Our project explored user control options for both mobile application, voice and control via existing voice interfaces.

The Omni system would be installed in homes or apartment units and work in the background.

My Role: User Research, Interaction Design, Prototyping, and Evaluation.

Team Size: 3

What was our challenge?

Goals of the Project

  • Determine, how users might effectively interact with a machine-learning powered smart home system via voice and touch screen interfaces.
  • Find out what users’ feelings were about privacy and security of the system.
  • Determine if voice is a useful method of interaction, and if so, in which scenarios.

Key Business Requirements

Initially, we collected business requirements and goals from the company stakeholders, and conducted a cognitive walkthrough on the existing conceptual prototype.

    Business objectives

  • Interaction through touch screen interaction (wall-mounted touch screen and mobile application).
  • Voice control capability.

    • Stakeholder goals

  • For this product to gain credibility with people and eventually become ubiquitous.
  • True “smart” home, where the home learns and functions with a person’s life.

Research

Background Research

Conducted background research, beginning with a literature review to familiarize ourselves with the product space, and a competitive analysis to determine which existing products may potentially complement or compete with Omni.

Reviewed articles and existing research on smart technology, voice control, and home automation space.

Field Research

Identifying the user needs

We conducted an online survey to collect information about users’ thoughts and expectations about smart and connected homes. Then, conducted contextual interviews in participants’ homes to learn more about their interactions within their own homes.

Through our research we want to answer the following questions:

  • How do potential or current smart technology users view, value, and think about smart technology, machine learning, and home automation?
  • What are the barriers or issues users encounter when using or considering using smart technology, machine learning, or home automation?
  • Who are the users within this space?

Survey

We conducted an online survey and received 27 responses. The survey was divided into five sections: demographics, information about the home, connectivity and devices, smart technology people use, and opinions on security and privacy.

Respondents were asked to indicate on a scale of 1 to 5, with 1 being “strongly disagree” and 5 being “strongly agree.”

Survey key findings

Aggregate results by exporting online results to excel sheet

  • Participants were interested in their smart lighting, smart thermostats, smart locks, smart plugs, smart appliances, security systems, and home assistants (like Amazon Echo or Google Home).
  • Majority of the participants showed interest in smart home automation.
  • Participants were concerned about keeping their information private.
  • Respondents gave a neutral response regarding whether or not they thought that smart devices would keep their information private. Few indicated a strong preference.
  • People were concerned about using cameras within home.

Contextual Interviews

We conducted six unstructured contextual interviews, in which we observed and interviewed participants in their homes in an effort to gather information.

Procedure

  • Asked a series of questions
     

  • Utilized their answers spun from those questions
     

  • Asked to show their smart device and how they use.
     

  • Frustrations on smart device
     

  • Opinions on machine learning and home automation

Contextual Interview Key Findings

Users want time-saving technology

Security/privacy are very important to users.

Compatibility with other smart devices is a key consideration.

Usability issues with voice interface.

Research Analysis

Affinity Diagram

This technique was used to identify key patterns and information. Took notes from the interviews and survey and built from the bottom up.

Affinity Diagram findings

In general, users are generally interested in existing smart home devices, but are more interested in technology that can help them save time. Other findings are as follows:

  • People are interested in smart technology for entertainment, home control, and security.
  • For those with smart devices, users use their smart devices to control the environment of their home.
  • Most users have non-smart devices that have an opportunity to become connected devices.
  • Users find that a lot of voice-controlled virtual assistants, like Alexa or Siri, which aren’t consistently good at voice recognition that often causes problems.
  • Connectivity can sometimes be a pain for users, with or without smart devices.
  • Our product should account for different households.
  • Users had varying opinions about Omni.
  • Many users of smart technology consider themselves to be tech-savvy.
  • People mostly feel positive about smart technology, but don't necessarily feel that it's critical.
  • People are not really concerned about devices knowing their data, but home security is an important factor.
  • Users do not always choose products for themselves. Some receive them as gifts or buy the first product available, before competing products are released.
  • Generally, the pricing point of the smart device is an important factor.
  • Setup of smart devices should be relatively easy, and should be something users can do themselves.
  • Routines are an important consideration, especially when considering machine learning.

Personas

Key user characteristics and requirements

  • Target users for this product fall into three categories: extremely smart technology expert, mainstream enthusiast, and smart technology novice.
  • People are interested in technology that makes life easier and does not interfere with their daily routines.
  • People find voice control to be frustrating when they have to repeat themselves or use very specific, non-natural language.
  • People have different levels of experience with voice interfaces, from no experience to daily use.
  • Privacy and security are major considerations for any home control system.

Day-in-the-Life

This model was used to create a daily view within one target user’s space.

User Requirements

  • Our solution needs to be easy to set up and easy to customize.
  • Voice control must be seamless with our product idea, and must be less of a hassle.
  • Our solution must consider all the different connectable items in a home, outside of just the lights in the thermostat.
  • Our design solution should consider usage inside an apartment home, as well as usage in a single-family home.
  • Our product should MAYBE have an option for a camera, and be trustworthy.
  • Our design solution should consider how the product can become ubiquitous.

Conceptual Design

Brainstorm session to identify different scenarios

Concept 1: Smart Home Augmentation

Experience consists of interaction with a mobile app, a wall switch, and voice

  • High-tech light switch that appears normal on the outside, to keep intuitiveness. Tech inside allows for motion tracking, sensors, “invisible” experience.
  • Mobile app for setup/config.
  • Could sense phone location, wearable location, etc.
  • Compatibility across devices.
  • Automation without “machine learning".

Use Cases:

  • Wake up in the middle of night and light turns on to 10% unless otherwise commanded.
  • Leave the house and all things are turned off.

Concept 2: Control Center

Experience consists of interaction with a mobile app, a wall switch, and voice

Experience consists of:

  • Larger wall screen that acts a centralized “control center”.
  • Smaller touch screens that replace light switches.
  • Screens are motion sensing.
  • Voice set-up.
  • NO Mobile application.

Wall screens allow for control, permissions, configuration; users can create “scenes”.

Use Cases:

  • Lights turn on when the screen detects motion.
  • Commands are given through voice interface to operate smart devices.

Concept 3: Electrical System Integration

- Technology is placed inside or on a wall, possibly inside or near the home’s electrical panel.

- Turns non smart items around the home into smart items.

Experience consists of:

Voice interaction

  • Home assistant (Echo, Google Home, etc).
  • OR virtual assistant (Siri, Google Now, Cortana).

Mobile application or desktop site.

  • Simple.
  • Used for setup and configuration, and to allow for integration of the mobile phone’s virtual assistant.
  • NO wall screens.

Selected Concept for Prototyping

We chose the Electrical System Integration concept (Concept 3) to guide our detailed design and prototyping activities. As we considered all three possibilities, this particular concept seemed to be an excellent fit for our target audience.

We wanted to focus on something that is familiar with our users,and find a way to bring the Omni product into their daily lives without too much interruption. In choosing to use a system integration concept, Omni would have no need to create another voice interface. It would make use of any connected home control’s native voice functions, and communicate through it. This similar function could also be used in something like a wall or mobile interface.

User-Centered Design

  • Our personas, which were based on the field research conducted, indicated that the solution needed to appeal to a variety of users with a range of experience levels with voice interfaces.
  • We designed our solution based on two of the personas we created (the “novice smart tech user” and the “mainstream enthusiast”).

Brainstorm session to explore design concept

Wireframes

Low Fidelity Solution

  • A low-fidelity prototype was created to represent concepts, flow, features, and functionalities of Omni product. It was designed to allow us to get quick feedback from target users to determine if the selected design concept was likely to be acceptable to target users.
  • The low-fidelity prototype supported both voice and mobile interactions.

How did we create it?

  • Balsamiq Mobile Wireframing

  • Invision Mobile UI Prototyping

  • Alexa: Simon SaysVoice Audio. Did recordings and played with respective tasks.

Low Fidelity Prototyping

Link to Interactive Prototypes: https://projects.invisionapp.com/share/E8NWJHZUM3J#/screens/318355671_Routine

Routine

Sound

Low-Fidelity Testing Insights

Key information from feedback on this solution fidelity

  • Participants indicated more of a preference to configure more tasks in a mobile application.
  • Participants those who owned smart devices tended to conduct each task using scripted language, while the participant with no experience used natural language to complete the task. Further testing of this was conducted during our high-fidelity user testing.
  • Usage of a mobile application for configuration was strongly hinted at, which may give us an opportunity to test more than just a voice prototype.

Based on user feedback from the low-fidelity prototype, we altered our solution and used both voice and mobile interactions in our high-fidelity prototype.

High Fidelity Solution

Solution is mostly voice using existing home/voice assistant technologies. Created more detailed mobile screens for each task in order to compare how users interacted using a voice interface vs. a mobile application. All front end interactions through known voice assistant technology and via a mobile interface. We focused more on creating scenarios for testing, rather than the entire app.

How did we create it?

  • Storyline Voice UI Prototyping

  • InVision Mobile UI Prototyping

  • Amazon Echo Dot Bluetooth Speaker

High Fidelity Prototype

The main features of the prototypes are:

  1. Setup a Bedtime Routine.
  2. Behavior Pattern Recognition.
  3. Sound Recognition.
  4. Customized Routine.

High Fidelity Prototype Mobile

Link to Interactive Prototype https://invis.io/8HOXOU9MS6D

High Fidelity Prototype Voice

Prototype Testing Results

Procedure

Research Method

Usability testing with usage of A/B testing methods.

Testing Location

Indianapolis, IN in team members’ homes within controlled and quiet environments.

Testing Sessions

Participants were asked to complete a series of four tasks, each with two subtasks (one primarily using the mobile app, and one using primarily voice). The order of the sub-tasks was alternated with each participant.

Results: Overall Data Analyzation

  • Mobile is the preferred method for complex tasks.
  • Overall user expectations for the prototype were met, and even exceeded for the mobile app.
  • Based on observation and data, all tasks seemed easy to complete.
  • Pattern recognition is an interesting feature, but is also an unfamiliar concept.
  • Users experienced in voice control interacted differently than those with less experience in voice control.
  • Switching from mobile to voice was met with mixed feelings by participants.

What Worked?

  • Participants liked how quickly and easily they could create or edit routines and patterns with the prototype.
  • Participants liked the convenience that pattern recognition provides.
  • Participants preferred using the mobile application instead of the voice interface for the tasks that required more steps.
  • Participants were typically able to complete the tasks more quickly with the mobile application.
  • Overall, participants thought the entire prototype as pretty simple and easy to use.

What Didn’t Work?

  • Participants had concerns about the privacy and security of their data.
  • Some participants experienced frustration due to the limited functionality of the prototype.
  • The color scheme and font sizes of the mobile, while visually interesting, made the interface harder to read.

Other Findings

  • Participants indicated that pattern recognition is an interesting feature, but is an unfamiliar concept that requires more education.
  • Technical issues with the voice prototype presented some challenges, but users were still able to complete the tasks.
  • There were differences in the way users experienced with voice control interacted with the system, versus users who were not as experienced with voice control.

A few quotes from testing….

  • “I don’t like the voice thing….[Task 4 is] just easier to edit on [the mobile app] to just edit it….I don’t have to talk to it and wait for a reply….voice takes too long. ”(P5)

  • “Personally, I don’t like using my voice quite as much. I’m a visual person that way. But there was that one [Task 3, sound] I thought was easier….maybe sometimes. Majority of the times I personally prefer the mobile device.” (P3)

  • “I prefer not having [machine learning] always active...” (P2)

  • “I think for everyday routines it’s good because it might suggest things for me...which I might not have thought of, which is good...From a security perspective, I’m a little worried because it knows my routine, it knows when I get home. So if a hacker gets into my system then they’re gonna know my whole routine.” (P1)

Conclusions

  • Both the mobile application and voice interface could provide added value for users on the Omni platform.
  • Voice is likely to be acceptable for tasks involving a relatively small number of steps.
  • Acceptability of the voice interface decreases for tasks requiring a larger number of steps.

Recommendations

  • Restricting complex tasks to touch screen interfaces instead of voice.
  • Consider giving visual feedback or confirmation via the mobile application once a task is completed via a voice interface.
  • Where possible, reduce the number of steps required to complete tasks.

Lessons Learned

  • Communication with clients
  • Implementation of methods for gathering requirements.
  • Collaboration within our team.
  • Voice interface prototyping.

Next Steps

- Collect additional feedback about the UI design, particularly readability of the new color scheme.

- Conduct further user testing in a smart home environment.

Continue research in the following areas:
  • Voice control interface, including the use of additional existing platforms.
  • Omni’s auto-prediction features.
  • Machine learning features.
  • The concept of transitioning between mobile and voice Interfaces.

Mobile Prototype Demo

Video Prototype Demo

Next