CLIENT: Personal Project

TIMELINE: July - August 2018

ROLE: UX, User Research

TOOLS: Amazon Mechanical Turk, www.usertesting.com

Case Study

This is a User Research project for Smule, a social network for creating, sharing, and discovering music content, and connecting people with passion for music from all over the world. I worked at Smule for seven years as a Lead QA Engineer, and observed how the company’s most popular app, Sing! literally grew from nothing to the number one free singing app in the Appstore. This user research project consists of four different types of user testing - heuristic evaluations, in-person usability testing, remote usability testing, and good old fashioned survey.

Problem Definition

Some of the most important UX flows in terms of driving user engagement are the tutorial flow and the Explore screen that helps users discover music content outside of their network. In May of 2018, the following problem was identified based on the analytics: after six weeks of using the app, usage of the Explore screen dropped by half. The conclusion the product team made was this.

Current UX experience on the Explore screen is not useful and engaging.

That is the problem I chose to investigate in this project.

User testing Strategy

So, how would we tackle these problems?

We would like to do user testing on the registration, tutorial and Explore screen flows because user is taken to the Explore screen straight after the FTUX (first time user experience). It is crucial to perform user testing methods that are both behavioral and attitudinal, as well as quantitative and qualitative. I decided to use the following quantitative behavioral methods: heuristic evaluations, in-person usability testing, and remote usability testing. In addition, I did a qualitative attitudinal method - a survey. 

Heuristic Evaluations

I walked through the FTUX, registration and the Explore screen flows making notes on anything that violates Nielsen’s Heuristics. The main research question here was:

Are there any obvious heuristic violations that negatively influence user’s first impression of the app and further engagement with it?

7 heuristic violations were observed during the evaluation.

1. On the tutorial screen, user is not presented with the Search option, which doesn’t match common mental model about singing karaoke.

2. If, while registering, a user enters an email that already exists in the system, they get to the Welcome screen that looks very similar to the registration screen, which can be very confusing. Users report being stuck. 

3. If a user enters an invalid email while logging in, she gets a cryptic error message.

4. The tutorial flow is interrupted by two full screen modals one of which appears rather pushy.

5. During the tutorial flow, user is presented with two almost identical screens, which is confusing. Also, the CTA buttons on the second screen say Sing, which is, in fact, referred to as Join elsewhere.

6. When searching for a song on the Explore screen, a user is presented with Songs, Invites and Recordings, and can be confused about the meaning of each.

7. Playlists don’t reflect common mental model about what a “playlist” is: they list user performances and not the actual songs.

In-person Usability testing

After doing the heuristic evaluations, I had a much clearer idea about the possible tasks I would like to use during the in-person usability testing. The purpose of user testing is to understand how well users can accomplish the task flow of creating a new account, going through the onboarding flow, discovering new content on the Explore screen, and expanding their current network. 

Research questions for this activity were the following:

1. Are there any usability problems that could be causing the Explore screen usage to drop off by half after Week 6?

2. Can users perform main tasks in the app, as well as exploring content, singing and joining, social actions on the Explore screen?

Usability testing containing 6 tasks was done with 3 users who met the requirements of the screener.

RESEARCH FINDINGS

Four of the heuristic evaluation findings got validated during in-person usability testing.

Two additional problems were uncovered as well.

2/3 users expected playlists to be customizable and editable. As of right now the app generates playlists based on the selected genres during the tutorial. There is no way to add new playlists or remove the existing ones.

3/3 users appeared confused about when to start singing, especially during the Join flow. Instructions on when to sing are visible only during the 4-5 seconds that the song loads. 

Remote Usability Testing

So far I identified four reoccurring themes. I decided that one more round of usability testing is needed before making conclusions. Three more participants on www.usertesting.com were hired to bring more light to the problem. Test goals, tasks and research questions remained the same.

RESEARCH FINDINGS

All six problems that were uncovered so far were validated during the remote usability testing. One more issue came up which is:

2/3 users would like to go back and sing the tutorial song again if not satisfied with the song or their performance but the app doesn’t allow it.

After three rounds of testing I created a summary of the themes that were reoccurring.

Survey

So far the conclusions I had was that there are indeed some usability issues in the tested flows that can explain why use engagement is going down. Besides usability issues, however, there might be issues with the content itself. To test this hypothesis, I constructed a survey that was sent to 301 users on Amazon’s​ ​Mechanical​ ​Turk platform and that contained 7 multiple choice questions and 1 free response question.

Research questions for the survey were:

1. Based on the current content on the tutorial screen, how satisfied and engaged are current or potential users after the FTUX flow?

2. What content on the tutorial screen will make current or potential users be more engaged with the app?

SURVEY FINDINGS

The survey uncovered three problems with the current content during the tutorial flow (other problems didn’t get validated based on the analytics).

Finding 1

41% of the survey respondents reported that they didn’t like the 4 genre limitation on the genre selection screen. 

Finding 2

When presented with the current song selection from the Unlock a song screen, only 16% of the respondents said they know and can sing any of the 12 songs, while 26% said they don’t know and won’t sing most of the songs. 

Finding 3

Traditional karaoke songs like Total Eclipse of the heart and American Pie appeared more engaging to the respondents. 29% of the respondents reported that they know all the songs and can sing them while 41% reported that they will sing half of these songs, which are much higher numbers than for the current song selection. Only 7.6% of the respondents said they don’t know and won’t sing most of the songs.

Design Recommendations

Let’s map the findings from all four methods of our research in a cost-benefit analysis chart that will help stakeholders evaluate the importance of fixing each problem.

For each of the 10 identified problems design solutions were developed. For example:

Problem

The app generates playlists based on the selected genres during the tutorial. There is no way to add new playlists or remove the existing ones. Users expect playlists to be customizable and editable.

Solution

I recommend adding an ability to add/remove existing playlists on the Explore screen. It can be a “+” button to the right of the Playlist sections, for example. After tapping on “+”, users will be taken to Songbook and prompted to select the songs they want to add to a playlist. They can create a new name for the playlist. They can even see their playlists on the Profile screen, next to Channel, Invites and other tabs. 

Further Testing

For the follow-up testing three proposals were created.

Proposal 1

Conduct usability testing for the newly redesigned Playlist functionality. Some of the possible further research questions are these:  

• Will the ability to modify playlists increase user engagement with the content of the app?

• What are user’s mental models when it comes to the definition of the “playlist” and its modification?

Proposal 2

Conduct an ethnographic research with the power users. The research questions here are the following:  

• Are there any fun features that users want to see in the app that will make them more engaged?

• Are there currently any show stoppers for the power users that they want us to hear about and fix?

Proposal 3

Perform further testing of the app music content using focus groups. Our main research question is the following: 

• What is the best ratio of new, trending music content and the old, more traditional karaoke content that can satisfy the needs of younger users and users over 35 years old?

Final Notes

During this project, I came to the realization that user research is definitely one of the most important areas in UX as it helps stakeholders uncover their blind spots. After testing the app for years, I never felt the need to modify playlists, for example. When a user who used the app before came to the session, despite the screener requirements, I had to disqualify the results of the testing. Designing survey questions that are unbiased in any way was a challenge as well.

View my other projects on Behance!