What we’ve learnt in Digital Product Research: adapting research techniques

The Digital Product Research team at Co-op Digital has been exploring what the future might look like for the wider Co-op. We’re about to move onto a new phase of work, so this is a good time to write up some of the things we’ve learnt.

Our team of 6 has been learning by doing. We’ve looked at things like Life after work, Everything is connected, Financial freedom through early planning. Our most recent work was about energy and co-operation: Collective switching for good. You can read about more of our experiments at dpr.coop.co.uk.

This is the first in a series of posts covering design principles and ways of working that have emerged in the last 12 months or so.

Getting out there matters

Our research started out much as you might expect: we spoke to Group colleagues, Co-op members and spent time with people in research labs. But we quickly became aware we were spending too much time in artificial settings. Research is best done in the context of the problem you’re trying to understand, so we made sure we got out of the office.

This took us to lots of different places — sometimes with a discussion guide that outlined areas of interest, sometimes with a digital prototype people could interact with.

Photograph show the back of Sophy's head as she rings the door bell hoping to speak to someone to do user research.

Hoping to understand how people use energy in their home, we took a trip to Lichfield and knocked on doors. We’ve looked for jobs at the Uber offices in a bid to understand a bit about what it’s like to be a driver. We also wandered down a high street to talk to shop owners about their relationship with other traders and with their customers. Doing all these things gave us greater confidence in the direction we’d take the projects.

Reflecting and adapting

We hit prototype testing fatigue after following GV’s Sprint method for a few weeks. We started to reflect on what the GV Sprint method offered us. We found it wasn’t providing enough insight into people’s problems, motivations and feelings. One of our experiments, Protecting your stuff, really highlighted that failing. The prototype was good, in lots of ways, and so was the idea of insurance based on trust within your community. But it didn’t explore people’s behaviour as a part of a real community in the context of our proposition.

We weren’t getting under the skin of the problem.

This sort of misstep led us to rethink how we thought about researching with prototypes. How might we prototype communities? How might we understand the mechanics of group behaviour to enable co-operation on a Job To Be Done?

Photograph of a group of people standing in a farm kitchen where the team thought about prototyping communities.

Our farm visit is an example of where we’ve given this approach a try. Read more on that experiment here: Cheaper, greener energy through smart behaviour.

Research is a team sport

We’ve each had ideas on how we might get closer to the user. Reading research papers helped our understanding of switching energy providers. And we used targeted Google Ads to get people to a website and used an Intercom chat widget where we could speak to them, in real time, at their point of need.

All of this was a collective effort.

Rather than have a dedicated user researcher, every member of the team has been involved in the research which is great. (Looking back, if we’d had a researcher, they might have helped make sure the time we spent with people clearly pointed back to the assumption we were trying to prove or disprove).

We’ve found that when every member of the team gets involved in the research process, they can understand people more and design our proposition better. As user researcher Simon Hurst says, getting each team member involved “ensures they design with the user, and not themselves in mind”.

We’ve poked and prodded along without a user researcher on the team and I wonder whether this has forced us to think differently. Learning from others — like GV’s Sprint method, or best practice led by an embedded researcher — is a good place to start, but there’s a lot to say for taking that baseline and adapting it to the specific problem at hand.

We’re also lucky to have a community of user researchers to guide us when needed. Research is integral to what we do and the onus is on us to question how we use it. We’ll keep doing that, as we move on to our next project.

James Rice
Interaction designer

How much do you know about your connected devices?

The Digital Product Research (DPR) team at Co-op Digital is exploring new products and services. We’ve been trying out Google Ventures’ Design Sprint, a framework that encourages teams to develop, prototype and test ideas in just 5 days.

Recently, we’ve looked at connected devices; everyday objects that communicate between themselves or with the internet. It’s a running joke that people don’t read terms of service documents, they just dart down the page to the ‘accept’ button so how much do they really understand about what they’ve signed up for?

Many connected devices are doing things people might not expect, like selling your personal data, or they’re vulnerable to malevolent activities, like your baby monitor being hacked. These things don’t seem to be common knowledge yet but when they start getting more coverage we expect there to be a big reaction.

A right to know what connected devices are doing

In the DPR team, we have a stance that the Co-op shouldn’t express an opinion on whether what a device is doing is good or bad. We’re just interested in making the information around it accessible to everyone so that people can decide for themselves.

In our first sprint we looked at how people relate to the connected devices they have in their homes. We found that though the people we interviewed were reluctant to switch them off at first, or to disable the ‘smart’ functionality, they were open to learning about what their devices are doing.

Influencing the buying decision

With that in mind, we looked at an earlier point in the buying process. We mapped the buying journey.

Mapping the buying journey on a whiteboard. Shows customers want to buy a TV. They research products by reading expert reviews, user reviews, looking on retailer websites and asking friends. Then they make a decision.

What if journalists and reviewers of connected devices were encouraged to write about privacy and security issues? Maybe this could satisfy our aim to influence consumers. If manufacturers knew that their terms and conditions would be scrutinised by reviewers and read by potential customers, maybe they’d make them more transparent from the start.

Our prototype

We made a website in a day and named it Legalease. The purpose of the website was to gather research. It was a throwaway prototype that wouldn’t be launched. It wasn’t Co-op branded so we could avoid any preconceptions. The site showed product terms and conditions and made it easy for reviewers to identify privacy and security clauses that could be clearer.

Shows a screenshot of Legalease prototype. The page shows an LG smart TV and highlights some of the T&Cs. Eg, 'please be aware that if your spoken word includes personal or other sensitive info, it will be captured if you use voice-recognition features'. Page shows someone's comment below: 'and then what happens to it? is it transmitted anywhere?'

The product page showed ‘top highlighted’ parts of the privacy policy ranked by votes. Annotations called into question the highlighted passage.

Shows a screenshot of another tab on the same page as first screenshot. This tab shows the T&Cs in full and contributors can highlight and comment on parts.

Another page showed the ‘full text’ – the full privacy policy document with annotations. The idea is that anybody who’s interested in this sort of thing can create an account and contribute. We imagined a community of enthusiasts would swarm around the text and discuss what they found noteworthy. This would become a resource for product reviewers (who in this case were our user research participants) to use in their reviews.

We interviewed reviewers

We spoke to a mixture of journalists and reviewers from publications like the Guardian and BBC and lesser known review sites like rtings.com. We got to understand how they write their stories.

Objectivity versus subjectivity

We found that what they write can be anywhere on the scale of objective to subjective. For example, a reviewer at rtings.com used repeatable machine testing to describe product features while a writer for The Next Web was able to introduce their own personal and political slant in their articles.

Accuracy

We found that the accuracy of their article was important to them. They’d use their personal and professional contacts for corroboration and often go to the source to give them chance to reply.

Sensationalism is winning!

We’re in danger of ‘fake news’. One of our research participants said:

“Now, with everything being on the internet, it’s pretty easy for someone who just has a couple of mates to throw stuff together on a blog and it look very persuasive.”

We found that they used a mixture of analytics and social media to measure their impact. There was no mention of being concerned with the broader impact their articles might have in terms of whether or not people bought the products based on certain aspects of what they wrote about.

Reviewers thoughts on our product

Some of our research participants made comparisons with websites that have similar structure and interactions like Genius and Medium. The annotations on the Legalease prototype highlighted ambiguity in the terms and conditions but our participants didn’t find that useful – they expected more objectivity. They were also concerned about the validity of the people making the annotations and said that lawyers or similar professionals would carry more weight and authority.

How ‘Co-op’ is the idea?

Our participants thought our prototype was open, fair and community-spirited so it reflects Co-op’s values. There were question marks around whether older organisation like Co-op can reinvent themselves in this way, though.

Reviewing security as well as features

Security and privacy are starting to show up more often in:

But after our research we don’t think reviewers would use something like a Legalease site to talk about security and privacy. Some of the journalists we spoke to thought their readers didn’t care about these issues, or that people are resigned to a lack of privacy. One said:

“People tend to approach tech products with blind faith, that they do what they say they do.”

Connecting the abstract with the real world

Our participants told us their readers are bothered by being bombarded by targeted ads and being ‘ripped off’. This leads us to consider exploring how to connect the more abstract issues around data protection and privacy to these real-world manifestations of those issues. Then we should explain why these annoying things keep happening — and in plain, everyday language.

James Rice
Product designer