Why we run tests and trials

At the beginning of the year I worked on a project with 3 of our Food stores. We were looking at using a third party to replace our current home delivery service. I’m a user researcher so I spoke to colleagues and customers about their experiences with the product we were trying out.

One day, I received an email from one of the store managers asking me to visit his store. He said they’d been having technical difficulties and they’d like some support.

But my job isn’t to provide technical support – it’s to understand why someone needs it.

It dawned on me that this store manager thought the point of putting a new product in his store for a short time was to see how well the team could use it – as if we were examining the team’s ability to fit in around this new thing, rather than seeing how well the new thing met colleague and customer needs. His email suggested he thought trials are a top-down, ‘you-must-use-this-thing’ situation, not something collaborative or something that his team could and should influence.

Setting things straight

Neither tests nor trials are about seeing whether the people using the new thing have the skills or expertise to use it. They’re about finding out whether the new thing is good enough and meets the needs of the people who’ll use it. Both help us make sure that the thing we’re building, or buying in the case of trials, will benefit the business and meet user needs.

When we test

If we’re designing something to solve a problem for colleagues or customers, for example a website to help Food store colleagues find out how to do something in their store quickly and easily, it’s essential we speak to those people and see them use the thing we’re designing as soon as possible. It helps us make sure we build the right thing.

We start with research so that we understand the problem or colleague or customer need; we make assumptions about how we could fix the problem or answer the need, and we build the simplest thing we can. Then we get it in front of users – the people we’re building it for. We make improvements based on their feedback and get a new version back in front them again for more feedback. And so on.

We focus tests on either a specific set of features or specific points of interactions in a service. By testing different things continuously, we begin to understand which features work and which don’t. And when we start to get things right, we invite more people to use it.

When we use trials

Sometimes there’s a product that fits our business needs already on the market so there’s not always a significant benefit in developing our own. Instead, we ask a small number of colleagues and/or customers to try this product so we can see how it fits within the service Co-op wants to provide. If it’s a good fit, we’ll make it available to more colleagues and/ or customers.

A participant’s role is so important

My experience is that many participants don’t understand the purpose of tests and trials. I fear that if they are told by someone at head office to use a thing for x-amount of time, their feedback might not be completely honest. I think we can work harder to help users throughout the organisation understand why we test and trial and the importance of their feedback.

Starting here…

Honest feedback is useful. Sugar-coated feedback isn’t

If your Co-op workplace is chosen to be part of a test or trial, it’s because we want to learn from you, our users. Building or finding the right thing is a conversation. It’s 2-way. Good design is collaborative meaning that you, our users, shape what we build. Everyone – managers, team leaders, assistants – can, and should, influence this.  

All feedback is valuable, even if the feedback is “this new thing you’ve asked us to use makes my work life way more difficult. And it always breaks”. For the Digital team to do our jobs properly we need to be aware of the problems so we can go back and build or buy something that you’ll actually find useful.

Tips for digital teams testing or running trials

Communicate clearly and regularly with the users you’re working with. Be clear about:

  • the purpose of tests and trials in general (maybe link to this post)
  • what you’re testing or trialling with them
  • why you’re testing or trialling ie, the difference this new product could make to the bigger picture
  • how important honest feedback is and the role it’ll play in shaping a product or service
  • that there are no right and wrong answers
  • the anonymity users can expect in any evidence we share – we don’t report back to management about specific users

Better communication will mean we’re all on the same page. And that’ll mean we’ll build better services.

Let us know how we can run tests and trials better in the comments.

Gillian MacDonald
User researcher

We’re building a user research lab

Co-op Digital is building an in-house user research lab on the lower ground floor of The Federation. It will be available to all Co-op teams, the Federation community and, eventually, the wider world.

We’re calling it Fed Lab.

What’s a user research lab?

A user research lab is a physical space. It’s usually 2 rooms: one room that looks like a lounge but with cameras and microphones installed, and another that looks like a regular meeting room but with a very large TV. The ‘lounge’ is for researchers and participants to do research tasks, like co-designing something or doing a task on a website. And the ‘meeting room’, called a viewing room, is where the team sit and watch the research happening live.

Labs help whole teams see first-hand how their designs are working or not working. This way, a team can learn together and work on changing designs and interactions there and then. This fast response to live user research is what makes a lab such an important tool in agile delivery.

Why we’re building a lab now

Co-op Digital’s remit is to create new digital products, services and platforms; pioneer new ways to cooperate online; and help our existing businesses make the most of digital. To do this, we put users and their needs first, and help teams across the Co-op do the same.

Until now, we’ve been using external labs for lab-based user and market research, but we know from past experience that having a dedicated in-house research space significantly ups the ante on how much time teams and stakeholders spend with users.

And we believe the more time we spend with users, the better.

In the long run, the lab will help us offer teams across the Co-op a complete in-house research service. As a result, we’ll be able to build consistency in research across the business, maintain knowledge and build on it in-house, and be more financially efficient and even profitable.

But it’s not just about us. The Federation, is a fast-growing community of like-minded digital businesses and innovators. Fed Lab will give these organisations access to the tools they need to learn more about their users too.

Where we’re up to

We’ve completed the discovery and design phase of the lab, we’ve chosen the technology, and we expect to have the lab up and running in the summer. We’re pioneering a new solution, so everything needs more care and time.  

In building the lab, we’re planning on using cutting-edge, military-grade technology that will not only offer a top-end experience for people in (and out of) the lab, but also support us in maintaining our data privacy standards and help us conform to GDPR.

As the project takes shape, we’ll share much more about the technology and the design. To keep up to date with our progress, subscribe to the Co-op Digital blog. For more information, get in touch with Kate Towsey.

Kate Towsey
User research operations

Matching our research approach to the project

We’ve been drafting our user research principles recently and one idea to come out of the sessions was that:

User researchers shouldn’t fall back on a single research method just because we know it.

It got me thinking about my latest project at Co-op Digital and how interviewing people in a lab may have been easier for us but our insights wouldn’t have been anywhere near as valuable.

My point is: it’s easy to stick with a research method because it’s familiar or we feel confident using it, but different projects demand different approaches and user researchers must think carefully about choosing the most suitable one.

Matching the research approach to the project

I’ve been working on a ‘later life planning’ team, looking at how we can help people plan for the future.

We wanted to have conversations with people around:

  • what planning for the future means in practice
  • their hopes for the future
  • any plans they have in place

But talking about wills, funerals, loss and what might happen to us in the future can be scary and emotional – so much so that it’s a conversation lots of people avoid having. I quickly realised that our research approach needed to be carefully and sensitively planned.

We use research labs regularly and they can be brilliant. They help the whole team witness the research first-hand and the controlled environment allows the researcher to focus on the interview rather than on logistics.

But labs can be quite clinical.

The white walls and huge two-way mirror don’t foster a comfortable, relaxed environment. I wanted the people we spoke to to be comfortable. This is something that needed to be more on their terms.

Researching in context

Photograph of hands cupping a mug at a dining table. Glasses in shot as well as a plate of chocolate biscuits.Apart from putting them at ease, the decision to visit people in their homes came from the desire to understand a wider context. What environment are people in when they have these conversations and make these decisions?

And home visits were great. Talking to someone surrounded by photos of their grandchildren, or with their pets bouncing around, helped us to understand what’s really important to them. By allowing us to see a little bit of how they live, our research participants gave us insights that we might not have got if we’d spoken over the phone or they’d talked to us in a lab.

For example, we saw:

  1. People struggling to find certain documents, despite them telling us that everything was in one place. This indicates that they might not have their plans as organised as they’d made out. We’re less likely to have found this out over the phone or in a lab.
  2. People’s expressions and body language while they had candid conversations with loved ones. One valuable insight was seeing the sense of urgency on a wife’s face when she spoke about needing to replace her husband’s expired life assurance plan. Her expression gave us an idea of what an important and worrying issue this was for her. Their body language show us how much importance each of them placed on different parts of their existing plan.  

Seeing things first-hand was a good reminder that people’s lives are messy. Anything we design or build needs to consider this.

Challenges with timing and practicalities

It took a long time to plan and prepare for the home visits. The logistics of travelling, getting lost, finding somewhere to park and finding the right spot to put the GoPro so we could record the interview was sometimes tricky. And by the time we’d introduced ourselves, talked through consent, set up the GoPro, things felt quite rushed. Next time, I’ll allocate time for these things or chat over the phone before the interview to establish a relationship and cover the basics in advance.

It’s tricky to involve the whole team

I’ve found it’s easier to get the whole team involved when we’re speaking to people in the lab – it’s one place, one day. But we carried out home visits over 2 weeks making it more difficult to pin everyone down.

We discussed who we wanted to speak to and what we’d like to find out as a team beforehand. This then fed into the plan and discussion guide. Product manager Sophia Ridge and designer Matt Tyas were able to come to the various interviews but making sure the whole team heard the voice of the interviewee and got the same insight was difficult.

We all came together to watch the home visit videos and I asked everyone to take notes as they would in a lab setting. I’d hoped we’d sort the findings and uncover themes and insights together. But 2 videos in, people were pulled onto other work. Next time, I’ll take the team out of our working space and ask them to leave their laptops behind.

Research community, how do you do it?

We’d be interested to hear how and why you’ve chosen to step outside the lab for different projects, and whether you think you got more useful insight from it. Leave a comment below.

Vicki Riley
User researcher

Vicki Riley on user research at Co-op Digital

We’re recruiting user researchers.

If you want to help digital teams build the right thing, and if learning about how people behave and why sounds interesting, you might be a good fit. Have a look our job description for more details.

Our user research team come from really varied backgrounds. Here’s Vicki’s story.

(Transcript) Vicki Riley: I’ve been a user researcher for just coming up to a year now, I’ve been at the Co-op for 2 years. Originally I worked on the analytics and optimisation team. That involved looking at the the data so Google Analytics to find out what people were doing on our websites. But being a user research now I can really delve into why people are behaving in a certain way and shape the products that we’re building based on that.

User research is about understanding the behaviour of people. So rather than asking people what they would do we would prefer to observe them. So with colleagues that could involve shadowing in-store, spend a day in the life of a colleague who works in one of our food stores to really get to the bottom of what they’re trying to do, what their pain points are and shape what we’re building based on what we’re seeing rather than just what we’re hearing.

A huge part of user research is kind of getting out of the building, going out to the places that our users are working or spending time. There are a lot of assumptions in Angel Square or in any business there are assumptions and without speaking to the people who use your services or use your products you’ll never really find out what’s happening, you’ll never really get to the bottom of it.

I love working with designers and interaction designers, content designers. They’re really involved in the research and they bring a different perspective to things. They have different backgrounds, different experiences, different knowledge so it
really helps to have a diverse group of people doing the research analysing the

I’m learning something new pretty much every week because I’m working on different teams and moving around a lot learning from people who’ve done user research but 10, 20 years so for my personal development it’s been brilliant this last year I’ve learned more than I have in my entire career.

I’ve wanted to work at the Co-op for a long time ever since I’ve finished University. They’re a company that really makes a difference in the community and I think user research has an opportunity to shape that, to shape what we do in the future.

Vicki Riley
User researcher


Simon Hurst: life as a user researcher at Co-op Digital

(Transcript) Simon Hurst: I’m Simon Hurst, I’m a user researcher here at the Co-op working on Membership.

So, Membership is key to the Co-op, it’s a lot what the Co-op is. So my role entails sort of challenging assumptions that the business might have about customers or about members. So it’s understanding how people want to engage with the Co-op in the 21st century. I’m a fairly experienced user researcher now so I’ve been through digital transformation in government. I was originally first worked on the first Department for Work and Pensions service that went live.

The ability to go and do that all over again with Co-op Digital and to help a lot of
people who were coming to user research quite new and to help them along. So I think one of the best parts of my job is mentoring people as well, so I’m sort of mentoring someone at the minute and I’ve just trained someone as a user research.

So being able to share sort of things I’ve learned as user researcher with other people and equally we’ve got a good mix of people from different digital backgrounds. So even amongst the community of user researchers, there’s people with different skill sets to me I can learn from as well. So it’s just a really good mix. And stuff we’re trying to do is genuinely trying to improve people’s lives and help people.

The things I’m looking forward to right now are really starting to influence more and more how much user research is listened to, so really getting it properly embedded now. So we’ve got the roots there we now need to build on that and so it’s making sort of links in the wider Co-op to sort of share user research findings as opposed to sort of people directly on the product teams. And trying to find other user researchers to come and join us, who can join in with that and helping to develop user researchers here.

Co-op was the only other job I’ve ever applied for apart from government in 20 years, since I left college, so I have no plans on going anywhere so it’s it’s really nailing it here I think is what I want to do.

Simon Hurst
User researcher

Find out about opportunities to work with us.

Making the move to user research

User research helps make products and services that work for the people who use them. It takes loads of different forms including lab sessions and interviews, onsite visits and analysing data but, regardless of its form, it must be present throughout the design process. And even after the thing is live.  

Moving into a user research role

I’ve worked at the Co-op for just under 2 years. I originally joined the Analytics and Optimisation team, but for the last 10 months I’ve been a user researcher at Co-op Digital.

User research really appealed to me because it’s about listening to users as well as looking at data. My old role was heavy on the quantitative side of things: I evaluated data collected from user journeys and improved the experience for users. Good user researchers consider both quantitative and qualitative research so I’ve been working on my qualitative research skills. Now I feel even better equipped to help teams design the right thing.

User research at Co-op Digital

I applied for a user research role after seeing the work that our now Head of User Research James Boardwell and the team were doing with wills. The multidisciplinary team was working in an agile way to build a digital service to make it simpler and quicker for Co-op customers to get a will.

I saw how both data and qualitative research fed into the design process. User research formed the basis for discussions and the team could test ideas, put them in front of people and iterate them quickly. The whole team came to user research sessions so that everyone saw first-hand how users behaved when we put prototypes in front of them and asked questions. The team analysed the themes that came out of the sessions together which meant that everyone had a similar idea about where the design was heading.

Everything moved so quickly and decisions were based on things that the team had seen or heard. At each show and tell the team knew so much more than the week before – they’d added another piece to the jigsaw. They’d started small and built the right thing, quickly. I loved watching their progress.

My first taste of user research

Supporting James was my first experience as a user researcher. I joined the Wills team during a sprint focused on increasing the number of people making it to the confirmation page. I already had good experience in this from my previous role but here I also got to see James talking to people, showing them the prototype and doing qualitative research in lab sessions.

The data I’d collected told us what was happening with real people using the website, and James’ conversations with people told us why it was happening. The data showed that the exit rate from the ‘Your details’ page was disproportionately high. Qualitative research told us that people felt uncomfortable giving their personal details before knowing exactly what the service offered. Changing the order of the pages, so, giving the user more upfront information, resulted in more people completing the form.

The 2 kinds of insight complemented each other. You can read more about this in James’ post, User research and sample sizes.

Learning how user research works in a product team

I spent 6 months working with the Membership team too. User research gives us the chance to test things to make sure we’re doing the right thing for users. This way, any decisions we make are better informed.

Working on Membership opened my eyes to other ways of doing research too. It’s not just about interviews. We:

  • used qualitative website feedback and quantitative analytics to compare what users told us with what they actually do
  • visited stores to find out what our members and customers talk to colleagues about
  • spoke directly to members

It’s about analysing all available resources.

Leading my first project

Photograph of a user research session. Shows 10 members of the Electrical discovery team talking about and analysing what they've seen in the user research lab.

For the last 2 months I’ve been leading the user research on a discovery in our Electrical business. This project has helped me learn a lot about how user research informs service design through techniques like customer journey mapping and service blueprints. Service design is a fairly new way of thinking at Co-op Digital so leading this project was sometimes challenging, but we’ve got a strong user research community at Co-op Digital and support and advice was always available if I needed it.

Hard work, but worth it

I think the biggest challenge for a user researcher is using all of their observations and data to find the need, and working with the team to translate these into things we can work on.

User research encourages teams to take a more balanced approach to design. It changes the way teams work and brings the business and digital sides of things together. It’s a way to stop people jumping to conclusions about what’s ‘right’ because we’re using evidence to make decisions. And ultimately, that’s going to work better.

If learning about how people behave and why sounds interesting and you want to help teams build the right thing, quickly and cost-effectively, get in touch with James Boardwell or leave a comment on the blog.

Vicki Riley
User researcher

Small is beautiful. User research and sample sizes

At the Co-op, we use both qualitative and quantitative approaches to make decisions about products. This post is about doing qualitative research with a small-ish numbers of users – and why that’s useful.

As a rule of thumb, qualitative and quantitative approaches are useful for different things:

  • If you want to understand the scale of something (such as how many users do X or Y, or how much of something is being used), use quantitative methods, like surveys.
  • If you want to understand why people do something and how they do it, qualitative methods such as interviews or seeing how users behave with a given task (user tests) are better.  

User research isn’t a one off event. It’s a process. By researching with a handful of users at a time, iteratively, and supported by data on user behaviour, we build better digital products and services.

How many users are enough?

We don’t need to observe many users doing something to identify why they’re behaving a certain way. Jakob Neilsen, a usability expert, found through research with Tom Landauer that 5 users is sufficient. More than 5 and your learning diminishes rapidly and “after the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new”. Here’s Neilsen’s graph of these diminishing returns:

Graph shows percentage of usability problems found on the y axis and number of test users on the x axis. the graph sows that we find 100% of usability problems with a relatively small number of test users.

Source: Jakob Neilsen

Analysing user data and user research findings are complementary in developing digital products and services. Data can help identify issues to then test with users, but it can also run the other way. In user research at the Co-op, we’ll often see things while doing user research which we’ll then investigate with data. It works both ways.

Screen Shot 2017-03-09 at 11.26.18

There’s cumulative value in cycles of research

The cycle of user research shown in the diagram is how product teams work at the Co-op. We typically iterate in weekly or fortnightly cycles.

For example, the Membership team has a rhythm of fortnightly cycles. These are often focused on discrete aspects of Membership. These research cycles accumulate learning over time. They create an understanding of Membership and of users needs. Cumulatively, this gives clarity to the whole user journey.

During the last 10 months, the Membership team have surveyed 674 users and interviewed 218. The value of this research accrues over time. The team has learnt as they’ve developed the service and iterated on the findings, getting to know far more than if they’d done the research in one block of work.

That’s why observing relatively few users doing a task, or speaking to a handful of users explaining something they’ve done, is enough to provide confidence in iterating a product and to continue to the next test. This is especially true when user research is used together with data on user behaviour and even more so when it’s done regularly to iterate the product.

Error-prone humans are there in quantitative research too

It’s not uncommon for people to give more weight to quantitative data when they’re making decisions. Data is seen as being more factual and objective than qualitative research: “you only spoke to 10 people, but we have data on thousands…!”

Data hides the error-prone human because humans are invisible in a spreadsheet or database. But even though they’re hidden, the humans are there: from the collection of the data itself and the design of that collection, to the assumptions brought to the interpretation of the data and the analysis of it.

All data is not the same

Survey data, based on responses from users, is distinct from data collected on behaviour through Google Analytics or MixPanel. Poor survey design produces misleading insights.

Getting useful behavioural data from a user journey is dependent on setting up the right flows and knowing what to track using analytics software.  Understanding what constitutes ‘good’ data and how to apply it is something we’re working on as a community of user researchers at the Co-op.

Research is a process, not a one-off

Digital product teams usually have a user researcher embedded. They can also draw on the skills and experience of the conversion and optimisation team and their quantitative and statistical skills and tools. The user researcher gets the whole product team involved in user research. By doing this, they gain greater empathy for and understanding of their users (and potential users).

This diagram shows some of the methods we use to help us make good product decisions and build the right thing to support users:

Screen Shot 2017-03-09 at 11.36.18

As user researchers our craft is working out how and when to deploy these different methods.

Part of the craft is choosing the right tool

Let’s take an example from a recent project I was involved in, Co-op wills, where we used both quantitative and qualitative research.

We had customer data from the online part of the service and analysed this using a tool called MixPanel. Here’s part of the journey, with each page view given a bar with corresponding number of visitors:

Screen Shot 2017-03-09 at 15.38.11

From this, we could determine how many users were getting to a certain page view of the wills service, and where they were dropping out.

The data let us see the issue and the scale of what’s happening, but it doesn’t give us a sense of how to resolve it.

What we didn’t know is why people were dropping out at different parts of the journey. Was it because they couldn’t use the service, or didn’t understand it, or because they needed information to get before they could complete?

To help us understand why people were dropping out, we used user data to create hypotheses. One of our hypotheses was that “users will be more likely to complete the journey if we start capturing their intent before their name and email address” ie, show them the service before asking them to commit.

Through user research with small numbers of users we found a series of different reasons why people were behaving in ways Mixpanel had showed us, from confusion over mirror wills to uncertainty about what the service involved, to requiring more information.

We only got this insight through speaking to, and observing users, and getting this insight allowed us to design ways to fix it.

It’s not an exact science – and that’s OK

Research is not an exact science. Combined with user data, user research is a process of understanding the world through the eyes, hands and ears of your users. That’s why it’s central to the way we’re building things at the Co-op.

James Boardwell
User researcher