Recently, in his post How we’ve helped users understand Membership, user researcher Simon Hurst said that “it’s fine to ‘fail’ as long as you do it quickly, learn from it and make changes to make things better.” It made me think about my most recent example of failing fast and how useful it was for the ‘more members’ part of the Membership team to do a quick, inexpensive trial so we could test an idea.
The problem with temporary cards
You can become a member by signing up online. You register your details, pay your £1 and your Co-op Membership card is sent to you through the post. You can also sign up in our food stores. You pay you £1 and you receive a temporary card to use there and then. The idea is that you’ll go online to register your temporary card later.
However, our user research and data show this isn’t what’s happening. 58% of temporary cards we’ve sold haven’t been registered. This is a problem because:
- around £1 million of 5% reward is sitting in a pot, and can’t be spent until the temp cards are registered
- we can’t get in touch with customers to let them know the balance they have because their temp card isn’t registered
- until they register the card, customers can’t access all the member benefits. For example, they can build up their rewards but they can’t spend them or choose a local cause to support
To try and increase the number of temporary cards being registered we ran a few trials in stores. We dubbed one of these ‘the flyer test’.
Encouraging temporary card holders to register
Here’s our hypothesis:
To test this hypothesis we asked colleagues on tills in 10 stores to watch out for customers who were swiping a temporary card. When they spotted this happening, we asked them to hand those customers a flyer which had a call to action on it: ‘register your temp card’. The flyer also explained the benefits of registering the card to try and nudge people into registering.
We included a vanity URL so we could track how many people registered their cards after receiving a flyer. Simple.
We had our hypothesis and agreed our test. Our first failure was cracking the logistics of designing, printing, delivering leaflets across the country. That was hard, and so was making sure our store colleagues understood why we were doing this. This was our first learning: there are colleagues across the business great at doing this, and working with them is better than working alone.
We hadn’t fixed anything. And that’s hard to take
We sent flyers to 10 stores across the country and asked them to hand them out for the next 4 weeks. We put Google Analytics tracking in place and we decided on our measure of success: 10 visits to the URL a week, with 50% of those going on to register their card.
The test went live and we eagerly refreshed the Google Analytics report each morning waiting to see an improvement in temporary card registration. There were none. Nobody was visiting our URL.
We called the test stores. Maybe they hadn’t been handing the flyers out? Turns out they had. And what’s more, colleagues liked them because the flyers were an easy, concise way to tell customers why they should register their cards.
But they weren’t working for customers.
Over 4 weeks, 35 people visited the URL, and 3 of those people registered their cards. We hadn’t hit our measures. The test had failed.
We learnt lots, quickly
The trial taught us that:
- People don’t naturally move from a physical thing (a flyer in a shop) to a digital thing (our website). Even if you spell out all the really great reasons why they should. If moving from physical to digital was a natural thing for people to do, they probably would have already registered their temporary card.
- Involving wider team members early on is important because they may have ideas, sometimes tried and tested ones, about how to get stuff done.
- We should test an idea from as many angles as we can before we go ahead and roll it out further. We based our hypothesis on user research, then came up with an idea that we thought would test it. If we had looked at the data as well, we would have seen that there are only around 50 active temporary cards per store, and that these cards are only seen around around twice a month. So…
- Targeting active temporary cards isn’t the best way to solve the wider problem.
Learning a lesson cheaply, and on a small scale
We often say it’s okay to fail, but it’s still disappointing when you’ve put time and effort into something. You start picking it apart. Maybe we picked the wrong stores? Or the wrong time of year? Or the wrong colour flyer?
No, those things don’t matter – our idea just wasn’t that great.
Failing is ok, as long as you recognise when to let your idea go and move onto tackling a problem another way. So yes, we failed but we only failed in 10 shops, not all 3,000. We didn’t spend much money, we didn’t inconvenience our users and we were open about how the tests were performing in our weeknotes and our show and tells.
Most importantly we learnt enough to inform where we should focus our efforts next.
We’re moving away from encouraging users to do something towards giving them the tools they need to do it there and then – our next trial will test if customers would register their temporary cards on a tablet in store.
Digital business analyst