data science

Taking a content design approach to how AI could help our colleagues

Co-op
A colleague in a Co-op store is standing in the aisle of a store holding a small handheld device. She is inputting information into the handheld device. There are jars of jam and containers of coffee on the shelves behind her.


Our ‘How do I’ (HDI) website was created by content designers pair-writing with store and operational colleagues. The aim was to provide operational policy information, in a way that was easy to understand, in a busy store environment.

Store colleagues rely on ‘How Do I’ to comply with legal regulations and maintain high standards of customer service. Colleagues tell us it’s useful, but difficult to find some information quickly. Our Content Design and Data Science teams worked together to test how using generative artificial intelligence (AI) and a large language model (LLM) could help.

It proved to be a great opportunity to learn from how content designers can work with teams who want to make the most of AI capability.

Taking a content design approach

As a Content Design team at Co-op, we create content that is evidence-based, user-focussed, and based on shared standards to meet our commercial goals. We want to keep these content design principles at the centre of our approach to AI generated content.

The teams designed a process that combined a Co-op built AI and a Microsoft LLM. It means that when a user enters a query, a Co-op built AI system looks at a copy of our ‘How do I’ website and finds the information that is most likely to be relevant. It takes this data and the original question, and feeds it all to a Microsoft LLM. The LLM then generates a response and passes it back to the user as an answer.

How the AI works

There are a number of illustrations to show a process of how the AI works in steps. Illustration 1: hands using a phone: Colleague types a question into AI HDI Illustration 2: a screen with a magnifying glass and options: AI search engine looks up relevant information. from HDI. Keyword and semantic search. Passes the question and relevant info to LLM Illustration 3: Letters LLM in a file: LLM generates a response and sends it back to the AI Illustration 4: Mobile showing a list :Answer is provided to colleague

All of the content on the ‘How Do I’ (HDI) website was created and designed according to content design principles. As a result of the way LLMs work, without content design expertise, LLMs generate new content that is not subject to the same rigorous user-focussed design processes.

We needed to test how the AI was working to make sure it does not give misleading, unclear or inaccurate information. We analysed search data and worked with colleagues to identify the common queries they search for. This helped us to build an extensive list of test questions covering a wide range of operational, legal and safety related themes.

Testing and analysing the AI responses

When we tested the AI system with questions, we used the language our colleagues used. We asked simple questions and complex questions. We included spelling mistakes and abbreviations, then we analysed the AI system responses.

We took a content design approach and used our content guidelines to assess the responses. Validating the accuracy of responses included fact checking against the original ‘How Do I’ content to understand whether the AI had missed or misinterpreted anything.

We used this analysis to create a number of recommendations for how to improve the content of the AI responses.

Accuracy

Almost all the AI system responses provided information that was relevant to the question. But analysis showed it sometimes gave incorrect, incomplete or potentially misleading information. ‘How do I’ contains a lot of safety guidance, so to avoid risk for our colleagues, customers and business, we needed to make sure that any responses are always 100% accurate.

Accessibility

The initial AI system responses were hard to read because they were stripped of their original content design formatting and layout. Some of the responses also used language that sounded conversational, but added a lot of unnecessary words. LLMs tend towards conversational responses, which can result in content that is not accessible. It does not always get the user to the information they need in the simplest way.

Language

The AI did not always understand some of our colleague vocabulary. For example, it struggled to understand the difference between ‘change’ meaning loose coins, and ‘change’ meaning to change something. It did not understand that ‘MyWork’ referred to a Co-op colleague app. This meant it sometimes could not give relevant answers to some of our questions.

Using content design to improve the AI

Our Content Design team is now working with our data science team to explore how we can improve the AI system’s responses. We’re aiming to improve its accuracy, the language the AI uses, and reduce unnecessary dialogue that distracts from the factual answers. We’re also exploring how we can improve the formatting and sequencing of the AI responses.

This collaborative approach is helping us to get the most out of the technology, and making sure it is delivering high quality, accessible content that meets our users needs.

Based on the content design recommendations, our data science team have made changes to instructions that alter parameters for the AI, which is also known as ‘prompt engineering’. This affects the way the AI system breaks down and reformats information. We’re experimenting with how much freedom the AI has to interpret the source material and we’re already seeing huge improvements to the accuracy, formatting and accessibility of the responses.

Impact of the innovation of this AI work

“The ‘How Do I’ project has been hugely innovative for the Co-op. Not only in the use of the cutting edge technology, but also in the close cross-business collaboration we needed to find new solutions to the interesting new problems associated with generative AI. We’ve worked closely with Joe Wheatley and the Customer Products team, as well as colleagues in our Software Development, Data Architecture and Store Operations teams. We’ve been able to combine skills, experience and knowledge from a wide range of business areas and backgrounds to build a pioneering new product designed with the needs of store colleagues at its core.”

Joe Wretham, Senior Data Scientist

The future of AI and content design

AI has so many possible applications and its been exciting to explore them. This test work has also shown the critical role content design has in making sure we are designing for our users. AI can create content that is appears to make sense and is natural sounding, but the content needs to help users understand what they need to do next, quickly and easily.

Content designers understand users and their needs. This means understanding their motivations, the challenges they face, their environment, and the language they use. The testing we’ve done with the ‘How do I’ AI system shows that AI cannot do this alone, but when AI is combined with content design expertise, there are much better outcomes for the user and for commercial goals.

The content design team at Co-op have been exploring how they can balance current content design responsibilities with exploring skills and new areas for development in AI.

Blog by Joe Wheatley

Find out more about topics in this blog:

What we learnt from talking to our members about data

3 Comments

On the Friday before our AGM, we held an event at Federation House so we could continue the conversation about how the Co-op uses and shares members’ data. We wanted to invite our members to help us shape our data policy in person. It was an open invitation and 63 people took the time to chat to us.

We ran 3 workshops to find out:

  1. How the Co-op compares to other businesses when it comes to being trusted with data.
  2. What data are people willing to share for social benefit and commercial benefit.
  3. What types of usage of data people are happy with.

Being trusted with data

IMG_5889

In this workshop, we asked people about the organisations they trusted (or didn’t) to use their data, with examples from banks, telecoms, member organisations, broadcasters to bring the subject alive. Depending on their personal experiences with those organisations, people have very different views around who they trust to hold their data securely and use that data sensibly.

An important thing we learnt was that the organisations people trust with their data have very clear reasons for why they hold different data and how they use it. Some people thought that, as the Co-op, we might end up with large amounts of data from across our different businesses (insurance, food, electrical and legal services). They wanted us to be clear about how we use those different types of data. So, as we build new data stores, we need to make sure that we’re careful and transparent when sharing members’ data across the Co-op.

Willingness to share

IMG_5906

Our second workshop asked if members were OK with sharing different types of data with the Co-op – information like their age, gender, salary, religious beliefs. We split the group into 2. One half was thinking about sharing data for commercial reasons, to improve our products and services. The other half was thinking about sharing data for social reasons, like community research.

We gave the groups a scale of how comfortable they were with sharing, from “Not at all” to “Yes please”.  People have many different opinions and different worries, concerns or reasons for sharing. Most people made up their minds with a firm “no” or “yes”, but the reasons for their answer varied widely, and some people changed their mind as the group debated the issues.

What does this tell us? Well, if we’re going to be trusted with holding more data, we’re going to have to give people choices around how their data might be used, both within the Co-op and externally.

Play your consent right

The third workshop was a game where people voted about whether they would give consent to companies to use their data for specific purposes. For example, if people would consent to the Co-op using their habits of purchasing pet food from Co-op stores to let them know about special offers in pet insurance.

Feelings were pretty much summarised by 2 responses:

  1. “I’m an individual – don’t assume what I’m interested in by age, postcode, gender.”
  2. “Maybe I shouldn’t provide more data about myself in case I miss out on special offers that the Co-op targets at particular people.”

We’ll need to think about how to use data to help people find the things we think they are most interested in, whilst not precluding people from other offers.

IMG_5914

Hearing from data experts

We live-streamed the opening discussions and you can watch the keynote and panel discussions on our Youtube channel.

A massive thank you to our experts who gave up their time to travel to Manchester, speak, answer questions and mingle throughout the event. So thank you:

What’s next

It’s not too late to join the conversation because we’ll keep talking to our members about data. Last week we published a post called Speaking to our members about how their personal data is used. We’re going to look at and analyse the feedback we got from the survey more closely to pick up more themes and add what we found out at this event

In the coming weeks, we’ll join Data Leaders to plan improving how we use data across the Co-op. We will also be working with our colleagues in Data Protection. And of course, we’ll discuss all of this with our Members’ Council and advisory boards.

Rob McKendrick
Head of Data Engineering

Our data team is growing, and we’re looking for talented people to join us in a number of different roles. Find out more about working for Co-op Digital.

Looking for different ways members can get involved with Co-op

Co-op2 Comments

Recently we started looking at different ways in which our members can get involved with our Co-op and share their opinions. Some examples of what members have supported so far are:

To get involved, members simply go to Co-op Membership and find something they’d like to join in with. 

Screen shot of the local causes areas of our website
Co-op Membership 

Not everyone is confident with digital channels and there’s a perception that it’s only young people who will embrace things like this. It would be a shame if that was the case, because the invitation to participate is open and relevant to all our members.

To make sure that we’re including as many our of members as possible we’re keeping an eye on the sign-up data. Using this we’re learning what we can do better.

This data is handled anonymously, sensitively and securely. This is about us using data for the benefit of our members to help us to be an inclusive Co-op.

I had a look at some of the data and plotted the following histograms of the ages of Members who signed up for the opportunities listed at the start of this post:

An image showing co-op Member ages and sign up's to our Join In initiative

To me, this data tells us we’re attracting members from a wide range of ages. The different opportunities themselves are appealing to different ages too. There’s a lot more to be said about this data but I’ll leave it there for now and welcome readers to comment.

Alex Waters
Data Science

 

Hello to Steve Fisher

Co-op

Following on from my last blog post where I welcomed Kevin Humphries to the team, I’m pleased to introduce Steve Fisher to CoopDigital as head of data projects.

Picture of Steve Fisher - head of data projects

If you’ve been following the CoopDigital blog then you will know that our goal is to become the membership organisation that is most trusted with data. In which data, in its broadest sense is something that can create more value for members and their communities.

Steve will help us to do that. With over 20 years extracting insights from very large and complex data sources, most recently with Oliver Wyman where he was the head of data engineering.

Working with cross functional teams across Co-op, Steve will ensure that colleagues that need to make decisions have access to the data and insight they need in a way that makes it really easy for them to use and they can get the greatest value from.

You’ll hear more from Steve in the next few months. Welcome Steve.

Catherine Brien
Data Science Director