I joined the Digital Engineering team in February as the Principal Service Manager. We’ve had a busy few months designing how we’re going to run the new digital services, the first of which being the Local Causes application website which forms part of our new Co-op Membership.
A different way of working
Coming from a background of traditional IT systems with on premise infrastructure, I knew from the moment I walked onto the 13th floor of 1 Angel Square that I’d have to start thinking differently. Every wall, surface and window was covered in sketches, Kanban boards and post its and the floor was buzzing with energy.
As a service team, our job is make sure the systems and services keep running, whether that’s by handling incidents, tackling problems or making sure that changes don’t cause outages or new issues. And that last one has been one of our biggest challenges. In the new digital teams, the pace is much faster than anything we’d been used to. Previously we’d been used to handling one or two big changes a month. The digital teams were aiming to release daily – we needed a different way of working.
The systems are quite complex with lots of different moving parts using lots of different technologies. Some front-end components like the website are built using new tools and technologies that build in automated deployment and automated regression testing. Other components down the stack are slower moving – they haven’t been built using these tools and require more manual intervention. We needed to build a change process that didn’t cause a bottle neck to releasing new features but at the same time would give us enough assurance that changes to one component weren’t going to cause issues up and down the stack (and at the same time making sure we’re not drowning in admin that doesn’t add value.)
Whilst we’re still keeping in line with the Co-op’s core change policies, we’ve tweaked the way we work to enable us to handle the higher volumes of change. At the moment whilst there’s a lot going on, we’re having daily Change Approval Board (CAB) meetings to make sure representatives from each of the components are aware of the changes going on across the whole ecosystem. This way we’re catching any potential conflicts and we’ve seen some great challenges between the teams to make their changes safer by improving their testing or deployment approaches.
We’re trialling different ways to make sure everyone knows what’s going on, from post-its on a whiteboard to venturing into the world of chat ops with a shared calendar integrated into a Slack channel. And as we go, we’re collecting feedback from all of the different teams – What’s not working? How can we make it more efficient? How could we tackle the admin differently?
We hit a great milestone this month – over 13 days we hit an average of one successful deployment a day to the Co-op Local Community Fund. Whilst we’re not quite at full continuous delivery levels yet we’ve learnt a lot in getting this far.
Principal Service Manager