The Post Office’s Horizon IT system should “never have seen the light of day” and bosses at supplier Fujitsu allowed it to be rolled out into the Post Office network despite being told it was not fit for purpose, according to a senior developer who worked on the project before it went live.
Horizon is the system at the heart of the scandal that saw hundreds of subpostmasters wrongly accused of fraud and false accounting until a 2019 High Court case proved the Post Office IT was at fault. Horizon is a counter-top sales and accounting system, commissioned by the government to automate Post Office services. It went live in Post Office branches in 1999 and an updated version is still in use today.
The developer, who has not previously talked publicly about his experiences on the project, told Computer Weekly that in the months leading up to its launch, Horizon’s problems were well known inside Fujitsu.
“Everybody in the building by the time I got there knew it was a bag of s**t”, he said. “It had gone through the test labs God knows how many times, and the testers were raising bugs by the thousand.”
The senior developer said he was contracted to work on the Horizon project between 1998 and 2000, at one point holding the job title Horizon Epos [electronic point of sale] development manager. He has asked to remain anonymous, but is prepared to give sworn witness statements to solicitors acting for subpostmasters in their ongoing appeals against past convictions.
The developer has also asked Computer Weekly to pass his contact details to the government’s Post Office Horizon IT inquiry, chaired by former High Court judge Wyn Williams.
Central to his allegation is that Horizon’s Epos system was initially built with “no design documents, no test documents, no peer reviews, no code reviews, no coding standards”.
He told Computer Weekly: “To my knowledge, no one on the team had a computer science degree or any degree-level qualifications in the right field. They might have had lower-level qualifications or certifications, but none of them had any experience in big development projects, or knew how to do any of this stuff properly. They didn’t know how to do it.”
In 2019, a High Court judge ruled that Horizon was “not remotely reliable” for the first 10 years of its existence. The judge found the IT system was prone to throwing up errors that could and did affect individual subpostmasters’ branch accounts. As Computer Weekly has been reporting since 2009, the Post Office held postmasters liable for these discrepancies. Some were prosecuted and imprisoned. Others lost their jobs and life savings.
Largest commercial IT system in Europe
When it was rolled out, Horizon was described by Fujitsu as the “largest non-military IT system in Europe”. About 40,000 Horizon terminals were installed in all Post Offices across the UK. The user interface was a touchscreen and keyboard linked to a PC under the counter which ran on the Windows NT operating system. Branch PCs were connected via ISDN to a back-end mainframe. The Fujitsu-designed Epos software on the PCs was written onto an off-the-shelf system called Riposte.
Our source said the big flaw in Horizon was the way data was being written to Riposte.“Riposte wasn’t really a database, it was a messaging system based on an XML structure where you write messages down into the message store, and then Riposte took care of replicating them,” he said.
“The first thing that you should always do with a system like that is design and agree a data dictionary and a message library repository, basically to say: these are the messages that are allowed to be written to the message store and they all provide the following function.
“It’s almost like an API [application programming interface] so that you have a list of allowed messages that can all be written to the correct format with the correct content.
“You should also have a layer of software that lies on top of the message store that checks that any application above it which is trying to write a message, conforms to the agreed data dictionary. Otherwise, you can just write freestyle to the message store, which is what they were doing. There was no application interface in there, no agreed data catalogue or anything.”
Computer Weekly also spoke to a former Fujitsu employee who worked in the Horizon Service Support Centre from 2001. The support engineer – who also wants to remain anonymous – recognised this new description of Horizon’s badly built message store, adding: “Our job was to fix these problems as they arose. We all knew the code wasn’t fit for purpose and needed rewriting. The data dictionary was still being added to when I got there.”
Senior managers were aware
The most serious allegation raised by the developer is that senior managers at Fujitsu were aware that an important element of the Horizon system did not function correctly and could not be fixed.
For the first 10 years of Horizon’s existence, transaction and account data was stored on terminals in each branch before being uploaded to a central database via ISDN. Our source says this part of the system simply did not work.
“The cash account was a piece of software that sat on the counter NT box, asleep all day,” he said. “At the end of the day, or a particular point in the day, it came to life, and it ran through the message store from the point it last finished. It started at a watermark from yesterday and combed through every transaction in the message store, up until the next watermark.
“A lot of the messages in there were nonsense, because there was no data dictionary, there was no API that enforced message integrity. The contents of the message were freehand, you could write whatever you wanted in the code, and everybody did it differently. And then, when you came back three weeks later, you could write it differently again.”
He gave an example of a message stored previously when a customer bought a stamp. It was feasible that a new message for buying a stamp weeks later could be slightly different.
“When the cash count came along, it found a message it was not expecting and either ignored it, tripped up, or added something it shouldn’t be adding,” he said.
In 2015, Computer Weekly reported another anonymous source who identified the cash writing program as a possible cause of serious problems. He told us the Post Office was warned about the risk of data corruption on the bespoke asynchronous communication system which sent messages between branches and the central Horizon set-up.
Speaking to Computer Weekly in 2015, the anonymous source told us: “The asynchronous system did not communicate in real time, but does so using a series of messages that are stored and forwarded, when the network connection is available. This means that messages to and from the centre may trip over each other. It is perfectly possible that, if not treated properly, messages from the centre may overwrite data held locally.”
Four years later, former Fujitsu engineer Richard Roll wrote in a witness statement to the High Court: “The issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at branch level.”
Roll’s evidence, which was accepted by the judge, suggests that the problems with Horizon identified by our source had not been dealt with by the time the system went live.
Other experts familiar with Horizon that Computer Weekly approached have also supported the developer’s claims.
The developer said he made his superiors at Fujitsu aware of the extent of the Epos system problems, telling them explicitly that the cash account needed to be scrapped.
“I broke it down and said: you can keep these bits at a push if you have to,” he said. “But that bit in the middle, these bits of the engine, the gearbox, you need to throw them away and rebuild them. Starting with the cash account. You’ve got to throw the cash account away and you’ve got to rewrite it.”
Bone of contention
Our source said Fujitsu’s unwillingness to deal with this central problem became a bone of contention with him. He said he was offered more responsibility on the project but refused unless the cash account was rebuilt from scratch. This, he claimed, resulted in him being moved off the Epos side of the project to become Horizon’s LFS [Logistics Feeder Service] team development manager.
Computer Weekly contacted Fujitsu, asking for a response to the developer’s allegations. A spokesperson for the company said: “As a long-term partner to UK public and private sector organisations, we are dedicated to supporting our customers, our employees and the people they serve in the UK. We provided detailed responses to all questions raised by the House of Commons BEIS Select Committee and are continuing to cooperate with the ongoing Post Office Horizon IT inquiry.”
The Post Office said it would not be appropriate to comment on individual allegations outside of the current independent inquiry and the courts.
The Department for Business, Energy and Industrial Strategy (BEIS), which holds government responsibility for the Post Office, said it had “established an independent, judge-led inquiry to ensure that lessons are learned, and that concrete changes take place at Post Office Ltd. If anyone has relevant information, we would encourage them to provide it to the Post Office Horizon IT inquiry.”
The Williams inquiry into the Post Office Horizon IT scandal is accepting evidence until 25 February and is due to report in the summer.
A Computer Weekly investigation in 2009 first revealed that subpostmasters, who run Post Office branches, were being blamed for unexplained financial losses, which they claimed were caused by errors made by the Horizon system. The Post Office denied this, and many subpostmasters were subsequently prosecuted for theft and false accounting, with prison sentences, community service, criminal records and heavy fines among the injustices they suffered as a result.
It has become one of the biggest miscarriages of justice in UK history.
