Friday 7 June 2019

Horizon trial day 17: PEAKs and troughs

Jason Coyne
Who would be an expert witness? I don't know what Jason Coyne was getting for sitting in the witness box of Court 26 for the last four days, but it wasn't enough.

One thing I have learned from my experience of reporting every day of the first (common issues) trial in this litigation is that it is dangerous to draw conclusions from what you see and hear in court. 

Furthermore I am not privy to all the information both sides have got at their fingertips, I haven't been given the joint statements and nor yet (you'll forgive me) have I had an opportunity to pore over each of Jason Coyne's 200+ page reports

So what follows is my impression of what happened in court today from listening to Mr Coyne's final day of cross-examination. And my impression is that Mr Coyne took an absolute battering.

The morning session was just one win after another for the Post Office QC, Mr Anthony de Garr Robinson.

Mr de Garr Robinson started with the list of 29 bugs in Horizon which Mr Coyne had asserted caused a discrepancy with a lasting impact. Then he set about steadily demolishing them.

Phantom transactions

First up was phantom transactions. There were two examples of phantom transactions which Mr Coyne had found originating in 2000 and 2001. None since.

Mr de Garr Robinson read out the Fujitsu report (known as a PEAK) into the 2000 example. The transactions were loaded onto the system by the Subpostmaster who then pressed a button which suspended the session.

So "these were transactions that were actually
 typed into the system. They didn't appear out of nowhere" and "the postmaster
 accidentally must have touched "suspend" with her hand
 or something with the result that the session was suspended for that period of time.

Mr Coyne agrees, but says it was the registering of that basket of transactions which was done by Horizon, not the user.

"I see", says Mr de Garr Robinson "So what you are suggesting is that if the system automatically commits an uncommitted basket after a period of time, that's evidence of phantom transactions? That's evidence of a bug in Horizon that needs to be corrected?"

"Yes", says Mr Coyne.

Mr de Garr Robinson directs Mr Coyne to Angela van den Bogerd's witness statement which notes this is a design quirk of Horizon. If a bunch of products sit in a basket for long enough on the screen Horizon will turn them into a sale automatically.

"So this isn't evidence of Horizon going wrong, is it?
" asks Mr de Garr Robinson. "It is an example of Horizon doing what it was supposed to do."

"It is evidence of the system doing something
 without the user choosing to do it." retorts Mr Coyne.

But that is not the point. It is not a bug in the system, and Mr Coyne describes it as this in his report.

Lasting losses

Mr Coyne then starts to dig himself a hole, claiming he cited it as a bug because he hadn't seen Angela van den Bogerd's witness statement explaining Horizon's little quirk.

Mr de Garr Robinson is ready for this. "You did." he says, and directs Mr Coyne to the section of his report entitled Angela van den Bogerd in which he records responding directly to her evidence about phantom transactions.

Mr de Garr Robinson tries again. He asks Mr Coyne to accept this is Horizon working as designed and not a bug.

Mr Coyne does acknowledge this, but points out that the experience would be confusing for the user.

Mr de Garr Robinson responds: "If we were looking at a table which was a
 table of areas where users might get confused, then we
 wouldn't be having this conversation. We are having this conversation because you have added
 this piece of evidence as evidence in support of the
 proposition, firstly, that there are bugs in Horizon
 and, secondly, that they cause losses, lasting losses to postmasters."

Having destroyed the PEAK listed in 2000 as an example of a phantom transaction, Mr de Garr Robinson then attempts to euthanise the only other example from 2001, suggesting it wasn't a phantom transaction at all, but an example of Horizon's default timeout quirk of turning a basket of transactions into a sale.

Mr Coyne pushed back quite a lot on this, but Mr de Garr Robinson asked him see how the PEAK progresses. We see, in another box, a note from a senior Fujitsu engineer, Pat Carroll, which makes it clear he has investigated thoroughly and concluded it was user error.

"Would it be fair to suggest
", asks Mr de Garr Robinson "that the best judgment to trust in relation to what was
 going on here is Mr Carroll's judgment?... Do you think you are in a position now, in 2019, looking at this PEAK nearly 18 years ago, to say that Mr Carroll, with the familiarity he had with the system and with the information that he had at his fingertips, he must have been wrong?"

Mr Coyne says on the evidence he can't be sure that's right. Mr de Garr Robinson reformulates his question: "Do you think you now, with the information you have,
 are in a position to say that Mr Carroll was wrong?"

Mr Coyne accepted he couln't. So why, Mr de Garr Robinson wanted to know, hadn't he mentioned the conclusions of Mr Carroll into his report? Surely that's what a fair, independent expert should be doing.

Mr Coyne accepted perhaps he should have done so.

A heavy beating

And on it went. PEAKs and KELs (Known Error Logs) which Mr Coyne had cited to support his evidence of bugs causing lasting impact on Subpostmaster branch accounts were examined. On each occasion he was forced to accept that it was at the least reasonable to accept that in most cases there was a bug which was spotted and resolved as part of Horizon's counter-measures, or no bug at all.

Whilst I don't want to suggest that Mr Coyne was not making valid points, and pushing back in some significant ways throughout the morning, the overall impression I got was that he was taking a heavy beating.

But that was nothing as to what happened next. Mr Coyne already been grilled on a PEAK which described a recovery failure. Some data had gone missing, leaving a branch with a discrepancy. According to Mr de Garr Robinson's reading of the PEAK log (and don't forget I couldn't see any of these PEAKs or KELs referred to today) the issue was resolved when the recovery failure was flagged by the system, Fujitsu got involved, called the branch and between the Subpostmaster, Fujitsu and the Post Office it all got resolved.

Mr Coyne agreed it would be reasonable to assume, looking at the evidence, that the bug was recognised and the Subpostmaster was recompensed, but he also made the point that the recovery failure caused a discrepancy.

Then things really started to unravel. Mr de Garr Robinson noted Mr Coyne was very keen to point out a problem, but not the fix. He had chosen to list this PEAK as one of his 29 examples of a bug causing a lasting discrepancy.*

"First of all" said Mr de Garr Robinson, "there's not evidence of a bug, but more importantly... the evidence constituted by this
 PEAK demonstrates that if there were any discrepancy caused, it would have been picked up in the process."


When asked why he had included it on his list, Mr Coyne said not everything on the list of 29 bugs was a list of 29 bugs which created a lasting discrepancy. It was a list of 29 bugs errors and defects which had an impact on branch accounts.

Mr de Garr Robinson asked Mr Coyne why it had appeared in a list on a joint statement under the title "distinct bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in branch accounts"*

It was a question to which there was no real answer.

After lunch the situation went into freefall. Mr de Garr Robinson wanted to know how many bugs out of the 29 listed in the joint statement were not actually "bugs" causing a "lasting discrepancy in branch accounts".

Mr Coyne had decided he was pretty sure 21 were, so eight weren't.
Tony de Garr Robinson QC, (r)

Entirely reasonably, Mr de Garr Robinson asked Mr Coyne to list them. Over the next excruciating hour (and feel free to read this in the transcript) the judge, Mr de Garr Robinson and eventually even Patrick Green, QC for the claimants, looked to Mr Coyne as he tried to work out which of the 29 lasting bugs were the eight which weren't.

The judge made a list, Mr de Garr Robinson made a list. They compared notes. The numbers didn't match up with Mr Coyne's new total. They went back to Mr Coyne. Mr de Garr Robinson, who was no doubt enjoying himself, returned to the central question - how could the defence and, indeed the claimants have been led to believe Mr Coyne thought there were 29 bugs that caused a lasting discrepancy in branch accounts?

Total humiliation

"Everybody on both sides of the court
 thought that was the position." he said "How could that possibly be the case?  How can your own counsel, Mr Coyne, have made that mistake?

The judge intervened - noting that the "lasting discrepancy" line came from both Dr Worden and Mr Coyne: "On the basis it is a joint statement,
 Mr de Garr Robinson", he warned "I think you have to be quite
 careful about how you put the question.

Quick as a flash Mr de Garr Robinson produced his trump card. Not only was the lasting discrepancy phrase in the joint statement, it was in the claimants' opening statement.

Which Mr Coyne then admitted he had probably never read.

It was a total humiliation.

At three o'clock Mr de Garr Robinson all but wiped the blood off his sword and handed it to his equerry. He turn to the judge:

"My Lord, I wonder whether it would be a convenient moment to break so that I could look at the
 list and decide...

"what you are going to do.
" finished the judge.

".... what I'm going to do." repeated the QC, possibly slightly dazed.

The spring in the step from the Post Office legal team as they marched out of court was very noticeable.

After the break Mr de Garr Robinson put to one side the heady atmosphere of the previous session and got back down to business.

He used the 35 minutes he had left of his four day cross examination to look at the Dalmellington bug, which affected Post Offices with outreach branches.

By the end of the 35 minutes he had demonstrated to Mr Coyne from Post Office internal investigations that of the 108 branches known to be affected, by the time the bug was discovered, 104 of them had noticed the dodgy transactions and reversed them themselves or had transaction corrections issued automatically by Horizon. Of the remaining four, two were manual errors caused by someone typing in a bag number twice and two suffered discrepancies of a pound or less and so weren't investigated.

Mr Coyne accepted, in not so many words, that whilst there was clearly a bug, it was very unlikely that anyone was out of pocket.

And so four long days of cross-examination came to an end. What little evidence Mr Coyne had found of serious Horizon errors causing lasting discrepancies to branch accounts had been tested, almost to destruction.

The coda

During re-examination, Mr Green for the claimants asked Mr Coyne about a number of area he'd been tested on over the course of the last four days. The issue ringing through loud and clear was disclosure.

We know the Post Office has fought tooth and nail to avoid disclosing anything it felt irrelevant to the trial. It meant that PEAKs and KELs were were handed over, but the Post Office refused to provide any search terms which might have helped Mr Coyne find errors in those PEAKs and KELs. They refused to give him any information which might have allowed him to investigate particular problems at claimants branches.

In fact, Mr Green took him to more than one document where he was warned off any interest in the claimants by the Post Office who told him "an attempt to obtain
 documents containing information that could potentially be tied to individual cases... is not the purpose of the Horizon trial."

Mr Green also asked Mr Coyne about mysterious documents known as MSCs, OCPs and OCRs (I think of them as quarks to the protons and electrons that are KELs and PEAKs). On seeing the KELs and PEAKs, Mr Coyne had asked in July last year to see their related MSCs, OCPs and OCRs. He got the MSCs in December and the OCPs and OCRs in January, a week before he delivered his final report.

Mr Green wondered about the way the MSCs were given to him.

Mr Coyne replied: "the MSC database was
 a singular database with all the data in it but the way
 that it has been provided to us... it
 has been split into three different Excel spreadsheets... there's no easy way to try and interrogate across the three spreadsheets. They aren't all the same
 length, so you can't put them together. One has lots of columns going across and the other
 one has literally hundreds of thousands of columns going
 down. So it is practically impossible to use for the purposes of analysis.

What were they like to work with, Mr Green wondered?

"Terrible." Mr Coyne replied.

Mr Green opened the subject out: "Mr Coyne, can I ask you whether you have been busy since
       the time that the Horizon trial should have ended, had
 it gone to plan and not been derailed?
JC: "Yes, yes, I have got a number of other projects."
PG: "Have you had a give oral evidence in any other cases?"
JC: "Yes, I have had to give oral evidence in a case at Manchester and I have a number of other non-contentious projects where I'm helping companies go live with big systems.
PG: "How have you found the timing of your receipt of
 documents for the purposes of preparing your reports?
JC: "It has been very problematic.
PG: "How does it compare to other cases you have been in?"
JC: "I don't think there has been another case where I have experienced this level of... the disclosure being
 provided in dribs and drabs, especially after making
 a specific request...  last year because it
       was quite clear then the types of documents that would
 be required.

Next week is going to be interesting, that's for sure.


* Although it was a joint statement the Post Office IT expert only accepted 12 of them.

Live tweets of the day can be found here. I'll be back live tweeting here from Tuesday at 10.30am.

If you can, please help keep this crowdfunded public-interest journalism project going by chucking a few quid in the tip jar below. Contributors who give £20 or more will start receiving regular "secret" emails which have all the info and gossip about this litigation as it makes its way through the courts.

If you want to see how much I have raised to date and how I've spent it, click here.
If you want to find out more about the underlying story, click here.
Choose an amount