Tuesday, 17 December 2019

Fisking the Horizon trial 1: the meaning of "robust".

Would you like to read 2423 words on the meaning of the word "robust"?

Now you can, below.

It was a big deal in the Post Office Horizon trial because the Post Office contended that Horizon is robust whereas the claimants thought it was only relatively robust.  On such things trials are won and lost, so getting a working definition of robust together was one of the first things the judge did.

The segment below comprises paragraphs 36 - 56 of the Horizon Issues judgment.

Language scholars and engineers may enjoy it:

The meaning of “robustness”

Turning to the disagreement about Issue 3, given the parties disagreed about whether the Horizon system is (or was) “robust”, it is a fairly elementary step to consider the meaning of that term, and how it is being used by the parties. Context is important so far as the meaning of the word “robust” is concerned. If someone is in robust health, it usually is taken to mean that they are healthy, or even very healthy. A robust exchange of views can be a polite way of referring to an argument. Given the importance of the concept to the Horizon system, its prominence in the Post Office’s defence of the system, and its express inclusion (admittedly in inverted commas) in the Horizon Issues, I asked each side in the litigation during oral closing submissions for a reference from their pleadings or submissions for the meaning which they ascribed to the word. I referred to this as their benchmark definition. Robustness was referred to by both sides in the litigation in numerous places, but not always in the same precise terms, and clarity is to be welcomed.

The claimants answered this by reference to the remainder of Issue 3, namely “extremely unlikely to be the cause of shortfalls in branches” and explained that the claimants had found the word robust “difficult to define” other than by reference to this. This would mean therefore that it had no separate independent meaning other than as a summary of the longer second part of Issue 3. In other words, a robust system would be one that is extremely unlikely to be the cause of shortfalls in branches. The claimants also implicitly, if not expressly, criticised use of the term both by the Post Office in its pleadings and written submissions as being more aligned to public relations than as a performance standard.

The Post Office asked for some time to provide the reference that I requested. Given the meaning of “robust” is so central in the Post Office’s defence of the Horizon system, I granted the Post Office the time that was requested.

The Post Office subsequently, after the trial ended, submitted a short document entitled “the Post Office’s case on the meaning of robustness”. This was not what was intended when I sought a reference from the Post Office to their definition, and the document submitted went rather further and made wider ranging submissions. The document did state, so far as the meaning of the word is concerned, the following:
“In Post Office’s submission, the meaning of robustness is a matter for expert opinion. Robustness is a well-established concept in the IT industry and is the subject of academic study: see para. 361 of Post Office’s written closing”.

I do not consider that the meaning of words is a matter for expert opinion. The two experts in this case are IT experts, not experts in linguistics or the meaning of language. However, the meaning of robustness within the field of IT is, arguably, a matter upon which the experts’ opinions should be considered, not least because they were applying that term to their expert exercise. The Post Office also relied upon the 1st Experts’ Joint Statement which in respect of Issue 3 stated the following as agreed:

“There are different dimensions of robustness, such as robustness against hardware failure, software defects and user error. The robustness of the system also depends on the processes around it.

Robustness does not mean perfection; but that the consequences of imperfection must be managed appropriately. If the extent of imperfection is too high, this would be very difficult to do which would imply less robustness.

Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its lifetime. The level of robustness may have increased or decreased as the system was changed.

The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this indicates any lack of robustness.”

In the areas of disagreement in this Joint Statement, each expert provided the following. Mr Coyne stated (inter alia):

“For the purposes of addressing the robustness of Horizon, I have applied the following definition of robustness:

‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling.’ ”

In consideration of the likelihood of Horizon to be the cause of shortfalls in branches, Horizon is not determined to be robust in this regard because:

(a) it contained high levels of bugs, errors and defects as set out under Issue 1 above which created discrepancies in the branch accounts of Subpostmasters;

(b) it suffered failures of internal mechanisms which were intended to ensure integrity of data;

(c) the system did not enable such discrepancies to be detected, accurately identified and/or recorded either reliably, consistently or at all;

(d) the system did not reliably identify ‘Mis-keying’, which is inevitable in any system with user input, and did not reliably have in place functionality to restrict users from progressing a mis-key;

(e) it required numerous processes and workarounds to be in place to allow Fujitsu to modify data already recorded by Horizon, which would not be required in a “robust” system; and/or

(f) there were weaknesses and risks of errors and other sources of unreliability within Horizon.”

(italics present in original)

Dr Worden stated in the same Joint Statement:

“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The term 'robust' is not, as implied in para 3.1 of the outline, either ill-defined or a piece of IT public relations. Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.

In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?

In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr Coyne implying that Horizon fell short of its robustness objectives.

In my current preliminary opinion, Horizon is a highly robust system, and this has important implications for the other Horizon issues, notably issue 1.”

It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition, and expressly said it was not adequate. In any event, the meaning of any word – even “robust”, or “robustness” – ought to be capable of description by the parties themselves. Although on its face it did not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr Coyne’s text, in other words that part of the text that contained his definition of robustness (which was in italics in the 1st Joint Statement). Obviously if the parties (or their experts) could agree the definition to be applied so far as the Horizon System is concerned, that ought to be identified. I therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne, and if not, what its alternative definition was.

This led to a further document being received from the Post Office dated 18 July 2019. It referred to the passage in the 1st Joint Statement (which is quoted at [40] above) as “the agreed definition”. That rather overlooks that Mr Coyne identified the definition of robustness which he was applying, and Dr Worden expressly disagreed with this in the same Joint Statement under the heading “Areas of Disagreement”, and stated “the definition of ‘robust’ proposed above by Mr Coyne is not adequate, for the reasons given below”. It also overlooks that in the 3rd Joint Statement, paragraph 3.1 had an agreed entry which stated the following:

“Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved.

From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events.

This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”

(emphasis added)

Later in the same document of 18 July 2019 the submission was made by the Post Office that “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. It was also submitted that “As Post Office understands it, this is what Mr Coyne meant when in his comments in [the 1st Joint Statement] he defined robustness as “the ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.

This was precisely the definition which Dr Worden, in his areas of disagreement on the 1st Joint Statement, described as “inadequate”. The end position therefore is as follows.

The claimants found “robustness” difficult to define in the abstract and tied it in with the other wording of Horizon Issue 3; a robust system would be “extremely unlikely to be the cause of shortfalls in branches”. That however is a consequence of how a robust system would operate, not a definition of what robustness means.

The Post Office defined it as follows: “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. The Post Office was also prepared to accept Mr Coyne’s italicised definition in the 1st Joint Statement, namely ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.

Mr Coyne applied the definition he set out in italics in the 1st Joint Statement, quoted in the immediately preceding paragraph of this judgment and at [41] above.

Dr Worden’s definition was as follows:

“Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.

In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?”

The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of Dr Worden.

The Post Office made submissions in paragraph 3(b) of the written submissions dated 18 July 2019 on robustness that stated that Mr Coyne cannot have intended to exclude the effect of countermeasures when he considered the concept of robustness. I shall return to this topic when dealing with countermeasures. This is because some of the countermeasures considered by Dr Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which both parties agree are outside of the Horizon System).

I do however accept the Post Office’s submissions that there is not a great or material difference in the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’ submission that robustness is difficult to define. Dr Worden defined robustness by using what he termed as “the dimensions of robustness”. It is rather circular to describe the meaning of robustness as being “robust against” some particular risk. Although Mr Coyne provided his definition in the 1st Joint Statement, the statement by Dr Worden that this was “inadequate” may only have been aimed at the entirety of Mr Coyne’s entry in the areas of disagreement, as effectively accepted by the Post Office in their most recent written submissions on the subject. Whether that is an explanation of the lack of agreement in the Joint Statement, I also agree with the Post Office that Mr Coyne’s definition is not materially different to that used by Dr Worden.

Robustness is indeed an engineering concept. It means the ability of any system to withstand or overcome adverse conditions. A robust system is strong and effective in all or most conditions. The robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences; this is the same meaning as how robustness was described in the Post Office’s written submissions dated 18 July 19. Robustness does not mean perfection.

The exercise necessary above, to arrive at the definition of robustness in [54] above, is not judicial pedantry. Given the central importance of robustness to the disputes about the Horizon System, and the Horizon Issues, it is in my judgment essential. It is mildly surprising, given how central the assertion of robustness has been to the Post Office’s defence of the Horizon System, that Dr Worden’s interpretation of the term has been relied upon so heavily by the Post Office, given the term was used by the Post Office for some years prior to his involvement.

However, regardless of that passing observation, I find that both experts correctly understood what robustness in fact means, and applied the definition at [54] above in considering their expert evidence. I will return to the expert evidence in some detail later in the judgment, including in the Technical Appendix. 

Peer demands judge-led inquiry into Post Office Horizon fiasco

Lord Arbuthnot
James, now Lord Arbuthnot, former MP for North East Hampshire, reacted to yesterday's High Court judgment by calling for judge-led inquiry into the Horizon fiasco. In a statement issued this morning he said:

“The subpostmasters have been vindicated in every respect.  It is an excellent Christmas present, but won at great cost.  The cost falls partly on the taxpayer but also heavily on the subpostmasters themselves, who will have their damages reduced by the amount the litigation funders will (justifiably) deduct.

“Now that these battles are being won, it is time to turn our attention to how it all came about and went so far.  We need an inquiry and, since the Post Office has repeatedly given inaccurate information including to me, it needs to be led by a judge.  It may be that the best person to conduct the inquiry would be the judge who already has such extensive knowledge of the details, Sir Peter Fraser.  He has done much of the work already."

Lord Arbuthnot led the cross-party parliamentary group of MPs who came together after so many Subpostmasters in their constituencies contacted them begging for help. He remains a member of the All Party Parliamentary Group which looks into matters relating to the Post Office, led by the Labour MP Gill Furniss.

On another note, the Law Gazette has picked up on Sir Peter Fraser's decision to refer the evidence of  Fujitsu employees in the Horizon trial to the Director of Public Prosecutions. There are cans and worms everywhere in this story.

*************************
If you have enjoyed reading the content on this website, I would be grateful if you could chuck a few quid in the tip jar below. Contributors who give £20 or more get regular "secret" emails which have all the info and gossip about this litigation and its wider repercussions. Click here to see how and where your money is being spent.

Choose an amount

They did it.

What a judgment. What an announcement.

I have spent many days of my life in Court 26 of the Rolls Building. Days I will never get back. Some were tedious, many were fascinating, a few were genuinely dramatic, but none held a candle to what happened yesterday afternoon.

As we know from the press statement on Wednesday last week, this litigation was supposed to be dead. Settled with an apology and £58m in compensation from the Post Office. Well - today's judgment re-energised the corpse and now the judge appears to have set in motion a Frankenstein's monster which could lay about the "nation's most trusted brand" and the IT firm which operates the Post Office's Horizon computer system, Fujitsu.

Court 26 is a big, badly designed room. It is around the size of a basketball court. Whilst it has generous amounts of space compared to most courtrooms, it is unsuited to coping with the dozens of claimants, their families, supporters, journalists, lawyers and observers who packed in there at 1.30pm on Monday 16 December 2019 to witness the second trial judgment being handed down.

Malicious prosecution

The first interesting discussion was about the possibility of claimants going after the Post Office for malicious prosecution. Normally with a settlement of this nature, claimants are prohibited from taking the defendant to court for any reason ever ever again, but the claimants' QC Patrick Green wanted the judge to make an order that claimants who have criminal prosecutions against their name should not be disbarred from pursuing the Post Office over criminal matters, simply because they have signed up to a civil settlement. After seeking agreement from the Post Office's QC Owain Draper (which, it transpired, was a formality), the judge made the order.

Then the judge made an announcement which stood outside of the 177,211 word findings he had just handed down. He told the parties and the court he had "grave concerns" about the evidence of the Fujitsu employees; so much so that he felt the veracity of evidence provided by Fujitsu employees in a number of Post Office prosecutions of Subpostmasters needed to be properly scrutinised. To that end, he would be supplying a dossier to the Director of Public Prosecutions for further investigation.

No one was expecting this. I was sitting with journalists from Computer Weekly, the Daily Mail and the Press Association and we all looked at each other, thinking pretty much the same thing. Fujitsu is now in the game.

The judgment

But why? How? I still haven't read every word of the Horizon trial judgment, so there may be some mitigating comments, but here are the choice quotes:

"The Post Office’s approach to evidence, even despite their considerable resources which are being liberally deployed at considerable cost, amounts to attack and disparagement of the claimants individually and collectively, together with the wholly unsatisfactory evidence of Fujitsu personnel such as Mr Parker."

Who is Mr Parker? Glad you asked:

"Mr Stephen Parker is... the Head of Post Office Application Support. He is therefore a very senior person. He first started work on what was then called the Royal Mail Group Account in 1997, which was before the introduction of Horizon. He has continued to provide support to the Post Office Account in the various roles he has occupied at Fujitsu throughout the whole of Horizon’s life, by which he meant both Legacy Horizon and Horizon Online."

So he, under oath, would tell the judge, the truth, right?

"Mr Parker chose specifically to give the impression in his 1st witness statement that Fujitsu did not have the power (the word Mr Parker expressly chose) to inject transactions into the counter at branches, even though he knew that it did. This paints him in a very poor light as a credible witness."

What did the judge think about Mr Parker's evidence, when exposed to cross-examination?

"I consider that Mr Parker, and the team who assisted him, sought to portray the Horizon system – Legacy Horizon and Horizon Online – in a light as favourable as possible to Fujitsu, regardless of its own internal evidence to the contrary, and regardless of the facts."

And the judge's conclusion about Fujitsu in general?

"Fujitsu do not,... appear to me to have properly and fully investigated.. myriad problems, nor did Fujitsu categorise such incidents correctly. They also seem to have moved away, in their investigations, from concluding that there were any issues with the software wherever it was possible for them to do so, regardless of evidence to the contrary, an approach that has been carried into the Fujitsu evidence for the Horizon Issues trial."

What about the Post Office?

They come in for the sort of pasting we have, perhaps, got used to. Their approach:

"has amounted, in reality, to bare assertions and denials that ignore what has actually occurred… It amounts to the 21st century equivalent of maintaining that the earth is flat.”

“A theme contained within some of the internal documents is an extreme sensitivity (seeming to verge, on occasion, to institutional paranoia) concerning any information that may throw doubt on the reputation of Horizon, or expose it to further scrutiny."

There may be more in parts of the judgment I have not yet reached, but this, to me, is the uncontrovertable conclusion which vindicates everything campaigning Subpostmasters have been saying for years:

"It was possible for bugs, errors or defects of the nature alleged by the claimants to have the potential both (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, and also (b) to undermine the reliability of Horizon accurately to process and to record transactions as alleged by the claimants.... Further, all the evidence in the Horizon Issues trial shows not only was there the potential for this to occur, but it actually has happened, and on numerous occasions." [my italics]

Alan Bates was the lead claimant in the litigation and founder of the Justice for Subpostmasters' Alliance. His dogged determination to see the Post Office held to account is well documented. He is not a man who likes the spotlight, but I am sure he would have been at court yesterday if he were not recovering from a recent hospital visit. He sent through this statement from his hideout in Wales:

Alan Bates
"This judgment, like that of the Common Issues trial vindicates everything we have been saying for years. The real problem we have been left with is the unrecovered expenses which we have had to incur to pursue the litigation and which include considerable litigation financing fees, all of which have devoured most of the £58m damages, leaving little left to be shared between the group.

"It would seem, from some recent excellent research work Eleanor Shaihk undertook, that successive governments have failed in their statutory duty to oversee and manage Post Office and this is something that we are planning to ask our MPs to raise next year.  If it turns out to be correct, we will be wanting to recover everything we have had to spend doing the job government should have done.

"It isn't over yet, just the end of another chapter."

There will be a number of people who will want to thank Alan for what he has achieved. I wish him a speedy convalescence.

Media interest

After the judgment was handed down, outside court, there was quite the scene. Subpostmasters giving interviews en masse. Banners, photo ops, the lot. I saw a couple of TV cameras there, and whilst I haven't watched it yet, I am told this story finally made it onto the BBC ten o'clock news.

I interviewed every claimant I could persuade to speak to me, then I got a cab to Broadcasting House where I went into the BBC Radio 4 studios to do a hit for Evan Davis's PM programme (listen here, 45 minutes into the show). After that I spoke to Rebecca Jones on the BBC News Channel.
You can also listen to a lengthy interview I conducted this morning with one of the BBCs best presenters, Dotun Adebayo, on BBC Radio 5 live's Up All Night. It starts 33 minutes into the programme.

Also, if I could offer a plug for my current employers, may I recommend you tune in to Channel 5 News at 6.30pm today (Tuesday) 17 December, where I will be discussing this story further.

I hope to have read the whole of the judgment in its entirety by then. And I will post up everything I find on this website.

Final note

There are many takeaways from the second trial judgment day. The main one is that the claimants were apparently told in a pre-judgment meeting with their solicitors that the most they can expect to get from the £58m settlement is £8m - £11m. This is the sum that will be left after the legal fees have been paid and the litigation funders have taken their cut.

Watching what the Criminal Cases Review Commission does next will be interesting. Will it refer all 35 cases it's looking at to the Court of Appeal?

How is the Director of Public Prosecutions going to react to being sent a dossier of evidence by a judge about the quality of evidence from Fujitsu used in criminal prosecutions by the Post Office?

Will there be a public inquiry? The journalist Tony Collins has already explained how it could come about. Don't be surprised if various parliamentarians start calling for one.

UPDATE: At 8.38am this morning Lord Arbuthnot called for a judge-led inquiry. It'll be interesting to see if his Conservative colleagues in the new administration are minded to listen.

*************************
If you have enjoyed reading the content on this website, I would be grateful if you could chuck a few quid in the tip jar below. Contributors who give £20 or more get regular "secret" emails which have all the info and gossip about this litigation and its wider repercussions. Click here to see how and where your money is being spent.


Choose an amount