This is the transcript of Day 14 of the Horizon trial, the second trial in the Bates and others v Post Office group litigation, held at the High Court's Rolls Building on Tuesday 4 June 2019.
Jason Coyne, independent IT expert for the claimants spent the day being cross-examined by the Post Office's QC, Anthony de Garr Robinson.
My write-up of the day is here.
Mr Coyne's reports can be found here.
Transcript follows:
Jason Coyne, independent IT expert for the claimants spent the day being cross-examined by the Post Office's QC, Anthony de Garr Robinson.
My write-up of the day is here.
Mr Coyne's reports can be found here.
Transcript follows:
Tuesday, 4 June 2019
(10.30 am)
MR DE GARR ROBINSON: May it please your Lordship, can
I start by thanking you for kindly agreeing to adjourn
this hearing on Wednesday of next week to suit my
personal circumstances.
MR JUSTICE FRASER: Not at all.
MR DE GARR ROBINSON: I would also like to thank my learned
friend who I know has difficulties the following week as
a result. I would like that to be put on record because
I'm most grateful.
MR JUSTICE FRASER: Can I therefore check. We are sitting
four days this week, which is all going to be the
claimants' expert being cross-examined by you, and then
we are sitting the three days next week identified in
the email of yesterday?
MR DE GARR ROBINSON: My Lord, I believe so, yes.
MR JUSTICE FRASER: And that is Dr Worden being
cross-examined by Mr Green.
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: The dates, times etc for closings etc
remain unaffected?
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: Thank you.
MR DE GARR ROBINSON: My Lord, one other matter I should
mention. Dr Worden's supplemental statement, there has
been no substantive engagement between the experts on
that statement, as I understand it.
MR JUSTICE FRASER: Which statement?
MR DE GARR ROBINSON: I should say his report. The report
that was sent to your Lordship a couple of weeks ago if
your Lordship recalls.
MR JUSTICE FRASER: But you are not making an application?
MR DE GARR ROBINSON: I have no application to make and
I was going to tell your Lordship that.
MR JUSTICE FRASER: Well, that's one of the points which --
I know you suggested there was nothing unconventional
about what Dr Worden had done but his covering email
explained what application you would or would not be
making. I don't know if you have seen that covering
email, it is the one that was forwarded on by the court.
MR DE GARR ROBINSON: I am sure I have seen it, my Lord, but
not recently.
MR JUSTICE FRASER: All right. But the situation is you are
not making an application?
MR DE GARR ROBINSON: I have no application to make.
MR JUSTICE FRASER: Understood. Is there anything else that
you would like to mention?
MR DE GARR ROBINSON: My Lord, no.
MR JUSTICE FRASER: There is something I have to mention, it
has probably been somewhat lost in the mists of time
given the circumstances of March, but you were in the
middle of doing a privilege redaction review.
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: Has that now been completed?
MR DE GARR ROBINSON: My Lord, yes. There were documents
were sent over to Freeths a number of weeks ago now in
accordance with your Lordship's deadline.
MR JUSTICE FRASER: I wonder if at some point, not now, you
could just give me a summary -- I think there were 30
documents you were looking at?
MR DE GARR ROBINSON: I certainly will, my Lord.
MR JUSTICE FRASER: If you could just give me a summary of
the number of documents there were, which went into
which category and which were disclosed, I would be very
grateful.
MR DE GARR ROBINSON: I will have to go back and review --
MR JUSTICE FRASER: I expected you would so I was not
expecting an answer.
MR DE GARR ROBINSON: Would your Lordship like that done
this week? Perhaps we could do it on Friday after --
MR JUSTICE FRASER: It doesn't have to be this week.
MR DE GARR ROBINSON: I'm grateful.
MR JUSTICE FRASER: The best thing actually, why not if it
could just be distilled into writing and then maybe send
an email on Monday.
MR DE GARR ROBINSON: My Lord, yes, that will be done.
MR JUSTICE FRASER: Then the other thing, which is for both
the parties, not just you. I'm producing judgment
number 5, which is the written reasons for all of the
rulings last week. It is substantially finished. It
resolves the outstanding point on the order about
detailed assessment. I'm hoping to send it out by the
end of today, depending on how long it takes me finally
to proof read it. If it is not at the end of today it
will be first thing tomorrow morning, and then based on
what that says I will just expect the order to be
produced in due course.
I do not think there's any other housekeeping,
unless you have got some, Mr Green?
MR GREEN: My Lord, no.
MR JUSTICE FRASER: Thank you very much,
Mr de Garr Robinson.
MR DE GARR ROBINSON: My Lord, there is one minor change to
the agreed reports between the experts.
MR JUSTICE FRASER: The joint statements?
MR GREEN: The joint statements. It is at {D1/5/10}.
MR JUSTICE FRASER: Which joint statement is this?
MR GREEN: This is the fourth joint statement of
4th March 2019.
MR JUSTICE FRASER: Yes.
MR GREEN: In the light of considering some further
documents the experts have agreed that at 11.1, where it
says "evidence from several PEAKs indicates that
whenever Fujitsu", it should be "usually", qualifying
that. And we will produce for your Lordship a revised
statement of that sentence that the experts are happy
with that makes --
MR JUSTICE FRASER: They have both agreed that, though?
MR GREEN: They have agreed that orally this morning, so
that's why we don't have a written updated version, but
we will produce a written version reflecting that
agreement as soon as we can.
My Lord, may I now call Mr Coyne.
MR JUSTICE FRASER: Yes.
MR JASON PETER COYNE (sworn)
Examination-in-chief by MR GREEN
MR JUSTICE FRASER: Have a seat, Mr Coyne, if you would.
MR GREEN: My Lord, there are hard copies of Mr Coyne's
statement arriving shortly. He is without them at the
moment, for which we apologise, but they will be here
shortly. He says he is content to proceed with them on
the screen.
Mr Coyne, you have made two reports?
A. Yes, that is correct.
Q. The first appears to be at {D2/1/1}.
A. Yes.
Q. If we go to page {D2/1/167} of that report, is that your
signature?
A. Yes it is.
Q. And do you believe the contents of that statement to be
true?
A. Yes, I do.
Q. Then you served a revised version of your supplemental
report?
A. Yes, I did.
Q. And if we look at {D2/4.1/1}.
A. Yes.
Q. If we go please to page {D2/4.1/266} of that report, is
that your signature?
A. It is indeed.
Q. Does that reflect the corrections that you made on
6th March?
A. Yes.
Q. Subject to those corrections now included in that
revised report, do you believe the contents to be true?
A. I do.
Q. There are also four joint reports.
MR JUSTICE FRASER: Joint statements.
MR GREEN: Joint statements which are at {D1/1/1}.
A. Yes.
Q. And {D1/2/1}.
A. Yes.
Q. {D1/4/1}.
A. Yes.
Q. And {D1/5/1}.
A. Yes, that is correct.
Q. Could we just look, please, at {D1/5/10}, at 11.1.
A. Yes.
Q. You will have heard the qualification. Is that
something that you have spoken to Dr Worden about?
A. Yes, I have spoken to him this morning just before the
start of court.
Q. And what have you agreed about that?
A. We have agreed that the word "whenever" overstates the
position and it would be better to say "usually".
Q. I'm very grateful. Subject to that, is there anything
in those reports that you would wish to change or depart
from?
A. No.
MR GREEN: I'm most grateful. My Lord, I have no further
questions.
MR JUSTICE FRASER: Mr de Garr Robinson.
Cross-examination by MR DE GARR ROBINSON
MR DE GARR ROBINSON: Mr Coyne, if I could ask you first of
all to go to bundle {C1/1/1}. This is a list of the
Horizon Issues.
A. Yes, I have that.
Q. And the first set of issues are headed "Bugs, errors and
defects in Horizon", for shorthand I will call them
bugs.
A. Yes.
Q. I would like to look at four issues in particular.
Issue 1:
"To what extent was it possible or likely for bugs
..."
Of the nature alleged at certain paragraphs of the
generic particulars of claim and referred to in certain
paragraphs of the generic defence.
"... to have the potential to cause apparent or
alleged discrepancies or shortfalls relating to
subpostmaster's branch accounts or transactions, or ...
to undermine the reliability of Horizon accurately to
process and to record transactions as alleged ..."
In certain paragraphs of the generic particulars of
claim. You see the references there to the generic
pleading?
A. Yes.
Q. I take it you read the relevant paragraphs of those
pleadings?
A. Yes.
Q. Issue 3, if we could just read that:
"To what extent and in what respects is the Horizon
System 'robust' and extremely unlikely to be the cause
of shortfalls in branches."
At the bottom there's more references to paragraphs
of the generic pleadings. You have read those as well,
haven't you?
A. Yes, I have.
Q. In order to clarify what these issues actually mean?
A. In order to provide the context of the questions.
Q. I'm grateful. Then paragraph (4):
"To what extent has there been potential for errors
in data recorded within Horizon to arise in (a) data
entry, (b) transfer or (c) processing of data in
Horizon?"
Then over the page at {C1/1/2} at issue (6):
"To what extent did measures and/or controls that
existed in Horizon prevent, detect, identify, report or
reduce to an extremely low level the risk of ..."
Those specified errors.
I am sure you will agree the issues I have just read
out are very important issues for the purposes of these
proceedings?
A. Yes.
Q. Would you agree they are probably the most important
issues for the purposes of these proceedings?
A. I'm not sure whether they are the most important but
they certainly are --
Q. They are of central importance, yes?
A. Yes.
Q. I would like to look briefly at the context that you had
regard to when interpreting them. Could I ask you now
to go to the pleadings bundle which is at bundle
{C3/1/1}. These are the generic particulars of claim
and I'm going to go to a few paragraphs. First of all,
page {C3/1/8} of that document and paragraphs 22 to 24.
22:
"Prior to disclosure and expert evidence, the
Claimants are unable to provide detailed particulars of
bugs, errors or defects which were or may have been the
cause of any discrepancies or alleged shortfalls
attributed to them by the Defendant, but will be able to
plead further thereto following disclosure or the
provision of information relating thereto by the
Defendant."
Skipping over the last sentence.
Then 23:
"However, the Claimants aver that there were a large
number of software coding errors, bugs or defects which
required fixes to be developed and implemented. There
were also data or data packet errors. There was a
frequent need for Fujitsu to rebuild branch transaction
data from backups, giving rise to the further risk of
error being introduced into the branch transaction
records."
Then there is a reference to the known error log.
A. Yes.
Q. Then various specific things are relied upon in
paragraph 24 which I would ask you to just cast your eye
down so you can see what's being alleged.
A. Mm. Yes.
Q. {C3/1/9}.
So the context in which these issues arise is the
overall question as to whether the problems which are
alleged here affected the claimants, created false
shortfalls in their accounts, would you agree with that?
A. Yes.
Q. If we can then move on to the generic defence and that's
at bundle {C3/3/1}, behind divider 3.
If we can go to page {C3/3/21}. Picking it up at
paragraph 50, just in the individual subparagraphs, this
is what the defendants are saying in response to that
case. (1):
"All IT systems experience software coding errors or
bugs which require fixes to be developed and
implemented. As is noted in paragraph 53 and 54 below,
there are robust measures in place in Horizon for their
detection, correction and remediation."
MR JUSTICE FRASER: I think we jumped forward a bit quickly.
Can we go back a page, please.
MR DE GARR ROBINSON: It's paragraph 50.
MR JUSTICE FRASER: Yes, you were reading from 50 and
page 21 was on the screen, but before you finished
reading we jumped to page 22.
MR DE GARR ROBINSON: I see. I'm grateful, my Lord. I'm
sorry, I was not looking.
MR JUSTICE FRASER: That's understood, don't worry.
MR DE GARR ROBINSON: Then it says at (2):
"All IT systems involving the transmission of data
over the internet experience data or data packet errors
during transmission and such systems routinely have
protective measures in place to prevent such errors
creating any difference between the data transmitted and
the data received and retained by the recipient.
Horizon has robust controls making it extremely unlikely
that transaction data input in a branch would be
corrupted when being transferred to, and stored in, Post
Office's data centre in a manner that would not be
detected and remedied."
So you will see that here Post Office are not saying
it didn't happen, but what they are saying is there is
a risk of it happening but for each occasion on which
data is input it is extremely unlikely that the data
would be corrupted. Do you see that?
A. I do see that.
Q. If we could move on to page {C3/3/22} now, picking it up
at paragraph 53, it is said:
" ... it is a truism that errors or bugs in an IT
system and data or data packet errors have the potential
to create errors in the data held in that system.
However Horizon has at all material times included
technical control measures to reduce to an extremely low
level the risk of an error in the transmission,
replication and storage of the transaction record data.
Then there is a list of the current measures.
Again, not saying that it didn't happen. There is
a risk but in any given case that risk is extremely low,
do you see that?
A. I can see the words that are being said, yes.
Q. So robustness -- we can move on to paragraph 54 on page
{C3/3/23}:
" ... in addition to the technical controls referred
to above, there are several operational procedures and
practices conducted by Post Office and subpostmasters
that serve to increase the reliability of the data
stored in the central data centre as an accurate record
of the transactions entered on branch terminals.
Then there is a list of those. So you will see that
what's being said is that the robustness of the system
is said to be affected both by technical measures and by
human measures. Do you see that?
A. Yes, I can see that. That is what is said, yes.
Q. Then over the page at {C3/3/24} and paragraph 55:
" ... Post Office admits that, like all other IT
systems, Horizon is not a perfect system which has never
had any errors or bugs. However, as indicated in
paragraphs 53 and 54 above, it has robust systems in
place to identify them, fix them and correct their
consequences (if any)."
Were those the paragraphs that you looked at when
you were deciding how to interpret the issues that we
have read to each other?
A. Yes.
Q. Before you decided on the approach that you would adopt,
you reflected on the issues thrown up by those
paragraphs, yes?
A. Yes.
Q. They made it clear, didn't they, that the critical issue
was not whether it is possible or likely for bugs to
have the potential to cause false shortfalls in branch
accounts over the life of Horizon, that was in fact
admitted. The critical issues were the extent to which
it was and is likely that bugs cause false shortfalls in
any given set of accounts. Do you accept that?
A. I don't really see the difference between the two
statements that were made there. The question was
whether it was possible or likely and the extent to
which it was possible or likely.
Q. Well, let's -- given that it is admitted that there have
been occasions where these things have happened in the
life of Horizon, to ask is it possible or likely that it
has happened within the life of Horizon is a rather
uninteresting question, isn't it?
A. But along with the statement that defects were
acknowledged was a statement that there has only been
three defects.
Q. No, Mr Coyne. I ask you to address my question, please.
If you could focus on what I asked you.
Given it was admitted that there have been occasions
when these things have happened, to answer the question:
is it possible or likely that these things have happened
by saying "yes" isn't a very helpful answer, is it?
A. If the answer was just simply "yes" and with no further
context then I agree that that would ...
Q. The critical issue that's raised by the four issues that
I have read to you is the word "extent" and the word
"likelihood" and the word "risk", isn't it?
A. They certainly are critical elements that would need to
be reviewed, yes.
Q. What you were being asked to do by those issues is to
give your opinion to the court on the extent to which it
was and is likely that bugs caused shortfalls in any
particular set of accounts?
A. Well, across the accounts of subpostmasters, yes.
Q. The extent to which various measures in place reduce
the risk faced by any given subpostmaster when doing
a transaction or when producing a particular set of
accounts, yes?
A. Yes.
Q. That's the critical question on which the court needs
your assistance, would you agree?
A. It is one of the critical questions, yes.
Q. Could you identify a question which you say is more
critical than that in the context of these proceedings?
A. I haven't taken a view on -- I have taken each of the
Horizon Issues that's put to me with equal priority.
I haven't set out to decide a prioritisation for them.
Q. Very well, I understand. But you will then at least
agree with me this far: that in asking you to give your
opinion on the issues we have read, what you're being
asked to do is to advise the court on the extent to
which problems had happened -- or problems were likely
to happen in relation to any given situation, the extent
to which those problems were likely to have been caught
by countermeasures in that situation, and the overall
question of the extent to which Horizon is robust and
extremely likely to be the cause of a shortfall in that
given situation. Would you agree with that?
A. I don't believe countermeasures were asked in the
questions, but broadly with what you say, yes --
Q. When I say countermeasures, you can take me as saying
controls and measures if you want --
A. Okay.
Q. Subject to that correction, you would agree that that
was the essential question being raised in the four
issues that I read to you?
A. I believe they were the questions that were asked and
I believe that in providing my reports I seek to address
those. I have answered those.
Q. Very good. So you accept these issues are of practical
importance to this trial?
A. Yes.
Q. So that the court's judgment in this case, when the time
comes later on to try claims by particular claimants,
that the accounting shortfalls which he or she is
complaining about was actually caused by Horizon?
A. Sorry, could you put that question again, please.
Q. I may have made a grammatical mistake in my question.
MR JUSTICE FRASER: I'm not really sure that's a question
for the expert anyway.
MR DE GARR ROBINSON: Very good. But important questions
are: when a claimant did a transaction in the period
covered by a particular set of accounts containing
a disputed shortfall, what risk did he face of Horizon
creating errors in the recording, transmission or
storage of records of those transactions, and how likely
was it that Horizon calls a false shortfall in those
accounts? Would you agree that those questions, the
Horizon Issues I have read to you, the resolution of
those issues will be of assistance in deciding that
question?
A. No, I don't believe that that's something that we have
been asked to calculate or assess.
Q. No, I'm not asking you, Mr Coyne -- I'm not suggesting
to you that you have been asked to decide on whether any
particular claimants' claim is right or not, what I'm
suggesting to you is that the context in which these --
given the context in which these issues arose -- were
drafted, and given the pleadings by reference to which
they were drafted, it was obvious that the purpose of
those issues was to assist the court so that it could
use the judgment that will be produced in this trial as
a basis for making ultimate decisions in ultimate breach
claims by claimants?
A. In a later trial?
Q. Yes.
A. Yes, I was aware of that.
Q. Isn't that the main reason why we are here?
A. Well, it is certainly a reason why we are here, yes.
Q. To enable the court to make useful findings as to the
general likelihood of any transaction being wrongly
recorded in a particular case?
A. Yes.
Q. And as to the general likelihood of any particular set
of accounts containing false shortfalls because of
Horizon?
A. I don't believe either of those two statements appeared
in the issues for the technical experts, did they?
Q. So you don't believe that when you were asked in the
Horizon Issues -- let me see if I can find them -- at
{C1/1/1}:
"To what extent was it possible or likely for bugs
[and so on] to have the potential to cause apparent or
alleged discrepancies or shortfalls?"
A. Yes.
Q. You don't think you were being asked to give an opinion
as to the extent of the likelihood of that happening in
any given case?
A. Yes, but I have not placed any focus on any particular
branch accounts for any subpostmasters or claimants.
Q. But you weren't focused on the entire life of Horizon.
There are two ways that one could construe the question
"To what extent was it possible or likely for bugs" and
so on. One is was it possible or likely for bugs to
have done that thing at any time during the life of
Horizon? And I think we have already agreed that the
answer to that is trivial because we know it already.
A. Sorry, I do not think I agree with that. I think
because we are talking of situations, bugs, errors and
defects that occurred throughout the life of Horizon,
you do have to look at the entire life cycle from when
it was first used by subpostmasters all the way up to
the near current date.
Q. I'm not suggesting that you don't have to look at
whether bugs have arisen during the life of Horizon.
What I'm suggesting to you is the practical purpose of
these issues, the question you are being asked, is to
what extent is it likely that those bugs are going to
cause shortfalls in any given case?
A. Yes, I agree with that.
Q. Thank you. So the focus was not just on the number of
bugs in Horizon, the focus was on bugs which were likely
to affect branch accounts, yes?
A. Yes.
Q. And also on whether there were -- I'm going to call them
countermeasures, I hope that's not controversial, which
were likely to catch and correct their impact, yes?
A. Yes.
Q. Either straightaway or after some time?
A. Typically if you were going to design a countermeasure
you would want to have it finding the defect
straightaway. If it was only going to find it some time
later it would be of limited value.
MR JUSTICE FRASER: Mr de Garr Robinson, just on your term
"countermeasures".
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: I understood that to have a specific
meaning within the IT world. Are you using it in
a technical meaning or are you using it as a generic
term?
MR DE GARR ROBINSON: I'm using it as a term to refer to
measures and controls in existence, both technical and
human measures and controls, which are designed or have
the effect of identifying bugs in Horizon or identifying
the consequences of bugs in Horizon so that they can be
looked at and fixed.
MR JUSTICE FRASER: So that's the way counsel is going to
use the term "countermeasures".
A. Thank you, my Lord.
MR DE GARR ROBINSON: And this trial, the issues that we
have been discussing, are designed to enable the experts
to grapple with that question, yes? Whether
countermeasures are likely to catch and correct the
impact of any bugs that may have arisen in the system?
A. Agreed.
Q. And countermeasures aren't limited to a process of
stopping the bug as soon as it appears. Sometimes
countermeasures only catch the symptoms or the
consequences of a bug, you would accept that, wouldn't
you?
A. Typically countermeasures are a design aspiration. When
you sit down to design a computer system and what
controls are in place, then you would consider what
controls are required, where to position them. Should
it be the case that the controls fail do you have
a human aspect doing a secondary check? It could go all
the way to employing auditors to re-check figures. You
could go on ad infinitum with --
Q. So you accept, then, that these controls or
countermeasures, whatever you would like to call them,
some of them have the effect of stopping the bug in its
track. Well, some of them have the effect of stopping a
bug arising in the first place?
A. The best way of dealing with defects is to ensure that
they don't arise in the first place --
Q. And --
A. Sorry, if you would allow me to just finish.
Q. Sorry.
A. So what you would do is throughout your testing regimes
you would flush out all of the defects at that point in
time, and you would put in controls to ensure that if
a defect does trigger, there is a way that you are
alerted to it and it is fixed as soon as possible.
Q. Yes. So one of the objectives is to have a testing
regime which picks up as many bugs as possible?
A. Indeed.
Q. But I think you accept that it is impossible to have
a testing regime that picks up all of them?
A. It is impossible to have a testing regime that picks up
all of them and that is why you have a process of
categorisation and you would allow a product to go live
if -- one of the terms that's used -- all severity 1 and
severity 2 defects have been dealt with. Severity 3
might be largely cosmetic and you would allow a system
to go live with cosmetic defects.
Q. That's not quite what I'm asking you. You accept there
are going to be bugs which arise in the operation of any
IT system which would simply evade any test that any
human being can devise?
A. There's always going to be bugs that are unknown to be
within the system at the point that you go live and they
are only discovered sometimes weeks afterwards,
sometimes years afterwards.
Q. Yes. And then -- so that's one line of defence. Then
there are other lines of defence when a bug does arise
because it hasn't been tested for. There are principles
such as defensive programming, and so on, which try and
stop the bug working its way through the system and
producing false results at the end?
A. Defensive programming is typically to stop the bug ever
occurring. It isn't the trigger mechanism; what happens
after the bug is triggered.
Q. Isn't one of the aspects of defensive programming having
little units of programmes which require data to be
transferred from one programme to another programme to
another programme, and every time you transfer from one
unit to the other checks are done to ensure the figures
are correct?
A. Yes, to ensure that a bug doesn't have an impact on the
user --
Q. So one aspect of defensive programming is not just to
prevent bugs, it is to ensure the bug doesn't
propagate --
A. Yes.
Q. -- when it manifests itself and produces an error in
data, yes?
A. Yes.
Q. So we have one set of measures which is designed to
prevent bugs occurring in the first place?
A. Mm.
Q. We have other measures which are designed to prevent
bugs causing problems in data at the user end, if I can
put it that way. But again no system of countermeasures
is perfect, there will always be bugs that get through?
A. Agreed. I completely agree.
Q. Then you have other measures which are designed to pick
up the problems which have been caused at the user end.
You have measures that are designed to assist in the
identification of the fact that there is a problem in
data such that it needs to be looked at and corrected if
it is wrong?
A. Yes, that's typically called impact management. So you
understand what the impact might be of the bug that you
have seen and you work out a process of resolving the
consequential impact of it.
Q. That's precisely -- and of course I'm reminded that many
of those countermeasures also pick up and correct for
user errors as well. They can sometimes have a dual
effect?
A. Yes, you can design the software in such a way that it
reduces the amount of user errors or the potential user
errors. One such that's appeared in this case is about
double typing high value cheques when they are remmed
into the system. So if it is a high value, potentially
you should type that in twice rather than in once.
Q. All of these things that we have just been discussing in
general terms is precisely what the concept of
robustness addresses, isn't it, in the IT industry?
A. Yes.
Q. That a system isn't merely reliable when conditions are
normal -- I think the technical term is nominal -- but
robust performance takes over if and when conditions are
abnormal or off nominal, when they become adverse and
errors are created, yes?
A. Yes, you would want a system to fail safely. So it is
acknowledged that there could be failures and it is what
you do when that system fails.
Q. So robustness is the very concept which underlies the
issues we have been discussing for the last half hour,
yes?
A. Yes.
Q. I think that's what you agree in the joint statement.
If we could go to the joint statement, it is at bundle
{D1/1/8}. This is paragraph 2.3, issue 3.
This is what you agreed with Dr Worden before any
reports were produced at all:
"There are different dimensions of robustness, such
as robustness against hardware failure, software defects
and user error. The robustness of the system also
depends on the process around it.
"Robustness does not mean perfection; but that the
consequences of imperfection must be managed
appropriately."
A. Yes.
Q. Then over the page {D1/1/9} you suggest your own
interpretation of robustness. You say:
" ... I have applied the following definition of
robustness:
"'... namely, the ability of a system to perform
correctly in any scenario, including where invalid
inputs are introduced, with effective error handling.'"
So those adverse conditions you refer to, that would
include bugs in the system itself, wouldn't it?
A. Yes.
Q. So withstanding them would include managing their
consequences appropriately, yes?
A. Yes.
Q. I think that's also what you say in the third joint
statement that was agreed some time later. That is at
{D1/4/2}. It is the second paragraph:
"From our experience of ..."
A. Sorry, I do not think I have the right --
Q. Sorry, it is {D1/4/2}.
MR JUSTICE FRASER: If you look at your screen you will see
what the witness is seeing.
MR DE GARR ROBINSON: It is {D1/4/2}.
At the bottom of the page, do you see that?
A. Yes, I do.
Q. Second sentence of the second paragraph:
"We agree that 'robust' does not mean infallible and
therefore horizon has and will continue to suffer
faults. Robustness limits the impact of those faults
and other adverse events."
A. Yes.
Q. If you go over the page to page {D1/4/4},
paragraph 3.13, third box down, third row down:
"Countermeasures work by limiting the impact of
Horizon bugs/error and defects on branch accounts.
"Countermeasures do not always eliminate the effects
of adverse events (they are not perfect) ..."
Do you agree with that?
A. Yes.
Q. No IT system ever has perfect countermeasures?
A. I agree.
Q. But they are often effective in the area where they are
deployed and that's why they become basic elements of
practical IT system design.
Incidentally, isn't that what Dr Worden was saying
in the first statement? If we can go back to the first
joint statement, which is {D1/1/9}, and look at page 9
of that statement, you will see in the bottom half of
the screen we have got Dr Worden's comments?
A. Yes.
Q. So you indicate your definition of robustness in the
top, and below he says:
"The definition of 'robust' proposed above by
Mr Coyne is not adequate for reasons given below. The
term 'robust' is not, as implied in para 3.1 of the
outline, either ill-defined or a piece of IT public
relations. Robustness (which is closely related to
resilience) is an engineering objective, and large parts
of project budgets are devoted to achieving it."
Would you agree with that?
A. Yes.
Q. "it receives its meaning in the phrase 'robust against
... [some risk or threat]' ..."
Would you agree with that?
A. Yes.
Q. "... and there are a large number of risks that business
IT systems need to be robust against - such as hardware
failures ..."
If we go over the page {D1/1/10}:
" ... communications failures, power cuts,
disasters, user errors or fraud. These are the
dimensions of robustness."
Do you agree with that?
A. Yes.
Q. "In all these dimensions, robustness does not mean 'be
perfect'; it means 'address the risks of being
imperfect'. The extent of robustness is to be
interpreted as: in how many dimensions was Horizon
robust? And: in each dimension, how large were the
remaining risks?"
Would you agree with that?
A. That is a little imprecise. I don't really understand
the reference to the dimensions. The dimensions aren't
defined.
Q. He talks about dimensions before, hardware failures --
he talks about different risks that are faced by the
system. But let's not get bogged down in dimensions.
The critical question, could I suggest to you, in
robustness, is how large were the remaining risks after
the relevant countermeasures have been brought into
account? Would you agree with that?
A. I'm not sure I would agree that that is the critical
question, no. Certainly consideration does need to be
given to how, once a risk triggers, it is dealt with.
Q. Let me get this straight. You are not agreeing that
when determining whether a system, an IT system, is
robust that a critical question is how large are
the risks remaining after you have had regard to the
countermeasures?
A. But when you talk about how large are the risks, that
quantification is something which I don't believe is
possible. It is not possible to quantify risk in such
a way.
Q. So please explain, Mr Coyne, how else would you grapple
with the question of robustness? You agree robustness
is an important concept?
A. Yes.
Q. You agree it is deployed very frequently in the IT
industry?
A. Yes.
Q. It is a subject of academic study, isn't it?
A. Yes.
Q. And there are university courses on it?
A. There may well be, I'm not aware --
Q. Yes. So given that, and given that the essential
purpose of this doctrine is to determine how well
systems guard against problems caused by -- it could be
human error, it could be bugs, it could be any adverse
conditions of that sort, isn't it obvious, doesn't it
follow as night follows day, that the ultimate question
being wrestled with by the concept of robustness is how
well are the risks faced by a system guarded against?
In other words, what are the risks remaining after you
have taken the countermeasures into account?
A. Yes, and typically that analysis, what you talk about
there, is conducted in the design process of a system at
the start where you would then go to build a system
after you are satisfied that it has all the design
characteristics and control measures in place. So it
achieves a particular level that you are satisfied with
and then you go to actually build the solution. That is
different to then having a system already in place and
looking back at it and trying to decide what its level
of robustness was.
Q. Sorry, I'm not sure I understand what you are saying,
Mr Coyne. Are you saying it is possible to define
a level of robustness that you want when you are
designing the system, but it is not possible to decide
whether you have achieved that level after you've built
it?
A. What I'm saying is when you set out to build the system
and you have the design processes and control measures,
you will take a view whether you are satisfied with the
level of risk within that build. It won't have
a number, it will be subjective, but there will be
a decision on whether there is the appropriate level of
robustness -- to use the phrase that we have termed
here -- and if everybody is agreed at that level of
robustness the build will then take place. But it is
a subjective view.
Q. I'm interested in your suggestion, Mr Coyne, that there
won't be any numbers. You say you are a qualified
PRINCE2 practitioner, yes?
A. Yes.
Q. I have seen a number of PRINCE2 documents and we can
take you to some tomorrow if this is necessary. But
what those documents make very clear is that in engaging
in the process of determining the risks -- managing the
risks to a given project, you do absolutely attach
numbers to particular risks. You have graphs, don't
you? You draw graphs up and you assign values to
particular risks?
A. Typically each risk will have an impact rating, often
between 0 and 5 but it could be different, and there is
a likelihood of that impacting, and typically you would
multiply one by the other and that gives you a number.
But that is far too simplistic for something at the
level of analysis that we are talking about here.
Q. What I'm suggesting to you, Mr Coyne, is previously you
said when you are designing a system and considering
the risk to which you wanted the system to be subject,
the level of risk which you were willing to accept, it
is a subjective view and it doesn't involve any numbers.
What I'm suggesting to you, Mr Coyne, is it absolutely
does involve numbers and indeed the very process you
have just described shows it involving numbers. Would
you agree with that?
A. Right, so what I set out with regard to numbers there
was a very simplistic way of measuring isolated aspects
of risk. That is used in PRINCE2. But you can't
necessarily scale up looking at simple isolated risks
and apply it to something retrospectively such as
Horizon.
Q. You are raising two completely different issues. The
first issue is the retrospective issue.
A. Right.
Q. And the second issue is numbers.
A. Right.
Q. You accept, don't you, that when designing a system you
absolutely do -- you identify the risks which you want
the system to guard against and you assign numbers to
the adequacy of the risk to which the system is guarding
against and you perform calculations with a view to
ascertaining whether the system is the kind of system
that you want, yes?
A. No, not really. When you are talking about
countermeasures or controls, they will be functional
design aspects and there will typically be a list of all
functional design aspects and nonfunctional design
aspects.
The types of risks that are monitored through the
design and implementation of an IT project are often
things such as delay, the lack of resources, the failure
of hardware, and it is those aspects that you would
attribute a numerical value, and then also a numerical
value which is likelihood.
So yes, they are both in the same area as risk, but
one is dealing with very simplistic isolated elements
and the other is dealing with the whole operation of
a system.
Q. You have not said that before, so far as I'm aware. You
are suggesting that PRINCE2 can only involve simple
calculations, it doesn't involve quite elaborate
calculations involving a wide array of risk
calculations?
A. I'm not saying that you can't make PRINCE2 more
elaborate. Typically in my experience people will
measure risk within PRINCE2 between 0 and 5, so it is
not very granular, but there's nothing to say that you
couldn't have some more scientific way of doing it.
Q. Are you suggesting that doesn't happen in practice,
because I would like to suggest to you it does, and
there are quite complex projects where there are quite
elaborate calculations made. Would you accept that?
A. I am sure there may well be.
Q. Just to very briefly talk about your retrospective
point. If you are brought in to design a new element to
a system, around an existing system, and you are being
asked to calculate the risk of that system not operating
accurately in collaboration with the legacy system
that's remaining, when performing that calculation risk
you will look at the past, won't you? You will look at
how the legacy system operates, how reliable the outputs
are out of the legacy system, all sorts of things which
are based upon looking back at retrospective events,
yes?
A. Yes, I would certainly look if there has been any
historic defects within the system, because in the
example that you are giving it is a system that is
already live that is being extended. So I would look
back at how the good the testing has been in the past,
whether bugs have escaped the testing. I would look at
how many defects there have been in any given launch or
release. I would look at how, when bugs are detected,
they are dealt with, what the mitigation of those would
be. There would be an assemblance of all of those
elements.
Q. Thank you. So what you describe as an assemblance would
be an examination, a review, of the data relating to
past performance --
A. Yes.
Q. -- with a view to ascertaining a risk that you face in
the future, yes?
A. It would certainly help set the context. It might well
be that simply looking backwards at a project, whilst it
might be helpful to set the context, it might well be
that the way you are going to approach this new project
is different, it might have different vendors that are
involved, the supporting mechanism might be completely
different, which might be a factor in the scenario. We
have Atos that were involved in certain aspects of the
support, we have Fujitsu that were involved in certain
aspects of the support. Post Office had a supporting
function and had a reconciliation function but that may
well have changed over time. So it is important to
understand what processes were in place at various
times.
Q. Yes. And forming that understanding, can I suggest to
you that invariably you will be looking at past
performance?
A. Yes.
Q. Even, for example, if you are bringing in a new system
that is designed to be compatible with an old system, as
well as looking at the data relating to the past
performance of the old system you would also look at the
data relating to the past performance of the new system
if there is some, wouldn't you?
A. Yes.
Q. So quite often when performing what you describe as
a prospective risk analysis you would often look at
historical data and look at historical performance,
wouldn't you?
A. Yes.
Q. I'm very grateful.
If we can now move to bundle {D3/1/11}, this is
Dr Worden's first report. If I could pick it up at the
bottom of the page, paragraph 48. Dr Worden has just
listed three of the issues that we have just been
discussing, and then he says in paragraph 48:
"In my opinion the most important of these is
issue 3 which encompasses a large and mature area of
modern IT practice."
Do I apprehend from the evidence you have already
given that in your opinion the most important of these
issues is not issue 3?
A. I don't see any as being more important than others.
Q. You would accept, though, that issue 3 encompasses
a large and mature area of IT practice, yes?
A. Yes.
Q. Then he says:
"Nearly all business IT systems need to be robust --
as the business depends on them -- and there is a large,
mature and tested set of techniques for achieving
robustness."
Do you agree with that?
A. Yes. It is a generic statement of the industry rather
than this particular project.
Q. Then if we could move on to page {D3/1/96} in the same
document. Paragraph 374. He sets out there -- I will
ask you to read it to yourself -- points that are
similar to the points that we have seen in the first
joint statement and he adds at the end:
"Robustness of IT systems is a large and mature
topic."
And I understand you would agree with that?
A. Yes.
Q. So what this all shows is that there is a well
understood body of practice and learning on how to
achieve robustness, yes?
A. Say that again, please?
Q. There is a well understood body of practice and learning
on how to achieve robustness in IT systems?
A. Yes.
Q. And it is possible to benchmark a system by reference to
other comparable systems which are deployed in the
industry, yes?
A. Yes.
Q. You say that I think in your first statement. Perhaps
if we could look at that {D2/4.1/154}, page 154,
paragraph 5.99.
A. Yes.
Q. I'm so sorry, Mr Coyne, I'm taking you all over the shop
and I'm wasting time. I do apologise. Something seems
to have gone wrong in my note. Could you give me one
moment?
A. Certainly. (Pause)
Q. Yes, let's go back to {D2/1/78}. It is page 78, as
I indicated first of all, and it is paragraph 5.91. You
say -- this is in your first report:
"... I have estimated the likely level of the
robustness of Horizon and benchmarked this against
industry standards based upon a review of the evidence
available including the known error log (KEL) and PEAK
system."
A. Yes.
Q. Your conclusion, having done that benchmarking exercise,
is that Horizon is robust, isn't it?
A. I said relatively robust, yes.
Q. Well, let's look at exactly what you say. It is bundle
{D1/4/2}, paragraph 3.1:
"From our experience of other computer systems ..."
This is at the top of the second paragraph.
A. Yes.
Q. "... Horizon is relatively robust."
A. Yes.
Q. So that's your considered view about the Horizon system,
isn't it?
A. Yes.
Q. On the basis of all the evidence that you had seen?
A. Yes. In my first report I stated that. In my second
report I explained that in light of the evidence that
had been disclosed between the two reports, that the
Horizon system wasn't as robust as I initially
considered.
Q. But your overall view after that process is set out in
the passage we just looked at?
A. It is.
Q. Your view is Horizon is relatively robust, and I would
like to explore that if I could. We agreed that
robustness is a mature topic which is the subject of
academic study and has well understood practical
principles. Would you agree?
A. Yes.
Q. You say that from your experience of other systems
Horizon is relatively robust. So you are saying that
you have considered comparable systems of which you have
experience?
A. Yes.
Q. These are systems with which you are familiar?
A. Indeed, yes.
Q. And you have compared Horizon's robustness with the
robustness of those systems, and compared with those
systems Horizon is relatively robust, yes?
A. Yes.
Q. Let me tell you what I think you are saying here and you
will correct me if I'm wrong. It has a good level of
robustness compared with some systems?
A. Yes.
Q. In the top quartile, maybe?
A. That's not the assessment that I believe I have made.
Q. Certainly in the top half, would you agree?
A. No, I don't think I want to be drawn on where it is
positioned.
Q. If I were to tell you that when I go running I'm
relatively fast, isn't that what I would be saying, that
I'm faster than most people I run with?
A. Yes. For context, I have worked on a number of systems
in banking, in manufacturing, and Horizon compares well
with those systems, with the context that we have added
here that it is not free from defects and the impact of
those defects is very important to consider but it is
relatively robust --
Q. So --
A. Sorry, if I could just add to that.
Q. Yes, of course.
A. What must be considered is that we are talking about
Horizon as it is today and the processes that are in
place today, but there has been quite a long journey to
get to the position that it is now --
Q. I'm going to ask you about that later, Mr Coyne, but you
have made the point and I understand it.
A. Okay.
Q. If I understand you correctly, you are saying that if
there is a spectrum of comparable systems in the world
today from 0 to 10, Horizon is towards the upper end.
You don't want to commit yourself to whether it is in
the top 25%, but it is towards the 10 end rather than
the 0 end?
A. Yes, but what you have got to take care -- and
benchmarking against systems that are in similar
contexts. There are of course systems that are dealing
with flag management, I used to work at
British Aerospace looking at their military aircraft
programming. There was a whole different spectrum of
robustness that was required there. I have also worked
in health care where you are talking about life critical
systems. Horizon doesn't compare anywhere near those
sorts of systems. What I'm talking about is Horizon
when benchmarked against systems of similar context, so
in retail, in banking and things like that.
Q. And these are systems where there are large numbers of
users?
A. Yes.
Q. And a great complexity in the transactions being
performed?
A. Yes.
Q. And these are systems which have measures and controls
to achieve robustness?
A. That aim to achieve robustness, yes.
Q. Could you just define which sectors you are talking
about as being comparable for the purposes of your
judgment?
A. So point of sale systems, something where a transaction
is taking place over the counter, is certainly
comparable. Banking has elements of it, where automated
payments are being transferred to different
organisations, so that is certainly comparable as well.
Stock control, things like that.
Q. And you are talking about large businesses with lots of
users and lots of complexity, yes?
A. Yes, at a similar scale to what is seen in Horizon.
Q. We have been talking about measures and controls or
countermeasures. Dr Worden identified 18 of them in his
first report, didn't he?
A. Mm.
Q. You will see that he says he has defined three letter
acronyms for each of them?
A. Yes.
Q. And he says specifically, doesn't he, that most of these
acronyms are not commonly used in the industry?
A. I think he did say that, yes.
Q. But in your second report you suggest that he might be
giving the false impression that they are commonly used
in the industry, do you remember saying that?
A. Certainly in my report I point out that they aren't used
in the industry.
Q. Let's see what you actually say. It is at bundle
{D2/4.1/141}, paragraph 5.67:
"Whilst Dr Worden acknowledges these acronyms are
mostly not used in industry, he has used them throughout
his report which gives the impression they have a
standard meaning and scope."
Do you think that is a fair thing to say? He
specifically says in his report that they don't.
A. Well, I actually think that if you do attribute acronyms
to things there is a danger that the perception is that
they are read as being industry standard terms.
Q. Even in circumstances where he specifically says that's
not the case on at least two occasions in his report?
I mean why did you feel it necessary to say that,
Mr Coyne, in circumstances where Dr Worden had already
made it clear?
A. I'm not sure.
Q. Anyway, you go through his 18 countermeasures over the
page {D2/4.1/142}, there is a table that goes on, and it
is fair to say you disagree whether one of them is
a standard countermeasure, you disagree that manual
workarounds count as a standard countermeasure?
A. Yes, I don't believe that a manual workaround can be
a --
Q. But for others you accept they are typical standard
features in IT systems design?
A. Yes, certainly when you are designing a system, some of
the aspirational features that you want to build in
there would be contained within these controls.
Q. Mr Coyne, you have used the word "aspirational" several
times now and I'm just wondering whether you're trying
to drum home a theme. If you want to make a point about
aspiration I don't want to stop you making it, but you
do throw this word in quite a lot to your answers and
I'm wondering what point you're trying to make by the
word "aspiration"?
A. I'm just pointing out that when you start a project and
you build something, you go through a workshop typically
of setting out what your design aspirations are. That
is just a term I use when I'm conducting workshops with
people going through things.
Q. I see. Your judgment on the robustness of the Horizon
system takes all these countermeasures that Dr Worden
refers to -- with the possible exception of
workarounds -- into account, doesn't it?
A. Yes.
Q. It is necessary to do so to form a judgment on
robustness, isn't it?
A. I'm not sure that you need to consider -- I'm not sure
that you need to decide that this is the subset of
factors that you are going to consider before you come
up with a calculation of robustness.
Q. Mr Coyne, didn't one of the Horizon Issues specifically
ask you to consider whether controls and measures in
Horizon reduced certain problems to an extremely low
level of risk?
A. It certainly -- yes, it did.
Q. And I think we have established, it has taken an hour to
do this, I think we have already established that when
making an overall judgment of robustness you both
consider how the system operates and then consider how
the countermeasures designed into the system also
operate?
A. Yes.
Q. With a view to coming to an overall decision as to the
robustness of the system?
A. Yes.
Q. Thank you. So in forming a judgment about robustness it
is necessary to form a judgment also, isn't it, about
how the countermeasures, what Dr Worden calls the
robustness countermeasures, how those countermeasures
were designed and operated in practice, yes?
A. Yes.
Q. A key element of robustness is considering how a system
limits the extent of the impact in this context, in the
context of the Horizon system, and the issues we are
considering, namely, does Horizon cause shortfalls in
branch accounts. A key element of the robustness
question is considering how the Horizon system limits
the extent of any impact on branch accounts, yes?
A. Yes. Certainly you need to establish what the impact is
and, secondly, you should take steps that if it is ever
going to happen again you reduce that impact.
Q. So in forming a judgment on robustness you have first of
all to see -- let's take a bug. A bug happens and the
first question you ask is: did that bug or could that
bug have had an impact on branch accounts?
A. Yes.
Q. Then you form a judgment on whether and to what extent
the countermeasures in place in the Horizon system would
have enabled that impact to be identified and fixed,
yes?
A. No, I think you would look at whether it did -- whether
any countermeasure or control did prevent it from having
a consequential impact, not whether it should have.
Q. Well, whether it would have?
A. Or whether it would have.
Q. You don't consider whether it would have, you consider
whether it did?
A. Well, if it did there would be evidence that it did. It
would be documented.
Q. But in some cases it will be blindingly obvious, won't
it, Mr Coyne? Let me give you an example, remming.
A Post Office example. Remming in and remming out.
Money is sent from Chesterfield to a branch.
Chesterfield has a record of the money it sends over.
The branch receives the money and types in -- or usually
it is a barcode actually, but it records on the Horizon
system how much money the branch has received.
You are aware, aren't you, that there is
an automatic system that checks Chesterfield's figures
against the branch's figures?
A. Yes.
Q. Are you suggesting that every time over the last
20 years there has been a discrepancy between
Chesterfield's figures and the branch's figures, are you
suggesting that it is necessary for you to see the
evidence of what happened next, of whether a transaction
correction was sent and what the evidence was in
relation to that branch? Are you really suggesting
that's necessary?
A. No, because in a typical scenario where the systems
operate as they should, as they are designed, then the
positioning of the countermeasures that you have put in
place would pick up on that so that would work
absolutely fine. In the scenario where the system
doesn't operate as expected there is a bug, error or
defect or communication fault, then a different set of
scenarios will likely be encountered and it is
understanding then what happens that is important.
Q. Let me get this straight. You are suggesting that there
are cases when you can take it as read, the situation is
such that you can take it as read that a problem will be
identified and it will be fixed. But you are suggesting
there are other situations where you can't take it as
read where specific evidence is needed, is that right?
A. Yes.
Q. Okay. But nonetheless, although you say that is
necessary, it is your considered opinion that the
Horizon system that you now see is robust, yes?
A. Relatively robust, yes.
Q. Thank you.
If I could look at the third joint statement again
which is at -- let's go to your first report, actually,
which is {D2/1/26}. At paragraph 3.7 you say:
" ... whilst the present-day version of Horizon,
supported by manual human support may now be considered
as relatively robust in the spectrum of computer systems
used in businesses today it has undergone major
modifications ..."
And in forming that judgment, if I can go down to
paragraph 3.9, you say:
"Fundamental in determining the robustness of
Horizon is gaining an understanding of the Post Office
(and Fujitsu) manual business processes applied when
determining and handling the effects of bugs ..."
So may I take it that the judgment that you record
in paragraph 3.7 is based upon an understanding that you
have gained of the processes you describe in
paragraph 3.9?
A. Well, one of the challenges is that we do have
an understanding of business processes for many parts of
the Horizon support system, we don't have it for all,
and one of the key elements is when it comes to
reconciliation, so the recovery of a situation that
occurs when a bug, error or defect has had an impact on
branch accounts, there is a need for reconciliation and
we don't have that part. So the closed loop, we are
unable to close that loop.
Q. I'm finding it difficult to understand what you are
saying. Are you saying that you are in a position to
make a judgment that Horizon is robust or are you saying
that you aren't?
A. I'm saying that based on the information that's
available it is relatively robust.
Q. Thank you.
A. What I'm also saying is subject to other documentation
or other parts of the process that has not been
disclosed at this stage, the situation may improve or it
may well be a worse situation.
Q. Mr Coyne, if I can suggest to you that makes no sense
whatsoever. You have made a judgment. You have already
confirmed I think more than once --
A. Yes.
Q. -- that it is your considered opinion that the Horizon
system as it exists at the moment, with all its
countermeasures, is robust?
A. Yes.
Q. In those circumstances you can't be suggesting, can you,
that there's evidence out that there that you haven't
seen that may mean that that judgment is completely
wrong?
A. I don't see why that can't be the position. There may
well be documentation and there is documentation that
must exist with regard to reconciliation within
subpostmasters' accounts after a bug, error or defect
has occurred and outstanding the outcome of that will
have an impact on the statement of robustness.
Q. Mr Coyne, when you talk about reconciliation you are
talking about the reconciliation that occurs between
transactions as transferred from the Horizon system into
Post Office systems and those transactions then being
compared with the transactions that are recorded by
banks, financial institutions and other Post Office
clients, is that what you are talking about?
A. No, sorry, specifically what I'm talking about here is
when a bug, error or defect occurs, as we have seen
here, there is the identification of a discrepancy on
a postmaster's account, and that's in the evidence.
What we then don't see is how, if at all, that was dealt
with.
Q. Just to be absolutely clear, you are not suggesting that
you think that when a bug was identified and
discrepancies were spotted in accounts, you are not
saying you think the accountancy discrepancies were not
dealt with, you are not saying that, are you?
A. No, I'm not saying that. What I'm saying is they were
identified typically by Fujitsu, and I don't know what
the process was, and Fujitsu will say that they don't
know because it appears in all the documents, they don't
know what Post Office does to correct that. That is a
missing part of the evidence. So I'm not saying I think
Post Office do nothing about it, I'm simply saying
I don't know what that process was.
Q. Right. You are not suggesting that you think, on the
basis of documents you have seen, that it is likely that
Post Office do nothing about it? You are not saying
that, are you?
A. No, I'm saying the opposite. It is likely that they do
something about it. I'm saying there is no evidence
available to --
Q. I'm grateful. Then that explains what I was trying to
explore with you. That explains why you feel able to
judge, as you do in paragraph 3.7, that Horizon is
relatively robust. On the basis of the evidence you
have seen --
A. Yes.
Q. -- including the evidence you have seen of the Fujitsu
processes and other documents including Post Office
processes, you are happy, and it is your considered
view, that Horizon is relatively robust?
A. Yes.
Q. Thank you. Now let's get back to what it means --
MR JUSTICE FRASER: Mr de Garr Robinson, we are going to
have a break at some point. Is now a good time?
MR DE GARR ROBINSON: It is a perfect time.
MR JUSTICE FRASER: We will come back in at 11.55 am.
Mr Coyne, you are in the middle of giving your
evidence. I am sure you have heard me say this before.
You are not to speak to anyone about the case and this
applies throughout the week because you are going to be
there all week. I know you know it, I might bore you
senseless by repeating it every time we stop. You don't
have to stay in the witness box but don't talk to
anybody else about the case.
We will come back in at 11.55 am.
(11.51 am)
(A short break)
(12.00 am)
MR DE GARR ROBINSON: Mr Coyne, getting back to what it
means for Horizon to be relatively robust. You have
described the systems which you say are comparable to
which you have compared Horizon. The systems that you
are talking about, they are usually in the private
sector, I apprehend?
A. No. No, across --
Q. Okay, across both. But the kind of businesses you are
talking about, banking, retail, those are private
sector?
A. Yes.
Q. Those sorts of systems have quite exacting requirements
of robustness, don't they?
A. Yes.
Q. No system can be perfect, everyone agrees that, but the
large and complex businesses that use these sorts of
systems, they don't have much tolerance for widespread
errors in data entry, processing or storage, do they?
A. No, that is right. They certainly strive to remove that
wherever possible.
Q. Given the number of transactions and the importance of
handling and accounting those transactions properly,
those businesses require the overwhelming majority of
their transactions to be handled and accounted for
properly, don't they?
A. They do indeed, yes.
Q. They can only afford for a tiny proportion to suffer
from lasting errors, yes?
A. Yes.
Q. Air traffic control and medical machines, frankly they
can't afford any errors at all.
A. Indeed.
Q. But the kind of systems we are talking about, their
tolerance for errors is very low, yes?
A. Certainly it will be fractions of a percent of the total
transaction, whatever the transaction might be, yes.
Q. Any significant level of non-transient errors would be
quite a serious threat to their businesses, wouldn't it?
A. Yes.
Q. It would be a threat to their customer base, a threat to
their manageability, a threat to their manpower
requirements, having to run around dealing with
problems, a threat to their operating costs, a threat to
their profitability, and given that businesses operate
in a competitive world where other businesses are happy
to displace them, it could be a threat to their very
survival, would you agree?
A. It could be. They are quite generic statements but
I don't disagree with any of them.
Q. Those are the businesses to which you have compared
Horizon, yes?
A. Yes.
Q. So when you say the Horizon system is relatively robust,
you are saying that in the overwhelming majority of
cases where conditions are both normal and adverse it
works reliably, yes?
A. Yes, the vast majority of all the transactions that flow
through the Horizon system will work successfully.
Q. And although there are occasions when it doesn't, those
represent only a tiny proportion of the work that it
does, correct?
A. Yes, it will be a small fraction of the work that it
does, yes.
Q. Even within that tiny proportion, a large majority of
the problems thrown up are picked up and corrected by
the various systems in place that are designed to do
precisely that?
A. Certainly many of the ones that go wrong for one reason
or another appear to be picked up. There are examples
where they don't appear to have been picked up and
examples which appear to have an impact.
Q. That is a very important opinion, isn't it, in the
context of this case?
A. Yes.
Q. For the purposes of this trial?
A. Yes.
Q. That being the case, Mr Coyne, I wonder why you haven't
actually spelt out that in your reports or your joint
statement?
A. I believe that I have. Have I not?
Q. It may be unfair, Mr Coyne, but my sense is that in both
your reports you tried to give a rather different
impression, an impression of a system which is beset by
problems, the sheer volume of which means it cannot
sensibly be regarded as reliable. Would that not be
fair?
A. I don't believe that that's fair, no.
Q. Let's have a look at your first report. If we could go
to bundle {D2/1/25}. This is the report -- your first
report that you produced in mid-October. And if we look
at paragraph 3.1 --
MR JUSTICE FRASER: I do not think we are on the right page
on the common screen.
MR DE GARR ROBINSON: {D2/1/25}.
A. Yes.
Q. You recite the agreement in paragraph 3.1 contained in
joint statement 1?
A. Mm.
Q. You say in paragraph 3.2:
"Each discovered bug ... could have remained
unresolved in Horizon for varying periods of time."
And you add that there is a possibility of other
bugs.
A. Mm.
Q. Then you say this in 3.3:
"The sheer volume of known error logs and
reconciliation reports confirm the wide-ranging extent
of the impact of such bugs ... This evidence
demonstrates that such bugs ... would undermine the
reliability of the Horizon system to accurately process
and record transactions."
That seems quite an ominous statement to make given
the propositions that you have just agreed with me
a minute and a half ago, Mr Coyne. Do you not see
a tension?
A. Yes, I think I do actually. At that point in time we
had a document set that consisted primarily of known
error logs. I think the PEAKs had only just been
disclosed a few days --
Q. Weeks?
A. Sorry?
Q. Would it not be weeks?
A. I don't know exactly how many.
Q. I'm sorry, I interrupted you. Please carry on.
A. Perhaps if we establish what the date was. I think we
do mention it in the reports.
Q. Yes. Please carry on. I interrupted you and
I shouldn't have.
A. So what we are saying there is the full picture was yet
to be revealed and that it may well undermine the
reliability of Horizon.
Q. So are you suggesting, Mr Coyne, that when you produced
your first statement you were doubtful about whether it
was robust?
A. Mm.
Q. And it is when you produced your second statement, when
you had more opportunity to look at the PEAKs and so on,
that you formed the impression that it was robust after
all?
A. I had a different concern when it came to create my
supplemental report because there was the discovery of
far more defects than it was originally said existed.
When putting together my first report I thought the
analysis had already been done to establish that there
were only three defects. I discovered a fourth defect
but then shortly afterwards discovered many more. It
was also the case that I had seen a number of reports
about improvement of the processes that was in place.
But then it transpires that when we saw some of the
witness statements that those processes, those
improvements in processes, hadn't been adopted.
So there was a number of other factors that were
brought in place that confirmed that there was large
elements of unreliability. That doesn't take away my
overriding statement that the Horizon system is
relatively robust.
Q. Mr Coyne, one of us is confused and it might be me.
I had taken you to paragraph 3.3 and suggested to you
that the impression, the strong impression given by
paragraph 3.3 was that Horizon wasn't very robust at
all. My understanding of your answer, and it may have
been a false understanding on my part, was you said,
well, at that stage we only had the PEAKs for a few
days, and the impression I got was you were saying it
was only later I realised that Horizon was relatively
robust. So I suggested -- I asked you about that. And
then what you actually said was, no, no, in my later
report I thought Horizon was less robust.
So at the time of this report your view as to the
robustness of Horizon was at its zenith, was it?
A. Could you clarify what you mean when you say zenith.
Q. It was at its peak. This was at a time when you thought
Horizon was very robust and it is only later that you
then downgraded it to relatively robust?
A. There's different aspects of it. This is the
complication with robustness, it isn't an entity that
can be measured by one simple set of criteria.
Q. All I'm trying to ascertain with you, Mr Coyne, is why
on the one hand you were saying at the time of this
report you thought that Horizon was relatively robust,
you felt able to include -- to say what you said in
paragraph 3.3, which gives a completely different
impression. I would like you to explain how you felt it
appropriate to do that.
A. What I said there is correct, that the bugs, errors and
defects that we see, when they trigger they would
undermine the reliability of Horizon and did undermine
Horizon's reliability.
Q. What you say is -- look at the first sentence:
"The sheer volume of known error logs and
reconciliation reports confirm the wide-ranging extent
of the impact of such bugs ..."
A. Yes.
Q. So first of all you say that there is a huge volume of
KELs showing bugs with branch impacts, do you?
A. Typically the KELs show that PEAKs exist that have had
branch impacts, but the KEL is the knowledge document.
Q. You appear to be saying there, Mr Coyne, and it may be
that I'm not being clear enough. Let me be absolutely
clear. You appear to be saying there that there is
a large volume of KELs which demonstrate bugs in the
Horizon system which had branch impacts. Is it the case
that at the time you wrote this report you had seen
a large number of KELs showing bugs with branch impacts?
A. Right, it is important to understand -- in order to
answer that, it's important to understand the
relationship between KELs and PEAKs. A KEL is
a document which is often updated with new knowledge
that will enable Fujitsu's support department to better
support the users as and when they find a particular
defect on the system. Contributing to that KEL is
a number of PEAKs. There might only be one occurrence
of a PEAK that leads to a KEL, there might be twenty or
thirty different PEAKs that contribute to a KEL. So the
fact that we had the KELs at that point in time and
those KELs indicated problems, and there were a number
of PEAKs that hadn't yet been fully examined and I set
out that in my report, but of the PEAKs that had been
examined there were PEAKs that suggested impact on
branch accounts.
Q. Let me ask my question again but I will do it in
a different way. Let's cross out the word "and
reconciliation reports" in the first sentence so it now
reads:
"The sheer volume of [KELs] ... confirm the
wide-ranging extent of the impact of such bugs ..."
So you are referring to a large volume. It's not
just a number, a significant number, you're saying
a large volume. It's a substantial body of KELs --
A. Yes.
Q. -- confirming the wide-ranging extent of the impact of
bugs.
A. Yes.
Q. Is that the case, that you had seen a large volume of
KELs indicating the wide-ranging extent of the impact of
bugs?
A. Yes.
Q. How many?
A. A number are listed in this report but I don't know
precisely what the number is.
Q. And when you say impacts, you are talking about branch
impacts, aren't you?
A. Yes.
Q. So you are saying there was a large number of KELs which
you had already seen at the time of this report that
confirmed that there were bugs, wide-ranging bugs -- or
confirmed bugs which had a wide-ranging impact on branch
accounts?
A. Yes. Certainly within the KELs there was reference to
discrepancies which appeared to be on branch accounts.
Q. Mr Coyne, I would like to suggest to you that in the
whole of this report that goes on for over 200 pages,
you refer to very few KELs that actually confirm the
existence of branch impacts?
A. There is a table I believe at the back of this report
that sets out the number of KELs.
Q. What page would that be? I may not have it in hard
copy.
A. Appendix G.
Q. What page?
A. 200, I believe.
MR JUSTICE FRASER: {D2/1/213} is appendix G. Could you
give me one second, please. (Pause)
Go ahead, they are just going to sort out my
screens. The common screen is fine, it's my custom
screens.
MR DE GARR ROBINSON: Mr Coyne, I have looked at the bug
table in the second joint statement of the experts.
A. Yes.
Q. The bug table in joint statement 2 is a table which sets
out the bugs that you have identified which you believe
show evidence of branch impacts, yes?
A. Yes.
Q. That table, if my calculations are correct, only refers
to four bugs that you had identified in your report in
addition to the three bugs that had been accepted by --
already disclosed by Post Office.
Now, what I would like to understand is do you
accept in principle that that's the position?
A. I don't I don't know if that's the number. That is a
number you have calculated, haven't you?
Q. Yes.
A. I haven't done the calculation.
Q. There are 29 bugs in the bug table.
A. Yes.
Q. Many of those bugs were only identified either by
Dr Worden in his report or by you in your second report,
yes?
A. I believe Dr Worden only identified the three bugs that
had already been identified.
Q. That's not -- perhaps we will come to that in due
course, Mr Coyne. In fact, Dr Worden -- I think the
number may be nine, something like nine bugs he had
identified in his first report that are now to be found
in the joint statement. Did you not appreciate that was
the case?
A. I don't know whether that's correct or not.
MR JUSTICE FRASER: But the KELs you were being asked about
volume, they are the ones listed in appendix G?
A. Yes.
MR DE GARR ROBINSON: What I'm suggesting to you, Mr Coyne,
is very few of these KELs have turned up -- or the bugs
to which you say these KELs relate, very few of them
have turned up in the joint statement bug table.
A. Right, okay. I'm not sure that is correct.
Q. I'm suggesting to you, Mr Coyne, that you had not seen
a large volume of KELs showing branch impact -- showing
bugs with branch impacts at the time you wrote this
report?
A. Well, I believe a number of these KELs that are
contained within this appendix have impact.
Q. Are you suggesting that all of these KELs relate to bugs
which had branch impact?
A. Many do, but what you have to remember about the KEL is
the KEL doesn't specifically relate to the branch.
That's typically the PEAK.
Q. Let's not talk about -- I'm just trying to analyse what
you said in paragraph 3.3 of your report. You said the
sheer volume of KELs confirm the widespread branch
impact. And I'm suggesting to you, Mr Coyne, that you
had not seen a huge volume of KELs which confirmed
branch impact at the time. You just hadn't.
A. That paragraph would be better if it said: the large
number of KELs that I have reviewed and a portion of the
PEAKs that have recently been disclosed and I have
reviewed dot dot dot, and carry on with that paragraph.
That would be nearer --
Q. If we look at page 214, for example, it starts with "Pin
pad", pin pad errors. So you have six pin pad errors.
{D2/1/214]
A. Yes.
Q. You are not suggesting those pin pad errors had branch
impact, are you? Those pin pad errors meant the branch
couldn't do business with a particular customer. They
didn't actually create an error in branch accounts, did
they?
A. It is possible they might. We would have to go through
and look at the detail.
Q. So you are suggesting that there are errors in pin pads
which are covered by KELs which have created branch
impacts, are you?
A. Yes.
Q. Because what's curious is I don't believe there are any
of those in the bug table in JS2.
A. Yes, I believe there was the scenario where it was
remembering the account of the last person that should
have been paid. So was it the allpay defect? The
payment that should have gone to other beneficiaries was
all sent to allpay instead because of a defect?
Q. I will have to look at that later.
Going back to page 25, paragraph 3.3, you explained
that you think you had seen already, by the time of your
first report, a large volume of KELs showing widespread
branch impact {D2/1/25}, and you say that it is not the
KELs so much as the PEAKs that refer to them, is that
how you put it?
A. KELs indicate that there was a defect and it is
typically the PEAKs that record the particular branch
impact. So by looking at either you can tell there was
a problem, but it is only when you look at both the KEL
and the related PEAKs that you understand the scope of
the problem.
Q. Wouldn't it be fair to say though that when you look at
a KEL, generally speaking the KEL which deals with it --
and KELs deal with all sorts of things, including things
that are not bugs at all. But when you look at a KEL,
generally the KEL will explain what the impact of the
problem is, the nature, the practical consequence of the
problem that has to be dealt with, would you accept
that?
A. Yes. It might, for example, say there is a bug and we
believe this is going to cause a discrepancy and we have
spotted it in eight branches, but it won't tell you what
the impact on those eight branches is. For that you
have to go to the particular PEAK.
Q. Generally speaking then, if there is a KEL that
addresses a bug that has a branch impact, generally it
will tell you?
A. Generally speaking.
Q. On the vast majority of occasions, yes?
A. I'd prefer to stick with generally speaking. I don't
know if it is the vast majority.
Q. Okay. You then say in paragraph 3.3 -- let's cross out
"known error logs", you now say:
"The sheer volume of ... reconciliation reports
confirm the wide-ranging extent of the impact of such
bugs ..."
So you are suggesting here that reconciliation
reports are good evidence of bugs in Horizon which had
branch impacts, are you?
A. Yes, there was a response to a question that I asked
about reconciliation, I don't know exactly how it was
phrased, but I asked for the number of reconciliation
problems that required manual intervention, and the
answer I was given was that it was hundreds of thousands
per week, I think was the --
Q. 10,000 a week that have to be F99ed, do you remember
that?
A. Yes.
Q. What is a reconciliation report?
A. Where there has been a problem within the system for one
reason or another and the transaction has not proceeded
to completion as it would originally have expected.
Q. Let me put to you what a reconciliation report is just
to save some time. When a transaction is done in
Horizon and the basket is committed to the system, the
transaction goes into the branch accounts, yes? That
transaction goes into the branch database.
A. Mm.
Q. And then copies of those transactions are then
transmitted through systems such as the TPS system,
could you explain what the TPS system is?
A. Transaction processing system.
Q. What is the TPS system? What does it do?
A. It sends -- well, it decides what payments need to go to
what particular beneficiaries.
Q. Is that right? Doesn't it transmit information relating
to these transactions that had been committed to the
BRDB to Post Office's back-end systems?
A. That is right.
Q. And isn't it Post Office's back-end systems, like
POLSAP, like Credence and so on, isn't it those systems
that then do the steps that are needed to ensure that
payments are made by financial institutions and so on,
is that right?
A. Yes, there is a harvesting process which will pick up
various transactions and will put them in certain
buckets, and then there are various processes that goes
on with those.
Q. And the harvesting process is a process by which --
well, what happens is transactions go into the TPS
system and they are then propagated into POLSAP and
other management information systems that are maintained
by Post Office, yes?
A. Records of those will do, but then a number of the
transactions will then go on to -- for example, if it is
a bill that's being paid it will go on to agree that
payment with whoever the party is.
Q. What happens then is -- generally the financial
institutions making payments have received messages
direct from things like the pin pad?
A. Yes.
Q. And what then happens is the financial institution has
its own record of the transactions it thinks are being
done?
A. That is right.
Q. And Post Office has its record of the transactions it
thinks are being done?
A. Yes.
Q. Those records are maintained on Post Office's systems?
A. Agreed.
Q. And then there is a reconciliation process by which
those systems are automatically reconciled to see
whether they agree with each other?
A. They should be automatically reconciled, yes.
Q. That is what you describe as reconciliation, and that
process generates reconciliation reports, doesn't it?
A. Yes.
Q. And when there are exceptions, then in those
circumstances there are reports indicating transactions
where some of the information which Post Office has
about them, it differs from some of the information that
the financial institution has about them?
A. Yes, something has gone wrong with the process.
Q. Yes. And it might be that some attributes are missing
that have nothing to do with the basic transaction
details?
A. It could be any aspect of it --
Q. It could be time stamps, it could be anything.
A. It could be value, it could be anything.
Q. It could even be that there has been a delay in the
financial institution or POLSAP receiving the necessary
information, couldn't it?
A. It could. I think it is safe to say that something has
gone wrong, the process hasn't operated as it was
designed, and an intervention needs to take place to
correct the transaction.
Q. What happens is a reconciliation process is
automatically undertaken, exception reconciliation
reports are generated which demonstrate where there are
exceptions that need to be looked at?
A. I'm not sure a reconciliation process is automatically
undertaken.
Q. Mr Coyne, when you have 3 million transactions that are
on the Bank of Ireland's books and 3 million
transactions that are on Post Office's books, you don't
have a human being comparing those two sets of
transactions, do you?
A. No, but that's when it is working as expected. When
something has gone wrong this is the reconciliation
process that I'm talking about there.
Q. We are getting bogged down in a way that I'm finding
quite surprising. The process by which lists of
transactions are reconciled is automatic, isn't it? It
is electronic?
A. Yes, when everything is working fine it is automated.
Nobody needs to get involved.
Q. The process of reconciliation is the process of
comparison between the two sets of data?
A. Yes.
Q. That process is electronic, isn't it?
A. Yes.
Q. And then what happens is there is a report generated on
a regular basis which identifies reconciliation
exceptions where for some reason, for the reasons we
have already discussed, there isn't a full
reconciliation and something has to be looked at?
A. Yes.
Q. We have discussed some of the reasons for that?
A. Yes.
Q. And that's where a manual element can become involved?
A. Yes.
Q. Sometimes there can be one error which explains 1,000 or
2,000 reconciliations exceptions?
A. Possibly, yes.
Q. And there will be a delay in the transaction getting
through to the bank, or it will be when it did go
through to the bank there was a time stamp missing so
the two didn't match. There could be something like
that that could explain an awful lot of them?
A. There could be a range of different things that are
wrong with the transaction.
Q. And that's when someone has to go and look to see if
there is a problem?
A. Yes, and that is the number that I'm referring to
because it was suggested that that was in the many
thousands.
Q. But I don't understand, Mr Coyne, why you think that the
existence of reconciliation reports confirms the
wide-ranging extent of the impact of bugs in Horizon
which are having an impact on branch accounts?
A. I have worked and designed banking systems, stock
broking systems. I have never seen the need for tens of
thousands of transactions per week to have a human
intervention. That suggests that something is going
wrong. It is working outside of process on a larger
scale than I would have expected.
MR JUSTICE FRASER: Can I just ask you, at {Day14/77:17} of
today's transcript you said "it was suggested" that it
was in the many thousands. Can you just tell me, "it
was suggested" doesn't really help me because I don't
know who suggested what to whom.
A. Sorry, my Lord. I asked the question as part of
a request for information.
MR JUSTICE FRASER: Yes.
A. And the response that I was given by Post Office's
lawyers was -- I believe it was ten thousand.
MR JUSTICE FRASER: So that is what you are referring to?
A. Yes.
MR JUSTICE FRASER: All right.
MR DE GARR ROBINSON: Now Mr Coyne, there are many, many,
many possible explanations for reconciliation exceptions
that have nothing to do with bugs in Horizon causing
shortfalls in branch accounts, aren't there?
A. There could be, yes.
Q. There's no reason to think -- the fact that there are
any scale of reconciliation exceptions does not
demonstrate, does not show or confirm, that there are
any bugs which have an impact on branch accounts, does
it?
A. It certainly could do if the system is having that level
of failure to reconcile then there is something wrong.
It is operating outside of the process it should
operate --
Q. Well, I will -- how is it that you are able to judge
that, Mr Coyne? Do you know what range the system was
designed for?
A. No, but I have experience of similar systems and I don't
believe that such a high level of manual reconciliation
would be tolerated or should be expected.
Q. But the possible causes of those reconciliation
exceptions could involve all sorts of things such as
phone lines from the Post Office to the financial
institutions, or the financial institutions' own
operating systems, couldn't they?
A. So that means that that is going wrong each week.
Q. The reason why I'm asking you about this is you are
suggesting that the fact that there are reconciliation
reports, large numbers of reconciliation exception
reports, let's call them, is itself confirming that
there are bugs in Horizon which have an impact on branch
accounts, and my suggestion to you, Mr Coyne, is that
doesn't confirm anything of the sort?
A. Well, it suggests rather than confirms.
Q. And where there is a reconciliation exception which is
looked into, what happens -- and a view is taken --
a view will then be taken by Post Office and the
financial institution as to what the correct position
would be?
A. Mm.
Q. If the view is taken that the correct position is that
the branch account should be corrected, what happens
then?
A. What should happen is a transaction correction should be
created by --
Q. And would you accept that when a transaction correction
is sent it is more likely than not that the transaction
correction will correct an error that's happened rather
than create one, yes?
A. It is more likely. There certainly have been scenarios
where transaction corrections have caused a problem but
it is certainly more likely that it would correct it,
yes.
Q. What I suggested to you about paragraph 3.3, Mr Coyne,
is this paragraph seems designed to give an impression
that Horizon isn't robust, whereas on the very next page
you say that it is. What do you say to that suggestion?
A. Well, I'm setting the context before providing my
summary that it was relatively robust.
Q. What I'm finding difficult to understand, Mr Coyne, is
you say -- you refer to a wide-ranging extent of the
impact on branches of bugs, and then over the page you
say but Horizon is relatively robust, and I would just
like you to explain how is it you felt able to do that?
A. Because as I have agreed with Dr Worden, robust doesn't
mean that there won't be an impact. The two aren't
mutually exclusive.
Q. You have also agreed with me, Mr Coyne, that in the
context in which you are using the term relatively
robust in paragraph 3.7, it means that there's only
a tiny number, a tiny proportion --
A. I didn't say tiny.
Q. Really? Is that not precisely what we were discussing
before?
A. I don't believe I'm using --
Q. When we were talking about using comparable systems and
how they have a low risk tolerance?
A. I wouldn't characterise it as tiny. I probably said
a fraction of a percentage.
Q. Okay.
MR GREEN: My Lord, the evidence -- the word "tiny" was put
and he said a small fraction. That's what the
transcript says.
MR DE GARR ROBINSON: Yes. And do you think -- my
suggestion to you, Mr Coyne, is that in paragraph 3.3
you're actually trying to give an impression which is
seeking to undermine actually what is a very helpful
conclusion that you have set out in paragraph 3.7?
A. That certainly wasn't the way it was written. It was
constructed to set the context for it.
Q. Let me also suggest, Mr Coyne, that in this report you
also tried to give the impression that the concept of
robustness has no meaning and it's impossible to
measure, would you agree with that?
A. I do agree that it is impossible -- it is very
subjective and it is impossible to come up with
a quantifiable measurement, yes.
Q. Now let's go back to the joint statement you made
shortly before you produced this report. It is at
{D1/1/8}, I'm going to page 8.
Now, I think you have maintained in the evidence you
have just given that paragraph 3.3 where you talk about
the sheer volume of things happening is consistent with
Horizon being relatively robust as compared with its
comparators, yes?
A. Mm.
Q. So the fact that there is such a sheer volume is not
inconsistent with it being relatively robust, is that
right?
A. Yes.
Q. Okay. Well, we have already read what the experts
agreed in paragraph 2.3. If we could go over the page
to the disagreement boxes on page {D1/1/9}, there at the
top you have got your definition of robustness.
A. Yes.
Q. Then just below that you say:
"in consideration of the likelihood of Horizon to be
the cause of shortfalls in branches, Horizon is not
determined to be robust in this regard because:
"(a) It contained high levels of bugs ... which
created discrepancies in the branch accounts."
So now I'm really confused, Mr Coyne. On the one
hand your judgment on 16th October was that actually
Horizon is relatively robust and a very small proportion
of transactions go wrong as a result of bugs, but here
you appear to be saying the opposite. Did you have
a change of mind between agreeing joint statement 1 and
your first report?
A. Joint statement 1 was constructed before the end of the
first report.
Q. Joint statement 1 was on 4 September.
A. Yes.
Q. Your report was on 16th October. I will be corrected if
those dates are wrong.
A. Right.
Q. Did you change your mind between 4 September and
16th October?
A. What I did is that whilst looking at the factors in the
round I considered Horizon to be relatively robust,
whereas my initial opinion was that it was not robust.
Q. So you did change your mind, is that right?
A. Yes.
Q. And what made you change your mind, please?
A. It was just a consideration of all aspects in the round.
Q. Well, Mr Coyne, in your -- a joint statement is
an important document, you have given evidence many,
many times before, haven't you?
A. Yes.
Q. And you will appreciate this is as important in many
ways as an expert report, correct?
A. It was a very early document, it was when we were
finding our way around the evidence that was available.
Q. Well, you wouldn't express an opinion unless it was your
considered opinion, is that right, in a joint statement?
A. That is true.
Q. So if, for example, you felt that things didn't look
good but you hadn't looked at all the evidence and you
wanted to withhold judgment, you wouldn't express
an opinion at that stage, would you? Or if you did, you
would express it in a very provisional and equivocal
way, wouldn't you?
A. Yes.
Q. But that's not what you do here, Mr Coyne, is it? And
I would like you to explain why, please.
A. Well, I mean the reasons for that are set out below in
(a) to (f).
Q. So you had seen all these things. You had seen a high
level of bugs which created discrepancies in branch
accounts, had you, already by that stage?
A. We had certainly seen the KELs at this stage. I'm not
sure whether the PEAKs had been disclosed to us at this
point in time.
Q. No, they hadn't. So you had seen the KELs, and from the
KELs alone you felt able to judge, did you, that there
was a high level of bugs that created discrepancies in
branch accounts?
A. Well, the KELs have a reference within them to the
PEAKs.
Q. Which you hadn't seen at that stage?
A. No, but we had seen that there's references to PEAKs.
Q. What's the significance of that in the context of my
question?
A. Because it is the KEL that would indicate how many,
often, of that defect that led to the KEL being created
had surfaced.
Q. So you accept that KELs are actually good -- a KEL will
be good evidence of whether there is a bug that creates
a branch impact. That's your view, is it?
A. It is an indicator but you can't use that document
alone. You have to look at the PEAKs as well.
Q. But Mr Coyne, it seems to me that in paragraph (a) you
are using that document alone. You hadn't seen any
PEAKs at that stage, had you?
A. No, we hadn't seen the PEAKs, but we knew that PEAKs
must have existed and that there are references to them.
Q. So what? Why does the fact that you know there are
PEAKs you haven't seen enable you to make a judgment
that you would not be able to make simply by looking at
the KEL?
A. The KEL talks about how the situation that occurs with
the bug, error or defect should be dealt with, and by
reading that you get an indication that there is or
there must be or there likely is a defect within the
system. And if the KEL refers to, for example, ten
different PEAKs then that is an indication, although you
can't be certain, that there has been ten occurrences of
that.
Q. So what you are saying is that if you see a KEL which
refers to a bug, you will be able to tell from that KEL
in the main whether the bug has an impact on branch
accounts or not?
A. There will be sometimes an indication whether there is
an impact on branch accounts. We typically will not
know what branch it is.
Q. You say "sometimes"; what I'm suggesting to you,
Mr Coyne, and I think you know what I'm suggesting to
you, is KELs are actually a good source of seeing
whether there is a bug which has branch impact. You may
not know the details of branch impact but it is a good
way of telling -- if there is a bug that is considered
in a KEL that has a branch impact it is likely to say
that, isn't it?
A. It will often say there might be a discrepancy, yes.
The way that these documents are created is somewhat
inconsistent. You can read a KEL and it will give you
all the information that you need. It will relate to
a branch, it will point out the discrepancy, and that
KEL on its own might be helpful. You will read another
KEL and there's very little information within it. You
will need to go to the five PEAKs that relate to it to
even start to understand what the impact was. So I'm
not saying that KELs aren't helpful documents. I'm
saying that they need to be read in context.
Q. Let's move on from that subject.
MR GREEN: My Lord, I didn't want to interrupt.
MR JUSTICE FRASER: Are you going to the joint statement?
MR GREEN: Yes. They have agreed on page 26 {D1/2/26} at
0.3 what the shortcomings of those documents are, so it
is surprising to hear a different --
MR JUSTICE FRASER: I'm aware of that, but if
Mr de Garr Robinson wants to spend his time exploring
it, I'm not going to stop him.
MR DE GARR ROBINSON: Mr Coyne, at this stage you had only
seen the KELs and you were in a position to make
a judgment that there were a high level of bugs that
created discrepancies, were you? That was your
considered view then just on the basis of seeing KELs?
A. As I pointed out, whilst I had only seen KELs I had seen
references to a number of PEAKs as well.
Q. Then after 4 September PEAKs were disclosed?
A. Some PEAKs were disclosed.
Q. A large number of PEAKs were disclosed?
A. Yes.
Q. Something like 200 and something thousand, is that
right? I'm not sure there were any other material
documents disclosed between 4 September and 16th October
that are relevant to the this question. Can you think
of any?
A. I'm not sure. I did try and map out all the various
disclosures that have taken place but that doesn't
appear --
Q. The ones that I'm aware of are the PEAKs.
A. Right.
Q. Clearly you had a change of mind between 4 September and
16th October. Would I be right in thinking that on
seeing the PEAKs, that caused you to change your mind?
A. Certainly the PEAKs do help to set the context about how
problems are dealt with. So they would help, yes.
Q. You have plainly had seen something good between
4 September and 16th October and what I'm trying to
explore with you, Mr Coyne, is what that good thing was.
What had you seen that made you change your mind?
A. Certainly the PEAKs would be part of that,
an understanding better of the processes that are in
place.
Q. So would you accept you were rather hasty in expressing
the judgment you express here on page 9? {D1/1/9}
A. I'm certainly content with my opinion as expressed in my
report. Yes, it would probably be better for me not to
have expressed that particular opinion there so early.
Q. Do you think it would have been helpful in your report
to have explained why you changed your mind? It was
a very surprising volte-face. Do you think it might
have been helpful to allow everyone to understand what
it was that was good about the system that allowed you
to form the view that Horizon was robust?
A. "Relatively robust" was the term.
Q. Yes, relatively robust.
A. Yes, I mean --
Q. The reason why I ask is you see in joint statement 1 you
are resolutely negative about Horizon. In your report
on 16th October you say Horizon is relatively robust but
the rest of your report says negative things about
Horizon. There's very little that's said that is good
about Horizon.
It is difficult to resist the impression, Mr Coyne,
that in your first report you were trying to say bad
things about Horizon and you were not interested in
saying anything that was good.
A. I reject that.
Q. Would that be fair?
A. No.
Q. What is it that was good about Horizon that caused you
to change your mind and where do you describe that,
where do you consider that in your first report?
A. What was helpful was to understand the support process
in more detail to understand how things such as fault
determination is done, albeit it is only
an understanding of how it is done within Fujitsu, to
understand that process more. So that was an improving
position for me.
Q. I see. And your judgment on reviewing the PEAKs was
that Fujitsu actually had quite a good support process,
is that right?
A. Yes, I mean the support process that Fujitsu operate,
and again this changes over time so it is very difficult
to pick a point in time and understand what obligations
they had, what roles and responsibilities they
fulfilled, but certainly by the time they become aware
that somebody believed there was a problem, so it hit
SSC, the support centre, third line support, the process
they had of determining whether there was a fault
appears to be a reasonably good process. So there's
obvious weaknesses, how does the fault get to them in
the first place? And that's --
Q. When you say obviously weaknesses, why is it obvious
that there were such weaknesses?
A. No, no, what I'm saying is the obvious weakness in the
opinion is we don't know what happens before it gets to
Fujitsu, but then once it comes into Fujitsu and they
determine that, yes, there has been a problem, or there
has not been a problem, and they believe there is
a discrepancy, you will see quite often that there is
a line at the bottom of the PEAK saying, hand this back
to Post Office for them to do whatever they need to do
to branch accounts. So the process is then handed over.
So we don't know happens afterwards and we don't
know what happens before but the process with Fujitsu,
certainly SSC, appears to be a reasonably good process.
Q. So what you found when you read the PEAKs was that when
a call got referred to the SSC, either because it is
a subpostmaster call, or perhaps it might be automatic,
it might be from the MSU because of a reconciliation
issue, your view was that Fujitsu was quite good at
spotting if there was a problem in Horizon, is that
right?
A. Yes.
Q. And it was quite good at making sure that problem was
fixed, yes?
A. Yes.
Q. And it was quite good at identifying the consequences of
a problem and informing Post Office as to what those
consequences were, yes?
A. I don't know whether they informed Post Office. In
their own records you would often see comments such as
corrections will need to be made. There is a number of
references to say we do not know what Post Office's
process will be to deal with this but it does need
dealing with.
Q. But it is fair to infer, isn't it, and I suggest that
you have inferred from all the PEAKs you have seen, that
where there is a bug that has been identified that
appears to have had an impact on branch accounts,
Fujitsu are quite good at identifying the branches that
had been affected by that bug?
A. They are quite good. They will often take quite a long
time in identifying which branches are affected. Such
as Dalmellington, Fujitsu were not asked to get involved
in that for a number of years after it had been
impacting a branch account. When they did get involved
they quite quickly established all the historic impact
of that.
Q. Just to go back to my question. Fujitsu are quite good
at identifying the branches that have been affected by
those kind of bugs?
A. In the main, yes.
Q. Thank you. And it is fair to infer that Fujitsu then
tells Post Office what those effects are?
A. Well, that's the bit that I don't know because I don't
know what the process is. I do not think we have had
sight. There was some very, very -- there was some
disclosure made only very recently which are the BIMS
documents, and I think the BIMS documents are in part
a communication between Fujitsu and Post Office, setting
out what the likely impact might be. There's also OCR
and OCP documents.
Q. That's not relevant to this, though, is it?
A. It is a communication between Fujitsu and Post Office --
Q. I see.
A. -- saying that changes were going to be made. So it may
well be relevant to this.
Q. But it is right, isn't it, that Fujitsu, from what you
have seen of the PEAKs, will work out the branches which
have been affected by any bug that they identify, yes?
A. That is what they set about to do, yes.
Q. And you don't assume -- you don't infer, do you, that
having identified those branches Fujitsu then keep that
information to themselves? You infer that that
information is passed onto Post Office don't you?
A. I think so. There is certainly one reference in a PEAK
where it says I suggest we don't tell the branch about
this, but I'm not sure whether that's we won't tell
Post Office about it, it is more keeping it from the
branch rather than Post Office --
Q. And that's the only PEAK of any kind that you have ever
found of that sort, isn't it?
A. Yes, I believe so.
Q. So we have one PEAK over 20 years that says something
like that. In those --
A. Sorry, we should be careful. I haven't had eyes on all
of the 200,000 PEAKs.
Q. Yes. So all of those things that we have been talking
about you consider, and you considered by the time of
your 16th October report, were quite good?
A. Yes.
Q. What I don't understand, Mr Coyne, is why you didn't say
any of that in your report. Would it not have been
a balanced thing to do?
A. Perhaps. Looking back at the report, possibly.
Q. Would it not have been helpful to explain the good
aspects that you had spotted in the system as well as
the bad ones?
A. Possibly.
Q. Did you have a reason for wanting to keep it back?
A. No, not at all.
Q. The impression I get, Mr Coyne, I'm sorry to put this to
you, is that your first report was designed to give
a very poor impression about Horizon --
A. No.
Q. -- and that if you had included what you've just
described to me it would have given a much less poor
impression of Horizon.
A. I mean, as I say in my report, we'd been asked to
identify specific bugs, errors and defects and the
nature of that type of assignment is to drill in and
find occurrences of that. Talking about all the good
things that could and should happen, either generically
in the industry or that Post Office might do this year
as a result of various improvements in place, doesn't
really in my opinion provide much assistance.
Q. But isn't it a necessary part of the judgment you make
as to whether Horizon is robust? How can you make
a judgment about that without taking it into account?
A. I think you can take it into account, but to spend pages
of text talking about all the various good things,
I don't see there's any value in that.
Q. Well, Mr Coyne, can I ask you -- put it this way, what's
the value of doing this: saying in your joint statement
on 4 September that Horizon really isn't robust because
it has a high level of all sorts of bad things, six
weeks later producing a report by which time your view
has changed diametrically, and not mentioning what it
was that you had seen in the intervening time that
caused you to change your mind? Wouldn't it be obvious
that you needed to explain that to the court to help it
make its own determination?
A. Firstly, I don't believe that my opinion has changed
diametrically. It is a spectrum of robustness, as
I tried to explain this morning, and I believed that
Horizon has gone further up on the spectrum. There's no
complete change of direction here.
Q. I see. Just looking at page {D1/1/9} you say:
"... Horizon is not determined to be robust in this
regard because:
"(a) it contained high levels of bugs ... which
created discrepancies in the branch accounts ..."
At that stage -- we are going back to 4 September --
how many bugs had you identified which created
discrepancies in branch accounts?
A. More than the three that had been originally brought to
my attention.
Q. But three is not high level and nor is four. You say
"high levels of bugs". What did you mean by the
expression "high levels of bugs"?
A. Certainly in the tens.
Q. In the tens?
A. Yes.
Q. So we are talking about a system which has operated over
20 years, yes?
A. Mm.
Q. Which undertakes something like 47 million transactions
a week. So we are talking about 49 billion transactions
over a 20-year period, something like that. It is the
right ballpark, isn't it?
A. Mm.
Q. We are talking about two different versions of Horizon,
and you are saying that a number of bugs in the tens
over that period is properly to be characterised as
"high levels of bugs"?
A. Well, that is what had been identified at this point in
time, but you have got to remember my expectation was
set that I was investigating the three defects that had
occurred by people that had been involved in the
investigation with this system for a long time. So to
move from three to a larger number than that initially
suggested that there must be some quite poor processes
in place if they can't identify bugs, errors and defects
which, for myself, being involved in the process for
only a number of weeks, quickly identified them --
Q. I'm really sorry, I'm sorry to stop you, Mr Coyne, but
I really need to ask you, first of all, did anyone ever
tell you there were only ever three bugs that had ever
occurred in Horizon? Has anyone ever said that to you?
A. I think it was said -- I think in part of the pleadings,
part of the legal documents, I believe it was pointed
out that there was three defects which had impacted
branch accounts.
Q. Has anyone ever told you there were only three defects,
that those were the only? Defects because I suggest to
you that no one has ever said that to you, including in
the pleadings.
A. That was certainly my original expectation when
I started.
Q. I see. Then as a result of looking at these KELs you
realised there were more than three?
A. Yes.
Q. And I think you have suggested that at this time you
took the view that there were in the range of tens of
bugs, is that right?
A. Yes. There was certainly a large number of KELs that
warranted further investigation because they were
indicative of having an impact on branch accounts.
Q. So hold on. You had seen some KELs that might be
consistent with branch impacts but you were not sure
yet, is that right?
A. Yes. As I explained before, KELs relate to PEAKs, and
really in order to confirm the position you need to read
the PEAKs.
Q. So these were not KELs that you knew created branch
account discrepancies, so let's lay those to one side,
shall we. You are making a claim here that there is
a high level of bugs that created discrepancies in
branch accounts and I would like you, if you can
remember, to tell me, at that time on 4 September, what
was the level of bugs you had found that created
discrepancies in branch accounts?
A. That would be in the tens.
Q. In the tens?
A. Yes.
Q. I mean, Mr Coyne, looking at your bug table, and it may
be that my approach to it is all wrong, in the bug table
in JS2, 29 bugs are identified, correct?
A. Yes.
Q. And of those 29 a very considerable number were only
identified either in Dr Worden's report or in your
second report, and the number that remain in JS2 from
your first report is a relatively small number, far less
than ten?
A. Right, okay. I haven't done the maths so I don't know
if that calculation is actually correct, but it is
certainly the case that with the additional context with
the provision of the PEAKs that a number of KELs that
I initially thought led to branch accounts then appeared
not to, and there was additional PEAKs that indicated
branch account impact so we brought them back in again.
And that's the importance of reviewing the documents
side by side.
Q. So are you saying -- and this is the last question I
will ask before we break for lunch -- that at the time
you made this report you thought there were more bugs,
branch account affecting bugs in Horizon than you
thought at the time you made your first joint statement,
or less?
A. No, I think the number is probably around the same.
I think I ended up with 28 or 29 ...
Q. That's your second joint statement. I'm asking you
about your first joint statement. Between 4 September
when you produced joint statement 1 and your first
report in the middle of October, had your view as to how
many branch affecting bugs there were gone up or gone
down?
A. No, it had gone down.
Q. Because of the PEAKs that you saw?
A. Yes.
MR DE GARR ROBINSON: Thank you. My Lord, is this
a convenient moment?
MR JUSTICE FRASER: Yes, we will come back at 2.05 pm.
Remember what I said, Mr Coyne.
(1.04 pm)
(The short adjournment)
(2.05 pm)
MR DE GARR ROBINSON: Mr Coyne, we have been looking at your
first statement of 4 September and I would like now to
move on properly to 16th October. This is the time of
your first report.
Can we look at {D2/1/1} which is the front page of
your report. You say by this time you had examined
a number of KELs and PEAKs which had recently been
disclosed?
A. Yes.
Q. Just looking at page 1 of your report, I see that you
were assisted by four people?
A. Yes.
Q. Could you explain who those four people are?
A. They are all members of my team that helped me with
various projects, both contentious and non-contentious
projects.
Q. They work for IT Group, do they, your company?
A. Yes.
Q. What are their qualifications, their specialities within
IT Group?
A. So Siobhan Forster specialises in databases. She
graduated after doing a forensics -- a technology
forensics degree from Preston and has worked with me for
a number of years. Chris Raske is more to do with the
roles and responsibilities side of my business, so he
understands what various parties do do or should do.
Jamie Smith is one of the researchers that we often use
on projects, and that's the same for Patrick Grant.
Patrick Grant has worked as an implementer of a number
of different computer systems, a programmer, and he has
worked for us for a couple of years now.
Q. So in relation to the preparation of your reports, what
roles did they play? What did they do for you?
A. They were operating typically as a researcher. So
because of the volume of documents, I was asking them to
identify certain themes. They would review -- I would
parcel up a number of the documents that I would search
for in the disclosure platform and they would try and go
through and identify whether there was any potential
relevance in those documents so that I could have a look
at those and decide whether they should be included in
the report or not.
Q. Were these people working full-time leading up to the
preparation of the first report?
A. No. The majority of these people were brought in
towards the end when we were receiving the PEAKs,
towards the end.
Q. So they were engaged in searching the PEAKs and
presumably searching other documents as well, were they?
A. Yes, all of the documents were on a disclosure review
platform, very similar to the one that we see here. We
all sit in the same office, so we would meet each
morning and I would ask for various categories of
documents to be searched for and found and gathered
together.
Q. Would I be right in thinking -- you say they weren't all
working full-time on the case for you, were some of them
spending more of their time on the case than others?
A. Yes, certainly Siobhan Forster worked on the case more
often than not.
Q. Would it be fair to say she spent most of her time on
the case while you were --
A. No, she certainly had other matters that she was dealing
with, but she worked on it more often than not.
Q. What instructions did you give them? What did you tell
them to look for?
A. Typically I'd conduct the first tranche of searching
through the material and we would identify materials to
do with particular types of discrepancy that were being
discussed within the documents, and then I would sit
down with one of these chaps, we would go through the
PEAK or the KEL or whatever the document might be, and
I would ask them to try and assemble all of the other
documents that would be in that family. So it might be
other PEAKs that report the same thing, it might be the
KEL, we had a few of the BIMS documents, so to identify
them, to put together all the package of information.
That in itself was quite a task.
Q. Would it be fair to say that you were generally asking
them to find problems, to find evidence of problems in
Horizon, things that had gone wrong?
A. Yes. Well, sorry, I had already identified the theme
that I wanted them to look for but that was typically
around a specific type of defect that might be occurring
around a particular time and I was asking them to look
at documents around that time or that contained
a similar theme.
Q. So if we go back to the joint statement, having said we
could move on. If we can look at {D1/1/2}. We are back
on 4 September again. At this time were those four
working with you? Did you already have their
assistance?
A. No, I think it was just myself and Siobhan at this time.
Q. So the other three came in later?
A. They would have always been in my office.
Q. But for the purposes of --
A. Yes.
Q. If we look at the agreement. I'm sorry, let's move on
to page {D1/1/3}. It says:
"Each expert's approach to writing his report, and
to this joint memorandum ... could broadly be one of
three possible approaches:
"a) To focus mainly on negative points found in the
disclosed documents about where Horizon may have fallen
short.
"b) To focus mainly on those aspects of Horizon
which were intended to achieve robustness and
reliability, and the evidence implying that they
succeeded."
Or:
"c) To provide the court with a clear foundation for
understanding the design and operation of Horizon; then,
building on that foundation, to provide a balanced
assessment of the ways in which Horizon succeeded ..."
A. Mm.
Q. Now, that's what you agreed with Dr Worden. If I could
just then go down to what you describe as areas of
disagreement. Second sentence:
"The issues are about how Horizon and its
interactive components operated and the processes
employed by Post Office and Fujitsu in supporting these
systems and the data within.
"Whilst my report will take a balanced approach, it
is the case that many of the issues require a deep focus
on the occurrences of bugs ... as well as the potential
for modification of transactional data. Whilst context
will be provided as to how Horizon should work in
typical circumstances, it is the non-typical operation
where focus will be placed."
That is right, isn't it, that you focused on things
going wrong, that's what you were looking for?
A. Yes, I looked for things that have actually gone wrong
and then followed the process through to find out how
they were dealt with.
Q. When you engaged in that process, you have already
described the team of people that you had assisting you.
Did anyone else help you? Did anyone else put forward
documents for your consideration that might be included
in the report?
A. No.
Q. Did you speak to any individual claimants, for example,
and get any information from them?
A. No.
Q. Going back to you and your team, you obviously by
16th October couldn't review all the PEAKs you had been
given, you had been given over 200,000, so you used
intelligent search techniques, didn't you?
A. Yes.
Q. Presumably you used those techniques for the whole
corpus of documents, not just PEAKs, but the KELs and
other documents as well, is that right?
A. That is correct.
Q. What other documents did you search across, can you
remember, at that time?
A. I don't have a list of all the documents. There is
a list in my first report, but all the documents all
went into the same platform, so it will have been every
document that had been formally disclosed at that point
in time.
Q. Thank you. Your report says you are of IT Group, when
you describe yourself at the beginning.
A. Yes.
Q. Can I ask what your position there is?
A. Director.
Q. As I understand it, correct me if I'm wrong, IT Group
offers a disclosure and search capability, doesn't it?
A. It does indeed.
Q. I read from your website, it is not in the bundles:
"IT Group's powerful e-disclosure and digital
investigation solution, Intella Connect, removes that
headache by enabling you to intelligently search, filter
and review large volumes of electronic data with speed
and ease."
Is that a fair description of the service that
IT Group provides?
A. It is indeed.
MR JUSTICE FRASER: Forgiving the split infinitive.
MR DE GARR ROBINSON: I'm sorry, my Lord?
MR JUSTICE FRASER: I said forgiving the split infinitive,
"to intelligently".
MR DE GARR ROBINSON: I do not think it is possible to
forgive. Others may disagree.
So is it right that you manage the e-discovery team
in IT Group?
A. Yes. I am the director that is responsible for that --
Q. So it is fair to say, isn't it, that you know quite
a lot about intelligent searching of documents?
A. Yes.
Q. And you have no difficulty, I'm not saying it is easy,
in searching through, for example, 200,000 PEAKs in lots
of very clever ways that I am sure I couldn't think of?
A. Yes.
Q. Would that also be true of your team of assistants?
A. Yes.
Q. And they would have used these intelligent search
techniques to go away and find the sort of documents
that you wanted them to find?
A. They would.
Q. At your daily meetings?
A. Yes.
Q. I imagine that your search facility is far more powerful
than searching for words and symbols within a certain
distance from each other? That's the kind of thing I'm
used to.
A. We didn't employ any artificial intelligence, we only
used the normal standard search techniques.
Q. Would I be right in thinking that with your experience,
you and your team have developed all sorts of tricks of
the trade with searches to find -- to unearth things
that the rest of us laymen might find more difficult to
find?
A. I'm not sure which tricks of the trade you are referring
to. We do see ourselves as being quite advanced in what
we can find within documents, yes.
Q. Could you give me an idea of some of the intelligent
searches that you would have used in a case of this
sort?
A. So typically we were looking at documents that were
talking about discrepancies, that would be talking about
errors, bugs, defects, imbalances, things like that.
Q. You and your team would search for these documents and
then they would be reviewed, would they?
A. Yes, we would typically try to understand -- because
often you will find one document but then you need to
understand where that document sits on the timeline, so
then you may have to go back a number of documents or
forward a number of documents -- we have the ability to
put them effectively in chronological order -- to try
and see who within the organisation was talking about
these themes.
Q. Presumably you would review the documents that you
found, but members of your team, when they found
documents that they thought you were looking for, they
would review them and then they would bring them to you
for your review as well if they thought they were good
ones?
A. Yes. It would be a process of them applying a tag to
a document. So essentially they were ruling out
irrelevant documents.
Q. Would they do their own reviewing and then provide you
with documents that they thought were suitable or did
you review everything that they tagged?
A. There was documents that, as the process went on, they
become better versed with spotting things that I may
find of interest within the document, but initially
I would ask them just to rule out documents that were
clearly irrelevant and I would review the balance of
that, but as things improved they were spotting themes
within documents, tagging it for my attention, and then
I would see the tag and would be able to go through
that, and then each day we would refine that process.
Q. Would it be fair to say that the total number of
documents that you and your team reviewed is larger than
the total number of documents that you reviewed?
A. Yes.
Q. I would be right in thinking, wouldn't I, that the
documents that you refer to in your report are all
documents that you yourself had reviewed?
A. Yes.
Q. Could you give me an indication -- the number of
documents the whole team reviewed was higher than the
ones you reviewed. Could you give me an idea of how
many documents your entire team would have reviewed in
this process?
A. I wouldn't have that number, I'm sorry.
Q. If we then go in the same document to {D2/1/83}, please?
MR JUSTICE FRASER: Do you want to stay in the first joint
statement?
MR DE GARR ROBINSON: I'm so sorry, I thought I was in JC1.
It is {D2/1/83}.
Just looking at paragraph 5.114, you say:
"Regarding the extent of potential errors within
Horizon I have analysed 5114 Horizon ... (KELs) to
determine the scope of potential bugs or 'PEAKs' ... Of
these 5114, I have found that 163 contain PEAKs that
could be of significant interest and of these [67] are
referred to in the report."
A. 76.
Q. I'm so sorry, 76. So you personally looked at 5,114
KELs, did you?
A. KELs and PEAKs, is that not? Sorry, yes. Yes.
Q. So you looked at them, so your team would have looked at
more?
A. They may have looked at the same number but they
possibly looked at more, yes.
Q. On top of those KELs you looked at PEAKs, as well?
A. Mm.
Q. If we go back to paragraph 1.28 at page {D2/1/20}. It
may be you will agree with this without having to look
at it. At this stage on 16th October -- it is 1.28 on
page 20 and you say:
"Of the [218,000-odd] PEAKs disclosed by POL I have
in the interest of expediency, used intelligent search
techniques to initially review those that might
specifically relate to branch accounts. I have
therefore reviewed 1,262 in the limited timeframe
allowed since disclosure."
Would I be right in thinking that your team would
have reviewed a considerable number more than that?
A. Yes.
Q. Just to be clear, the focus of your search was what?
Was it branch account problems?
A. Yes.
Q. Were there any other problems?
A. The searches were surrounding discrepancies but, of
course, a search such as that will come up with lots of
things that are largely irrelevant. But yes, branch
account problems.
Q. And would that 1,262 PEAKs you referred to there be on
top of the 163 PEAKs that you found having regard to the
5,114 KELs you read?
A. Sorry, could you ask that question once more.
Q. If you remember back at paragraph 114, I'm asking this
question in a very maladroit way, you said: I looked at
5,000-odd KELs and I found that 163 contained PEAKs that
could be of significant interest?
A. Yes.
Q. I get the impression therefore that you looked at the
KELs and then, having regard to the KELs, you then
looked at the PEAKs referred to in those KELs?
A. Yes.
Q. And you found 163 that you thought were interesting and
you refer to 76?
A. Yes.
Q. The 163, was that part of the 1,262 that you regarded --
that you found that might specifically relate to branch
accounts?
A. It is likely to be a subset of that.
Q. Thank you.
A. Because what you find is when you look in a KEL and it
lists a number of PEAKs, you go to have a look at the
PEAKs. Sometimes it is a PEAK of interest but it
actually doesn't contain any of the words you were
originally searching for, but once you have read it you
understand why it is of interest.
Q. Yes. If we move forward three and a half months, so we
are now at the beginning of February, and you produce
your second statement which is very long and that's at
{D2/4.1/16}. If we could look at that document at
page 16, please. You say in paragraph 3.20:
"I have now reviewed more of these records using
text search criteria and filtering. This has enabled me
to address some issues more thoroughly and has enabled a
more in-depth analysis in relation to the extent of the
Horizon Issues and the overall robustness ..."
So would I be right in thinking that you had looked
at more KELs?
A. Yes.
Q. And I think it must be obvious you looked at more PEAKs,
as well?
A. Yes.
Q. Had you looked at more other documents as well?
A. Yes.
Q. Could you give some indication of the sort of other
documents you were looking at in the intervening time?
A. There was a number of OCRs and OCPs but I do not think
we had the full family of them at this point in time.
There was a number of the design documents, the process
documents. There was a lot of reports about how
feedback of the support of issues was being fed back to
Post Office. There was documents on many themes.
Q. How many more KELs -- it may be that you can only
estimate, it may be impossible to estimate -- do you
think you had looked at by this time?
A. It would be a crazy guess. I don't know. I honestly
don't know.
Q. What about PEAKs? How many more PEAKs do you think you
had reviewed by this time? I would suggest it is likely
you looked at a lot of them but that may be unfair.
A. It will be a lot of them. There wasn't a linear review
where I sat and started and went next, next, next. That
wasn't how it happened. But after searching for
a document, deciding it was of interest, and then
picking out some of the themes and then searching for
those themes, you might identify conceivably another
five hundred documents of which you might flick through
those very, very quickly just to see whether they are of
any interest, they are in the right date range.
Q. Did your team, the four people that's mentioned that we
have discussed, did they help you at any time during
this three and a half month period?
A. Any members of the team? Sorry?
Q. Did the whole team, all four members, help you at any
time?
A. Running up to Coyne 1, the first report, then it was the
team members that you see there. For the second report
it was just Siobhan Forster.
Q. Did she spend a great deal of her time helping you
during that period?
A. Yes.
Q. She was doing intelligent searches as well?
A. Yes.
Q. And she was reviewing PEAKs and other documents as well?
A. In order to bring them to my attention. The process was
very similar all the way through really.
Q. I see. Would I right in thinking that the focus during
that period was again PEAKs and KELs and other documents
indicating problems with branch accounts?
A. We were also looking at elements in Dr Worden's report.
So there will have been searches trying to understand
more about themes surrounding his opinion.
Q. I see. Have you reviewed further documents since you
produced your second report on 1st February?
A. Yes, I think I'm right to say there has been documents
that have been disclosed since then. Yes.
Q. Have you -- let's take it in stages, have you reviewed
more PEAKs since then?
A. I think there was a small disclosure of about 1,200
PEAKs that have been reviewed.
Q. What about OCPs and OCRs, have you reviewed a number of
those?
A. Yes.
Q. Could you give an estimate of how many you have looked
at?
A. It will be thousands but I don't know what the precise
number would be.
Q. MSCs as well, the three databases of MSCs, you've
reviewed a number of those as well, haven't you?
A. MSCs were a very significant challenge because they were
disclosed in three separate Excel sheets, none of which
appeared to marry up with each other, so it was a task
because there were literally millions of lines of code
in that. So I have searched within those documents.
Q. The OCPs and the OCRs and the MSCs, what have you
searched for? FAD codes, that kind of thing?
A. I have searched for the word "FAD", because that might
indicate that it is describing something for
a particular branch, but I have not searched for a FAD
code as such.
Q. So you have looked for FAD because that is a reference
to a branch. Each branch has its own FAD code, doesn't
it?
A. FAD is an indicator that the document might be to do
with a branch, but you do have to be very careful with
that because simply searching for "FAD" returns hundreds
of documents and you actually find that it is just
a pro forma document with the word FAD in it and there
is a space next to it and there's nothing in it.
Q. Presumably this is where intelligent searches come in,
which I would be --
A. Absolutely.
Q. -- incapable of doing.
A. Absolutely.
Q. And what other things were you looking for when you went
through OCPs and OCRs and MSCs?
A. We used various searches for "subpostmaster" that would
indicate that there was potentially dialogue with the
subpostmaster, to see whether there was any advice given
to the subpostmaster or what type of information was
being requested from the subpostmaster. So all the
variations of subpostmaster were used as search terms.
Q. Thank you, Mr Coyne. We now have a sense of what you
looked at when you produced your first report and what
additional documents you looked at when you produced
your second report. We've already spent some time
talking about your first report but there are some extra
questions I would like to ask you about it.
If we could go to {D2/1/26} paragraph 3.7 which is
at page 26. You say:
"With regards to Issue 3, whilst the present-day
version of Horizon supported by manual human support may
now be considered as relatively robust in the spectrum
of computer systems used in businesses today it has
undergone major modifications in its history. It is
likely that in 1999 when it was first commissioned, and
in 2010 when it was significantly upgraded (to Horizon
Online), it was less robust."
So you are saying there that the level of robustness
may have varied over time?
A. Yes.
Q. And you are focusing in particular on the early days of
Legacy Horizon and the early days of Horizon Online?
A. Yes.
Q. Why do you focus on those periods?
A. Because I believe it is an incontrovertible fact that
there was a larger number of problems after the launch
of the first version of Horizon and after the upgrade to
Horizon Online. Certainly I was in court when shortly
after the launch of Horizon Online there was talk about
the red alert or the red care situation, shortly after
Horizon Online, where it needed some intensive care
because of problems.
Q. Do you mean during the pilot project when only a very
small number of branches were actually using
Horizon Online? Is that what you are referring to?
A. Certainly there was problems during that period but it
was real branches that were being used. It wasn't
a testing environment or anything like that.
Q. Would I be right in thinking, Mr Coyne, that you have
chosen the wording of that paragraph quite carefully?
A. I would like to think that I choose all of my words
carefully.
Q. Very good. That's fair. I note that you are not saying
here that Horizon was definitely not robust at some
earlier time. That's deliberate, isn't it?
A. Yes.
Q. You are merely saying that it is relatively robust now,
it may have been less robust at some earlier dates, but
you are not saying it wasn't relatively robust on those
earlier dates, are you?
A. No, I'm not. That supports what I have attempted to
assist the court with all along. It isn't a binary
situation that it is either robust or not robust, it is
a spectrum, and throughout those periods it was less
robust further to the left of the spectrum.
Q. Even during those early periods in 1999/2000 when
Legacy Horizon was rolled out, and in 2010 when
Horizon Online was rolled out, would I be right in
thinking it is not your judgment that it was not
relatively robust even then?
A. I think that is quite a difficult question to answer
because that would require very specific focus on those
periods and a very specific benchmarking of the position
at those two periods which I haven't conducted
specifically.
Q. You say that you have conducted the benchmarking now?
A. Yes.
Q. Are you suggesting that you haven't conducted any
benchmarking in relation to any earlier period?
A. Certainly I have looked at the number of defects that
have been in existence and the number of requests to
support throughout the whole life of Horizon and there
are peaks -- sorry, that is a bad word.
Q. Spikes.
A. Spikes after the launch of the Legacy version of Horizon
and then after the upgrade to Horizon, and I completely
understand why with any new system and with any major
upgrade there's going to be an increase in problems.
That's why I have said during those periods the system
will have been less robust because there was problems
being ironed out both in the technology -- the
programming -- the networking and the support processes.
All these problems were being ironed out.
Q. So outside those periods, how long did those periods
last? Are we talking about 1999 to 2000 for
Legacy Horizon, is that the sort of period you are
talking about?
A. No, I mean the actual period extends, it is a curve that
slowly drops over the course of the next couple of
years. So, again, it is not a binary: it becomes stable
at some point in time, the faults stop at some point in
time.
Q. So you are suggesting -- you have a graph in mind which
you are using as a basis for forming this judgment. Is
there a graph somewhere that indicates what you are
talking about?
A. There's certainly two images, I think one in my report
and one in Dr Worden's report, that show -- that have
correlation of spikes of activity around those two
areas.
Q. Could you tell me where in your report we will find this
graph? I'm afraid I'm not in a position to suggest
a page number. Do you remember where it is?
A. There is one in my first report at -- it is page 193.
Q. Thank you. A graph. Is this your first report or your
second report?
A. This is my first report.
Q. My page 193 is a figure 3.
MR GREEN: It is bundle page 206, internal 193.
MR DE GARR ROBINSON: I'm looking at the wrong numbering.
MR JUSTICE FRASER: Are we looking at your page 193 in the
top right-hand corner?
A. Sorry, my Lord, yes.
MR GREEN: Page 193 top right.
MR DE GARR ROBINSON: 193 at the top and 206 in the bundle
{D2/1/206}.
I'm looking at that graph. What is this a graph of?
A. So this is a graph of BIMS.
Q. Could you explain what they are?
A. They are business impact statements that are made
between Fujitsu and Post Office.
Q. It looks as if the spike had more or less finished by
about 2011. So would it be fair to say that if you are
treating this as evidence of less robustness, from about
2011 it stabilises, would that be right?
A. I think it probably could be argued that it stabilises
around 2013 but it is the curve that I attempted to
describe before.
Q. So that's the basis upon which you form that judgment
that it was less robust during these early periods, is
that right?
A. That is one of the graphs that illustrates my opinion,
yes.
Q. Now, if we go back to -- and is there a similar -- have
you done a similar graph for Legacy Horizon or is there
nothing?
A. I believe Dr Worden has a similar graph that goes back
to Horizon.
MR JUSTICE FRASER: You said there were two, I think, one in
your report and one in Dr Worden's report.
A. Yes.
MR JUSTICE FRASER: You think he has got one as well?
MR DE GARR ROBINSON: This may be wrong but it may be at
{D3/1/187}.
MR JUSTICE FRASER: Let's have a look.
MR DE GARR ROBINSON: There is a table of loss amounts by
year. Do you see that? Is that the one you have in
mind?
A. No, I do not think it is. I think there is a different
one.
MR JUSTICE FRASER: Is it {D3/1/129}? I'm just doing this
to try and help Mr de Garr Robinson, I'm not
interfering. Have a look at D3/1/129.
MR DE GARR ROBINSON: Is that the bundle reference?
MR JUSTICE FRASER: That is the bundle reference. Could you
call that up on the common screen, please. 129.
MR DE GARR ROBINSON: It is the same graph as at 187.
MR JUSTICE FRASER: Oh, is it the same?
MR GREEN: I think it is the same.
MR JUSTICE FRASER: I won't try and help any more.
MR GREEN: It has a different title but it is the same
thing.
MR DE GARR ROBINSON: Do you think you can --
A. Yes, if you give me a moment I can go through these
paper copies.
Q. It might be the next page. If we go to the next page on
the screen, is that it? No sorry.
MR GREEN: {D3/1/130}.
MR DE GARR ROBINSON: Can we look at page 130, please, in
this document. {D3/1/130}. Thank you. Is that it?
A. I think it is that actually, yes, because that
illustrates the same peak/spike, that illustrates the
same spike around Horizon.
Q. But it doesn't indicate any spike around Legacy Horizon,
does it?
A. Not for KELs, no.
Q. I'm wondering, Mr Coyne, whether this judgment is really
based upon your experience, that your experience of when
you roll out a new system you are going to have teething
problems and then, once it is rolled out, the position
stabilises. Is that really what your judgment is based
on?
A. Part of my judgment is based on my experience but it is
certainly the case, and we might not have a graph for
this, but when you look at the number of PEAKs over the
period, there is a spike in the PEAKs at the start of
Horizon.
Q. So just to backtrack slightly. What you are saying is
Horizon is robust, relatively robust now. Do you accept
that it has been robust for a significant period or
would you not be prepared to say that?
A. It has been relatively robust apart from the early years
of the two launches, Horizon Legacy and Horizon Online.
Q. So both Horizon Online and Legacy Horizon have been
robust for most of their lives but there were initial
periods where they were less robust?
A. Yes.
Q. In relation to those initial periods -- and do correct
me if I'm wrong -- you are not saying they definitely
weren't robust during that period, you are simply saying
they were materially less robust, is that right?
A. Yes.
Q. Thank you. That's very clear.
Let's go back to section 3 of your first report. If
we can go to {D2/1/26}, paragraph 3.10. Perhaps I don't
need to ask you about that in the light of what you have
just told me. Instead what I would like to do is to go
to page {D2/1/77} of the same document, please. You
will recall that I suggested to you that there was
a danger of forming an impression from your first report
that although you clearly state that Horizon is
relatively robust, you don't say anything good about it.
What you tend to do is to say bad things about it.
I suggested to you that was because you were knowingly
giving a poor impression of Horizon. Would that be fair
or would that be unfair?
A. That would be unfair.
Q. Why didn't you then say more about the good aspects of
Horizon that led you to the conclusion that it was
relatively robust?
A. Because this -- because the question of robustness was
in the context of would that robustness prevent or
reduce the risk of bugs, errors and defects causing
discrepancies. So the many good things about Horizon,
and I suppose if you are going to start talking about
good things, how far do you go? I don't know where you
stop with them. This, as I understand it, is about
whether any of the failures that have occurred within
Horizon have led to defects or discrepancies.
Q. But in the first joint statement you indicated that you
were going to adopt a balanced approach. I fully
understand why you are looking at problems, and problems
which were relevant to branch account impact. I'm not
suggesting you shouldn't have done that. But if you are
engaging in a properly balanced approach you have to
explain the good things with as much care and clarity as
you explain the bad things. And what I would like to
suggest to you, Mr Coyne, is that you devoted much more
attention to talking about the bad things than you ever
did talking about the good things?
A. Yes, I will accept that is fair.
Q. And why did you do that?
A. Because my understanding of the instruction is to try
and identify the risks that are associated with bugs,
errors and defects, and there are many other issues, but
if they had an impact or could have had an impact on
branch account.
Q. But --
A. Sorry. So in order to do that you have to focus on what
actually happened, is my understanding, and then from
understanding what has actually happened understand what
the impact of it is.
Talking about all the nice things in Horizon that
happen in the high 90% of the time I don't believe
delivers any real value to my instructions.
Q. But in deciding whether or not a problem that arose in
Horizon actually had a branch impact, one of the things
that you have to consider is whether the countermeasures
that surround the Horizon system operate effectively or
not?
A. I absolutely agree with that --
Q. And you accept that, don't you? You accept that it is
relevant to the judgment of robustness as to how good
the countermeasures were?
A. Yes. But the way to do that is to identify the fault in
the first place and then have a look at the
countermeasures, whether they were positioned correctly,
whether they worked or not.
Q. Can I suggest to you, Mr Coyne, that that's not what you
did in your first report at all. Most of your first
report was devoted to problems that actually didn't
relate to branch impacts at all, and when you did talk
about problems that related to branch impacts you didn't
consider whether the full panoply of countermeasures
would have corrected or reacted to or identified the
branch impacts you were talking about?
A. That could well be to do with not having all of the
documents that was requested at the time.
Q. Well, you didn't do it in your second report either, did
you? You did the same thing. It was more of the same
in your second report, wasn't it?
A. I don't believe that's the case, no.
Q. So you are suggesting in your second report you did
consider how good the support process was and you did
consider, where you had identified a bug that may have
a branch impact, you did consider the extent to which it
was likely that that impact would have been identified
and sorted out?
A. I spend a large amount of the second report addressing
specifically the countermeasures that Dr Worden raises,
paragraph by paragraph.
Q. And your overall view of those countermeasures, bearing
in mind that your judgment now is that Horizon is
relatively robust, yes?
A. Mm.
Q. Your overall view of those countermeasures is that they
are relatively good, yes?
A. I think it is a danger looking at them in the overall
because there is a number of them that I don't believe
are good. But broadly, if the countermeasure is
positioned correctly, in the vast majority of occasions
they should work.
Q. What I would like to suggest to you, Mr Coyne, is that
it follows from your overall judgment that Horizon is
relatively robust now. It follows that overall your
assessment of the countermeasures that Dr Worden has
identified is also that they are relatively good. Would
that be fair?
A. Yes. Each of the countermeasures that Dr Worden has
suggested are in place and at some point in time have
been in place within Horizon. We don't know when they
were in place within Horizon on a number of occasions
but they have been in place.
Q. Please answer my question, and let me be clearer about
it because it is quite important. What I'm suggesting
to you is that it is part and parcel of your judgment,
your overall judgment that Horizon is relatively robust,
that the countermeasures operating together, the
countermeasures that are in operation in relation to
Horizon, that they are relatively good as well. It is
not your view that operating together, taken as a whole,
the countermeasures are of poor quality or that there's
anything wrong with them, is it?
A. No, it is not.
Q. I'm grateful for that frank answer. It is fair to say,
isn't it, that your overall judgment is that the
countermeasures together, there may be individual
instances where you are not so sure, but overall your
judgment is that those countermeasures work well in
Horizon?
A. Yes, but you can't disregard the fact that the system
has changed throughout the last --
Q. Just --
A. Sorry --
MR JUSTICE FRASER: Hold on a second, I would like to hear
the answer, and then you can either re-put the question
or follow on.
Go on, Mr Coyne.
A. We can't disregard the fact that the system has changed
over the course of the last 20 years and whatever the
countermeasures may be today, they will likely be
different in Horizon Legacy, different countermeasures,
possibly the same aspiration to catch defects at certain
point in time, but they will be different, they will be
constructed differently and they will be positioned
differently. So we can't --
MR JUSTICE FRASER: Mr Coyne, just pause there. All right.
Mr de Garr Robinson is going to re-put his question if
he wants to.
MR DE GARR ROBINSON: I am. I'm talking about the period of
time relating to the current configuration of Horizon
and going back in time until the point in time after the
initial teething problems were resolved. You understand
what I mean?
A. All the way back to --
MR JUSTICE FRASER: Can we use years?
MR DE GARR ROBINSON: Let's go back to 2012 for the sake of
argument to make it safe, because I think that is a year
you suggested. So from 2012 to 2019 your view is that
Horizon is and has been relatively robust, correct?
A. Yes, I would say that, but I would be very clear that
robust does not mean that it hasn't suffered
significant --
Q. Everybody is agreed that robustness doesn't mean
perfect.
A. That's fine then, yes.
Q. Which is the point you are trying to make, is that
right?
A. Yes.
MR JUSTICE FRASER: What word were you going to add after
"significant", just out of interest?
A. Defects that have continued throughout the period.
MR DE GARR ROBINSON: So during that period, 2012 to 2019,
your judgment is, and you feel capable of making the
judgment, that Horizon is and has been relatively
robust, correct?
A. Yes.
Q. Now that judgment includes a judgment not only about the
electronic systems and so on, it includes a judgment
about everything including the countermeasure processes
surrounding Horizon, correct?
A. Yes. There has to be a number of caveats about that and
I explained before that I don't really know what happens
within Post Office to correct defects if Fujitsu has
spotted them. So I can't comment really on what that
process would be.
Q. But in relation to the -- nonetheless you have
sufficient information to allow you to form an overall
judgment as to robustness and your judgment is that
during that period the system itself and the
countermeasures around it were relatively robust?
A. Yes.
Q. The reason why I ask you that -- and just to be clear,
the same is true of the period Legacy Horizon, shall we
say from 2001 to 2010, would that be a fair period?
A. Yes. Perhaps one year after that.
Q. Okay, 2002 to 2010. During that period your view is
that Horizon was relatively robust?
A. Yes.
Q. And that included the countermeasures surrounding it and
supporting it?
A. Yes.
Q. My question to you, Mr Coyne, and we will be coming to
countermeasures later, but it is quite a striking fact
that if you read the section of your report that
addresses countermeasures, your second report, it goes
over dozens and dozens and dozens of pages in your
section 5 -- and as I say, we will come to it -- there
is no hint of your judgment that overall the
countermeasures during those periods have operated --
have been relatively good. Why is that?
A. I don't know. It certainly wasn't a conscious decision
to leave anything out, and --
Q. Could I suggest to you -- I'm sorry, I'm interrupting
you.
A. Sorry. And because I have found bugs, errors and
defects throughout that period, that is an illustration
that however good the countermeasures were, that they
were overcome at various points.
Q. We will come to that, I will be asking you quite a few
questions about whether the countermeasures were
overcome by the bugs that you have identified. But my
question to you, Mr Coyne, remains that you are quite
happy to go on at great length about how much you
disagree with Dr Worden and about all the problems you
found in relation to countermeasures. Nowhere do you
actually say what you have very fairly said to me in the
last half hour, namely that the countermeasures actually
during those long periods of time have worked quite
well. Why would you not say that given that you want to
give a balanced view?
A. I believe the view that I have given in the reports is
balanced --
Q. Where -- I'm so sorry, I'm speaking too quickly. Please
carry on.
A. I don't see that utilising many pages of text to talk
about the various aspects of Horizon's countermeasures,
which may well be similar to how Dr Worden set them out,
they may well operate in much different ways, we don't
really know, but we can observe that they are likely to
operate in those ways. I do not see there is a great
deal of value when it is a fact, and what I believe the
court was asking in the Horizon Issues was to address if
it is possible that they have failed. So it is the
failure side of it.
Q. I see. So you regarded it to be your job to indicate to
the court whether there were occasions on which Horizon
had failed. That was the essential endeavour that you
were engaged in when addressing Horizon Issues 1, 3, 4
and 6, was it?
A. I believe that a number of those Horizon Issues ask that
specifically.
Q. Didn't those issues ask specifically how likely -- what
was the extent of the likelihood of problems occurring
in Horizon so as to cause problems in --
A. Yes --
Q. -- branch accounts?
A. Yes, that is right, but that doesn't ask for me to point
out a list of all the possible countermeasures.
Q. But isn't it relevant to the judgment as to overall
likelihood how well the countermeasures work? I mean
Dr Worden recognised that was the position, that's why
he spent so many pages of his report explaining the
architecture of Horizon, explaining how the systems
worked and explaining how they operated in relation to
each other. Then in your response, in your second
report, what you did -- what I suggest that you did is
you sat in your armchair and took pot shots at various
points that he made without giving any sense as to
an overall judgment as to whether he was right or not
that the countermeasures did work relatively well?
A. No, I don't believe I did that. What I did was I looked
at the entire landscape of Horizon's operation,
identified the percentages were -- there were weaknesses
within the process, and then focused on that
specifically to find out whether there had actually been
any bugs, errors and defects, and then looked at why the
countermeasures or controls may or may not have worked
after that bug, error or defect had triggered. Because
the key is how the system operates when it doesn't
operate properly, not how to operates when it does
operate properly.
Q. I have been side-tracked, I have allowed myself to be
side-tracked.
We were going to look at page {D2/1/77} of your
first report and I'm about to ask you a question about a
paragraph on this page. The starting point of the
question is that you agreed with me that your judgment
is that Horizon, both Horizon Online and indeed
Legacy Horizon, for much of its life has been relatively
robust?
A. Mm.
Q. If we go to paragraph 5.88, you say:
"In my position as an expert I am unable to estimate
the level of the Horizon system's robustness. Given the
size and age of Horizon, I would however make the expert
assumption (based upon systems of similar magnitude),
that there are not many people who could. The sheer
enormity of the task to garner a thorough understanding
of the code which would be required to estimate
robustness is, in my opinion, nearly impossible."
So here you seem to be saying that it is actually
not possible to assess whether Horizon is robust or not?
A. No, to come up with a number. When I say the level, I'm
talking about a particular number or a percentage or
a ... (Pause)
Q. So what you are suggesting is that you don't have -- in
order to make an absolute judgment, to give an absolute
number of robustness, you would need to look at all the
code and look at everything else, is that right?
A. Yes, because there may well be a defect in the code on
one day which has a problem that eludes countermeasures,
and that code may then be fixed so it is in a different
position on day one than it is on day two.
Q. And you are saying that kind of enquiry is necessary in
order to make an absolute judgment of robustness?
A. In order to make an absolute judgment, yes.
Q. But it is not required in order to make a relative
judgment of robustness?
A. Yes.
Q. I would like you to explain why that would be the case.
Why would it not also be needed for a relative judgment
to be made, to benchmark against industry standards?
A. Because with a relative judgment you are considering all
factors in the round and it is still very subjective but
it is across multiple different factors that are
improving or getting worse.
Q. I understand you when you suggest that to make
a judgment about robustness involves an overall
assessment. That I understand. But my question to you
is why is it necessary to look at the code and look at
absolutely everything to engage in an impossible task to
make an absolute assessment of robustness, when it is
not necessary to do that in order to make a relative
assessment of robustness? I would just like you to
explain that if you could.
A. There is a number of different ways of assessing
robustness. By looking at the code at any one point in
time you could establish whether there are any bugs,
errors or defects in there. It would be a near
impossible task but it could be done depending on how
big the code base could be.
Q. How would it be possible? Just as testing is never
perfect. You have a set of code and you are about to
release it into live operation but you have to test it
first, you can test it as many ways as you can but you
are never going to spot every problem in it.
A. No, but --
Q. Are you suggesting that for the purposes of determining
whether a code is robust you could look at the code and
perform that judgment?
A. Yes, you certainly could look at the code and perform
that judgment because what you can do, and this has
often happened in the aircraft industry, they have
multiple different operating systems that control
different devices. So if there is a catastrophic
failure of one piece of code another piece of code will
take over.
So if by looking at the code you can establish that
is in there, then you can give a view of that being
completely robust, because if it fails in its entirety
there is another solution that can take over. So that
is how I can illustrate the code example.
Q. This is the last question I will ask and then I will
move on. If you are forming a judgment as to the
robustness of a system, don't you need to have regard to
the way it operated in operation?
A. Yes.
Q. Isn't that what you should be looking at, not millions
of lines of machine code? You are looking at how it
operates in practice, looking at how the countermeasure
operates and forming an overall judgment as to the
efficiency of the operation, aren't you?
A. That could be one part of it, yes.
Q. Wouldn't that be sufficient? Wouldn't be all you needed
to do?
A. No, because you will almost certainly be able to
identify a scenario where somebody or many people will
operate the system, the Horizon system, throughout
a year, or possibly even throughout the entire life
cycle of Horizon, and will never experience one problem
at all because they will not trigger the unusual
circumstances that unearths a bug. So if you went and
studied that one person you would ultimately conclude
the system is absolutely robust. I have watched this
person do 10,000 transactions per year for ten years and
it has never had a problem.
Q. What I'm attempting to explore with you, Mr Coyne, is
your attempt to explain why it is that on the one hand
you have reached your judgment that Horizon is
relatively robust, but on the other hand you can't form
a judgment on the level of robustness without looking at
all the code. You have suggested that it is to be
explained because one is a relative judgment whereas the
other one is an absolute judgment. And I'm asking you,
if it is necessary to look at the code in order to make
a decision about robustness, why isn't it equally
necessary to look at the code when making a judgment
about relative robustness? And I don't understand why
the distinction between relative and absolute robustness
should make the slightest difference.
A. Because we can look at the rest of the industry in
either banking or point of sale or stock control or all
the different factors that make up Horizon and look at
the types of defects that are seen in those industries
and that are tolerated in those industries and the
impact of those failures within those industries and it
is through that lens that I have said that Horizon is
robust relative to those other industries.
Q. Well, could I suggest to you, Mr Coyne, what you are
doing in your first report, in indicating that Horizon
is robust then suggesting that it isn't, then suggesting
it is impossible to tell that it is robust, what you are
trying to do is to devalue the judgment that you have
very fairly and very honestly made that Horizon is
robust. What you are trying to do in your report is to
give a negative impression, which is not balanced but is
designed to achieve a conclusion?
A. That's certainly what I was not trying to do, no.
MR DE GARR ROBINSON: Very well.
My Lord, is this a convenient moment?
MR JUSTICE FRASER: If it is convenient to you. Is it?
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: You are moving on to a new subject,
right.
We are going to have a short 10-minute break. The
usual rules apply. 3.20 pm.
(3.12 pm)
(A short break)
(3.24 pm)
MR DE GARR ROBINSON: Mr Coyne, before the break I was
suggesting to you that in your first report you were
seeking to give a negative impression about Horizon that
was not consistent with your genuine view that Horizon
was relatively robust and you have answered those
questions.
But I'm now going to explore another aspect of that
which is that in your report you display more interest
in pointing out the problems you found with Horizon than
with actually giving a fair assessment of the overall
position of Horizon and my questions for the rest of
this afternoon will be looking at that question.
I don't have time to take you to more than a few
examples which is unfortunate because there are so many
of them in your report. And that is a problem that
arises in cases of this sort. The documents you refer
to in your report, in fact both your reports, are often
long and involved with substantial complexities.
Do you accept that where an expert is referring to
a great number of documents of this sort which are all
being presented as supporting a view, giving
an impression of a certain state of affairs, it is
incumbent on the expert to ensure that he or she deals
with the document scrupulously fairly so as to avoid
giving any risk of a false impression?
A. Yes.
Q. Counsel won't have time to cross-examine the expert on
all of them, and the judge certainly won't have time to
go through all of them with a fine-tooth comb in his or
her own chambers afterwards, will they? So a judge is
relying on the expert to discharge their duty to ensure
that a fair view is being given of any document being
referred to.
A. Yes.
Q. Now, do you think you have done that in your reports?
A. Yes.
Q. The documents you've referred to in your reports, and
there are a great number of them, have you considered
them all carefully yourself before explaining the
inferences you draw from them or the constructions you
put upon them?
A. That was certainly my aim in including the documents in
the report, yes.
Q. Well, let's deal with a few examples. As I say, I can
only deal with a few. First of all, cost benefit
analysis. This is a theme of both your reports, that
bugs in Horizon are diagnosed and fixed on a cost
benefit basis. Is that right?
A. There is certainly reference within the documents about
looking at bugs and fixing them on a cost benefit basis.
Q. There is a claim made in both your reports that where
bugs that are known to cause shortfalls, those bugs will
not be diagnosed and fixed on a cost benefit basis by
Fujitsu, essentially?
A. Yes, and I think that that was one of the agreements
that Dr Worden and I arrived at in one of our joint
statements.
Q. We will talk about the joint statements in a moment.
I am sure you will agree that it is inevitable that
every business will take some decisions on a cost
benefit basis?
A. Yes, that is true. It must be considered here that
there is a different dynamic here between the user of
the system and the people who fix the defects. In
an organisation they may have their own internal
programmers and therefore they may not fix things on
a cost benefit analysis basis. If you have
a development team that's not your own resources but you
are paying for those resources then there could be
a different dynamic at play there.
Q. The point you are making you are applying to the fixing
of bugs in Horizon. You are saying, correct me if I'm
wrong, that the support process provided by the SSC
allowed relevant bugs to remain in the system in order
to save money?
A. Yes, and there is certainly a reference within
a document to that view being taken.
Q. Let's see how you develop the point. If we could go to
your first report {D2/1/27}, please. This is
paragraph 3.15:
"There are a range of measures and controls existing
in Horizon each designed to prevent, detect, identify,
report and reduce the risk of several multifaceted
errors. It is likely that during the life of Horizon
system that these measures and controls improved. It is
also reasonably likely that in the majority of cases the
measures and controls were successful."
Stopping there, Mr Coyne, I need to correct
something I put to you before the break.
I put to you that there wasn't a hint of anything
positive said about countermeasures in your report.
That's not fair. There are in the report a number of
occasions -- a number of sentences of this sort which
could be said to be positive statements. What I would
suggest to you, however, is that that sentence is
a dramatic understatement of the positive view that you
have of the Horizon countermeasures?
A. I don't believe that's the case. It is correct there is
a range of measures and I explain what they are designed
for.
Q. In the majority of cases. Your view is that it is only
in a fraction of a percentage of cases that these things
don't work, isn't that right? Isn't that what you said
to me this morning?
A. I said it would be a fraction, yes.
Q. A fraction of a percentage.
A. Yes, it is likely a fraction of a percentage.
Q. Very good.
A. But I do actually say there it is likely that during the
life of Horizon the measures and controls improved and
that in the majority of the cases the controls were
successful.
Q. Let's move on to the next sentence, Mr Coyne, because
having said that positive thing, which as I have
suggested to you is a dramatic understatement of your
actual view, you then immediately try to undermine even
that view because you then say:
"However, there is also evidence to indicate that a
cost/benefit analysis was applied to the fixing of
bugs/errors/defects and that the possibility of error
was not reduced as far as possible."
So you're saying, well, countermeasures are applied
in -- are likely to apply most of the time, but there is
this problem. And you are suggesting there is a problem
in the support process because the support process --
that's Fujitsu, the SSC -- appears only to have fixed
bugs on the basis of a cost benefit analysis.
You do see that that's what you are trying to do.
You are trying to weaken your positive statement in the
previous statement even further by relying on this cost
benefit and basis claim, is that right?
A. No, I'm trying to provide the full picture and context.
Q. By your use of the word "However", you are setting this
up as undermining even the limited positive statement
that you make in the previous sentence, aren't you?
A. No, I'm setting out that after making that statement you
have to consider what follows.
Q. You repeat this point in the third joint statement and
you refer to it half a dozen times in your second
report, don't you?
A. I do, but it is important because the question that was
asked was: was it reduced as far as possible?
Q. I'm sorry, where is that asked?
A. It is one of the Horizon Issues.
Q. I see.
A. So when you consider "as far as possible", cost benefit
shouldn't really be involved in that consideration. It
should be whatever is possible.
Q. Let me carry on with this line. What I understand you
to be saying here is that there is evidence that bugs
which had or may have had a non-transient effect on
branch accounts were sometimes not resolved on the
grounds that the costs to Fujitsu or the Post Office
outweighed the benefit to the subpostmasters. Is that
the claim you are making?
A. Yes, that's certainly what I have got from one of the --
there's certainly a document, I am sure I make
a reference to it, where there is a number of bugs or at
least one bug that's considered. The impact of that is
that it very rarely occurs and therefore it is not worth
investing the money in fixing it.
Q. Let's see how you do support it in your first report,
shall we? If we go to page {D2/1/97} at paragraph
5.161, you say:
"Whilst both Horizon and Horizon Online contain
a number of measures and controls designed to check
system integrity, these mechanisms have been shown to
have failed. This is a point agreed upon in the joint
statement. It has been identified that known
issues/bugs were often deferred and dealt with on a
cost/benefit basis."
Here you have actually increased the scope and, if
I may say so, ambition of the claim. You are now saying
it happened "often". Is that your considered view of
the documents you have seen?
A. Could I just turn back, please, to the paragraph before?
Q. Of course. {D2/1/97}
(Pause)
A. It might be unfair for me to say "often".
Q. I'm going to come to whether it is unfair, Mr Coyne, but
first of all I'm going to ask you to analyse the
evidence that you rely on in support of the claim.
Because it is fair to say you are not saying it happened
once or twice over 20 years, you are saying it said it
happened "often", and over a 20-year period "often"
would usually mean dozens if not, in the context of
Horizon, hundreds of times. Would you agree with that?
A. No, I would take "often" to be ten or 15 times.
Q. So ten or 15 times over 20 years, you think it is fair
to describe that as "often", without bearing in mind the
context of a system which has undertaken something like
49 billion transactions during that period. 10 or 15
times as opposed to 49 billion transactions?
A. But this doesn't relate to the number of transactions,
does it? It relates to changes to the system. I do not
think the two correlate.
Q. Is it your claim then that your understanding of the
documents you have read is that there have been 10 or 15
occasions when a decision has been made not to fix a bug
on a cost benefit basis. Is that your understanding of
the evidence you have seen?
A. There are certainly a number of them. It might be
incorrect of me to say "often". There are certainly
a number of them.
Q. Well, I'm going to suggest to you, just so you
understand where I'm coming from, Mr Coyne, that the
evidence gets nowhere near anything like that and that
if you didn't know it, you certainly ought to have done.
So you can expect some questions from me about that.
You repeat the claim in paragraph 5.199 at page
{D2/1/108} where you say:
"Whilst both Horizon and Horizon Online contain many
measures and controls for ensuring system integrity,
these mechanisms do/have failed. It has been identified
that known issues/bugs were often deferred and dealt
with on a cost/benefit basis."
So you repeat the word "often". This is a theme of
both your reports, isn't it, that this is something that
happened on a regular basis?
A. It is certainly something that has happened and I have
noted on a number of occasions.
Q. If we go to page {D2/1/98}, in support of the claim
that's made in 5.161 you give one piece of evidence and
that's a risk and compliance meeting minute of
18th September 2013.
A. Mm.
Q. Let's look at the minutes. The bundle reference is
{F/11/40}.
(Pause)
MR JUSTICE FRASER: Sorry, it is {F/1140/1}.
While the operator is doing that, I think this is
one of them that was subject to the review. But it
doesn't matter, this isn't a specific point on that.
But insofar as you review documents and produce versions
less redacted, can someone just make sure that the more
up to date version goes on the database rather than the
original one?
MR DE GARR ROBINSON: My Lord, yes. All the lesser redacted
versions I think are certainly in the trial bundles.
I stand to be corrected if that's not right. I don't
believe this is one of them --
MR JUSTICE FRASER: This might not be one of them.
MR DE GARR ROBINSON: -- in any event.
You will see, Mr Coyne, this is dated
18th September 2013, it is the Post Office Risk &
Compliance Committee.
A. Mm.
Q. And what you rely on is at page {F/1140/3}. So this is
what you are relying on in support of your contention
that bugs were fixed -- often fixed on a cost benefit
basis:
"Discussion:
"It was reported that following the recent
Ernst & Young external audit four risks been identified.
Three of the risks raised had been addressed however the
final risk, relating to the communication by Fujitsu of
changes made to the Horizon system, was still
outstanding.
"It was identified that it would cost over
£1 million to implement the mitigation being suggested
by the audit and that this was not proportionate to the
risk being managed."
So certainly an example of a cost benefit analysis
being applied.
"Decisions.
"The Committee agreed that the risk be accepted with
Dave Hulbert as the owner and Lesley Sewell being
ultimately responsible."
So that's the evidence you rely on for your
proposition.
A. That --
Q. And as you say in paragraph 5.161, Ernst & Young had
identified four risks. Three have been addressed but
the final would cost over £1 million to mitigate.
If we look at the risk itself, that is in
{F/1127/1}. This is a paper for the Risk & Compliance
Committee, and if we look at the background,
paragraph 2:
"2.1 The 2012/13 Ernst & Young IT audit found no
significant exceptions but did identify a small number
of improvement opportunities. Four high level
improvement opportunities were recorded. Three have
progressed and are either complete or in the process of
completing. For one, IT&C believe we have sufficient
process and mitigation in place to accept this risk.
This paper is to highlight this decision to the Risk &
Compliance Committee."
The next paragraph reads:
"2.2. The specific observation was with regard to
Change Management Monitoring control. The actual
observation read 'Management should make use of a system
generated list of changes in performing the monitoring
control to further enhance its effectiveness'.
"2.3 The risk being that changes may be made to the
system that are not approved and not found through
monitoring."
A. Yes.
Q. So what's being said here is there needs to be
a change -- or what's being suggested is that it would
be an improvement to the system if the change controls
applied within Post Office were made automatic so that
every time a change was being made to the system, this
would be automatically notified, electronically notified
to Post Office that it was happening. Do you see?
A. This is a very important aspect. So what this is saying
here is the current situation is that Fujitsu will make
a change to the Horizon system and it is not clear
whether Post Office will have either approved that
change or will be aware that that change has been made.
There is a weakness in the process that needs to be
addressed.
Now, on the basis --
Q. Wait, can I stop you there for a moment. Are you
suggesting Ernst & Young were saying there was
a weakness in the system that had to be addressed?
A. They are saying there needs to be a change to the -- a
change to the management monitoring control.
Q. Look back at paragraph 2.1, Mr Coyne:
"The 2012/13 Ernst & Young audit found no
significant exceptions but did identify a small number
of improvement opportunities."
Even looking at this document you can see that the
characterisation you have just adopted is simply not
right. These are improvement opportunities. These are
not problems that need to be addressed, are they?
A. From the text that's being said there about the changes
that need to be made then they are important.
Q. That's your judgment, is it?
A. Yes.
Q. It is not said anywhere that I have seen. Have you seen
it said somewhere in one of the documents?
A. I believe that there is more of an Excel spreadsheet
type layout to this document because this is the minutes
of the meeting that reviews the observations from
Ernst & Young. There is another Excel spreadsheet where
I think Ernst & Young provide their output.
Q. I see. So you are suggesting there is an Ernst & Young
document which doesn't suggest that this is
an improvement opportunity, it actually suggests it is
a real problem that needs to be addressed, is that your
contention?
A. Yes, I believe it sets out --
Q. Very good.
A. Sorry, I believe it sets out more context than what is
included here.
Q. When you wrote paragraph 5.161 you had read this
document, had you?
A. Yes.
Q. If we look at paragraph 3, I think you suggested before
that Post Office didn't know what changes were being
made and that's why it is so important. Is that your
logic?
A. Yes. What this is saying here is that there is a risk
that changes are made without being adequately informed
to the Post Office.
Q. We see the existing system described in paragraph 3:
"The Post Office Service Management team currently
monitor IT system changes on a monthly basis by cross
referencing known and approved changes against a list
produced by Fujitsu. E&Y observed that this could be
enhanced ..."
"Enhanced", not this is a serious problem that needs
to be remedied, they use the word "enhanced".
"... if the list was generated by the IT system
rather than by change records."
It is a relatively modest enhancement, isn't it,
wouldn't you agree?
A. I believe this is pointed out because there is the
possibility that the change records, which are a human
created document, may not have been completed adequately
and it would be better to have it flagged up that if the
system changes, an automated message goes to the
Post Office that the system has changed rather than rely
upon a paper system.
Q. We can all agree it would be better Mr Coyne.
A. Yes.
Q. The question is how serious the problem is that the
existing monitoring system is being applied.
If you look at the next paragraph:
"IT service management engaged with Fujitsu to
understand how this could be achieved and it was
concluded a very difficult and potentially expensive
approach to adopt this as all changes are recorded as
'events' within the IT system of which there are
multiple thousands per day with changes only being a
small percentage. The cost and difficulty in extracting
these specific change events on a regular basis would
outweigh the value in monitoring the change."
Do you see that?
A. Yes.
Q. One can see why someone might form that judgment, yes?
A. Yes, but if you break that down so that there are
multiple thousand a day, but the changes are only
a small percentage, so a small percentage of multiple
thousands is still quite a big number and that's changes
made to Horizon every day.
Q. Mr Coyne, you do seem to be deviating from the path that
I'm asking you questions about. We are not talking
about how many changes there were. We are simply
talking about how they are monitored and the existing
system is a human system for monitoring those changes
and what's being suggested is that it should become
an automatic system?
A. Yes.
Q. Then if we go over the page to {F/1127/2} the proposal
in paragraph 5 is:
"To continue with the existing process of monitoring
but to additionally raise this as a risk within IT&C and
to monitor any exceptions found through the existing
process. If exceptions are found then re-consider the
proposal from E&Y and assess the impact of the change
versus the benefit."
So a weighing up exercise is taking place and the
conclusion for the time being is: keep things as they
are but be aware of this potential improvement and
monitor in the meantime to see whether in fact it would
be a good idea to make it.
A. Yes.
Q. That's an entirely reasonable approach wouldn't you
agree?
A. Well, the approach is broadly saying let's carry on
doing what we do now and just look at it again in the
future.
Q. No. They are suggesting, first of all, carry on as we
are doing now which is a monitoring system. We have
a monitoring system; a suggestion is being made as to
how to improve it, but as well as carrying on with the
system, monitor the monitoring system to see if any
problems arise, if there are any exceptions and if there
are, then go back to the proposal of automatic
monitoring.
A. That is what it says.
Q. It is not possible for you sitting in your chair over
there to say that that was necessarily the wrong
decision, is it?
A. No, that's true.
Q. It is the kind of thing that a business would quite
sensibly want to do. They would like to assess whether
there is a problem or not before they spend such a vast
amount of money. You can't say that's intrinsically
unreasonable, can you?
A. No, it is not intrinsically unreasonable but it leaves
exposed a risk that could be dealt with but hasn't been
dealt with because of a cost benefit basis and that was
the observation that I made.
Q. But there is a much more important point, Mr Coyne,
which is that this has nothing to do with fixing bugs in
Horizon, does it?
A. Yes, it goes directly to fixing bugs in Horizon because
the changes that they are talking about making are
changes to Horizon.
Q. Mr Coyne, let me remind you of the sentence that you say
this document supports that's contained in
paragraph 5.161 of your expert report:
"It has been identified that known issues/bugs were
often deferred and dealt with on a cost benefit basis."
This proposal here and the decision that was made
from this proposal is not evidence of any known issue or
bug being deferred and dealt with in a cost benefit
basis, is it?
A. No. This isn't to do with the deferment of a bug, no.
MR JUSTICE FRASER: Mr De Garr Robinson, there is another
document that goes with this one and I'm slightly
concerned you might be putting some of your questions on
an incorrect basis. So just pause there one second.
If you could look at please -- and I don't want this
to go on the common screen -- {F/869/30}.
I don't want to set a hare running but I just feel
that if you look at {F/869/30} and then it is the third
column from the left and {F/869/31}, I think those are
the more detailed points that these conclusions you have
been taking the witness to go with. Now I might be
wrong.
MR DE GARR ROBINSON: I'm sorry, what am I supposed to be
looking at my Lord?
MR JUSTICE FRASER: The testing sample of 15 back end
changes, ten of which were in the counter and five of
which were manual. If you scroll down that column. Do
you have {F/869/30}?
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: If you start at that bullet point and
just read that column down to the bottom of that page
and then onto the next page leading to the conclusion
that begins "There is an increased risk...".
MR DE GARR ROBINSON: Based on a testing sample of 15 back
end changes?
MR JUSTICE FRASER: Yes, ten at the counter and 5 manual
changes. If you just read that column down to yourself
to the conclusion, "There is an increased risk...".
(Pause).
MR DE GARR ROBINSON: My Lord, yes. Is this the 2012/13
audit?
MR JUSTICE FRASER: No. I'll tell you what I'm going to do.
Mr Coyne could you go out for a minute please?
MR DE GARR ROBINSON: Is this the Ernst & Young management
letter of 2011?
MR JUSTICE FRASER: It is. Are you about to come onto it?
MR DE GARR ROBINSON: No, I'm not. This proposal~-- no I'm
not going into it.
(The witness withdraws)
MR JUSTICE FRASER: Hold on one second, Mr De Garr Robinson,
I am waiting for the witness to go out.
Now what would you like to say?
MR DE GARR ROBINSON: My Lord, this document relates to
a proposal for improvement that was made in the 2012/13
audit.
MR JUSTICE FRASER: I know.
MR DE GARR ROBINSON: My Lord I believe you are taking me to
the 2011 management letter.
MR JUSTICE FRASER: I know, but when the witness in part of
his answers to you says, "I think there is a document",
and he goes on to deal with it directly in his report
immediately after the passage you are asking him to, it
would help me hugely if you would make it clear with him
what the document is he says supports what he is saying
justifies his conclusion on the other document you are
putting to him.
MR DE GARR ROBINSON: I see.
MR JUSTICE FRASER: Do you see what I mean?
MR DE GARR ROBINSON: Yes. I must confess my focus was on
fixing bugs rather than his other answers.
MR JUSTICE FRASER: I entirely understand that. I'm just
faced with a witness who from time to time says: well,
there is a document, and it would be enormously helpful
to me if you could either, how shall I put it neutrally,
either get a tick in the box that that's a document or
a cross that there isn't such a document when you are
putting those questions to him.
MR DE GARR ROBINSON: I will try my Lord.
MR JUSTICE FRASER: I'm not saying --
MR DE GARR ROBINSON: Given the vast size of the bundles --
MR JUSTICE FRASER: I'm not saying you have to do it on
every single occasion, but on this particular occasion,
because the paragraph comes immediately afterwards, it
is much better to deal with it now than, for example,
store them all up until Friday evening at 4.30 when Mr
Green tries to re-examine him in ten minutes.
I'm attempting to explain to you why it is going to
be helpful to me.
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: Can we have the witness back in please.
(In the presence of the witness)
MR JUSTICE FRASER: Mr Coyne, don't read anything into that
at all. I just had to have an exchange about a
particular reference, that is all.
MR DE GARR ROBINSON: Mr Coyne, I have suggested to you that
this particular document has nothing to do with
deferring or not dealing with issues or bugs on a cost
benefit basis and I believe you have accepted that,
I hope you have. (Pause)
You have accepted that?
A. Yes, this particular document isn't about fixing bugs on
a cost benefit basis.
Q. You will have seen from the documents, these documents
that what was being considered by the Risk & Compliance
Committee was a proposal that was made in the Ernst &
Young audit of 2012/13 and we branched into that
question, although I must admit it wasn't a question
I was particularly interested in at that stage. But you
did say you thought you had seen a document. Is the
document you thought you had seen the one that's
referred to in 5.162 of your first report? That is at
{D2/1/98}. Might it be the Ernst & Young letter of
2011? Is that what you were thinking of?
A. Do you have that document?
Q. Yes. It is {F/869/1} that is the management letter. Is
that the document you were thinking of?
A. No, I think the way the document works is there is
a management letter and there is like a spreadsheet
version of the observations I believe.
Q. My learned friend is very kindly suggesting that it
might be at {F/1075.1/1}. Perhaps we can go to that.
MR JUSTICE FRASER: I think it is just a draft of the one
that I showed you.
MR DE GARR ROBINSON: This one is:
"IT component of management letter for the year
ended 31 March 2013".
Is that the document you had in mind?
A. Quite possibly. Can I just have a look at the --
Q. Perhaps we could scroll forward one page, first of all.
{F/1075.1/2}. There is an overview. Then there is
a series of detailed observations. Is that page 2?
There is a proposal about SAP payroll in paragraph 1
{F/1075.1/3}. In paragraph 2 a proposal about POLSAP
security parameters. If we go forward a page
{F/1075.1/4}. Paragraph 3, a proposal about POLSAP
again. Paragraph 4, "Change management monitoring
control".
A. Yes so that relates back --
Q. Is that what you were thinking of?
A. Yes. Sorry, can I just have a moment to read that to
remind myself. (Pause).
MR JUSTICE FRASER: If it doesn't jump out at you as the
document to which you were referring, please just say so
and Mr De Garr Robinson can move on.
A. This is certainly the document that the text in the
report refers to but there are documents that relate to
cost benefit analysis within I believe it is PEAKs.
MR DE GARR ROBINSON: That is entirely separate and I'm
going there, please don't think I'm going to leave you
out Mr Coyne. But when you said there was
a spreadsheet, is this the --
A. Yes, I think it is.
Q. I'm grateful. My Lord, I don't know if that assists
you?
MR JUSTICE FRASER: That's very helpful and I don't want to
knock you off your course or anything like that.
MR DE GARR ROBINSON: No, that's quite understandable.
So you have accepted that the evidence you cite for
the proposition "known issues" were often deferred and
dealt with on a cost benefit basis is no evidence for
that proposition at all. You have accepted that,
haven't you?
A. I accept that the reference that I have provided doesn't
go to a document that shows that but I have other
references that I will --
Q. You didn't have you -- say you think there is a PEAK and
we will come -- but you didn't have a PEAK in mind at
the time that you made your first report, did you?
A. Or I have not referenced it correctly when putting the
report together.
Q. Why is it you were able to give this example as evidence
of the claim that this often happened. How did it come
about? Did you not read the document --
A. No, there is more than one.
Q. Mr Coyne, you make a claim?
A. Yes.
Q. Then you immediately cite evidence in support of it?
A. Yes.
Q. And I'm asking you how it is you came to cite evidence
in support of it which is no evidence at all. Did you
not read the document?
A. I have certainly read the document. I believe it is
incorrectly referenced.
Q. What do you mean incorrectly referenced please, would
you explain that?
A. Yes. So I have had a note of the point of cost benefit
analysis and I have then searched for the appropriate
reference in my database of references and it looks like
I have referenced the wrong document, but it does
contain the words "cost benefit" and that's how the
mistake --
Q. Mr Coyne, isn't that exactly the sort of thing that you
accepted a few minutes ago that an expert should not do?
That it is incumbent upon an expert when citing
a document as evidence for a proposition to read the
document carefully to ensure that it is evidence for
that proposition?
A. Yes.
Q. Why did you not when going through your report read the
document carefully with a view to ensuring it was
evidence for the proposition?
A. With regard to this I have made a mistake in referencing
this document.
Q. Can I suggest to you that the mistake that you made is
that you were anxious to assert that fixes were deferred
and dealt with on a cost benefit basis and so what you
did was you found a document that talked about cost
benefit basis and you put it in?
A. I have found other documents but I have made a mistake,
I do accept that.
Q. Let's go forward to another document you cite.
I believe this is the only other document you cite in
your first report for this proposition. If we go to
paragraph 6.1 which is at page {D2/1/109}.
A. Sorry, I don't have that yet.
MR JUSTICE FRASER: Do you have your hard copy report in
front of you?
A. Yes, I do. Yes, I have that.
MR DE GARR ROBINSON: So is this evidence you have in mind?
Perhaps I could ask you to read paragraph 6.1 to 6.3.
(Pause).
Now in fairness to you you don't cite this document
as being in support of your claim that fixes were often
dealt with on a cost benefit basis, but was this
a document that you had in mind in supporting that
claim?
A. Yes.
Q. Well, let's look at that, shall we? The document is at
{F/1697/1}. This is called "Reconciliation and Incident
Management Joint Working Document". It is dated, the
current version, 18 March 2013.
If we could go forward to page {F/1697/9} of this
document to see its purpose. Perhaps I could ask you to
read section 1.1:
"What is a reconciliation?
"End to end reconciliation within HNG-X..."
That is Horizon Online, isn't it?
A. Yes.
Q. "... is the mechanism by which Post Office Ltd (POL) and
Post office Account (POA) establish which transactions
are complete and correct, and which are not. An
incomplete transaction is not necessarily a
Reconciliation error, but it might become one if it is
not completed in a timely manner. An incorrect
transaction is a Reconciliation error."
A. Yes.
Q. Over the page {F/1697/10}:
"Each and every reconciliation error is the result
of some system fault. That fault might, for example, be
a software bug fault (introduced through either design
or coding), a system crash, or a telephone line being
dug up. Such faults may affect transactions, thus it is
the job of Reconciliation Service to detect when and how
any transaction is affected by any system fault."
A. Yes.
Q. "A reported reconciliation error provides:
"A business impact in terms of an error report on
a transaction and;
"Evidence of a system fault that may need some
corrective action.
"It is acknowledged that not all system faults will
lead to corrective action as this is generally done on a
contractual and/or cost benefit basis."
A. Yes.
Q. So do you say that that is evidence for the proposition
that bug fixes in Horizon are dealt with or deferred on
a cost benefit basis?
A. Yes.
Q. What this document makes clear is that a system fault
has a very wide range of meanings, doesn't it?
A. Yes.
Q. On a fair reading it is a fault causing an error in the
systems by which transaction data are compared against
and reconciled with client transaction data?
A. Yes.
Q. It is a fault which it is the job of the reconciliation
service to investigate, to detect when and how the fault
was operated?
A. Yes.
Q. It is not a fault causing Horizon to have a false
transaction in the first place. That is the job of the
SSC to investigate, isn't it?
A. Yes.
Q. It is hardly surprising that the question whether
a system fault causing a reconciliation error is
corrected might involve a cost benefit analysis or
an analysis of who is responsible for what between the
client and the Post Office, is it?
A. I do not think the two go hand in hand. If you are
deciding whether to correct bugs on a cost -- sorry,
correct system faults, a subset of which is software
bugs or faults, depending on which is in the latest
version of the document, then you are deciding not to
fix those faults because it is either too expensive or
doesn't provide sufficient benefit.
Q. Mr Coyne, I'm suggesting to you that this document says
nothing about the approach adopted by the SSC to bugs in
Horizon causing discrepancies in branch accounts. Do
you not accept that?
A. Well, bugs in Horizon are fixed by Fujitsu.
Q. They are fixed by the SSC. They are not fixed by the
reconciliation service, would you accept that?
A. I don't believe that they are necessarily fixed by SSC.
I think they are identified by SSC.
Q. You are quite right, they are fixed by development when
the SSC passes it onto development.
A. Yes.
Q. But they are not fixed by the reconciliation service,
are they?
A. No they are not fixed by the reconciliation service but
the reconciliation service will report that there is
a problem; that will then be investigated and it will be
decided whether it is a software, bug or fault either
introduced through design or coding, whether it is
a system crash, whether it is a telephone line being dug
up. Somebody then will decide whether that system fault
should be fixed. But if it is a system fault that is
a software bug, it will be Fujitsu that fixes that even
though the fault might be found initially by the
reconciliation service.
Q. What I'm suggesting to you Mr Coyne is that you can't
reasonably infer that this passage of this working
document is talking about the SSC's and the fourth
tier's and the third tier's approach to the fixing of
bugs in Horizon?
A. I believe firmly it is. That's what it is saying.
Q. Very well. I suggest to you, Mr Coyne, that you
certainly can't infer that it was often decided by the
SSC or development not to fix bugs because the cost was
too great?
A. Well, what it actually says, this is generally done on
a contractual or cost benefit basis.
Q. If this were generally done there would be a significant
number of PEAKs recording that it had been done. It is
the kind of thing that would be recorded in a PEAK,
isn't it?
A. If the process is working properly, then it is likely
that there will be a PEAK identified for these, yes.
Q. If there were a significant number of PEAKs -- if this
were often done, there would be a significant number of
PEAKs evidencing the fact that it is done, correct?
A. There might only be one PEAK.
Q. Why might there only be one if it is often done
Mr Coyne?
A. Because if it becomes a problem that occurs repeated
times and there is a consideration of what the fix is,
Fujitsu won't raise tens or hundreds of PEAKs per day
all relating to the same fault. They will refer back to
the one single KEL that identifies the problem. That's
the purpose of PEAKs and KELs.
Q. If known issues are often deferred or dealt with on
a cost benefit basis, then there must be a significant
number of separate issues that are deferred or dealt
with on that basis; correct?
A. Yes.
Q. For each of those issues there will be at least one
PEAK, sometimes there will be more, considering that
aspect, yes?
A. There should be at least one PEAK or one KEL or probably
both.
Q. So if this is often done, there will often be known
issues in relation to which this is done so there will
often be PEAKs recording that this has been done;
correct?
A. No. If the fault is found again next week, Fujitsu go
back, they search for the reason for the fault, they
find that there is a KEL -- that is the purpose of the
KEL it is the knowledge base. So they will not then
raise another PEAK, they will see that it is going to be
fixed at some time in the future by that KEL.
Q. It may be you are not quite grasping the suggestion I'm
making to you. Are you suggesting that actually there's
only ever been one issue in relation to which this has
happened and that there's one KEL covering it happening
and therefore there will only be one PEAK covering it
happening, is that the logic of your position?
A. There is likely one KEL for every different
manifestation of the defect.
Q. Yes, so if it is being done often -- your claim is that
it is being done often -- then there will often be PEAKs
recording this decision being made. In fact there will
probably often be KELs as well, wouldn't there?
A. Yes but they don't duplicate a KEL every time a defect
happens.
Q. Mr Coyne, I'm struggling and it may be my fault. If
known issues are often dealt with on a cost benefit
basis, there must often be issues which are dealt with
on a cost benefit basis, correct?
A. Yes.
Q. And each of those issues will have their own KELs and
their own PEAKs, yes?
A. Yes.
Q. Perhaps I can put my point again. In circumstances
where we have a system in which Fujitsu often make
decisions to defer or not deal with known issues on
a cost benefit basis, there are going to be
a significant number of PEAKs and KELs recording that
being done, do you accept that?
A. I do but I do not think it is Fujitsu that is taking the
decision to do it on a cost benefit basis. If that is
the Post Office taking the decision then, yes, I agree
with that.
Q. Am I right in thinking that at the time you produced
this report you had not found any such PEAK because if
you had it would be in this report, wouldn't it?
A. I don't know whether I was aware of it at the time but
there are certainly PEAKs that talk about things being
deferred on a cost benefit basis, whether I had found it
at the time of this report or found it later I'm not
sure.
Q. I suggest to you, Mr Coyne, that you obviously hadn't
because if you had they would have been referred to in
this report, would they not?
A. Likely, yes.
Q. Right. So at the time you made this report is it likely
that you didn't know of any PEAKs in which this was
recorded, that your only evidence is the two documents
I have just taken you to?
A. This document is quite strong because it talks about the
corrective action generally being done on a cost
basis --
Q. Is it likely that when you made this statement the only
evidence you had to support it was the Risk and
Compliance committee meeting minutes and the document we
have just come to? I have had your commentary on
whether you think either of those documents is good
evidence. I'm simply asking you whether it is likely
that those were the only two pieces of evidence you had
at the time you made this report?
A. It is certainly possible that is the case, but I would
have to look at when the PEAKs arrived.
Q. You have just said there are PEAKs, plural, where this
has happened?
A. Yes.
Q. Now Mr Coyne I'm aware of one PEAK which might be said
to support a suggestion that there was one occasion on
which a cost benefit analysis was adopted and I'm going
to take you to it. It is referred to in your second
report. Are you aware of others that are not referred
to in your second report?
A. I believe there is more than one, yes.
Q. You clearly looked very hard for them. Would I be right
in inferring that that is the case? You looked very
hard for PEAKs to support this proposition, did you?
A. I don't believe there were any specific searches that
I conducted for that, no.
Q. So you didn't make any specific searches but you
stumbled over how many PEAKs to support the proposition?
A. A handful perhaps.
Q. I'm only aware of one, Mr Coyne. Might it be possible
for you to come to court tomorrow and tell me what the
others are?
A. Certainly, yes.
Q. Let's go to the one I'm aware of. Let's go to
{F/271/1}. This is a PEAK PCO120937. It is dated
20th May 2005. It is a customer call, at the top box of
the PEAK, details entered are:
"Summary: PM is stating that the system remmed out
some coin."
If we go down to the second line in the top box:
"PM is stating that the system remmed out some coin
its own and it produced no paper work, and they would.
Like to know if we can find out the Pouch number so they
can reverse it out."
Do you see that?
A. Yes.
Q. If we go over the page to {F/271/2} at 16th May, 12.01.
I won't read it out but you will see that Sarah English
is unable to recreate the problem. Do you see that?
A. At 12.01? Yes.
Q. Then if we go over the page to page {F/271/3} and this
time at the top of the page, 17th May, 13.55. This is
Gary Maxwell?
A. Yes.
Q. The last paragraph you will see he says:
"I have attempted to replicate this on our test
counter but could not abandon the session any other way
than by following the available menu options. In all
cases the session finished as expected without writing
any messages."
They can't recreate the problem, yes?
A. Yes.
Q. Often that's a reason for thinking that there may not be
a problem there. But often it may be that they are just
perplexed and one just doesn't know.
If one goes to {F/271/4} (Pause). 7.32 about three
boxes down, 13th June 2005. This is from Lionel Higman?
A. Higman I believe.
Q. Thank you.
"Is the frequency of occurrence for this problem
known? How severe are the problems caused? Is recovery
possible? Would it be sufficient to raise a KEL to
manage future occurrences more effectively?"
You see the question being raised there?
A. Yes.
Q. Then if you go down to 10.13. My note is terrible...
A. 10.33 might be relevant:
"Gary advises development have found a potential
bug. They want to know the scale of the problem as there
is potential risk in changing the code."
Q. Right. Thank you. For some reason I keep typing a 1
instead of a 3 in my notes. The concern being expressed
by development here is risk from making a change. Do
you see that?
A. Well, found a bug and then there is associated risk with
making the change. There are two elements.
Q. If we go back. I'm sorry I went too fast. If we go
back to page 3.
A. There was 10.13 on there that might be helpful as well.
Q. I'm sorry Mr Coyne, what were you saying?
A. If we go back to the page that we were just on at 10.13,
at the bottom.
MR JUSTICE FRASER: On 15th June?
A. 15th June, yes.
"First instance, check the PEAKs. The impact on the
office is that they will have an amount posted to the
suspense account ... equivalent to the remittance
amount. This will be evident until remedial action is
taken."
MR DE GARR ROBINSON: Stopping there. What does that mean?
A. So the suspense account is going to be likely in
balance. It is going to have an impact anyway.
MR DE GARR ROBINSON: It means that the postmaster is going
to know very well what happened, isn't it?
A. Well, the postmaster will have a suspense account that
doesn't balance. I do not think they will know what's
happened but they might know what the impact is.
Q. At any rate they will know there is a problem that needs
to be addressed, yes?
A. Did the postmaster raise the call on this one? Did they
ring up in the first instance reporting the fault?
Q. I believe the passage I read you -- yes, postmaster is
stating the system remmed out some coins, do you
remember?
A. Yes.
Q. The postmaster knew exactly what had happened. And he
says {F/271/5}:
"There is nothing the PM can do to correct this. The
sensible recovery option is for the NBSC to issue an
error notice for the remittance amount as in this case."
So that had happened in this case it would seem?
A. Yes, so that is a correction on the accounts.
Q. What we now call transaction corrections.
"The alternative would be for SSC to create the
necessary Riposte messages to cancel out the error
(currently being tested).
"A KEL has been created on the back of this call and
will be updated shortly.
"Given the frequency of the problem & the apparent
risk involved in introducing a code fix the KEL should
be adequate."
Do you see that? What's being presented there by
Gary Maxwell is that it is a very low frequency problem
and there's risk involved in introducing the code fix.
Do you know why there is a risk involved in
introducing a code fix?
A. There's inherent risk in any change to a system. So
introducing any fixes to code brings a risk with it.
Q. So up until now the debate is about risk of doing
a change compared with the benefit of doing the change,
bearing in mind it is such a rare problem. Do you see
that?
A. Yes.
Q. Then if we go down to 11.28 we see Mark Scardifield.
This is where the word "cost" gets mentioned:
"Weighing up the cost and risk of an attempted fix
against the fact that this has only been reported once,
I do not believe that we should make a code fix. If
further incidents of this problem are reported we can
review this decision. Gary has raised a KEL, so
returning for closure as "Published Known Error".
Do you see that?
A. Yes.
Q. So what we have is a debate which largely revolves
around the risk involved in making a fix in this part of
the system?
A. Yes.
Q. And what Mark Scardifield does is he brings in the word
"cost"?
A. Yes.
Q. Do you say that this PEAK is evidence for the
proposition that decisions about fixing problems were
often made on a cost benefit basis?
A. It certainly does indicate that.
Q. It indicates that what's happened is that the parties
are concerned primarily with risk and on that basis they
decide that they will leave the problem as it is. It
doesn't support the notion that decisions are often made
on a cost benefit basis, does it?
A. It is another reference to decisions being taken on
a cost benefit basis.
Q. It is one instance. It is a PEAK where the word "cost"
is referred to?
A. Yes and the other document that we talked about, things
are generally done on a cost benefit basis.
Q. Let's look at the KEL that was created for this?
A. Before we move on, just for context, can we just read
the top column there because it talks about what the
possible remedial actions are.
MR JUSTICE FRASER: Which page?
A. F/271 and it is the one above 10.14.
MR JUSTICE FRASER: If you look to the right, is it page 5?
A. Page 5 because it talks there that the alternative would
be for SSC to create the necessary Riposte message to
cancel out the error.
MR JUSTICE FRASER: You are going to have to give us both,
Mr De Garr Robinson and me, the date and the time please
of where you are reading.
A. Certainly, it is the very top box. It is above
15th June 2005.
MR JUSTICE FRASER: The one in the pale green at the very
top?
A. Yes. It starts with "... remittance amount as in this
case."
But of interest is the alternative remedial action
to fix the accounts is:
"... would be for SSC to create the necessary
Riposte messages to cancel out the error".
That is the insertion of messages in Riposte which
is relevant to other aspects that we are talking about.
MR DE GARR ROBINSON: Thank you. Can we now look at the KEL
that was created for this fix because what I want to
suggest to you, Mr Coyne, is that the focus of this
decision or the main reason for this decision was risk.
If we could look to {F/276/1} please. This is the
KEL that was created?
A. Yes.
Q. It says:
"The PM was attempting to do a remittance out ADC
for a bag of coins. After entering an invalid bar code
for the particular type of transaction the session was
aborted by pressing the 'home' key."
Then:
"Firstly, during the remittance out process pressing
'home' should not be an available option. The only way
of completing the session is to navigate the screen
prompts & cancel the session or return to Desktop and
bin the item on the sales stack.
"By aborting the session in this way the EPOSS
remittance items on the sales stack were incorrectly
committed. Normally these messages are only committed
once a successful barcode has been scanned/entered & at
the same time as the LFS remittance out message."
Then there is some computing gobbledygook:
"The PM will not be able to reverse the transaction
using existing reversal (because of the tran type) or
will they be able to use the pouch reversal function (as
no pouch id exists).
"The suspense account will be debited/credited by
the amount of the remittance. The office snapshot/cash
account report will continue to show the remittance
amount under unclaimed payments."
Their solution. It refers to Atos, but Atos wasn't
on the scene in 2005. So I apprehend that one big
word-processing change was made when Atos was involved
many years later but it says:
"Solution:
"Development have identified a possible bug in the
counter code. However, due to the frequency of the
problem & the risks involved in making the necessary
changes it has been decided that a code change will not
be made. Raise a call on EDSC. In the one case to date,
an error notice was issued to the office for the amount
of the remittance."
What we have here is a debate in a PEAK about risk
where there is a reference to cost and then we have
a KEL that has been produced as a result of that debate.
That KEL does not refer to cost, it just refers to risk.
Do you suggest, Mr Coyne, that this document, this
KEL and this PEAK together, constitute evidence that
deferring and dealing with known issues were often dealt
with on a cost benefit basis?
A. It is certainly another document that indicates that
cost was a consideration in whether to fix.
Q. Well this is the only document of which I'm aware, PEAK
or KEL, in which this point has been made. You say that
you are aware of others?
MR GREEN: My Lord, I have tried to let it run but I mean in
that actual section of his report at 5.166 on page 99 he
actually specifies the other PEAK.
MR JUSTICE FRASER: Give me the reference.
MR GREEN: It is 5.166.
MR JUSTICE FRASER: No, the bundle reference.
MR GREEN: Sorry, {D2/1/99} which is the next page of his
report in the same section.
MR DE GARR ROBINSON: This is his second report?
MR GREEN: No, in the first report where we were dealing
with the Ernst & Young stuff and you put to him only
documents and so forth, and in 5.166 he specifically
mentions a PEAK. It was put to him he hadn't looked at
any.
MR JUSTICE FRASER: Is this the one at {F/275.1/1}?
MR GREEN: It is my Lord, yes.
MR JUSTICE FRASER: Yes. Mr De Garr Robinson, I'm very
happy for you to look at that and resume with it in the
morning or do it now?
MR DE GARR ROBINSON: No, my Lord, I'm content to break
today.
MR JUSTICE FRASER: Okay. So Mr Green what was your point
on the way the question was put that it is the only
document?
MR GREEN: When you look back across the transcript --
MR JUSTICE FRASER: The transcript says --
MR GREEN: It is the only document I'm aware of.
MR JUSTICE FRASER:
"Question: This is the only document of which I am
aware, PEAK or KEL, in which this point has been made."
MR GREEN: And it has been going on for about 20 minutes.
And it is not really fair to Mr Coyne when one ought
to be aware of what he says in that very section of his
report on this point.
MR JUSTICE FRASER: Right, well --
MR DE GARR ROBINSON: My learned friend seems to be
suggesting that the document {F/275.1/1} is further
evidence of a decision to defer a fix on the basis of
a cost benefit analysis. Let's look at that, shall we,
can we look at {F/275.1/1} please. So this is dated
18th June 2005. The call logger is:
"Deleted user. ITU, SV&I."
I'm not sure what that means. POL have raised the
following incident:
"'A cash declaration was made in "Stock Balancing"
for the amount displayed on the Snapshot. When the Cash
Variance was checked afterwards a Gain of £45.05 was
displayed.
"The discrepancy was the cash value of the
transactions performed on the stock unit after rolling
into branch trading."
Then there is further information about it.
Could we perhaps go over the page {F/275.1/2}.
16th June at 15.59:
"I've attempted to recreate the situation which
occurred over a CA Rollover in this instance and the
injecting a transaction like the mails txns ...
Difference being that up here the trace has turned up
a transaction received ... and in the supplied audit log
nothing at all appears ..."
Then the next box says:
"After several days of attempting to recreate this
problem ... I have only seen one instance of this
problem."
Excuse me. (Pause)
Could we go back to the first page of this PEAK,
please {F/275.1/1}. You see the call type on the
left-hand side, Mr Coyne?
A. Yes.
Q. "Business Integration Testing Incidents/Defects."
This is the testing PEAK, isn't it?
A. Yes, I think that is right.
Q. Could you explain what a testing PEAK is?
A. It is a defect that has arisen on the system during
testing, so not in live operation.
Q. Does a testing PEAK like this, is it capable of
constituting evidence that known issues in the system
are dealt with on a cost benefit basis?
A. Certainly, if they are dealing with defects that are
found in testing on a cost benefit basis and therefore
don't fix the defect it will go out to live.
Q. Now, is it -- and it may be unfair to ask you this
question because you didn't raise it but now you have
seen this PEAK do you remember it?
A. I'm not sure I do. Can I look, please, at the end of
the ...
Q. I don't know how long it is.
MR JUSTICE FRASER: It is five pages.
MR DE GARR ROBINSON: Could we look at page 4, please.
I must confess I hadn't read 5.166 as being even
relevant to this point. {F/275.1/4}
A. It looks like it needs to go out to Escher to be fixed
rather than Fujitsu. It says at the bottom:
"This amounts to a declaration by POA that we are no
longer interested in obtaining a fix."
But it doesn't make a reference to a cost benefit.
Q. Thank you.
MR JUSTICE FRASER: Right. Is that a good --
MR DE GARR ROBINSON: I think we are at 4.30 pm.
MR JUSTICE FRASER: I think it probably is. Right.
A couple of things. Mr Coyne, you have some
homework, but generally when you are giving your
evidence tomorrow you have got in front of you, and
that's why I checked this afternoon, a hard copy of your
report?
A. Yes, I have.
MR JUSTICE FRASER: It is very tempting when
Mr de Garr Robinson is asking questions just to look at
the screen. It's probably a good idea also to flick to
your report but it is completely up to you whether you
do that or not.
And Mr Green, you might want to store points such as
that one up for re-examination, if you have any, because
you might find that upon mature reflection they are not
the points you thought they were when you jumped to your
feet, although I appreciate you have been restrained, or
you think you have been restrained today.
And Mr de Garr Robinson, 10.30 am tomorrow.
I assume you don't have any objection to the witness
looking on his knowledge database to do the homework
that you set him?
MR DE GARR ROBINSON: My Lord, of course not. Obviously he
won't be talking to anyone.
MR JUSTICE FRASER: No, I'm just about to remind him.
Mr Coyne, normally you won't have homework but it is
particularly important that you, not your assistant,
does that.
A. Absolutely, my Lord.
MR JUSTICE FRASER: If he finds anything what do you want
him to do in terms of bringing it with him or notifying
references? Are you happy to get that tomorrow morning
at 10.15?
MR DE GARR ROBINSON: My Lord, yes. I don't think I will be
going back to it immediately. It will be something to
be dealt with on another day.
MR JUSTICE FRASER: All right. In that case, Mr Coyne, if
you can just find them and bring -- I don't know whether
it is easy to print them off. If you do find anything
bring it with you and we will deal with that first thing
but you won't necessarily be asked questions about it
because Mr de Garr Robinson will want to have a look at
it. Is that clear?
A. It is indeed.
MR JUSTICE FRASER: Anything else? 10.30 am tomorrow.
Thank you very much.
(4.36 pm)
(The court adjourned until 10.30 am on
Wednesday, 5th June 2019)
(10.30 am)
MR DE GARR ROBINSON: May it please your Lordship, can
I start by thanking you for kindly agreeing to adjourn
this hearing on Wednesday of next week to suit my
personal circumstances.
MR JUSTICE FRASER: Not at all.
MR DE GARR ROBINSON: I would also like to thank my learned
friend who I know has difficulties the following week as
a result. I would like that to be put on record because
I'm most grateful.
MR JUSTICE FRASER: Can I therefore check. We are sitting
four days this week, which is all going to be the
claimants' expert being cross-examined by you, and then
we are sitting the three days next week identified in
the email of yesterday?
MR DE GARR ROBINSON: My Lord, I believe so, yes.
MR JUSTICE FRASER: And that is Dr Worden being
cross-examined by Mr Green.
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: The dates, times etc for closings etc
remain unaffected?
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: Thank you.
MR DE GARR ROBINSON: My Lord, one other matter I should
mention. Dr Worden's supplemental statement, there has
been no substantive engagement between the experts on
that statement, as I understand it.
MR JUSTICE FRASER: Which statement?
MR DE GARR ROBINSON: I should say his report. The report
that was sent to your Lordship a couple of weeks ago if
your Lordship recalls.
MR JUSTICE FRASER: But you are not making an application?
MR DE GARR ROBINSON: I have no application to make and
I was going to tell your Lordship that.
MR JUSTICE FRASER: Well, that's one of the points which --
I know you suggested there was nothing unconventional
about what Dr Worden had done but his covering email
explained what application you would or would not be
making. I don't know if you have seen that covering
email, it is the one that was forwarded on by the court.
MR DE GARR ROBINSON: I am sure I have seen it, my Lord, but
not recently.
MR JUSTICE FRASER: All right. But the situation is you are
not making an application?
MR DE GARR ROBINSON: I have no application to make.
MR JUSTICE FRASER: Understood. Is there anything else that
you would like to mention?
MR DE GARR ROBINSON: My Lord, no.
MR JUSTICE FRASER: There is something I have to mention, it
has probably been somewhat lost in the mists of time
given the circumstances of March, but you were in the
middle of doing a privilege redaction review.
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: Has that now been completed?
MR DE GARR ROBINSON: My Lord, yes. There were documents
were sent over to Freeths a number of weeks ago now in
accordance with your Lordship's deadline.
MR JUSTICE FRASER: I wonder if at some point, not now, you
could just give me a summary -- I think there were 30
documents you were looking at?
MR DE GARR ROBINSON: I certainly will, my Lord.
MR JUSTICE FRASER: If you could just give me a summary of
the number of documents there were, which went into
which category and which were disclosed, I would be very
grateful.
MR DE GARR ROBINSON: I will have to go back and review --
MR JUSTICE FRASER: I expected you would so I was not
expecting an answer.
MR DE GARR ROBINSON: Would your Lordship like that done
this week? Perhaps we could do it on Friday after --
MR JUSTICE FRASER: It doesn't have to be this week.
MR DE GARR ROBINSON: I'm grateful.
MR JUSTICE FRASER: The best thing actually, why not if it
could just be distilled into writing and then maybe send
an email on Monday.
MR DE GARR ROBINSON: My Lord, yes, that will be done.
MR JUSTICE FRASER: Then the other thing, which is for both
the parties, not just you. I'm producing judgment
number 5, which is the written reasons for all of the
rulings last week. It is substantially finished. It
resolves the outstanding point on the order about
detailed assessment. I'm hoping to send it out by the
end of today, depending on how long it takes me finally
to proof read it. If it is not at the end of today it
will be first thing tomorrow morning, and then based on
what that says I will just expect the order to be
produced in due course.
I do not think there's any other housekeeping,
unless you have got some, Mr Green?
MR GREEN: My Lord, no.
MR JUSTICE FRASER: Thank you very much,
Mr de Garr Robinson.
MR DE GARR ROBINSON: My Lord, there is one minor change to
the agreed reports between the experts.
MR JUSTICE FRASER: The joint statements?
MR GREEN: The joint statements. It is at {D1/5/10}.
MR JUSTICE FRASER: Which joint statement is this?
MR GREEN: This is the fourth joint statement of
4th March 2019.
MR JUSTICE FRASER: Yes.
MR GREEN: In the light of considering some further
documents the experts have agreed that at 11.1, where it
says "evidence from several PEAKs indicates that
whenever Fujitsu", it should be "usually", qualifying
that. And we will produce for your Lordship a revised
statement of that sentence that the experts are happy
with that makes --
MR JUSTICE FRASER: They have both agreed that, though?
MR GREEN: They have agreed that orally this morning, so
that's why we don't have a written updated version, but
we will produce a written version reflecting that
agreement as soon as we can.
My Lord, may I now call Mr Coyne.
MR JUSTICE FRASER: Yes.
MR JASON PETER COYNE (sworn)
Examination-in-chief by MR GREEN
MR JUSTICE FRASER: Have a seat, Mr Coyne, if you would.
MR GREEN: My Lord, there are hard copies of Mr Coyne's
statement arriving shortly. He is without them at the
moment, for which we apologise, but they will be here
shortly. He says he is content to proceed with them on
the screen.
Mr Coyne, you have made two reports?
A. Yes, that is correct.
Q. The first appears to be at {D2/1/1}.
A. Yes.
Q. If we go to page {D2/1/167} of that report, is that your
signature?
A. Yes it is.
Q. And do you believe the contents of that statement to be
true?
A. Yes, I do.
Q. Then you served a revised version of your supplemental
report?
A. Yes, I did.
Q. And if we look at {D2/4.1/1}.
A. Yes.
Q. If we go please to page {D2/4.1/266} of that report, is
that your signature?
A. It is indeed.
Q. Does that reflect the corrections that you made on
6th March?
A. Yes.
Q. Subject to those corrections now included in that
revised report, do you believe the contents to be true?
A. I do.
Q. There are also four joint reports.
MR JUSTICE FRASER: Joint statements.
MR GREEN: Joint statements which are at {D1/1/1}.
A. Yes.
Q. And {D1/2/1}.
A. Yes.
Q. {D1/4/1}.
A. Yes.
Q. And {D1/5/1}.
A. Yes, that is correct.
Q. Could we just look, please, at {D1/5/10}, at 11.1.
A. Yes.
Q. You will have heard the qualification. Is that
something that you have spoken to Dr Worden about?
A. Yes, I have spoken to him this morning just before the
start of court.
Q. And what have you agreed about that?
A. We have agreed that the word "whenever" overstates the
position and it would be better to say "usually".
Q. I'm very grateful. Subject to that, is there anything
in those reports that you would wish to change or depart
from?
A. No.
MR GREEN: I'm most grateful. My Lord, I have no further
questions.
MR JUSTICE FRASER: Mr de Garr Robinson.
Cross-examination by MR DE GARR ROBINSON
MR DE GARR ROBINSON: Mr Coyne, if I could ask you first of
all to go to bundle {C1/1/1}. This is a list of the
Horizon Issues.
A. Yes, I have that.
Q. And the first set of issues are headed "Bugs, errors and
defects in Horizon", for shorthand I will call them
bugs.
A. Yes.
Q. I would like to look at four issues in particular.
Issue 1:
"To what extent was it possible or likely for bugs
..."
Of the nature alleged at certain paragraphs of the
generic particulars of claim and referred to in certain
paragraphs of the generic defence.
"... to have the potential to cause apparent or
alleged discrepancies or shortfalls relating to
subpostmaster's branch accounts or transactions, or ...
to undermine the reliability of Horizon accurately to
process and to record transactions as alleged ..."
In certain paragraphs of the generic particulars of
claim. You see the references there to the generic
pleading?
A. Yes.
Q. I take it you read the relevant paragraphs of those
pleadings?
A. Yes.
Q. Issue 3, if we could just read that:
"To what extent and in what respects is the Horizon
System 'robust' and extremely unlikely to be the cause
of shortfalls in branches."
At the bottom there's more references to paragraphs
of the generic pleadings. You have read those as well,
haven't you?
A. Yes, I have.
Q. In order to clarify what these issues actually mean?
A. In order to provide the context of the questions.
Q. I'm grateful. Then paragraph (4):
"To what extent has there been potential for errors
in data recorded within Horizon to arise in (a) data
entry, (b) transfer or (c) processing of data in
Horizon?"
Then over the page at {C1/1/2} at issue (6):
"To what extent did measures and/or controls that
existed in Horizon prevent, detect, identify, report or
reduce to an extremely low level the risk of ..."
Those specified errors.
I am sure you will agree the issues I have just read
out are very important issues for the purposes of these
proceedings?
A. Yes.
Q. Would you agree they are probably the most important
issues for the purposes of these proceedings?
A. I'm not sure whether they are the most important but
they certainly are --
Q. They are of central importance, yes?
A. Yes.
Q. I would like to look briefly at the context that you had
regard to when interpreting them. Could I ask you now
to go to the pleadings bundle which is at bundle
{C3/1/1}. These are the generic particulars of claim
and I'm going to go to a few paragraphs. First of all,
page {C3/1/8} of that document and paragraphs 22 to 24.
22:
"Prior to disclosure and expert evidence, the
Claimants are unable to provide detailed particulars of
bugs, errors or defects which were or may have been the
cause of any discrepancies or alleged shortfalls
attributed to them by the Defendant, but will be able to
plead further thereto following disclosure or the
provision of information relating thereto by the
Defendant."
Skipping over the last sentence.
Then 23:
"However, the Claimants aver that there were a large
number of software coding errors, bugs or defects which
required fixes to be developed and implemented. There
were also data or data packet errors. There was a
frequent need for Fujitsu to rebuild branch transaction
data from backups, giving rise to the further risk of
error being introduced into the branch transaction
records."
Then there is a reference to the known error log.
A. Yes.
Q. Then various specific things are relied upon in
paragraph 24 which I would ask you to just cast your eye
down so you can see what's being alleged.
A. Mm. Yes.
Q. {C3/1/9}.
So the context in which these issues arise is the
overall question as to whether the problems which are
alleged here affected the claimants, created false
shortfalls in their accounts, would you agree with that?
A. Yes.
Q. If we can then move on to the generic defence and that's
at bundle {C3/3/1}, behind divider 3.
If we can go to page {C3/3/21}. Picking it up at
paragraph 50, just in the individual subparagraphs, this
is what the defendants are saying in response to that
case. (1):
"All IT systems experience software coding errors or
bugs which require fixes to be developed and
implemented. As is noted in paragraph 53 and 54 below,
there are robust measures in place in Horizon for their
detection, correction and remediation."
MR JUSTICE FRASER: I think we jumped forward a bit quickly.
Can we go back a page, please.
MR DE GARR ROBINSON: It's paragraph 50.
MR JUSTICE FRASER: Yes, you were reading from 50 and
page 21 was on the screen, but before you finished
reading we jumped to page 22.
MR DE GARR ROBINSON: I see. I'm grateful, my Lord. I'm
sorry, I was not looking.
MR JUSTICE FRASER: That's understood, don't worry.
MR DE GARR ROBINSON: Then it says at (2):
"All IT systems involving the transmission of data
over the internet experience data or data packet errors
during transmission and such systems routinely have
protective measures in place to prevent such errors
creating any difference between the data transmitted and
the data received and retained by the recipient.
Horizon has robust controls making it extremely unlikely
that transaction data input in a branch would be
corrupted when being transferred to, and stored in, Post
Office's data centre in a manner that would not be
detected and remedied."
So you will see that here Post Office are not saying
it didn't happen, but what they are saying is there is
a risk of it happening but for each occasion on which
data is input it is extremely unlikely that the data
would be corrupted. Do you see that?
A. I do see that.
Q. If we could move on to page {C3/3/22} now, picking it up
at paragraph 53, it is said:
" ... it is a truism that errors or bugs in an IT
system and data or data packet errors have the potential
to create errors in the data held in that system.
However Horizon has at all material times included
technical control measures to reduce to an extremely low
level the risk of an error in the transmission,
replication and storage of the transaction record data.
Then there is a list of the current measures.
Again, not saying that it didn't happen. There is
a risk but in any given case that risk is extremely low,
do you see that?
A. I can see the words that are being said, yes.
Q. So robustness -- we can move on to paragraph 54 on page
{C3/3/23}:
" ... in addition to the technical controls referred
to above, there are several operational procedures and
practices conducted by Post Office and subpostmasters
that serve to increase the reliability of the data
stored in the central data centre as an accurate record
of the transactions entered on branch terminals.
Then there is a list of those. So you will see that
what's being said is that the robustness of the system
is said to be affected both by technical measures and by
human measures. Do you see that?
A. Yes, I can see that. That is what is said, yes.
Q. Then over the page at {C3/3/24} and paragraph 55:
" ... Post Office admits that, like all other IT
systems, Horizon is not a perfect system which has never
had any errors or bugs. However, as indicated in
paragraphs 53 and 54 above, it has robust systems in
place to identify them, fix them and correct their
consequences (if any)."
Were those the paragraphs that you looked at when
you were deciding how to interpret the issues that we
have read to each other?
A. Yes.
Q. Before you decided on the approach that you would adopt,
you reflected on the issues thrown up by those
paragraphs, yes?
A. Yes.
Q. They made it clear, didn't they, that the critical issue
was not whether it is possible or likely for bugs to
have the potential to cause false shortfalls in branch
accounts over the life of Horizon, that was in fact
admitted. The critical issues were the extent to which
it was and is likely that bugs cause false shortfalls in
any given set of accounts. Do you accept that?
A. I don't really see the difference between the two
statements that were made there. The question was
whether it was possible or likely and the extent to
which it was possible or likely.
Q. Well, let's -- given that it is admitted that there have
been occasions where these things have happened in the
life of Horizon, to ask is it possible or likely that it
has happened within the life of Horizon is a rather
uninteresting question, isn't it?
A. But along with the statement that defects were
acknowledged was a statement that there has only been
three defects.
Q. No, Mr Coyne. I ask you to address my question, please.
If you could focus on what I asked you.
Given it was admitted that there have been occasions
when these things have happened, to answer the question:
is it possible or likely that these things have happened
by saying "yes" isn't a very helpful answer, is it?
A. If the answer was just simply "yes" and with no further
context then I agree that that would ...
Q. The critical issue that's raised by the four issues that
I have read to you is the word "extent" and the word
"likelihood" and the word "risk", isn't it?
A. They certainly are critical elements that would need to
be reviewed, yes.
Q. What you were being asked to do by those issues is to
give your opinion to the court on the extent to which it
was and is likely that bugs caused shortfalls in any
particular set of accounts?
A. Well, across the accounts of subpostmasters, yes.
Q. The extent to which various measures in place reduce
the risk faced by any given subpostmaster when doing
a transaction or when producing a particular set of
accounts, yes?
A. Yes.
Q. That's the critical question on which the court needs
your assistance, would you agree?
A. It is one of the critical questions, yes.
Q. Could you identify a question which you say is more
critical than that in the context of these proceedings?
A. I haven't taken a view on -- I have taken each of the
Horizon Issues that's put to me with equal priority.
I haven't set out to decide a prioritisation for them.
Q. Very well, I understand. But you will then at least
agree with me this far: that in asking you to give your
opinion on the issues we have read, what you're being
asked to do is to advise the court on the extent to
which problems had happened -- or problems were likely
to happen in relation to any given situation, the extent
to which those problems were likely to have been caught
by countermeasures in that situation, and the overall
question of the extent to which Horizon is robust and
extremely likely to be the cause of a shortfall in that
given situation. Would you agree with that?
A. I don't believe countermeasures were asked in the
questions, but broadly with what you say, yes --
Q. When I say countermeasures, you can take me as saying
controls and measures if you want --
A. Okay.
Q. Subject to that correction, you would agree that that
was the essential question being raised in the four
issues that I read to you?
A. I believe they were the questions that were asked and
I believe that in providing my reports I seek to address
those. I have answered those.
Q. Very good. So you accept these issues are of practical
importance to this trial?
A. Yes.
Q. So that the court's judgment in this case, when the time
comes later on to try claims by particular claimants,
that the accounting shortfalls which he or she is
complaining about was actually caused by Horizon?
A. Sorry, could you put that question again, please.
Q. I may have made a grammatical mistake in my question.
MR JUSTICE FRASER: I'm not really sure that's a question
for the expert anyway.
MR DE GARR ROBINSON: Very good. But important questions
are: when a claimant did a transaction in the period
covered by a particular set of accounts containing
a disputed shortfall, what risk did he face of Horizon
creating errors in the recording, transmission or
storage of records of those transactions, and how likely
was it that Horizon calls a false shortfall in those
accounts? Would you agree that those questions, the
Horizon Issues I have read to you, the resolution of
those issues will be of assistance in deciding that
question?
A. No, I don't believe that that's something that we have
been asked to calculate or assess.
Q. No, I'm not asking you, Mr Coyne -- I'm not suggesting
to you that you have been asked to decide on whether any
particular claimants' claim is right or not, what I'm
suggesting to you is that the context in which these --
given the context in which these issues arose -- were
drafted, and given the pleadings by reference to which
they were drafted, it was obvious that the purpose of
those issues was to assist the court so that it could
use the judgment that will be produced in this trial as
a basis for making ultimate decisions in ultimate breach
claims by claimants?
A. In a later trial?
Q. Yes.
A. Yes, I was aware of that.
Q. Isn't that the main reason why we are here?
A. Well, it is certainly a reason why we are here, yes.
Q. To enable the court to make useful findings as to the
general likelihood of any transaction being wrongly
recorded in a particular case?
A. Yes.
Q. And as to the general likelihood of any particular set
of accounts containing false shortfalls because of
Horizon?
A. I don't believe either of those two statements appeared
in the issues for the technical experts, did they?
Q. So you don't believe that when you were asked in the
Horizon Issues -- let me see if I can find them -- at
{C1/1/1}:
"To what extent was it possible or likely for bugs
[and so on] to have the potential to cause apparent or
alleged discrepancies or shortfalls?"
A. Yes.
Q. You don't think you were being asked to give an opinion
as to the extent of the likelihood of that happening in
any given case?
A. Yes, but I have not placed any focus on any particular
branch accounts for any subpostmasters or claimants.
Q. But you weren't focused on the entire life of Horizon.
There are two ways that one could construe the question
"To what extent was it possible or likely for bugs" and
so on. One is was it possible or likely for bugs to
have done that thing at any time during the life of
Horizon? And I think we have already agreed that the
answer to that is trivial because we know it already.
A. Sorry, I do not think I agree with that. I think
because we are talking of situations, bugs, errors and
defects that occurred throughout the life of Horizon,
you do have to look at the entire life cycle from when
it was first used by subpostmasters all the way up to
the near current date.
Q. I'm not suggesting that you don't have to look at
whether bugs have arisen during the life of Horizon.
What I'm suggesting to you is the practical purpose of
these issues, the question you are being asked, is to
what extent is it likely that those bugs are going to
cause shortfalls in any given case?
A. Yes, I agree with that.
Q. Thank you. So the focus was not just on the number of
bugs in Horizon, the focus was on bugs which were likely
to affect branch accounts, yes?
A. Yes.
Q. And also on whether there were -- I'm going to call them
countermeasures, I hope that's not controversial, which
were likely to catch and correct their impact, yes?
A. Yes.
Q. Either straightaway or after some time?
A. Typically if you were going to design a countermeasure
you would want to have it finding the defect
straightaway. If it was only going to find it some time
later it would be of limited value.
MR JUSTICE FRASER: Mr de Garr Robinson, just on your term
"countermeasures".
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: I understood that to have a specific
meaning within the IT world. Are you using it in
a technical meaning or are you using it as a generic
term?
MR DE GARR ROBINSON: I'm using it as a term to refer to
measures and controls in existence, both technical and
human measures and controls, which are designed or have
the effect of identifying bugs in Horizon or identifying
the consequences of bugs in Horizon so that they can be
looked at and fixed.
MR JUSTICE FRASER: So that's the way counsel is going to
use the term "countermeasures".
A. Thank you, my Lord.
MR DE GARR ROBINSON: And this trial, the issues that we
have been discussing, are designed to enable the experts
to grapple with that question, yes? Whether
countermeasures are likely to catch and correct the
impact of any bugs that may have arisen in the system?
A. Agreed.
Q. And countermeasures aren't limited to a process of
stopping the bug as soon as it appears. Sometimes
countermeasures only catch the symptoms or the
consequences of a bug, you would accept that, wouldn't
you?
A. Typically countermeasures are a design aspiration. When
you sit down to design a computer system and what
controls are in place, then you would consider what
controls are required, where to position them. Should
it be the case that the controls fail do you have
a human aspect doing a secondary check? It could go all
the way to employing auditors to re-check figures. You
could go on ad infinitum with --
Q. So you accept, then, that these controls or
countermeasures, whatever you would like to call them,
some of them have the effect of stopping the bug in its
track. Well, some of them have the effect of stopping a
bug arising in the first place?
A. The best way of dealing with defects is to ensure that
they don't arise in the first place --
Q. And --
A. Sorry, if you would allow me to just finish.
Q. Sorry.
A. So what you would do is throughout your testing regimes
you would flush out all of the defects at that point in
time, and you would put in controls to ensure that if
a defect does trigger, there is a way that you are
alerted to it and it is fixed as soon as possible.
Q. Yes. So one of the objectives is to have a testing
regime which picks up as many bugs as possible?
A. Indeed.
Q. But I think you accept that it is impossible to have
a testing regime that picks up all of them?
A. It is impossible to have a testing regime that picks up
all of them and that is why you have a process of
categorisation and you would allow a product to go live
if -- one of the terms that's used -- all severity 1 and
severity 2 defects have been dealt with. Severity 3
might be largely cosmetic and you would allow a system
to go live with cosmetic defects.
Q. That's not quite what I'm asking you. You accept there
are going to be bugs which arise in the operation of any
IT system which would simply evade any test that any
human being can devise?
A. There's always going to be bugs that are unknown to be
within the system at the point that you go live and they
are only discovered sometimes weeks afterwards,
sometimes years afterwards.
Q. Yes. And then -- so that's one line of defence. Then
there are other lines of defence when a bug does arise
because it hasn't been tested for. There are principles
such as defensive programming, and so on, which try and
stop the bug working its way through the system and
producing false results at the end?
A. Defensive programming is typically to stop the bug ever
occurring. It isn't the trigger mechanism; what happens
after the bug is triggered.
Q. Isn't one of the aspects of defensive programming having
little units of programmes which require data to be
transferred from one programme to another programme to
another programme, and every time you transfer from one
unit to the other checks are done to ensure the figures
are correct?
A. Yes, to ensure that a bug doesn't have an impact on the
user --
Q. So one aspect of defensive programming is not just to
prevent bugs, it is to ensure the bug doesn't
propagate --
A. Yes.
Q. -- when it manifests itself and produces an error in
data, yes?
A. Yes.
Q. So we have one set of measures which is designed to
prevent bugs occurring in the first place?
A. Mm.
Q. We have other measures which are designed to prevent
bugs causing problems in data at the user end, if I can
put it that way. But again no system of countermeasures
is perfect, there will always be bugs that get through?
A. Agreed. I completely agree.
Q. Then you have other measures which are designed to pick
up the problems which have been caused at the user end.
You have measures that are designed to assist in the
identification of the fact that there is a problem in
data such that it needs to be looked at and corrected if
it is wrong?
A. Yes, that's typically called impact management. So you
understand what the impact might be of the bug that you
have seen and you work out a process of resolving the
consequential impact of it.
Q. That's precisely -- and of course I'm reminded that many
of those countermeasures also pick up and correct for
user errors as well. They can sometimes have a dual
effect?
A. Yes, you can design the software in such a way that it
reduces the amount of user errors or the potential user
errors. One such that's appeared in this case is about
double typing high value cheques when they are remmed
into the system. So if it is a high value, potentially
you should type that in twice rather than in once.
Q. All of these things that we have just been discussing in
general terms is precisely what the concept of
robustness addresses, isn't it, in the IT industry?
A. Yes.
Q. That a system isn't merely reliable when conditions are
normal -- I think the technical term is nominal -- but
robust performance takes over if and when conditions are
abnormal or off nominal, when they become adverse and
errors are created, yes?
A. Yes, you would want a system to fail safely. So it is
acknowledged that there could be failures and it is what
you do when that system fails.
Q. So robustness is the very concept which underlies the
issues we have been discussing for the last half hour,
yes?
A. Yes.
Q. I think that's what you agree in the joint statement.
If we could go to the joint statement, it is at bundle
{D1/1/8}. This is paragraph 2.3, issue 3.
This is what you agreed with Dr Worden before any
reports were produced at all:
"There are different dimensions of robustness, such
as robustness against hardware failure, software defects
and user error. The robustness of the system also
depends on the process around it.
"Robustness does not mean perfection; but that the
consequences of imperfection must be managed
appropriately."
A. Yes.
Q. Then over the page {D1/1/9} you suggest your own
interpretation of robustness. You say:
" ... I have applied the following definition of
robustness:
"'... namely, the ability of a system to perform
correctly in any scenario, including where invalid
inputs are introduced, with effective error handling.'"
So those adverse conditions you refer to, that would
include bugs in the system itself, wouldn't it?
A. Yes.
Q. So withstanding them would include managing their
consequences appropriately, yes?
A. Yes.
Q. I think that's also what you say in the third joint
statement that was agreed some time later. That is at
{D1/4/2}. It is the second paragraph:
"From our experience of ..."
A. Sorry, I do not think I have the right --
Q. Sorry, it is {D1/4/2}.
MR JUSTICE FRASER: If you look at your screen you will see
what the witness is seeing.
MR DE GARR ROBINSON: It is {D1/4/2}.
At the bottom of the page, do you see that?
A. Yes, I do.
Q. Second sentence of the second paragraph:
"We agree that 'robust' does not mean infallible and
therefore horizon has and will continue to suffer
faults. Robustness limits the impact of those faults
and other adverse events."
A. Yes.
Q. If you go over the page to page {D1/4/4},
paragraph 3.13, third box down, third row down:
"Countermeasures work by limiting the impact of
Horizon bugs/error and defects on branch accounts.
"Countermeasures do not always eliminate the effects
of adverse events (they are not perfect) ..."
Do you agree with that?
A. Yes.
Q. No IT system ever has perfect countermeasures?
A. I agree.
Q. But they are often effective in the area where they are
deployed and that's why they become basic elements of
practical IT system design.
Incidentally, isn't that what Dr Worden was saying
in the first statement? If we can go back to the first
joint statement, which is {D1/1/9}, and look at page 9
of that statement, you will see in the bottom half of
the screen we have got Dr Worden's comments?
A. Yes.
Q. So you indicate your definition of robustness in the
top, and below he says:
"The definition of 'robust' proposed above by
Mr Coyne is not adequate for reasons given below. The
term 'robust' is not, as implied in para 3.1 of the
outline, either ill-defined or a piece of IT public
relations. Robustness (which is closely related to
resilience) is an engineering objective, and large parts
of project budgets are devoted to achieving it."
Would you agree with that?
A. Yes.
Q. "it receives its meaning in the phrase 'robust against
... [some risk or threat]' ..."
Would you agree with that?
A. Yes.
Q. "... and there are a large number of risks that business
IT systems need to be robust against - such as hardware
failures ..."
If we go over the page {D1/1/10}:
" ... communications failures, power cuts,
disasters, user errors or fraud. These are the
dimensions of robustness."
Do you agree with that?
A. Yes.
Q. "In all these dimensions, robustness does not mean 'be
perfect'; it means 'address the risks of being
imperfect'. The extent of robustness is to be
interpreted as: in how many dimensions was Horizon
robust? And: in each dimension, how large were the
remaining risks?"
Would you agree with that?
A. That is a little imprecise. I don't really understand
the reference to the dimensions. The dimensions aren't
defined.
Q. He talks about dimensions before, hardware failures --
he talks about different risks that are faced by the
system. But let's not get bogged down in dimensions.
The critical question, could I suggest to you, in
robustness, is how large were the remaining risks after
the relevant countermeasures have been brought into
account? Would you agree with that?
A. I'm not sure I would agree that that is the critical
question, no. Certainly consideration does need to be
given to how, once a risk triggers, it is dealt with.
Q. Let me get this straight. You are not agreeing that
when determining whether a system, an IT system, is
robust that a critical question is how large are
the risks remaining after you have had regard to the
countermeasures?
A. But when you talk about how large are the risks, that
quantification is something which I don't believe is
possible. It is not possible to quantify risk in such
a way.
Q. So please explain, Mr Coyne, how else would you grapple
with the question of robustness? You agree robustness
is an important concept?
A. Yes.
Q. You agree it is deployed very frequently in the IT
industry?
A. Yes.
Q. It is a subject of academic study, isn't it?
A. Yes.
Q. And there are university courses on it?
A. There may well be, I'm not aware --
Q. Yes. So given that, and given that the essential
purpose of this doctrine is to determine how well
systems guard against problems caused by -- it could be
human error, it could be bugs, it could be any adverse
conditions of that sort, isn't it obvious, doesn't it
follow as night follows day, that the ultimate question
being wrestled with by the concept of robustness is how
well are the risks faced by a system guarded against?
In other words, what are the risks remaining after you
have taken the countermeasures into account?
A. Yes, and typically that analysis, what you talk about
there, is conducted in the design process of a system at
the start where you would then go to build a system
after you are satisfied that it has all the design
characteristics and control measures in place. So it
achieves a particular level that you are satisfied with
and then you go to actually build the solution. That is
different to then having a system already in place and
looking back at it and trying to decide what its level
of robustness was.
Q. Sorry, I'm not sure I understand what you are saying,
Mr Coyne. Are you saying it is possible to define
a level of robustness that you want when you are
designing the system, but it is not possible to decide
whether you have achieved that level after you've built
it?
A. What I'm saying is when you set out to build the system
and you have the design processes and control measures,
you will take a view whether you are satisfied with the
level of risk within that build. It won't have
a number, it will be subjective, but there will be
a decision on whether there is the appropriate level of
robustness -- to use the phrase that we have termed
here -- and if everybody is agreed at that level of
robustness the build will then take place. But it is
a subjective view.
Q. I'm interested in your suggestion, Mr Coyne, that there
won't be any numbers. You say you are a qualified
PRINCE2 practitioner, yes?
A. Yes.
Q. I have seen a number of PRINCE2 documents and we can
take you to some tomorrow if this is necessary. But
what those documents make very clear is that in engaging
in the process of determining the risks -- managing the
risks to a given project, you do absolutely attach
numbers to particular risks. You have graphs, don't
you? You draw graphs up and you assign values to
particular risks?
A. Typically each risk will have an impact rating, often
between 0 and 5 but it could be different, and there is
a likelihood of that impacting, and typically you would
multiply one by the other and that gives you a number.
But that is far too simplistic for something at the
level of analysis that we are talking about here.
Q. What I'm suggesting to you, Mr Coyne, is previously you
said when you are designing a system and considering
the risk to which you wanted the system to be subject,
the level of risk which you were willing to accept, it
is a subjective view and it doesn't involve any numbers.
What I'm suggesting to you, Mr Coyne, is it absolutely
does involve numbers and indeed the very process you
have just described shows it involving numbers. Would
you agree with that?
A. Right, so what I set out with regard to numbers there
was a very simplistic way of measuring isolated aspects
of risk. That is used in PRINCE2. But you can't
necessarily scale up looking at simple isolated risks
and apply it to something retrospectively such as
Horizon.
Q. You are raising two completely different issues. The
first issue is the retrospective issue.
A. Right.
Q. And the second issue is numbers.
A. Right.
Q. You accept, don't you, that when designing a system you
absolutely do -- you identify the risks which you want
the system to guard against and you assign numbers to
the adequacy of the risk to which the system is guarding
against and you perform calculations with a view to
ascertaining whether the system is the kind of system
that you want, yes?
A. No, not really. When you are talking about
countermeasures or controls, they will be functional
design aspects and there will typically be a list of all
functional design aspects and nonfunctional design
aspects.
The types of risks that are monitored through the
design and implementation of an IT project are often
things such as delay, the lack of resources, the failure
of hardware, and it is those aspects that you would
attribute a numerical value, and then also a numerical
value which is likelihood.
So yes, they are both in the same area as risk, but
one is dealing with very simplistic isolated elements
and the other is dealing with the whole operation of
a system.
Q. You have not said that before, so far as I'm aware. You
are suggesting that PRINCE2 can only involve simple
calculations, it doesn't involve quite elaborate
calculations involving a wide array of risk
calculations?
A. I'm not saying that you can't make PRINCE2 more
elaborate. Typically in my experience people will
measure risk within PRINCE2 between 0 and 5, so it is
not very granular, but there's nothing to say that you
couldn't have some more scientific way of doing it.
Q. Are you suggesting that doesn't happen in practice,
because I would like to suggest to you it does, and
there are quite complex projects where there are quite
elaborate calculations made. Would you accept that?
A. I am sure there may well be.
Q. Just to very briefly talk about your retrospective
point. If you are brought in to design a new element to
a system, around an existing system, and you are being
asked to calculate the risk of that system not operating
accurately in collaboration with the legacy system
that's remaining, when performing that calculation risk
you will look at the past, won't you? You will look at
how the legacy system operates, how reliable the outputs
are out of the legacy system, all sorts of things which
are based upon looking back at retrospective events,
yes?
A. Yes, I would certainly look if there has been any
historic defects within the system, because in the
example that you are giving it is a system that is
already live that is being extended. So I would look
back at how the good the testing has been in the past,
whether bugs have escaped the testing. I would look at
how many defects there have been in any given launch or
release. I would look at how, when bugs are detected,
they are dealt with, what the mitigation of those would
be. There would be an assemblance of all of those
elements.
Q. Thank you. So what you describe as an assemblance would
be an examination, a review, of the data relating to
past performance --
A. Yes.
Q. -- with a view to ascertaining a risk that you face in
the future, yes?
A. It would certainly help set the context. It might well
be that simply looking backwards at a project, whilst it
might be helpful to set the context, it might well be
that the way you are going to approach this new project
is different, it might have different vendors that are
involved, the supporting mechanism might be completely
different, which might be a factor in the scenario. We
have Atos that were involved in certain aspects of the
support, we have Fujitsu that were involved in certain
aspects of the support. Post Office had a supporting
function and had a reconciliation function but that may
well have changed over time. So it is important to
understand what processes were in place at various
times.
Q. Yes. And forming that understanding, can I suggest to
you that invariably you will be looking at past
performance?
A. Yes.
Q. Even, for example, if you are bringing in a new system
that is designed to be compatible with an old system, as
well as looking at the data relating to the past
performance of the old system you would also look at the
data relating to the past performance of the new system
if there is some, wouldn't you?
A. Yes.
Q. So quite often when performing what you describe as
a prospective risk analysis you would often look at
historical data and look at historical performance,
wouldn't you?
A. Yes.
Q. I'm very grateful.
If we can now move to bundle {D3/1/11}, this is
Dr Worden's first report. If I could pick it up at the
bottom of the page, paragraph 48. Dr Worden has just
listed three of the issues that we have just been
discussing, and then he says in paragraph 48:
"In my opinion the most important of these is
issue 3 which encompasses a large and mature area of
modern IT practice."
Do I apprehend from the evidence you have already
given that in your opinion the most important of these
issues is not issue 3?
A. I don't see any as being more important than others.
Q. You would accept, though, that issue 3 encompasses
a large and mature area of IT practice, yes?
A. Yes.
Q. Then he says:
"Nearly all business IT systems need to be robust --
as the business depends on them -- and there is a large,
mature and tested set of techniques for achieving
robustness."
Do you agree with that?
A. Yes. It is a generic statement of the industry rather
than this particular project.
Q. Then if we could move on to page {D3/1/96} in the same
document. Paragraph 374. He sets out there -- I will
ask you to read it to yourself -- points that are
similar to the points that we have seen in the first
joint statement and he adds at the end:
"Robustness of IT systems is a large and mature
topic."
And I understand you would agree with that?
A. Yes.
Q. So what this all shows is that there is a well
understood body of practice and learning on how to
achieve robustness, yes?
A. Say that again, please?
Q. There is a well understood body of practice and learning
on how to achieve robustness in IT systems?
A. Yes.
Q. And it is possible to benchmark a system by reference to
other comparable systems which are deployed in the
industry, yes?
A. Yes.
Q. You say that I think in your first statement. Perhaps
if we could look at that {D2/4.1/154}, page 154,
paragraph 5.99.
A. Yes.
Q. I'm so sorry, Mr Coyne, I'm taking you all over the shop
and I'm wasting time. I do apologise. Something seems
to have gone wrong in my note. Could you give me one
moment?
A. Certainly. (Pause)
Q. Yes, let's go back to {D2/1/78}. It is page 78, as
I indicated first of all, and it is paragraph 5.91. You
say -- this is in your first report:
"... I have estimated the likely level of the
robustness of Horizon and benchmarked this against
industry standards based upon a review of the evidence
available including the known error log (KEL) and PEAK
system."
A. Yes.
Q. Your conclusion, having done that benchmarking exercise,
is that Horizon is robust, isn't it?
A. I said relatively robust, yes.
Q. Well, let's look at exactly what you say. It is bundle
{D1/4/2}, paragraph 3.1:
"From our experience of other computer systems ..."
This is at the top of the second paragraph.
A. Yes.
Q. "... Horizon is relatively robust."
A. Yes.
Q. So that's your considered view about the Horizon system,
isn't it?
A. Yes.
Q. On the basis of all the evidence that you had seen?
A. Yes. In my first report I stated that. In my second
report I explained that in light of the evidence that
had been disclosed between the two reports, that the
Horizon system wasn't as robust as I initially
considered.
Q. But your overall view after that process is set out in
the passage we just looked at?
A. It is.
Q. Your view is Horizon is relatively robust, and I would
like to explore that if I could. We agreed that
robustness is a mature topic which is the subject of
academic study and has well understood practical
principles. Would you agree?
A. Yes.
Q. You say that from your experience of other systems
Horizon is relatively robust. So you are saying that
you have considered comparable systems of which you have
experience?
A. Yes.
Q. These are systems with which you are familiar?
A. Indeed, yes.
Q. And you have compared Horizon's robustness with the
robustness of those systems, and compared with those
systems Horizon is relatively robust, yes?
A. Yes.
Q. Let me tell you what I think you are saying here and you
will correct me if I'm wrong. It has a good level of
robustness compared with some systems?
A. Yes.
Q. In the top quartile, maybe?
A. That's not the assessment that I believe I have made.
Q. Certainly in the top half, would you agree?
A. No, I don't think I want to be drawn on where it is
positioned.
Q. If I were to tell you that when I go running I'm
relatively fast, isn't that what I would be saying, that
I'm faster than most people I run with?
A. Yes. For context, I have worked on a number of systems
in banking, in manufacturing, and Horizon compares well
with those systems, with the context that we have added
here that it is not free from defects and the impact of
those defects is very important to consider but it is
relatively robust --
Q. So --
A. Sorry, if I could just add to that.
Q. Yes, of course.
A. What must be considered is that we are talking about
Horizon as it is today and the processes that are in
place today, but there has been quite a long journey to
get to the position that it is now --
Q. I'm going to ask you about that later, Mr Coyne, but you
have made the point and I understand it.
A. Okay.
Q. If I understand you correctly, you are saying that if
there is a spectrum of comparable systems in the world
today from 0 to 10, Horizon is towards the upper end.
You don't want to commit yourself to whether it is in
the top 25%, but it is towards the 10 end rather than
the 0 end?
A. Yes, but what you have got to take care -- and
benchmarking against systems that are in similar
contexts. There are of course systems that are dealing
with flag management, I used to work at
British Aerospace looking at their military aircraft
programming. There was a whole different spectrum of
robustness that was required there. I have also worked
in health care where you are talking about life critical
systems. Horizon doesn't compare anywhere near those
sorts of systems. What I'm talking about is Horizon
when benchmarked against systems of similar context, so
in retail, in banking and things like that.
Q. And these are systems where there are large numbers of
users?
A. Yes.
Q. And a great complexity in the transactions being
performed?
A. Yes.
Q. And these are systems which have measures and controls
to achieve robustness?
A. That aim to achieve robustness, yes.
Q. Could you just define which sectors you are talking
about as being comparable for the purposes of your
judgment?
A. So point of sale systems, something where a transaction
is taking place over the counter, is certainly
comparable. Banking has elements of it, where automated
payments are being transferred to different
organisations, so that is certainly comparable as well.
Stock control, things like that.
Q. And you are talking about large businesses with lots of
users and lots of complexity, yes?
A. Yes, at a similar scale to what is seen in Horizon.
Q. We have been talking about measures and controls or
countermeasures. Dr Worden identified 18 of them in his
first report, didn't he?
A. Mm.
Q. You will see that he says he has defined three letter
acronyms for each of them?
A. Yes.
Q. And he says specifically, doesn't he, that most of these
acronyms are not commonly used in the industry?
A. I think he did say that, yes.
Q. But in your second report you suggest that he might be
giving the false impression that they are commonly used
in the industry, do you remember saying that?
A. Certainly in my report I point out that they aren't used
in the industry.
Q. Let's see what you actually say. It is at bundle
{D2/4.1/141}, paragraph 5.67:
"Whilst Dr Worden acknowledges these acronyms are
mostly not used in industry, he has used them throughout
his report which gives the impression they have a
standard meaning and scope."
Do you think that is a fair thing to say? He
specifically says in his report that they don't.
A. Well, I actually think that if you do attribute acronyms
to things there is a danger that the perception is that
they are read as being industry standard terms.
Q. Even in circumstances where he specifically says that's
not the case on at least two occasions in his report?
I mean why did you feel it necessary to say that,
Mr Coyne, in circumstances where Dr Worden had already
made it clear?
A. I'm not sure.
Q. Anyway, you go through his 18 countermeasures over the
page {D2/4.1/142}, there is a table that goes on, and it
is fair to say you disagree whether one of them is
a standard countermeasure, you disagree that manual
workarounds count as a standard countermeasure?
A. Yes, I don't believe that a manual workaround can be
a --
Q. But for others you accept they are typical standard
features in IT systems design?
A. Yes, certainly when you are designing a system, some of
the aspirational features that you want to build in
there would be contained within these controls.
Q. Mr Coyne, you have used the word "aspirational" several
times now and I'm just wondering whether you're trying
to drum home a theme. If you want to make a point about
aspiration I don't want to stop you making it, but you
do throw this word in quite a lot to your answers and
I'm wondering what point you're trying to make by the
word "aspiration"?
A. I'm just pointing out that when you start a project and
you build something, you go through a workshop typically
of setting out what your design aspirations are. That
is just a term I use when I'm conducting workshops with
people going through things.
Q. I see. Your judgment on the robustness of the Horizon
system takes all these countermeasures that Dr Worden
refers to -- with the possible exception of
workarounds -- into account, doesn't it?
A. Yes.
Q. It is necessary to do so to form a judgment on
robustness, isn't it?
A. I'm not sure that you need to consider -- I'm not sure
that you need to decide that this is the subset of
factors that you are going to consider before you come
up with a calculation of robustness.
Q. Mr Coyne, didn't one of the Horizon Issues specifically
ask you to consider whether controls and measures in
Horizon reduced certain problems to an extremely low
level of risk?
A. It certainly -- yes, it did.
Q. And I think we have established, it has taken an hour to
do this, I think we have already established that when
making an overall judgment of robustness you both
consider how the system operates and then consider how
the countermeasures designed into the system also
operate?
A. Yes.
Q. With a view to coming to an overall decision as to the
robustness of the system?
A. Yes.
Q. Thank you. So in forming a judgment about robustness it
is necessary to form a judgment also, isn't it, about
how the countermeasures, what Dr Worden calls the
robustness countermeasures, how those countermeasures
were designed and operated in practice, yes?
A. Yes.
Q. A key element of robustness is considering how a system
limits the extent of the impact in this context, in the
context of the Horizon system, and the issues we are
considering, namely, does Horizon cause shortfalls in
branch accounts. A key element of the robustness
question is considering how the Horizon system limits
the extent of any impact on branch accounts, yes?
A. Yes. Certainly you need to establish what the impact is
and, secondly, you should take steps that if it is ever
going to happen again you reduce that impact.
Q. So in forming a judgment on robustness you have first of
all to see -- let's take a bug. A bug happens and the
first question you ask is: did that bug or could that
bug have had an impact on branch accounts?
A. Yes.
Q. Then you form a judgment on whether and to what extent
the countermeasures in place in the Horizon system would
have enabled that impact to be identified and fixed,
yes?
A. No, I think you would look at whether it did -- whether
any countermeasure or control did prevent it from having
a consequential impact, not whether it should have.
Q. Well, whether it would have?
A. Or whether it would have.
Q. You don't consider whether it would have, you consider
whether it did?
A. Well, if it did there would be evidence that it did. It
would be documented.
Q. But in some cases it will be blindingly obvious, won't
it, Mr Coyne? Let me give you an example, remming.
A Post Office example. Remming in and remming out.
Money is sent from Chesterfield to a branch.
Chesterfield has a record of the money it sends over.
The branch receives the money and types in -- or usually
it is a barcode actually, but it records on the Horizon
system how much money the branch has received.
You are aware, aren't you, that there is
an automatic system that checks Chesterfield's figures
against the branch's figures?
A. Yes.
Q. Are you suggesting that every time over the last
20 years there has been a discrepancy between
Chesterfield's figures and the branch's figures, are you
suggesting that it is necessary for you to see the
evidence of what happened next, of whether a transaction
correction was sent and what the evidence was in
relation to that branch? Are you really suggesting
that's necessary?
A. No, because in a typical scenario where the systems
operate as they should, as they are designed, then the
positioning of the countermeasures that you have put in
place would pick up on that so that would work
absolutely fine. In the scenario where the system
doesn't operate as expected there is a bug, error or
defect or communication fault, then a different set of
scenarios will likely be encountered and it is
understanding then what happens that is important.
Q. Let me get this straight. You are suggesting that there
are cases when you can take it as read, the situation is
such that you can take it as read that a problem will be
identified and it will be fixed. But you are suggesting
there are other situations where you can't take it as
read where specific evidence is needed, is that right?
A. Yes.
Q. Okay. But nonetheless, although you say that is
necessary, it is your considered opinion that the
Horizon system that you now see is robust, yes?
A. Relatively robust, yes.
Q. Thank you.
If I could look at the third joint statement again
which is at -- let's go to your first report, actually,
which is {D2/1/26}. At paragraph 3.7 you say:
" ... whilst the present-day version of Horizon,
supported by manual human support may now be considered
as relatively robust in the spectrum of computer systems
used in businesses today it has undergone major
modifications ..."
And in forming that judgment, if I can go down to
paragraph 3.9, you say:
"Fundamental in determining the robustness of
Horizon is gaining an understanding of the Post Office
(and Fujitsu) manual business processes applied when
determining and handling the effects of bugs ..."
So may I take it that the judgment that you record
in paragraph 3.7 is based upon an understanding that you
have gained of the processes you describe in
paragraph 3.9?
A. Well, one of the challenges is that we do have
an understanding of business processes for many parts of
the Horizon support system, we don't have it for all,
and one of the key elements is when it comes to
reconciliation, so the recovery of a situation that
occurs when a bug, error or defect has had an impact on
branch accounts, there is a need for reconciliation and
we don't have that part. So the closed loop, we are
unable to close that loop.
Q. I'm finding it difficult to understand what you are
saying. Are you saying that you are in a position to
make a judgment that Horizon is robust or are you saying
that you aren't?
A. I'm saying that based on the information that's
available it is relatively robust.
Q. Thank you.
A. What I'm also saying is subject to other documentation
or other parts of the process that has not been
disclosed at this stage, the situation may improve or it
may well be a worse situation.
Q. Mr Coyne, if I can suggest to you that makes no sense
whatsoever. You have made a judgment. You have already
confirmed I think more than once --
A. Yes.
Q. -- that it is your considered opinion that the Horizon
system as it exists at the moment, with all its
countermeasures, is robust?
A. Yes.
Q. In those circumstances you can't be suggesting, can you,
that there's evidence out that there that you haven't
seen that may mean that that judgment is completely
wrong?
A. I don't see why that can't be the position. There may
well be documentation and there is documentation that
must exist with regard to reconciliation within
subpostmasters' accounts after a bug, error or defect
has occurred and outstanding the outcome of that will
have an impact on the statement of robustness.
Q. Mr Coyne, when you talk about reconciliation you are
talking about the reconciliation that occurs between
transactions as transferred from the Horizon system into
Post Office systems and those transactions then being
compared with the transactions that are recorded by
banks, financial institutions and other Post Office
clients, is that what you are talking about?
A. No, sorry, specifically what I'm talking about here is
when a bug, error or defect occurs, as we have seen
here, there is the identification of a discrepancy on
a postmaster's account, and that's in the evidence.
What we then don't see is how, if at all, that was dealt
with.
Q. Just to be absolutely clear, you are not suggesting that
you think that when a bug was identified and
discrepancies were spotted in accounts, you are not
saying you think the accountancy discrepancies were not
dealt with, you are not saying that, are you?
A. No, I'm not saying that. What I'm saying is they were
identified typically by Fujitsu, and I don't know what
the process was, and Fujitsu will say that they don't
know because it appears in all the documents, they don't
know what Post Office does to correct that. That is a
missing part of the evidence. So I'm not saying I think
Post Office do nothing about it, I'm simply saying
I don't know what that process was.
Q. Right. You are not suggesting that you think, on the
basis of documents you have seen, that it is likely that
Post Office do nothing about it? You are not saying
that, are you?
A. No, I'm saying the opposite. It is likely that they do
something about it. I'm saying there is no evidence
available to --
Q. I'm grateful. Then that explains what I was trying to
explore with you. That explains why you feel able to
judge, as you do in paragraph 3.7, that Horizon is
relatively robust. On the basis of the evidence you
have seen --
A. Yes.
Q. -- including the evidence you have seen of the Fujitsu
processes and other documents including Post Office
processes, you are happy, and it is your considered
view, that Horizon is relatively robust?
A. Yes.
Q. Thank you. Now let's get back to what it means --
MR JUSTICE FRASER: Mr de Garr Robinson, we are going to
have a break at some point. Is now a good time?
MR DE GARR ROBINSON: It is a perfect time.
MR JUSTICE FRASER: We will come back in at 11.55 am.
Mr Coyne, you are in the middle of giving your
evidence. I am sure you have heard me say this before.
You are not to speak to anyone about the case and this
applies throughout the week because you are going to be
there all week. I know you know it, I might bore you
senseless by repeating it every time we stop. You don't
have to stay in the witness box but don't talk to
anybody else about the case.
We will come back in at 11.55 am.
(11.51 am)
(A short break)
(12.00 am)
MR DE GARR ROBINSON: Mr Coyne, getting back to what it
means for Horizon to be relatively robust. You have
described the systems which you say are comparable to
which you have compared Horizon. The systems that you
are talking about, they are usually in the private
sector, I apprehend?
A. No. No, across --
Q. Okay, across both. But the kind of businesses you are
talking about, banking, retail, those are private
sector?
A. Yes.
Q. Those sorts of systems have quite exacting requirements
of robustness, don't they?
A. Yes.
Q. No system can be perfect, everyone agrees that, but the
large and complex businesses that use these sorts of
systems, they don't have much tolerance for widespread
errors in data entry, processing or storage, do they?
A. No, that is right. They certainly strive to remove that
wherever possible.
Q. Given the number of transactions and the importance of
handling and accounting those transactions properly,
those businesses require the overwhelming majority of
their transactions to be handled and accounted for
properly, don't they?
A. They do indeed, yes.
Q. They can only afford for a tiny proportion to suffer
from lasting errors, yes?
A. Yes.
Q. Air traffic control and medical machines, frankly they
can't afford any errors at all.
A. Indeed.
Q. But the kind of systems we are talking about, their
tolerance for errors is very low, yes?
A. Certainly it will be fractions of a percent of the total
transaction, whatever the transaction might be, yes.
Q. Any significant level of non-transient errors would be
quite a serious threat to their businesses, wouldn't it?
A. Yes.
Q. It would be a threat to their customer base, a threat to
their manageability, a threat to their manpower
requirements, having to run around dealing with
problems, a threat to their operating costs, a threat to
their profitability, and given that businesses operate
in a competitive world where other businesses are happy
to displace them, it could be a threat to their very
survival, would you agree?
A. It could be. They are quite generic statements but
I don't disagree with any of them.
Q. Those are the businesses to which you have compared
Horizon, yes?
A. Yes.
Q. So when you say the Horizon system is relatively robust,
you are saying that in the overwhelming majority of
cases where conditions are both normal and adverse it
works reliably, yes?
A. Yes, the vast majority of all the transactions that flow
through the Horizon system will work successfully.
Q. And although there are occasions when it doesn't, those
represent only a tiny proportion of the work that it
does, correct?
A. Yes, it will be a small fraction of the work that it
does, yes.
Q. Even within that tiny proportion, a large majority of
the problems thrown up are picked up and corrected by
the various systems in place that are designed to do
precisely that?
A. Certainly many of the ones that go wrong for one reason
or another appear to be picked up. There are examples
where they don't appear to have been picked up and
examples which appear to have an impact.
Q. That is a very important opinion, isn't it, in the
context of this case?
A. Yes.
Q. For the purposes of this trial?
A. Yes.
Q. That being the case, Mr Coyne, I wonder why you haven't
actually spelt out that in your reports or your joint
statement?
A. I believe that I have. Have I not?
Q. It may be unfair, Mr Coyne, but my sense is that in both
your reports you tried to give a rather different
impression, an impression of a system which is beset by
problems, the sheer volume of which means it cannot
sensibly be regarded as reliable. Would that not be
fair?
A. I don't believe that that's fair, no.
Q. Let's have a look at your first report. If we could go
to bundle {D2/1/25}. This is the report -- your first
report that you produced in mid-October. And if we look
at paragraph 3.1 --
MR JUSTICE FRASER: I do not think we are on the right page
on the common screen.
MR DE GARR ROBINSON: {D2/1/25}.
A. Yes.
Q. You recite the agreement in paragraph 3.1 contained in
joint statement 1?
A. Mm.
Q. You say in paragraph 3.2:
"Each discovered bug ... could have remained
unresolved in Horizon for varying periods of time."
And you add that there is a possibility of other
bugs.
A. Mm.
Q. Then you say this in 3.3:
"The sheer volume of known error logs and
reconciliation reports confirm the wide-ranging extent
of the impact of such bugs ... This evidence
demonstrates that such bugs ... would undermine the
reliability of the Horizon system to accurately process
and record transactions."
That seems quite an ominous statement to make given
the propositions that you have just agreed with me
a minute and a half ago, Mr Coyne. Do you not see
a tension?
A. Yes, I think I do actually. At that point in time we
had a document set that consisted primarily of known
error logs. I think the PEAKs had only just been
disclosed a few days --
Q. Weeks?
A. Sorry?
Q. Would it not be weeks?
A. I don't know exactly how many.
Q. I'm sorry, I interrupted you. Please carry on.
A. Perhaps if we establish what the date was. I think we
do mention it in the reports.
Q. Yes. Please carry on. I interrupted you and
I shouldn't have.
A. So what we are saying there is the full picture was yet
to be revealed and that it may well undermine the
reliability of Horizon.
Q. So are you suggesting, Mr Coyne, that when you produced
your first statement you were doubtful about whether it
was robust?
A. Mm.
Q. And it is when you produced your second statement, when
you had more opportunity to look at the PEAKs and so on,
that you formed the impression that it was robust after
all?
A. I had a different concern when it came to create my
supplemental report because there was the discovery of
far more defects than it was originally said existed.
When putting together my first report I thought the
analysis had already been done to establish that there
were only three defects. I discovered a fourth defect
but then shortly afterwards discovered many more. It
was also the case that I had seen a number of reports
about improvement of the processes that was in place.
But then it transpires that when we saw some of the
witness statements that those processes, those
improvements in processes, hadn't been adopted.
So there was a number of other factors that were
brought in place that confirmed that there was large
elements of unreliability. That doesn't take away my
overriding statement that the Horizon system is
relatively robust.
Q. Mr Coyne, one of us is confused and it might be me.
I had taken you to paragraph 3.3 and suggested to you
that the impression, the strong impression given by
paragraph 3.3 was that Horizon wasn't very robust at
all. My understanding of your answer, and it may have
been a false understanding on my part, was you said,
well, at that stage we only had the PEAKs for a few
days, and the impression I got was you were saying it
was only later I realised that Horizon was relatively
robust. So I suggested -- I asked you about that. And
then what you actually said was, no, no, in my later
report I thought Horizon was less robust.
So at the time of this report your view as to the
robustness of Horizon was at its zenith, was it?
A. Could you clarify what you mean when you say zenith.
Q. It was at its peak. This was at a time when you thought
Horizon was very robust and it is only later that you
then downgraded it to relatively robust?
A. There's different aspects of it. This is the
complication with robustness, it isn't an entity that
can be measured by one simple set of criteria.
Q. All I'm trying to ascertain with you, Mr Coyne, is why
on the one hand you were saying at the time of this
report you thought that Horizon was relatively robust,
you felt able to include -- to say what you said in
paragraph 3.3, which gives a completely different
impression. I would like you to explain how you felt it
appropriate to do that.
A. What I said there is correct, that the bugs, errors and
defects that we see, when they trigger they would
undermine the reliability of Horizon and did undermine
Horizon's reliability.
Q. What you say is -- look at the first sentence:
"The sheer volume of known error logs and
reconciliation reports confirm the wide-ranging extent
of the impact of such bugs ..."
A. Yes.
Q. So first of all you say that there is a huge volume of
KELs showing bugs with branch impacts, do you?
A. Typically the KELs show that PEAKs exist that have had
branch impacts, but the KEL is the knowledge document.
Q. You appear to be saying there, Mr Coyne, and it may be
that I'm not being clear enough. Let me be absolutely
clear. You appear to be saying there that there is
a large volume of KELs which demonstrate bugs in the
Horizon system which had branch impacts. Is it the case
that at the time you wrote this report you had seen
a large number of KELs showing bugs with branch impacts?
A. Right, it is important to understand -- in order to
answer that, it's important to understand the
relationship between KELs and PEAKs. A KEL is
a document which is often updated with new knowledge
that will enable Fujitsu's support department to better
support the users as and when they find a particular
defect on the system. Contributing to that KEL is
a number of PEAKs. There might only be one occurrence
of a PEAK that leads to a KEL, there might be twenty or
thirty different PEAKs that contribute to a KEL. So the
fact that we had the KELs at that point in time and
those KELs indicated problems, and there were a number
of PEAKs that hadn't yet been fully examined and I set
out that in my report, but of the PEAKs that had been
examined there were PEAKs that suggested impact on
branch accounts.
Q. Let me ask my question again but I will do it in
a different way. Let's cross out the word "and
reconciliation reports" in the first sentence so it now
reads:
"The sheer volume of [KELs] ... confirm the
wide-ranging extent of the impact of such bugs ..."
So you are referring to a large volume. It's not
just a number, a significant number, you're saying
a large volume. It's a substantial body of KELs --
A. Yes.
Q. -- confirming the wide-ranging extent of the impact of
bugs.
A. Yes.
Q. Is that the case, that you had seen a large volume of
KELs indicating the wide-ranging extent of the impact of
bugs?
A. Yes.
Q. How many?
A. A number are listed in this report but I don't know
precisely what the number is.
Q. And when you say impacts, you are talking about branch
impacts, aren't you?
A. Yes.
Q. So you are saying there was a large number of KELs which
you had already seen at the time of this report that
confirmed that there were bugs, wide-ranging bugs -- or
confirmed bugs which had a wide-ranging impact on branch
accounts?
A. Yes. Certainly within the KELs there was reference to
discrepancies which appeared to be on branch accounts.
Q. Mr Coyne, I would like to suggest to you that in the
whole of this report that goes on for over 200 pages,
you refer to very few KELs that actually confirm the
existence of branch impacts?
A. There is a table I believe at the back of this report
that sets out the number of KELs.
Q. What page would that be? I may not have it in hard
copy.
A. Appendix G.
Q. What page?
A. 200, I believe.
MR JUSTICE FRASER: {D2/1/213} is appendix G. Could you
give me one second, please. (Pause)
Go ahead, they are just going to sort out my
screens. The common screen is fine, it's my custom
screens.
MR DE GARR ROBINSON: Mr Coyne, I have looked at the bug
table in the second joint statement of the experts.
A. Yes.
Q. The bug table in joint statement 2 is a table which sets
out the bugs that you have identified which you believe
show evidence of branch impacts, yes?
A. Yes.
Q. That table, if my calculations are correct, only refers
to four bugs that you had identified in your report in
addition to the three bugs that had been accepted by --
already disclosed by Post Office.
Now, what I would like to understand is do you
accept in principle that that's the position?
A. I don't I don't know if that's the number. That is a
number you have calculated, haven't you?
Q. Yes.
A. I haven't done the calculation.
Q. There are 29 bugs in the bug table.
A. Yes.
Q. Many of those bugs were only identified either by
Dr Worden in his report or by you in your second report,
yes?
A. I believe Dr Worden only identified the three bugs that
had already been identified.
Q. That's not -- perhaps we will come to that in due
course, Mr Coyne. In fact, Dr Worden -- I think the
number may be nine, something like nine bugs he had
identified in his first report that are now to be found
in the joint statement. Did you not appreciate that was
the case?
A. I don't know whether that's correct or not.
MR JUSTICE FRASER: But the KELs you were being asked about
volume, they are the ones listed in appendix G?
A. Yes.
MR DE GARR ROBINSON: What I'm suggesting to you, Mr Coyne,
is very few of these KELs have turned up -- or the bugs
to which you say these KELs relate, very few of them
have turned up in the joint statement bug table.
A. Right, okay. I'm not sure that is correct.
Q. I'm suggesting to you, Mr Coyne, that you had not seen
a large volume of KELs showing branch impact -- showing
bugs with branch impacts at the time you wrote this
report?
A. Well, I believe a number of these KELs that are
contained within this appendix have impact.
Q. Are you suggesting that all of these KELs relate to bugs
which had branch impact?
A. Many do, but what you have to remember about the KEL is
the KEL doesn't specifically relate to the branch.
That's typically the PEAK.
Q. Let's not talk about -- I'm just trying to analyse what
you said in paragraph 3.3 of your report. You said the
sheer volume of KELs confirm the widespread branch
impact. And I'm suggesting to you, Mr Coyne, that you
had not seen a huge volume of KELs which confirmed
branch impact at the time. You just hadn't.
A. That paragraph would be better if it said: the large
number of KELs that I have reviewed and a portion of the
PEAKs that have recently been disclosed and I have
reviewed dot dot dot, and carry on with that paragraph.
That would be nearer --
Q. If we look at page 214, for example, it starts with "Pin
pad", pin pad errors. So you have six pin pad errors.
{D2/1/214]
A. Yes.
Q. You are not suggesting those pin pad errors had branch
impact, are you? Those pin pad errors meant the branch
couldn't do business with a particular customer. They
didn't actually create an error in branch accounts, did
they?
A. It is possible they might. We would have to go through
and look at the detail.
Q. So you are suggesting that there are errors in pin pads
which are covered by KELs which have created branch
impacts, are you?
A. Yes.
Q. Because what's curious is I don't believe there are any
of those in the bug table in JS2.
A. Yes, I believe there was the scenario where it was
remembering the account of the last person that should
have been paid. So was it the allpay defect? The
payment that should have gone to other beneficiaries was
all sent to allpay instead because of a defect?
Q. I will have to look at that later.
Going back to page 25, paragraph 3.3, you explained
that you think you had seen already, by the time of your
first report, a large volume of KELs showing widespread
branch impact {D2/1/25}, and you say that it is not the
KELs so much as the PEAKs that refer to them, is that
how you put it?
A. KELs indicate that there was a defect and it is
typically the PEAKs that record the particular branch
impact. So by looking at either you can tell there was
a problem, but it is only when you look at both the KEL
and the related PEAKs that you understand the scope of
the problem.
Q. Wouldn't it be fair to say though that when you look at
a KEL, generally speaking the KEL which deals with it --
and KELs deal with all sorts of things, including things
that are not bugs at all. But when you look at a KEL,
generally the KEL will explain what the impact of the
problem is, the nature, the practical consequence of the
problem that has to be dealt with, would you accept
that?
A. Yes. It might, for example, say there is a bug and we
believe this is going to cause a discrepancy and we have
spotted it in eight branches, but it won't tell you what
the impact on those eight branches is. For that you
have to go to the particular PEAK.
Q. Generally speaking then, if there is a KEL that
addresses a bug that has a branch impact, generally it
will tell you?
A. Generally speaking.
Q. On the vast majority of occasions, yes?
A. I'd prefer to stick with generally speaking. I don't
know if it is the vast majority.
Q. Okay. You then say in paragraph 3.3 -- let's cross out
"known error logs", you now say:
"The sheer volume of ... reconciliation reports
confirm the wide-ranging extent of the impact of such
bugs ..."
So you are suggesting here that reconciliation
reports are good evidence of bugs in Horizon which had
branch impacts, are you?
A. Yes, there was a response to a question that I asked
about reconciliation, I don't know exactly how it was
phrased, but I asked for the number of reconciliation
problems that required manual intervention, and the
answer I was given was that it was hundreds of thousands
per week, I think was the --
Q. 10,000 a week that have to be F99ed, do you remember
that?
A. Yes.
Q. What is a reconciliation report?
A. Where there has been a problem within the system for one
reason or another and the transaction has not proceeded
to completion as it would originally have expected.
Q. Let me put to you what a reconciliation report is just
to save some time. When a transaction is done in
Horizon and the basket is committed to the system, the
transaction goes into the branch accounts, yes? That
transaction goes into the branch database.
A. Mm.
Q. And then copies of those transactions are then
transmitted through systems such as the TPS system,
could you explain what the TPS system is?
A. Transaction processing system.
Q. What is the TPS system? What does it do?
A. It sends -- well, it decides what payments need to go to
what particular beneficiaries.
Q. Is that right? Doesn't it transmit information relating
to these transactions that had been committed to the
BRDB to Post Office's back-end systems?
A. That is right.
Q. And isn't it Post Office's back-end systems, like
POLSAP, like Credence and so on, isn't it those systems
that then do the steps that are needed to ensure that
payments are made by financial institutions and so on,
is that right?
A. Yes, there is a harvesting process which will pick up
various transactions and will put them in certain
buckets, and then there are various processes that goes
on with those.
Q. And the harvesting process is a process by which --
well, what happens is transactions go into the TPS
system and they are then propagated into POLSAP and
other management information systems that are maintained
by Post Office, yes?
A. Records of those will do, but then a number of the
transactions will then go on to -- for example, if it is
a bill that's being paid it will go on to agree that
payment with whoever the party is.
Q. What happens then is -- generally the financial
institutions making payments have received messages
direct from things like the pin pad?
A. Yes.
Q. And what then happens is the financial institution has
its own record of the transactions it thinks are being
done?
A. That is right.
Q. And Post Office has its record of the transactions it
thinks are being done?
A. Yes.
Q. Those records are maintained on Post Office's systems?
A. Agreed.
Q. And then there is a reconciliation process by which
those systems are automatically reconciled to see
whether they agree with each other?
A. They should be automatically reconciled, yes.
Q. That is what you describe as reconciliation, and that
process generates reconciliation reports, doesn't it?
A. Yes.
Q. And when there are exceptions, then in those
circumstances there are reports indicating transactions
where some of the information which Post Office has
about them, it differs from some of the information that
the financial institution has about them?
A. Yes, something has gone wrong with the process.
Q. Yes. And it might be that some attributes are missing
that have nothing to do with the basic transaction
details?
A. It could be any aspect of it --
Q. It could be time stamps, it could be anything.
A. It could be value, it could be anything.
Q. It could even be that there has been a delay in the
financial institution or POLSAP receiving the necessary
information, couldn't it?
A. It could. I think it is safe to say that something has
gone wrong, the process hasn't operated as it was
designed, and an intervention needs to take place to
correct the transaction.
Q. What happens is a reconciliation process is
automatically undertaken, exception reconciliation
reports are generated which demonstrate where there are
exceptions that need to be looked at?
A. I'm not sure a reconciliation process is automatically
undertaken.
Q. Mr Coyne, when you have 3 million transactions that are
on the Bank of Ireland's books and 3 million
transactions that are on Post Office's books, you don't
have a human being comparing those two sets of
transactions, do you?
A. No, but that's when it is working as expected. When
something has gone wrong this is the reconciliation
process that I'm talking about there.
Q. We are getting bogged down in a way that I'm finding
quite surprising. The process by which lists of
transactions are reconciled is automatic, isn't it? It
is electronic?
A. Yes, when everything is working fine it is automated.
Nobody needs to get involved.
Q. The process of reconciliation is the process of
comparison between the two sets of data?
A. Yes.
Q. That process is electronic, isn't it?
A. Yes.
Q. And then what happens is there is a report generated on
a regular basis which identifies reconciliation
exceptions where for some reason, for the reasons we
have already discussed, there isn't a full
reconciliation and something has to be looked at?
A. Yes.
Q. We have discussed some of the reasons for that?
A. Yes.
Q. And that's where a manual element can become involved?
A. Yes.
Q. Sometimes there can be one error which explains 1,000 or
2,000 reconciliations exceptions?
A. Possibly, yes.
Q. And there will be a delay in the transaction getting
through to the bank, or it will be when it did go
through to the bank there was a time stamp missing so
the two didn't match. There could be something like
that that could explain an awful lot of them?
A. There could be a range of different things that are
wrong with the transaction.
Q. And that's when someone has to go and look to see if
there is a problem?
A. Yes, and that is the number that I'm referring to
because it was suggested that that was in the many
thousands.
Q. But I don't understand, Mr Coyne, why you think that the
existence of reconciliation reports confirms the
wide-ranging extent of the impact of bugs in Horizon
which are having an impact on branch accounts?
A. I have worked and designed banking systems, stock
broking systems. I have never seen the need for tens of
thousands of transactions per week to have a human
intervention. That suggests that something is going
wrong. It is working outside of process on a larger
scale than I would have expected.
MR JUSTICE FRASER: Can I just ask you, at {Day14/77:17} of
today's transcript you said "it was suggested" that it
was in the many thousands. Can you just tell me, "it
was suggested" doesn't really help me because I don't
know who suggested what to whom.
A. Sorry, my Lord. I asked the question as part of
a request for information.
MR JUSTICE FRASER: Yes.
A. And the response that I was given by Post Office's
lawyers was -- I believe it was ten thousand.
MR JUSTICE FRASER: So that is what you are referring to?
A. Yes.
MR JUSTICE FRASER: All right.
MR DE GARR ROBINSON: Now Mr Coyne, there are many, many,
many possible explanations for reconciliation exceptions
that have nothing to do with bugs in Horizon causing
shortfalls in branch accounts, aren't there?
A. There could be, yes.
Q. There's no reason to think -- the fact that there are
any scale of reconciliation exceptions does not
demonstrate, does not show or confirm, that there are
any bugs which have an impact on branch accounts, does
it?
A. It certainly could do if the system is having that level
of failure to reconcile then there is something wrong.
It is operating outside of the process it should
operate --
Q. Well, I will -- how is it that you are able to judge
that, Mr Coyne? Do you know what range the system was
designed for?
A. No, but I have experience of similar systems and I don't
believe that such a high level of manual reconciliation
would be tolerated or should be expected.
Q. But the possible causes of those reconciliation
exceptions could involve all sorts of things such as
phone lines from the Post Office to the financial
institutions, or the financial institutions' own
operating systems, couldn't they?
A. So that means that that is going wrong each week.
Q. The reason why I'm asking you about this is you are
suggesting that the fact that there are reconciliation
reports, large numbers of reconciliation exception
reports, let's call them, is itself confirming that
there are bugs in Horizon which have an impact on branch
accounts, and my suggestion to you, Mr Coyne, is that
doesn't confirm anything of the sort?
A. Well, it suggests rather than confirms.
Q. And where there is a reconciliation exception which is
looked into, what happens -- and a view is taken --
a view will then be taken by Post Office and the
financial institution as to what the correct position
would be?
A. Mm.
Q. If the view is taken that the correct position is that
the branch account should be corrected, what happens
then?
A. What should happen is a transaction correction should be
created by --
Q. And would you accept that when a transaction correction
is sent it is more likely than not that the transaction
correction will correct an error that's happened rather
than create one, yes?
A. It is more likely. There certainly have been scenarios
where transaction corrections have caused a problem but
it is certainly more likely that it would correct it,
yes.
Q. What I suggested to you about paragraph 3.3, Mr Coyne,
is this paragraph seems designed to give an impression
that Horizon isn't robust, whereas on the very next page
you say that it is. What do you say to that suggestion?
A. Well, I'm setting the context before providing my
summary that it was relatively robust.
Q. What I'm finding difficult to understand, Mr Coyne, is
you say -- you refer to a wide-ranging extent of the
impact on branches of bugs, and then over the page you
say but Horizon is relatively robust, and I would just
like you to explain how is it you felt able to do that?
A. Because as I have agreed with Dr Worden, robust doesn't
mean that there won't be an impact. The two aren't
mutually exclusive.
Q. You have also agreed with me, Mr Coyne, that in the
context in which you are using the term relatively
robust in paragraph 3.7, it means that there's only
a tiny number, a tiny proportion --
A. I didn't say tiny.
Q. Really? Is that not precisely what we were discussing
before?
A. I don't believe I'm using --
Q. When we were talking about using comparable systems and
how they have a low risk tolerance?
A. I wouldn't characterise it as tiny. I probably said
a fraction of a percentage.
Q. Okay.
MR GREEN: My Lord, the evidence -- the word "tiny" was put
and he said a small fraction. That's what the
transcript says.
MR DE GARR ROBINSON: Yes. And do you think -- my
suggestion to you, Mr Coyne, is that in paragraph 3.3
you're actually trying to give an impression which is
seeking to undermine actually what is a very helpful
conclusion that you have set out in paragraph 3.7?
A. That certainly wasn't the way it was written. It was
constructed to set the context for it.
Q. Let me also suggest, Mr Coyne, that in this report you
also tried to give the impression that the concept of
robustness has no meaning and it's impossible to
measure, would you agree with that?
A. I do agree that it is impossible -- it is very
subjective and it is impossible to come up with
a quantifiable measurement, yes.
Q. Now let's go back to the joint statement you made
shortly before you produced this report. It is at
{D1/1/8}, I'm going to page 8.
Now, I think you have maintained in the evidence you
have just given that paragraph 3.3 where you talk about
the sheer volume of things happening is consistent with
Horizon being relatively robust as compared with its
comparators, yes?
A. Mm.
Q. So the fact that there is such a sheer volume is not
inconsistent with it being relatively robust, is that
right?
A. Yes.
Q. Okay. Well, we have already read what the experts
agreed in paragraph 2.3. If we could go over the page
to the disagreement boxes on page {D1/1/9}, there at the
top you have got your definition of robustness.
A. Yes.
Q. Then just below that you say:
"in consideration of the likelihood of Horizon to be
the cause of shortfalls in branches, Horizon is not
determined to be robust in this regard because:
"(a) It contained high levels of bugs ... which
created discrepancies in the branch accounts."
So now I'm really confused, Mr Coyne. On the one
hand your judgment on 16th October was that actually
Horizon is relatively robust and a very small proportion
of transactions go wrong as a result of bugs, but here
you appear to be saying the opposite. Did you have
a change of mind between agreeing joint statement 1 and
your first report?
A. Joint statement 1 was constructed before the end of the
first report.
Q. Joint statement 1 was on 4 September.
A. Yes.
Q. Your report was on 16th October. I will be corrected if
those dates are wrong.
A. Right.
Q. Did you change your mind between 4 September and
16th October?
A. What I did is that whilst looking at the factors in the
round I considered Horizon to be relatively robust,
whereas my initial opinion was that it was not robust.
Q. So you did change your mind, is that right?
A. Yes.
Q. And what made you change your mind, please?
A. It was just a consideration of all aspects in the round.
Q. Well, Mr Coyne, in your -- a joint statement is
an important document, you have given evidence many,
many times before, haven't you?
A. Yes.
Q. And you will appreciate this is as important in many
ways as an expert report, correct?
A. It was a very early document, it was when we were
finding our way around the evidence that was available.
Q. Well, you wouldn't express an opinion unless it was your
considered opinion, is that right, in a joint statement?
A. That is true.
Q. So if, for example, you felt that things didn't look
good but you hadn't looked at all the evidence and you
wanted to withhold judgment, you wouldn't express
an opinion at that stage, would you? Or if you did, you
would express it in a very provisional and equivocal
way, wouldn't you?
A. Yes.
Q. But that's not what you do here, Mr Coyne, is it? And
I would like you to explain why, please.
A. Well, I mean the reasons for that are set out below in
(a) to (f).
Q. So you had seen all these things. You had seen a high
level of bugs which created discrepancies in branch
accounts, had you, already by that stage?
A. We had certainly seen the KELs at this stage. I'm not
sure whether the PEAKs had been disclosed to us at this
point in time.
Q. No, they hadn't. So you had seen the KELs, and from the
KELs alone you felt able to judge, did you, that there
was a high level of bugs that created discrepancies in
branch accounts?
A. Well, the KELs have a reference within them to the
PEAKs.
Q. Which you hadn't seen at that stage?
A. No, but we had seen that there's references to PEAKs.
Q. What's the significance of that in the context of my
question?
A. Because it is the KEL that would indicate how many,
often, of that defect that led to the KEL being created
had surfaced.
Q. So you accept that KELs are actually good -- a KEL will
be good evidence of whether there is a bug that creates
a branch impact. That's your view, is it?
A. It is an indicator but you can't use that document
alone. You have to look at the PEAKs as well.
Q. But Mr Coyne, it seems to me that in paragraph (a) you
are using that document alone. You hadn't seen any
PEAKs at that stage, had you?
A. No, we hadn't seen the PEAKs, but we knew that PEAKs
must have existed and that there are references to them.
Q. So what? Why does the fact that you know there are
PEAKs you haven't seen enable you to make a judgment
that you would not be able to make simply by looking at
the KEL?
A. The KEL talks about how the situation that occurs with
the bug, error or defect should be dealt with, and by
reading that you get an indication that there is or
there must be or there likely is a defect within the
system. And if the KEL refers to, for example, ten
different PEAKs then that is an indication, although you
can't be certain, that there has been ten occurrences of
that.
Q. So what you are saying is that if you see a KEL which
refers to a bug, you will be able to tell from that KEL
in the main whether the bug has an impact on branch
accounts or not?
A. There will be sometimes an indication whether there is
an impact on branch accounts. We typically will not
know what branch it is.
Q. You say "sometimes"; what I'm suggesting to you,
Mr Coyne, and I think you know what I'm suggesting to
you, is KELs are actually a good source of seeing
whether there is a bug which has branch impact. You may
not know the details of branch impact but it is a good
way of telling -- if there is a bug that is considered
in a KEL that has a branch impact it is likely to say
that, isn't it?
A. It will often say there might be a discrepancy, yes.
The way that these documents are created is somewhat
inconsistent. You can read a KEL and it will give you
all the information that you need. It will relate to
a branch, it will point out the discrepancy, and that
KEL on its own might be helpful. You will read another
KEL and there's very little information within it. You
will need to go to the five PEAKs that relate to it to
even start to understand what the impact was. So I'm
not saying that KELs aren't helpful documents. I'm
saying that they need to be read in context.
Q. Let's move on from that subject.
MR GREEN: My Lord, I didn't want to interrupt.
MR JUSTICE FRASER: Are you going to the joint statement?
MR GREEN: Yes. They have agreed on page 26 {D1/2/26} at
0.3 what the shortcomings of those documents are, so it
is surprising to hear a different --
MR JUSTICE FRASER: I'm aware of that, but if
Mr de Garr Robinson wants to spend his time exploring
it, I'm not going to stop him.
MR DE GARR ROBINSON: Mr Coyne, at this stage you had only
seen the KELs and you were in a position to make
a judgment that there were a high level of bugs that
created discrepancies, were you? That was your
considered view then just on the basis of seeing KELs?
A. As I pointed out, whilst I had only seen KELs I had seen
references to a number of PEAKs as well.
Q. Then after 4 September PEAKs were disclosed?
A. Some PEAKs were disclosed.
Q. A large number of PEAKs were disclosed?
A. Yes.
Q. Something like 200 and something thousand, is that
right? I'm not sure there were any other material
documents disclosed between 4 September and 16th October
that are relevant to the this question. Can you think
of any?
A. I'm not sure. I did try and map out all the various
disclosures that have taken place but that doesn't
appear --
Q. The ones that I'm aware of are the PEAKs.
A. Right.
Q. Clearly you had a change of mind between 4 September and
16th October. Would I be right in thinking that on
seeing the PEAKs, that caused you to change your mind?
A. Certainly the PEAKs do help to set the context about how
problems are dealt with. So they would help, yes.
Q. You have plainly had seen something good between
4 September and 16th October and what I'm trying to
explore with you, Mr Coyne, is what that good thing was.
What had you seen that made you change your mind?
A. Certainly the PEAKs would be part of that,
an understanding better of the processes that are in
place.
Q. So would you accept you were rather hasty in expressing
the judgment you express here on page 9? {D1/1/9}
A. I'm certainly content with my opinion as expressed in my
report. Yes, it would probably be better for me not to
have expressed that particular opinion there so early.
Q. Do you think it would have been helpful in your report
to have explained why you changed your mind? It was
a very surprising volte-face. Do you think it might
have been helpful to allow everyone to understand what
it was that was good about the system that allowed you
to form the view that Horizon was robust?
A. "Relatively robust" was the term.
Q. Yes, relatively robust.
A. Yes, I mean --
Q. The reason why I ask is you see in joint statement 1 you
are resolutely negative about Horizon. In your report
on 16th October you say Horizon is relatively robust but
the rest of your report says negative things about
Horizon. There's very little that's said that is good
about Horizon.
It is difficult to resist the impression, Mr Coyne,
that in your first report you were trying to say bad
things about Horizon and you were not interested in
saying anything that was good.
A. I reject that.
Q. Would that be fair?
A. No.
Q. What is it that was good about Horizon that caused you
to change your mind and where do you describe that,
where do you consider that in your first report?
A. What was helpful was to understand the support process
in more detail to understand how things such as fault
determination is done, albeit it is only
an understanding of how it is done within Fujitsu, to
understand that process more. So that was an improving
position for me.
Q. I see. And your judgment on reviewing the PEAKs was
that Fujitsu actually had quite a good support process,
is that right?
A. Yes, I mean the support process that Fujitsu operate,
and again this changes over time so it is very difficult
to pick a point in time and understand what obligations
they had, what roles and responsibilities they
fulfilled, but certainly by the time they become aware
that somebody believed there was a problem, so it hit
SSC, the support centre, third line support, the process
they had of determining whether there was a fault
appears to be a reasonably good process. So there's
obvious weaknesses, how does the fault get to them in
the first place? And that's --
Q. When you say obviously weaknesses, why is it obvious
that there were such weaknesses?
A. No, no, what I'm saying is the obvious weakness in the
opinion is we don't know what happens before it gets to
Fujitsu, but then once it comes into Fujitsu and they
determine that, yes, there has been a problem, or there
has not been a problem, and they believe there is
a discrepancy, you will see quite often that there is
a line at the bottom of the PEAK saying, hand this back
to Post Office for them to do whatever they need to do
to branch accounts. So the process is then handed over.
So we don't know happens afterwards and we don't
know what happens before but the process with Fujitsu,
certainly SSC, appears to be a reasonably good process.
Q. So what you found when you read the PEAKs was that when
a call got referred to the SSC, either because it is
a subpostmaster call, or perhaps it might be automatic,
it might be from the MSU because of a reconciliation
issue, your view was that Fujitsu was quite good at
spotting if there was a problem in Horizon, is that
right?
A. Yes.
Q. And it was quite good at making sure that problem was
fixed, yes?
A. Yes.
Q. And it was quite good at identifying the consequences of
a problem and informing Post Office as to what those
consequences were, yes?
A. I don't know whether they informed Post Office. In
their own records you would often see comments such as
corrections will need to be made. There is a number of
references to say we do not know what Post Office's
process will be to deal with this but it does need
dealing with.
Q. But it is fair to infer, isn't it, and I suggest that
you have inferred from all the PEAKs you have seen, that
where there is a bug that has been identified that
appears to have had an impact on branch accounts,
Fujitsu are quite good at identifying the branches that
had been affected by that bug?
A. They are quite good. They will often take quite a long
time in identifying which branches are affected. Such
as Dalmellington, Fujitsu were not asked to get involved
in that for a number of years after it had been
impacting a branch account. When they did get involved
they quite quickly established all the historic impact
of that.
Q. Just to go back to my question. Fujitsu are quite good
at identifying the branches that have been affected by
those kind of bugs?
A. In the main, yes.
Q. Thank you. And it is fair to infer that Fujitsu then
tells Post Office what those effects are?
A. Well, that's the bit that I don't know because I don't
know what the process is. I do not think we have had
sight. There was some very, very -- there was some
disclosure made only very recently which are the BIMS
documents, and I think the BIMS documents are in part
a communication between Fujitsu and Post Office, setting
out what the likely impact might be. There's also OCR
and OCP documents.
Q. That's not relevant to this, though, is it?
A. It is a communication between Fujitsu and Post Office --
Q. I see.
A. -- saying that changes were going to be made. So it may
well be relevant to this.
Q. But it is right, isn't it, that Fujitsu, from what you
have seen of the PEAKs, will work out the branches which
have been affected by any bug that they identify, yes?
A. That is what they set about to do, yes.
Q. And you don't assume -- you don't infer, do you, that
having identified those branches Fujitsu then keep that
information to themselves? You infer that that
information is passed onto Post Office don't you?
A. I think so. There is certainly one reference in a PEAK
where it says I suggest we don't tell the branch about
this, but I'm not sure whether that's we won't tell
Post Office about it, it is more keeping it from the
branch rather than Post Office --
Q. And that's the only PEAK of any kind that you have ever
found of that sort, isn't it?
A. Yes, I believe so.
Q. So we have one PEAK over 20 years that says something
like that. In those --
A. Sorry, we should be careful. I haven't had eyes on all
of the 200,000 PEAKs.
Q. Yes. So all of those things that we have been talking
about you consider, and you considered by the time of
your 16th October report, were quite good?
A. Yes.
Q. What I don't understand, Mr Coyne, is why you didn't say
any of that in your report. Would it not have been
a balanced thing to do?
A. Perhaps. Looking back at the report, possibly.
Q. Would it not have been helpful to explain the good
aspects that you had spotted in the system as well as
the bad ones?
A. Possibly.
Q. Did you have a reason for wanting to keep it back?
A. No, not at all.
Q. The impression I get, Mr Coyne, I'm sorry to put this to
you, is that your first report was designed to give
a very poor impression about Horizon --
A. No.
Q. -- and that if you had included what you've just
described to me it would have given a much less poor
impression of Horizon.
A. I mean, as I say in my report, we'd been asked to
identify specific bugs, errors and defects and the
nature of that type of assignment is to drill in and
find occurrences of that. Talking about all the good
things that could and should happen, either generically
in the industry or that Post Office might do this year
as a result of various improvements in place, doesn't
really in my opinion provide much assistance.
Q. But isn't it a necessary part of the judgment you make
as to whether Horizon is robust? How can you make
a judgment about that without taking it into account?
A. I think you can take it into account, but to spend pages
of text talking about all the various good things,
I don't see there's any value in that.
Q. Well, Mr Coyne, can I ask you -- put it this way, what's
the value of doing this: saying in your joint statement
on 4 September that Horizon really isn't robust because
it has a high level of all sorts of bad things, six
weeks later producing a report by which time your view
has changed diametrically, and not mentioning what it
was that you had seen in the intervening time that
caused you to change your mind? Wouldn't it be obvious
that you needed to explain that to the court to help it
make its own determination?
A. Firstly, I don't believe that my opinion has changed
diametrically. It is a spectrum of robustness, as
I tried to explain this morning, and I believed that
Horizon has gone further up on the spectrum. There's no
complete change of direction here.
Q. I see. Just looking at page {D1/1/9} you say:
"... Horizon is not determined to be robust in this
regard because:
"(a) it contained high levels of bugs ... which
created discrepancies in the branch accounts ..."
At that stage -- we are going back to 4 September --
how many bugs had you identified which created
discrepancies in branch accounts?
A. More than the three that had been originally brought to
my attention.
Q. But three is not high level and nor is four. You say
"high levels of bugs". What did you mean by the
expression "high levels of bugs"?
A. Certainly in the tens.
Q. In the tens?
A. Yes.
Q. So we are talking about a system which has operated over
20 years, yes?
A. Mm.
Q. Which undertakes something like 47 million transactions
a week. So we are talking about 49 billion transactions
over a 20-year period, something like that. It is the
right ballpark, isn't it?
A. Mm.
Q. We are talking about two different versions of Horizon,
and you are saying that a number of bugs in the tens
over that period is properly to be characterised as
"high levels of bugs"?
A. Well, that is what had been identified at this point in
time, but you have got to remember my expectation was
set that I was investigating the three defects that had
occurred by people that had been involved in the
investigation with this system for a long time. So to
move from three to a larger number than that initially
suggested that there must be some quite poor processes
in place if they can't identify bugs, errors and defects
which, for myself, being involved in the process for
only a number of weeks, quickly identified them --
Q. I'm really sorry, I'm sorry to stop you, Mr Coyne, but
I really need to ask you, first of all, did anyone ever
tell you there were only ever three bugs that had ever
occurred in Horizon? Has anyone ever said that to you?
A. I think it was said -- I think in part of the pleadings,
part of the legal documents, I believe it was pointed
out that there was three defects which had impacted
branch accounts.
Q. Has anyone ever told you there were only three defects,
that those were the only? Defects because I suggest to
you that no one has ever said that to you, including in
the pleadings.
A. That was certainly my original expectation when
I started.
Q. I see. Then as a result of looking at these KELs you
realised there were more than three?
A. Yes.
Q. And I think you have suggested that at this time you
took the view that there were in the range of tens of
bugs, is that right?
A. Yes. There was certainly a large number of KELs that
warranted further investigation because they were
indicative of having an impact on branch accounts.
Q. So hold on. You had seen some KELs that might be
consistent with branch impacts but you were not sure
yet, is that right?
A. Yes. As I explained before, KELs relate to PEAKs, and
really in order to confirm the position you need to read
the PEAKs.
Q. So these were not KELs that you knew created branch
account discrepancies, so let's lay those to one side,
shall we. You are making a claim here that there is
a high level of bugs that created discrepancies in
branch accounts and I would like you, if you can
remember, to tell me, at that time on 4 September, what
was the level of bugs you had found that created
discrepancies in branch accounts?
A. That would be in the tens.
Q. In the tens?
A. Yes.
Q. I mean, Mr Coyne, looking at your bug table, and it may
be that my approach to it is all wrong, in the bug table
in JS2, 29 bugs are identified, correct?
A. Yes.
Q. And of those 29 a very considerable number were only
identified either in Dr Worden's report or in your
second report, and the number that remain in JS2 from
your first report is a relatively small number, far less
than ten?
A. Right, okay. I haven't done the maths so I don't know
if that calculation is actually correct, but it is
certainly the case that with the additional context with
the provision of the PEAKs that a number of KELs that
I initially thought led to branch accounts then appeared
not to, and there was additional PEAKs that indicated
branch account impact so we brought them back in again.
And that's the importance of reviewing the documents
side by side.
Q. So are you saying -- and this is the last question I
will ask before we break for lunch -- that at the time
you made this report you thought there were more bugs,
branch account affecting bugs in Horizon than you
thought at the time you made your first joint statement,
or less?
A. No, I think the number is probably around the same.
I think I ended up with 28 or 29 ...
Q. That's your second joint statement. I'm asking you
about your first joint statement. Between 4 September
when you produced joint statement 1 and your first
report in the middle of October, had your view as to how
many branch affecting bugs there were gone up or gone
down?
A. No, it had gone down.
Q. Because of the PEAKs that you saw?
A. Yes.
MR DE GARR ROBINSON: Thank you. My Lord, is this
a convenient moment?
MR JUSTICE FRASER: Yes, we will come back at 2.05 pm.
Remember what I said, Mr Coyne.
(1.04 pm)
(The short adjournment)
(2.05 pm)
MR DE GARR ROBINSON: Mr Coyne, we have been looking at your
first statement of 4 September and I would like now to
move on properly to 16th October. This is the time of
your first report.
Can we look at {D2/1/1} which is the front page of
your report. You say by this time you had examined
a number of KELs and PEAKs which had recently been
disclosed?
A. Yes.
Q. Just looking at page 1 of your report, I see that you
were assisted by four people?
A. Yes.
Q. Could you explain who those four people are?
A. They are all members of my team that helped me with
various projects, both contentious and non-contentious
projects.
Q. They work for IT Group, do they, your company?
A. Yes.
Q. What are their qualifications, their specialities within
IT Group?
A. So Siobhan Forster specialises in databases. She
graduated after doing a forensics -- a technology
forensics degree from Preston and has worked with me for
a number of years. Chris Raske is more to do with the
roles and responsibilities side of my business, so he
understands what various parties do do or should do.
Jamie Smith is one of the researchers that we often use
on projects, and that's the same for Patrick Grant.
Patrick Grant has worked as an implementer of a number
of different computer systems, a programmer, and he has
worked for us for a couple of years now.
Q. So in relation to the preparation of your reports, what
roles did they play? What did they do for you?
A. They were operating typically as a researcher. So
because of the volume of documents, I was asking them to
identify certain themes. They would review -- I would
parcel up a number of the documents that I would search
for in the disclosure platform and they would try and go
through and identify whether there was any potential
relevance in those documents so that I could have a look
at those and decide whether they should be included in
the report or not.
Q. Were these people working full-time leading up to the
preparation of the first report?
A. No. The majority of these people were brought in
towards the end when we were receiving the PEAKs,
towards the end.
Q. So they were engaged in searching the PEAKs and
presumably searching other documents as well, were they?
A. Yes, all of the documents were on a disclosure review
platform, very similar to the one that we see here. We
all sit in the same office, so we would meet each
morning and I would ask for various categories of
documents to be searched for and found and gathered
together.
Q. Would I be right in thinking -- you say they weren't all
working full-time on the case for you, were some of them
spending more of their time on the case than others?
A. Yes, certainly Siobhan Forster worked on the case more
often than not.
Q. Would it be fair to say she spent most of her time on
the case while you were --
A. No, she certainly had other matters that she was dealing
with, but she worked on it more often than not.
Q. What instructions did you give them? What did you tell
them to look for?
A. Typically I'd conduct the first tranche of searching
through the material and we would identify materials to
do with particular types of discrepancy that were being
discussed within the documents, and then I would sit
down with one of these chaps, we would go through the
PEAK or the KEL or whatever the document might be, and
I would ask them to try and assemble all of the other
documents that would be in that family. So it might be
other PEAKs that report the same thing, it might be the
KEL, we had a few of the BIMS documents, so to identify
them, to put together all the package of information.
That in itself was quite a task.
Q. Would it be fair to say that you were generally asking
them to find problems, to find evidence of problems in
Horizon, things that had gone wrong?
A. Yes. Well, sorry, I had already identified the theme
that I wanted them to look for but that was typically
around a specific type of defect that might be occurring
around a particular time and I was asking them to look
at documents around that time or that contained
a similar theme.
Q. So if we go back to the joint statement, having said we
could move on. If we can look at {D1/1/2}. We are back
on 4 September again. At this time were those four
working with you? Did you already have their
assistance?
A. No, I think it was just myself and Siobhan at this time.
Q. So the other three came in later?
A. They would have always been in my office.
Q. But for the purposes of --
A. Yes.
Q. If we look at the agreement. I'm sorry, let's move on
to page {D1/1/3}. It says:
"Each expert's approach to writing his report, and
to this joint memorandum ... could broadly be one of
three possible approaches:
"a) To focus mainly on negative points found in the
disclosed documents about where Horizon may have fallen
short.
"b) To focus mainly on those aspects of Horizon
which were intended to achieve robustness and
reliability, and the evidence implying that they
succeeded."
Or:
"c) To provide the court with a clear foundation for
understanding the design and operation of Horizon; then,
building on that foundation, to provide a balanced
assessment of the ways in which Horizon succeeded ..."
A. Mm.
Q. Now, that's what you agreed with Dr Worden. If I could
just then go down to what you describe as areas of
disagreement. Second sentence:
"The issues are about how Horizon and its
interactive components operated and the processes
employed by Post Office and Fujitsu in supporting these
systems and the data within.
"Whilst my report will take a balanced approach, it
is the case that many of the issues require a deep focus
on the occurrences of bugs ... as well as the potential
for modification of transactional data. Whilst context
will be provided as to how Horizon should work in
typical circumstances, it is the non-typical operation
where focus will be placed."
That is right, isn't it, that you focused on things
going wrong, that's what you were looking for?
A. Yes, I looked for things that have actually gone wrong
and then followed the process through to find out how
they were dealt with.
Q. When you engaged in that process, you have already
described the team of people that you had assisting you.
Did anyone else help you? Did anyone else put forward
documents for your consideration that might be included
in the report?
A. No.
Q. Did you speak to any individual claimants, for example,
and get any information from them?
A. No.
Q. Going back to you and your team, you obviously by
16th October couldn't review all the PEAKs you had been
given, you had been given over 200,000, so you used
intelligent search techniques, didn't you?
A. Yes.
Q. Presumably you used those techniques for the whole
corpus of documents, not just PEAKs, but the KELs and
other documents as well, is that right?
A. That is correct.
Q. What other documents did you search across, can you
remember, at that time?
A. I don't have a list of all the documents. There is
a list in my first report, but all the documents all
went into the same platform, so it will have been every
document that had been formally disclosed at that point
in time.
Q. Thank you. Your report says you are of IT Group, when
you describe yourself at the beginning.
A. Yes.
Q. Can I ask what your position there is?
A. Director.
Q. As I understand it, correct me if I'm wrong, IT Group
offers a disclosure and search capability, doesn't it?
A. It does indeed.
Q. I read from your website, it is not in the bundles:
"IT Group's powerful e-disclosure and digital
investigation solution, Intella Connect, removes that
headache by enabling you to intelligently search, filter
and review large volumes of electronic data with speed
and ease."
Is that a fair description of the service that
IT Group provides?
A. It is indeed.
MR JUSTICE FRASER: Forgiving the split infinitive.
MR DE GARR ROBINSON: I'm sorry, my Lord?
MR JUSTICE FRASER: I said forgiving the split infinitive,
"to intelligently".
MR DE GARR ROBINSON: I do not think it is possible to
forgive. Others may disagree.
So is it right that you manage the e-discovery team
in IT Group?
A. Yes. I am the director that is responsible for that --
Q. So it is fair to say, isn't it, that you know quite
a lot about intelligent searching of documents?
A. Yes.
Q. And you have no difficulty, I'm not saying it is easy,
in searching through, for example, 200,000 PEAKs in lots
of very clever ways that I am sure I couldn't think of?
A. Yes.
Q. Would that also be true of your team of assistants?
A. Yes.
Q. And they would have used these intelligent search
techniques to go away and find the sort of documents
that you wanted them to find?
A. They would.
Q. At your daily meetings?
A. Yes.
Q. I imagine that your search facility is far more powerful
than searching for words and symbols within a certain
distance from each other? That's the kind of thing I'm
used to.
A. We didn't employ any artificial intelligence, we only
used the normal standard search techniques.
Q. Would I be right in thinking that with your experience,
you and your team have developed all sorts of tricks of
the trade with searches to find -- to unearth things
that the rest of us laymen might find more difficult to
find?
A. I'm not sure which tricks of the trade you are referring
to. We do see ourselves as being quite advanced in what
we can find within documents, yes.
Q. Could you give me an idea of some of the intelligent
searches that you would have used in a case of this
sort?
A. So typically we were looking at documents that were
talking about discrepancies, that would be talking about
errors, bugs, defects, imbalances, things like that.
Q. You and your team would search for these documents and
then they would be reviewed, would they?
A. Yes, we would typically try to understand -- because
often you will find one document but then you need to
understand where that document sits on the timeline, so
then you may have to go back a number of documents or
forward a number of documents -- we have the ability to
put them effectively in chronological order -- to try
and see who within the organisation was talking about
these themes.
Q. Presumably you would review the documents that you
found, but members of your team, when they found
documents that they thought you were looking for, they
would review them and then they would bring them to you
for your review as well if they thought they were good
ones?
A. Yes. It would be a process of them applying a tag to
a document. So essentially they were ruling out
irrelevant documents.
Q. Would they do their own reviewing and then provide you
with documents that they thought were suitable or did
you review everything that they tagged?
A. There was documents that, as the process went on, they
become better versed with spotting things that I may
find of interest within the document, but initially
I would ask them just to rule out documents that were
clearly irrelevant and I would review the balance of
that, but as things improved they were spotting themes
within documents, tagging it for my attention, and then
I would see the tag and would be able to go through
that, and then each day we would refine that process.
Q. Would it be fair to say that the total number of
documents that you and your team reviewed is larger than
the total number of documents that you reviewed?
A. Yes.
Q. I would be right in thinking, wouldn't I, that the
documents that you refer to in your report are all
documents that you yourself had reviewed?
A. Yes.
Q. Could you give me an indication -- the number of
documents the whole team reviewed was higher than the
ones you reviewed. Could you give me an idea of how
many documents your entire team would have reviewed in
this process?
A. I wouldn't have that number, I'm sorry.
Q. If we then go in the same document to {D2/1/83}, please?
MR JUSTICE FRASER: Do you want to stay in the first joint
statement?
MR DE GARR ROBINSON: I'm so sorry, I thought I was in JC1.
It is {D2/1/83}.
Just looking at paragraph 5.114, you say:
"Regarding the extent of potential errors within
Horizon I have analysed 5114 Horizon ... (KELs) to
determine the scope of potential bugs or 'PEAKs' ... Of
these 5114, I have found that 163 contain PEAKs that
could be of significant interest and of these [67] are
referred to in the report."
A. 76.
Q. I'm so sorry, 76. So you personally looked at 5,114
KELs, did you?
A. KELs and PEAKs, is that not? Sorry, yes. Yes.
Q. So you looked at them, so your team would have looked at
more?
A. They may have looked at the same number but they
possibly looked at more, yes.
Q. On top of those KELs you looked at PEAKs, as well?
A. Mm.
Q. If we go back to paragraph 1.28 at page {D2/1/20}. It
may be you will agree with this without having to look
at it. At this stage on 16th October -- it is 1.28 on
page 20 and you say:
"Of the [218,000-odd] PEAKs disclosed by POL I have
in the interest of expediency, used intelligent search
techniques to initially review those that might
specifically relate to branch accounts. I have
therefore reviewed 1,262 in the limited timeframe
allowed since disclosure."
Would I be right in thinking that your team would
have reviewed a considerable number more than that?
A. Yes.
Q. Just to be clear, the focus of your search was what?
Was it branch account problems?
A. Yes.
Q. Were there any other problems?
A. The searches were surrounding discrepancies but, of
course, a search such as that will come up with lots of
things that are largely irrelevant. But yes, branch
account problems.
Q. And would that 1,262 PEAKs you referred to there be on
top of the 163 PEAKs that you found having regard to the
5,114 KELs you read?
A. Sorry, could you ask that question once more.
Q. If you remember back at paragraph 114, I'm asking this
question in a very maladroit way, you said: I looked at
5,000-odd KELs and I found that 163 contained PEAKs that
could be of significant interest?
A. Yes.
Q. I get the impression therefore that you looked at the
KELs and then, having regard to the KELs, you then
looked at the PEAKs referred to in those KELs?
A. Yes.
Q. And you found 163 that you thought were interesting and
you refer to 76?
A. Yes.
Q. The 163, was that part of the 1,262 that you regarded --
that you found that might specifically relate to branch
accounts?
A. It is likely to be a subset of that.
Q. Thank you.
A. Because what you find is when you look in a KEL and it
lists a number of PEAKs, you go to have a look at the
PEAKs. Sometimes it is a PEAK of interest but it
actually doesn't contain any of the words you were
originally searching for, but once you have read it you
understand why it is of interest.
Q. Yes. If we move forward three and a half months, so we
are now at the beginning of February, and you produce
your second statement which is very long and that's at
{D2/4.1/16}. If we could look at that document at
page 16, please. You say in paragraph 3.20:
"I have now reviewed more of these records using
text search criteria and filtering. This has enabled me
to address some issues more thoroughly and has enabled a
more in-depth analysis in relation to the extent of the
Horizon Issues and the overall robustness ..."
So would I be right in thinking that you had looked
at more KELs?
A. Yes.
Q. And I think it must be obvious you looked at more PEAKs,
as well?
A. Yes.
Q. Had you looked at more other documents as well?
A. Yes.
Q. Could you give some indication of the sort of other
documents you were looking at in the intervening time?
A. There was a number of OCRs and OCPs but I do not think
we had the full family of them at this point in time.
There was a number of the design documents, the process
documents. There was a lot of reports about how
feedback of the support of issues was being fed back to
Post Office. There was documents on many themes.
Q. How many more KELs -- it may be that you can only
estimate, it may be impossible to estimate -- do you
think you had looked at by this time?
A. It would be a crazy guess. I don't know. I honestly
don't know.
Q. What about PEAKs? How many more PEAKs do you think you
had reviewed by this time? I would suggest it is likely
you looked at a lot of them but that may be unfair.
A. It will be a lot of them. There wasn't a linear review
where I sat and started and went next, next, next. That
wasn't how it happened. But after searching for
a document, deciding it was of interest, and then
picking out some of the themes and then searching for
those themes, you might identify conceivably another
five hundred documents of which you might flick through
those very, very quickly just to see whether they are of
any interest, they are in the right date range.
Q. Did your team, the four people that's mentioned that we
have discussed, did they help you at any time during
this three and a half month period?
A. Any members of the team? Sorry?
Q. Did the whole team, all four members, help you at any
time?
A. Running up to Coyne 1, the first report, then it was the
team members that you see there. For the second report
it was just Siobhan Forster.
Q. Did she spend a great deal of her time helping you
during that period?
A. Yes.
Q. She was doing intelligent searches as well?
A. Yes.
Q. And she was reviewing PEAKs and other documents as well?
A. In order to bring them to my attention. The process was
very similar all the way through really.
Q. I see. Would I right in thinking that the focus during
that period was again PEAKs and KELs and other documents
indicating problems with branch accounts?
A. We were also looking at elements in Dr Worden's report.
So there will have been searches trying to understand
more about themes surrounding his opinion.
Q. I see. Have you reviewed further documents since you
produced your second report on 1st February?
A. Yes, I think I'm right to say there has been documents
that have been disclosed since then. Yes.
Q. Have you -- let's take it in stages, have you reviewed
more PEAKs since then?
A. I think there was a small disclosure of about 1,200
PEAKs that have been reviewed.
Q. What about OCPs and OCRs, have you reviewed a number of
those?
A. Yes.
Q. Could you give an estimate of how many you have looked
at?
A. It will be thousands but I don't know what the precise
number would be.
Q. MSCs as well, the three databases of MSCs, you've
reviewed a number of those as well, haven't you?
A. MSCs were a very significant challenge because they were
disclosed in three separate Excel sheets, none of which
appeared to marry up with each other, so it was a task
because there were literally millions of lines of code
in that. So I have searched within those documents.
Q. The OCPs and the OCRs and the MSCs, what have you
searched for? FAD codes, that kind of thing?
A. I have searched for the word "FAD", because that might
indicate that it is describing something for
a particular branch, but I have not searched for a FAD
code as such.
Q. So you have looked for FAD because that is a reference
to a branch. Each branch has its own FAD code, doesn't
it?
A. FAD is an indicator that the document might be to do
with a branch, but you do have to be very careful with
that because simply searching for "FAD" returns hundreds
of documents and you actually find that it is just
a pro forma document with the word FAD in it and there
is a space next to it and there's nothing in it.
Q. Presumably this is where intelligent searches come in,
which I would be --
A. Absolutely.
Q. -- incapable of doing.
A. Absolutely.
Q. And what other things were you looking for when you went
through OCPs and OCRs and MSCs?
A. We used various searches for "subpostmaster" that would
indicate that there was potentially dialogue with the
subpostmaster, to see whether there was any advice given
to the subpostmaster or what type of information was
being requested from the subpostmaster. So all the
variations of subpostmaster were used as search terms.
Q. Thank you, Mr Coyne. We now have a sense of what you
looked at when you produced your first report and what
additional documents you looked at when you produced
your second report. We've already spent some time
talking about your first report but there are some extra
questions I would like to ask you about it.
If we could go to {D2/1/26} paragraph 3.7 which is
at page 26. You say:
"With regards to Issue 3, whilst the present-day
version of Horizon supported by manual human support may
now be considered as relatively robust in the spectrum
of computer systems used in businesses today it has
undergone major modifications in its history. It is
likely that in 1999 when it was first commissioned, and
in 2010 when it was significantly upgraded (to Horizon
Online), it was less robust."
So you are saying there that the level of robustness
may have varied over time?
A. Yes.
Q. And you are focusing in particular on the early days of
Legacy Horizon and the early days of Horizon Online?
A. Yes.
Q. Why do you focus on those periods?
A. Because I believe it is an incontrovertible fact that
there was a larger number of problems after the launch
of the first version of Horizon and after the upgrade to
Horizon Online. Certainly I was in court when shortly
after the launch of Horizon Online there was talk about
the red alert or the red care situation, shortly after
Horizon Online, where it needed some intensive care
because of problems.
Q. Do you mean during the pilot project when only a very
small number of branches were actually using
Horizon Online? Is that what you are referring to?
A. Certainly there was problems during that period but it
was real branches that were being used. It wasn't
a testing environment or anything like that.
Q. Would I be right in thinking, Mr Coyne, that you have
chosen the wording of that paragraph quite carefully?
A. I would like to think that I choose all of my words
carefully.
Q. Very good. That's fair. I note that you are not saying
here that Horizon was definitely not robust at some
earlier time. That's deliberate, isn't it?
A. Yes.
Q. You are merely saying that it is relatively robust now,
it may have been less robust at some earlier dates, but
you are not saying it wasn't relatively robust on those
earlier dates, are you?
A. No, I'm not. That supports what I have attempted to
assist the court with all along. It isn't a binary
situation that it is either robust or not robust, it is
a spectrum, and throughout those periods it was less
robust further to the left of the spectrum.
Q. Even during those early periods in 1999/2000 when
Legacy Horizon was rolled out, and in 2010 when
Horizon Online was rolled out, would I be right in
thinking it is not your judgment that it was not
relatively robust even then?
A. I think that is quite a difficult question to answer
because that would require very specific focus on those
periods and a very specific benchmarking of the position
at those two periods which I haven't conducted
specifically.
Q. You say that you have conducted the benchmarking now?
A. Yes.
Q. Are you suggesting that you haven't conducted any
benchmarking in relation to any earlier period?
A. Certainly I have looked at the number of defects that
have been in existence and the number of requests to
support throughout the whole life of Horizon and there
are peaks -- sorry, that is a bad word.
Q. Spikes.
A. Spikes after the launch of the Legacy version of Horizon
and then after the upgrade to Horizon, and I completely
understand why with any new system and with any major
upgrade there's going to be an increase in problems.
That's why I have said during those periods the system
will have been less robust because there was problems
being ironed out both in the technology -- the
programming -- the networking and the support processes.
All these problems were being ironed out.
Q. So outside those periods, how long did those periods
last? Are we talking about 1999 to 2000 for
Legacy Horizon, is that the sort of period you are
talking about?
A. No, I mean the actual period extends, it is a curve that
slowly drops over the course of the next couple of
years. So, again, it is not a binary: it becomes stable
at some point in time, the faults stop at some point in
time.
Q. So you are suggesting -- you have a graph in mind which
you are using as a basis for forming this judgment. Is
there a graph somewhere that indicates what you are
talking about?
A. There's certainly two images, I think one in my report
and one in Dr Worden's report, that show -- that have
correlation of spikes of activity around those two
areas.
Q. Could you tell me where in your report we will find this
graph? I'm afraid I'm not in a position to suggest
a page number. Do you remember where it is?
A. There is one in my first report at -- it is page 193.
Q. Thank you. A graph. Is this your first report or your
second report?
A. This is my first report.
Q. My page 193 is a figure 3.
MR GREEN: It is bundle page 206, internal 193.
MR DE GARR ROBINSON: I'm looking at the wrong numbering.
MR JUSTICE FRASER: Are we looking at your page 193 in the
top right-hand corner?
A. Sorry, my Lord, yes.
MR GREEN: Page 193 top right.
MR DE GARR ROBINSON: 193 at the top and 206 in the bundle
{D2/1/206}.
I'm looking at that graph. What is this a graph of?
A. So this is a graph of BIMS.
Q. Could you explain what they are?
A. They are business impact statements that are made
between Fujitsu and Post Office.
Q. It looks as if the spike had more or less finished by
about 2011. So would it be fair to say that if you are
treating this as evidence of less robustness, from about
2011 it stabilises, would that be right?
A. I think it probably could be argued that it stabilises
around 2013 but it is the curve that I attempted to
describe before.
Q. So that's the basis upon which you form that judgment
that it was less robust during these early periods, is
that right?
A. That is one of the graphs that illustrates my opinion,
yes.
Q. Now, if we go back to -- and is there a similar -- have
you done a similar graph for Legacy Horizon or is there
nothing?
A. I believe Dr Worden has a similar graph that goes back
to Horizon.
MR JUSTICE FRASER: You said there were two, I think, one in
your report and one in Dr Worden's report.
A. Yes.
MR JUSTICE FRASER: You think he has got one as well?
MR DE GARR ROBINSON: This may be wrong but it may be at
{D3/1/187}.
MR JUSTICE FRASER: Let's have a look.
MR DE GARR ROBINSON: There is a table of loss amounts by
year. Do you see that? Is that the one you have in
mind?
A. No, I do not think it is. I think there is a different
one.
MR JUSTICE FRASER: Is it {D3/1/129}? I'm just doing this
to try and help Mr de Garr Robinson, I'm not
interfering. Have a look at D3/1/129.
MR DE GARR ROBINSON: Is that the bundle reference?
MR JUSTICE FRASER: That is the bundle reference. Could you
call that up on the common screen, please. 129.
MR DE GARR ROBINSON: It is the same graph as at 187.
MR JUSTICE FRASER: Oh, is it the same?
MR GREEN: I think it is the same.
MR JUSTICE FRASER: I won't try and help any more.
MR GREEN: It has a different title but it is the same
thing.
MR DE GARR ROBINSON: Do you think you can --
A. Yes, if you give me a moment I can go through these
paper copies.
Q. It might be the next page. If we go to the next page on
the screen, is that it? No sorry.
MR GREEN: {D3/1/130}.
MR DE GARR ROBINSON: Can we look at page 130, please, in
this document. {D3/1/130}. Thank you. Is that it?
A. I think it is that actually, yes, because that
illustrates the same peak/spike, that illustrates the
same spike around Horizon.
Q. But it doesn't indicate any spike around Legacy Horizon,
does it?
A. Not for KELs, no.
Q. I'm wondering, Mr Coyne, whether this judgment is really
based upon your experience, that your experience of when
you roll out a new system you are going to have teething
problems and then, once it is rolled out, the position
stabilises. Is that really what your judgment is based
on?
A. Part of my judgment is based on my experience but it is
certainly the case, and we might not have a graph for
this, but when you look at the number of PEAKs over the
period, there is a spike in the PEAKs at the start of
Horizon.
Q. So just to backtrack slightly. What you are saying is
Horizon is robust, relatively robust now. Do you accept
that it has been robust for a significant period or
would you not be prepared to say that?
A. It has been relatively robust apart from the early years
of the two launches, Horizon Legacy and Horizon Online.
Q. So both Horizon Online and Legacy Horizon have been
robust for most of their lives but there were initial
periods where they were less robust?
A. Yes.
Q. In relation to those initial periods -- and do correct
me if I'm wrong -- you are not saying they definitely
weren't robust during that period, you are simply saying
they were materially less robust, is that right?
A. Yes.
Q. Thank you. That's very clear.
Let's go back to section 3 of your first report. If
we can go to {D2/1/26}, paragraph 3.10. Perhaps I don't
need to ask you about that in the light of what you have
just told me. Instead what I would like to do is to go
to page {D2/1/77} of the same document, please. You
will recall that I suggested to you that there was
a danger of forming an impression from your first report
that although you clearly state that Horizon is
relatively robust, you don't say anything good about it.
What you tend to do is to say bad things about it.
I suggested to you that was because you were knowingly
giving a poor impression of Horizon. Would that be fair
or would that be unfair?
A. That would be unfair.
Q. Why didn't you then say more about the good aspects of
Horizon that led you to the conclusion that it was
relatively robust?
A. Because this -- because the question of robustness was
in the context of would that robustness prevent or
reduce the risk of bugs, errors and defects causing
discrepancies. So the many good things about Horizon,
and I suppose if you are going to start talking about
good things, how far do you go? I don't know where you
stop with them. This, as I understand it, is about
whether any of the failures that have occurred within
Horizon have led to defects or discrepancies.
Q. But in the first joint statement you indicated that you
were going to adopt a balanced approach. I fully
understand why you are looking at problems, and problems
which were relevant to branch account impact. I'm not
suggesting you shouldn't have done that. But if you are
engaging in a properly balanced approach you have to
explain the good things with as much care and clarity as
you explain the bad things. And what I would like to
suggest to you, Mr Coyne, is that you devoted much more
attention to talking about the bad things than you ever
did talking about the good things?
A. Yes, I will accept that is fair.
Q. And why did you do that?
A. Because my understanding of the instruction is to try
and identify the risks that are associated with bugs,
errors and defects, and there are many other issues, but
if they had an impact or could have had an impact on
branch account.
Q. But --
A. Sorry. So in order to do that you have to focus on what
actually happened, is my understanding, and then from
understanding what has actually happened understand what
the impact of it is.
Talking about all the nice things in Horizon that
happen in the high 90% of the time I don't believe
delivers any real value to my instructions.
Q. But in deciding whether or not a problem that arose in
Horizon actually had a branch impact, one of the things
that you have to consider is whether the countermeasures
that surround the Horizon system operate effectively or
not?
A. I absolutely agree with that --
Q. And you accept that, don't you? You accept that it is
relevant to the judgment of robustness as to how good
the countermeasures were?
A. Yes. But the way to do that is to identify the fault in
the first place and then have a look at the
countermeasures, whether they were positioned correctly,
whether they worked or not.
Q. Can I suggest to you, Mr Coyne, that that's not what you
did in your first report at all. Most of your first
report was devoted to problems that actually didn't
relate to branch impacts at all, and when you did talk
about problems that related to branch impacts you didn't
consider whether the full panoply of countermeasures
would have corrected or reacted to or identified the
branch impacts you were talking about?
A. That could well be to do with not having all of the
documents that was requested at the time.
Q. Well, you didn't do it in your second report either, did
you? You did the same thing. It was more of the same
in your second report, wasn't it?
A. I don't believe that's the case, no.
Q. So you are suggesting in your second report you did
consider how good the support process was and you did
consider, where you had identified a bug that may have
a branch impact, you did consider the extent to which it
was likely that that impact would have been identified
and sorted out?
A. I spend a large amount of the second report addressing
specifically the countermeasures that Dr Worden raises,
paragraph by paragraph.
Q. And your overall view of those countermeasures, bearing
in mind that your judgment now is that Horizon is
relatively robust, yes?
A. Mm.
Q. Your overall view of those countermeasures is that they
are relatively good, yes?
A. I think it is a danger looking at them in the overall
because there is a number of them that I don't believe
are good. But broadly, if the countermeasure is
positioned correctly, in the vast majority of occasions
they should work.
Q. What I would like to suggest to you, Mr Coyne, is that
it follows from your overall judgment that Horizon is
relatively robust now. It follows that overall your
assessment of the countermeasures that Dr Worden has
identified is also that they are relatively good. Would
that be fair?
A. Yes. Each of the countermeasures that Dr Worden has
suggested are in place and at some point in time have
been in place within Horizon. We don't know when they
were in place within Horizon on a number of occasions
but they have been in place.
Q. Please answer my question, and let me be clearer about
it because it is quite important. What I'm suggesting
to you is that it is part and parcel of your judgment,
your overall judgment that Horizon is relatively robust,
that the countermeasures operating together, the
countermeasures that are in operation in relation to
Horizon, that they are relatively good as well. It is
not your view that operating together, taken as a whole,
the countermeasures are of poor quality or that there's
anything wrong with them, is it?
A. No, it is not.
Q. I'm grateful for that frank answer. It is fair to say,
isn't it, that your overall judgment is that the
countermeasures together, there may be individual
instances where you are not so sure, but overall your
judgment is that those countermeasures work well in
Horizon?
A. Yes, but you can't disregard the fact that the system
has changed throughout the last --
Q. Just --
A. Sorry --
MR JUSTICE FRASER: Hold on a second, I would like to hear
the answer, and then you can either re-put the question
or follow on.
Go on, Mr Coyne.
A. We can't disregard the fact that the system has changed
over the course of the last 20 years and whatever the
countermeasures may be today, they will likely be
different in Horizon Legacy, different countermeasures,
possibly the same aspiration to catch defects at certain
point in time, but they will be different, they will be
constructed differently and they will be positioned
differently. So we can't --
MR JUSTICE FRASER: Mr Coyne, just pause there. All right.
Mr de Garr Robinson is going to re-put his question if
he wants to.
MR DE GARR ROBINSON: I am. I'm talking about the period of
time relating to the current configuration of Horizon
and going back in time until the point in time after the
initial teething problems were resolved. You understand
what I mean?
A. All the way back to --
MR JUSTICE FRASER: Can we use years?
MR DE GARR ROBINSON: Let's go back to 2012 for the sake of
argument to make it safe, because I think that is a year
you suggested. So from 2012 to 2019 your view is that
Horizon is and has been relatively robust, correct?
A. Yes, I would say that, but I would be very clear that
robust does not mean that it hasn't suffered
significant --
Q. Everybody is agreed that robustness doesn't mean
perfect.
A. That's fine then, yes.
Q. Which is the point you are trying to make, is that
right?
A. Yes.
MR JUSTICE FRASER: What word were you going to add after
"significant", just out of interest?
A. Defects that have continued throughout the period.
MR DE GARR ROBINSON: So during that period, 2012 to 2019,
your judgment is, and you feel capable of making the
judgment, that Horizon is and has been relatively
robust, correct?
A. Yes.
Q. Now that judgment includes a judgment not only about the
electronic systems and so on, it includes a judgment
about everything including the countermeasure processes
surrounding Horizon, correct?
A. Yes. There has to be a number of caveats about that and
I explained before that I don't really know what happens
within Post Office to correct defects if Fujitsu has
spotted them. So I can't comment really on what that
process would be.
Q. But in relation to the -- nonetheless you have
sufficient information to allow you to form an overall
judgment as to robustness and your judgment is that
during that period the system itself and the
countermeasures around it were relatively robust?
A. Yes.
Q. The reason why I ask you that -- and just to be clear,
the same is true of the period Legacy Horizon, shall we
say from 2001 to 2010, would that be a fair period?
A. Yes. Perhaps one year after that.
Q. Okay, 2002 to 2010. During that period your view is
that Horizon was relatively robust?
A. Yes.
Q. And that included the countermeasures surrounding it and
supporting it?
A. Yes.
Q. My question to you, Mr Coyne, and we will be coming to
countermeasures later, but it is quite a striking fact
that if you read the section of your report that
addresses countermeasures, your second report, it goes
over dozens and dozens and dozens of pages in your
section 5 -- and as I say, we will come to it -- there
is no hint of your judgment that overall the
countermeasures during those periods have operated --
have been relatively good. Why is that?
A. I don't know. It certainly wasn't a conscious decision
to leave anything out, and --
Q. Could I suggest to you -- I'm sorry, I'm interrupting
you.
A. Sorry. And because I have found bugs, errors and
defects throughout that period, that is an illustration
that however good the countermeasures were, that they
were overcome at various points.
Q. We will come to that, I will be asking you quite a few
questions about whether the countermeasures were
overcome by the bugs that you have identified. But my
question to you, Mr Coyne, remains that you are quite
happy to go on at great length about how much you
disagree with Dr Worden and about all the problems you
found in relation to countermeasures. Nowhere do you
actually say what you have very fairly said to me in the
last half hour, namely that the countermeasures actually
during those long periods of time have worked quite
well. Why would you not say that given that you want to
give a balanced view?
A. I believe the view that I have given in the reports is
balanced --
Q. Where -- I'm so sorry, I'm speaking too quickly. Please
carry on.
A. I don't see that utilising many pages of text to talk
about the various aspects of Horizon's countermeasures,
which may well be similar to how Dr Worden set them out,
they may well operate in much different ways, we don't
really know, but we can observe that they are likely to
operate in those ways. I do not see there is a great
deal of value when it is a fact, and what I believe the
court was asking in the Horizon Issues was to address if
it is possible that they have failed. So it is the
failure side of it.
Q. I see. So you regarded it to be your job to indicate to
the court whether there were occasions on which Horizon
had failed. That was the essential endeavour that you
were engaged in when addressing Horizon Issues 1, 3, 4
and 6, was it?
A. I believe that a number of those Horizon Issues ask that
specifically.
Q. Didn't those issues ask specifically how likely -- what
was the extent of the likelihood of problems occurring
in Horizon so as to cause problems in --
A. Yes --
Q. -- branch accounts?
A. Yes, that is right, but that doesn't ask for me to point
out a list of all the possible countermeasures.
Q. But isn't it relevant to the judgment as to overall
likelihood how well the countermeasures work? I mean
Dr Worden recognised that was the position, that's why
he spent so many pages of his report explaining the
architecture of Horizon, explaining how the systems
worked and explaining how they operated in relation to
each other. Then in your response, in your second
report, what you did -- what I suggest that you did is
you sat in your armchair and took pot shots at various
points that he made without giving any sense as to
an overall judgment as to whether he was right or not
that the countermeasures did work relatively well?
A. No, I don't believe I did that. What I did was I looked
at the entire landscape of Horizon's operation,
identified the percentages were -- there were weaknesses
within the process, and then focused on that
specifically to find out whether there had actually been
any bugs, errors and defects, and then looked at why the
countermeasures or controls may or may not have worked
after that bug, error or defect had triggered. Because
the key is how the system operates when it doesn't
operate properly, not how to operates when it does
operate properly.
Q. I have been side-tracked, I have allowed myself to be
side-tracked.
We were going to look at page {D2/1/77} of your
first report and I'm about to ask you a question about a
paragraph on this page. The starting point of the
question is that you agreed with me that your judgment
is that Horizon, both Horizon Online and indeed
Legacy Horizon, for much of its life has been relatively
robust?
A. Mm.
Q. If we go to paragraph 5.88, you say:
"In my position as an expert I am unable to estimate
the level of the Horizon system's robustness. Given the
size and age of Horizon, I would however make the expert
assumption (based upon systems of similar magnitude),
that there are not many people who could. The sheer
enormity of the task to garner a thorough understanding
of the code which would be required to estimate
robustness is, in my opinion, nearly impossible."
So here you seem to be saying that it is actually
not possible to assess whether Horizon is robust or not?
A. No, to come up with a number. When I say the level, I'm
talking about a particular number or a percentage or
a ... (Pause)
Q. So what you are suggesting is that you don't have -- in
order to make an absolute judgment, to give an absolute
number of robustness, you would need to look at all the
code and look at everything else, is that right?
A. Yes, because there may well be a defect in the code on
one day which has a problem that eludes countermeasures,
and that code may then be fixed so it is in a different
position on day one than it is on day two.
Q. And you are saying that kind of enquiry is necessary in
order to make an absolute judgment of robustness?
A. In order to make an absolute judgment, yes.
Q. But it is not required in order to make a relative
judgment of robustness?
A. Yes.
Q. I would like you to explain why that would be the case.
Why would it not also be needed for a relative judgment
to be made, to benchmark against industry standards?
A. Because with a relative judgment you are considering all
factors in the round and it is still very subjective but
it is across multiple different factors that are
improving or getting worse.
Q. I understand you when you suggest that to make
a judgment about robustness involves an overall
assessment. That I understand. But my question to you
is why is it necessary to look at the code and look at
absolutely everything to engage in an impossible task to
make an absolute assessment of robustness, when it is
not necessary to do that in order to make a relative
assessment of robustness? I would just like you to
explain that if you could.
A. There is a number of different ways of assessing
robustness. By looking at the code at any one point in
time you could establish whether there are any bugs,
errors or defects in there. It would be a near
impossible task but it could be done depending on how
big the code base could be.
Q. How would it be possible? Just as testing is never
perfect. You have a set of code and you are about to
release it into live operation but you have to test it
first, you can test it as many ways as you can but you
are never going to spot every problem in it.
A. No, but --
Q. Are you suggesting that for the purposes of determining
whether a code is robust you could look at the code and
perform that judgment?
A. Yes, you certainly could look at the code and perform
that judgment because what you can do, and this has
often happened in the aircraft industry, they have
multiple different operating systems that control
different devices. So if there is a catastrophic
failure of one piece of code another piece of code will
take over.
So if by looking at the code you can establish that
is in there, then you can give a view of that being
completely robust, because if it fails in its entirety
there is another solution that can take over. So that
is how I can illustrate the code example.
Q. This is the last question I will ask and then I will
move on. If you are forming a judgment as to the
robustness of a system, don't you need to have regard to
the way it operated in operation?
A. Yes.
Q. Isn't that what you should be looking at, not millions
of lines of machine code? You are looking at how it
operates in practice, looking at how the countermeasure
operates and forming an overall judgment as to the
efficiency of the operation, aren't you?
A. That could be one part of it, yes.
Q. Wouldn't that be sufficient? Wouldn't be all you needed
to do?
A. No, because you will almost certainly be able to
identify a scenario where somebody or many people will
operate the system, the Horizon system, throughout
a year, or possibly even throughout the entire life
cycle of Horizon, and will never experience one problem
at all because they will not trigger the unusual
circumstances that unearths a bug. So if you went and
studied that one person you would ultimately conclude
the system is absolutely robust. I have watched this
person do 10,000 transactions per year for ten years and
it has never had a problem.
Q. What I'm attempting to explore with you, Mr Coyne, is
your attempt to explain why it is that on the one hand
you have reached your judgment that Horizon is
relatively robust, but on the other hand you can't form
a judgment on the level of robustness without looking at
all the code. You have suggested that it is to be
explained because one is a relative judgment whereas the
other one is an absolute judgment. And I'm asking you,
if it is necessary to look at the code in order to make
a decision about robustness, why isn't it equally
necessary to look at the code when making a judgment
about relative robustness? And I don't understand why
the distinction between relative and absolute robustness
should make the slightest difference.
A. Because we can look at the rest of the industry in
either banking or point of sale or stock control or all
the different factors that make up Horizon and look at
the types of defects that are seen in those industries
and that are tolerated in those industries and the
impact of those failures within those industries and it
is through that lens that I have said that Horizon is
robust relative to those other industries.
Q. Well, could I suggest to you, Mr Coyne, what you are
doing in your first report, in indicating that Horizon
is robust then suggesting that it isn't, then suggesting
it is impossible to tell that it is robust, what you are
trying to do is to devalue the judgment that you have
very fairly and very honestly made that Horizon is
robust. What you are trying to do in your report is to
give a negative impression, which is not balanced but is
designed to achieve a conclusion?
A. That's certainly what I was not trying to do, no.
MR DE GARR ROBINSON: Very well.
My Lord, is this a convenient moment?
MR JUSTICE FRASER: If it is convenient to you. Is it?
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: You are moving on to a new subject,
right.
We are going to have a short 10-minute break. The
usual rules apply. 3.20 pm.
(3.12 pm)
(A short break)
(3.24 pm)
MR DE GARR ROBINSON: Mr Coyne, before the break I was
suggesting to you that in your first report you were
seeking to give a negative impression about Horizon that
was not consistent with your genuine view that Horizon
was relatively robust and you have answered those
questions.
But I'm now going to explore another aspect of that
which is that in your report you display more interest
in pointing out the problems you found with Horizon than
with actually giving a fair assessment of the overall
position of Horizon and my questions for the rest of
this afternoon will be looking at that question.
I don't have time to take you to more than a few
examples which is unfortunate because there are so many
of them in your report. And that is a problem that
arises in cases of this sort. The documents you refer
to in your report, in fact both your reports, are often
long and involved with substantial complexities.
Do you accept that where an expert is referring to
a great number of documents of this sort which are all
being presented as supporting a view, giving
an impression of a certain state of affairs, it is
incumbent on the expert to ensure that he or she deals
with the document scrupulously fairly so as to avoid
giving any risk of a false impression?
A. Yes.
Q. Counsel won't have time to cross-examine the expert on
all of them, and the judge certainly won't have time to
go through all of them with a fine-tooth comb in his or
her own chambers afterwards, will they? So a judge is
relying on the expert to discharge their duty to ensure
that a fair view is being given of any document being
referred to.
A. Yes.
Q. Now, do you think you have done that in your reports?
A. Yes.
Q. The documents you've referred to in your reports, and
there are a great number of them, have you considered
them all carefully yourself before explaining the
inferences you draw from them or the constructions you
put upon them?
A. That was certainly my aim in including the documents in
the report, yes.
Q. Well, let's deal with a few examples. As I say, I can
only deal with a few. First of all, cost benefit
analysis. This is a theme of both your reports, that
bugs in Horizon are diagnosed and fixed on a cost
benefit basis. Is that right?
A. There is certainly reference within the documents about
looking at bugs and fixing them on a cost benefit basis.
Q. There is a claim made in both your reports that where
bugs that are known to cause shortfalls, those bugs will
not be diagnosed and fixed on a cost benefit basis by
Fujitsu, essentially?
A. Yes, and I think that that was one of the agreements
that Dr Worden and I arrived at in one of our joint
statements.
Q. We will talk about the joint statements in a moment.
I am sure you will agree that it is inevitable that
every business will take some decisions on a cost
benefit basis?
A. Yes, that is true. It must be considered here that
there is a different dynamic here between the user of
the system and the people who fix the defects. In
an organisation they may have their own internal
programmers and therefore they may not fix things on
a cost benefit analysis basis. If you have
a development team that's not your own resources but you
are paying for those resources then there could be
a different dynamic at play there.
Q. The point you are making you are applying to the fixing
of bugs in Horizon. You are saying, correct me if I'm
wrong, that the support process provided by the SSC
allowed relevant bugs to remain in the system in order
to save money?
A. Yes, and there is certainly a reference within
a document to that view being taken.
Q. Let's see how you develop the point. If we could go to
your first report {D2/1/27}, please. This is
paragraph 3.15:
"There are a range of measures and controls existing
in Horizon each designed to prevent, detect, identify,
report and reduce the risk of several multifaceted
errors. It is likely that during the life of Horizon
system that these measures and controls improved. It is
also reasonably likely that in the majority of cases the
measures and controls were successful."
Stopping there, Mr Coyne, I need to correct
something I put to you before the break.
I put to you that there wasn't a hint of anything
positive said about countermeasures in your report.
That's not fair. There are in the report a number of
occasions -- a number of sentences of this sort which
could be said to be positive statements. What I would
suggest to you, however, is that that sentence is
a dramatic understatement of the positive view that you
have of the Horizon countermeasures?
A. I don't believe that's the case. It is correct there is
a range of measures and I explain what they are designed
for.
Q. In the majority of cases. Your view is that it is only
in a fraction of a percentage of cases that these things
don't work, isn't that right? Isn't that what you said
to me this morning?
A. I said it would be a fraction, yes.
Q. A fraction of a percentage.
A. Yes, it is likely a fraction of a percentage.
Q. Very good.
A. But I do actually say there it is likely that during the
life of Horizon the measures and controls improved and
that in the majority of the cases the controls were
successful.
Q. Let's move on to the next sentence, Mr Coyne, because
having said that positive thing, which as I have
suggested to you is a dramatic understatement of your
actual view, you then immediately try to undermine even
that view because you then say:
"However, there is also evidence to indicate that a
cost/benefit analysis was applied to the fixing of
bugs/errors/defects and that the possibility of error
was not reduced as far as possible."
So you're saying, well, countermeasures are applied
in -- are likely to apply most of the time, but there is
this problem. And you are suggesting there is a problem
in the support process because the support process --
that's Fujitsu, the SSC -- appears only to have fixed
bugs on the basis of a cost benefit analysis.
You do see that that's what you are trying to do.
You are trying to weaken your positive statement in the
previous statement even further by relying on this cost
benefit and basis claim, is that right?
A. No, I'm trying to provide the full picture and context.
Q. By your use of the word "However", you are setting this
up as undermining even the limited positive statement
that you make in the previous sentence, aren't you?
A. No, I'm setting out that after making that statement you
have to consider what follows.
Q. You repeat this point in the third joint statement and
you refer to it half a dozen times in your second
report, don't you?
A. I do, but it is important because the question that was
asked was: was it reduced as far as possible?
Q. I'm sorry, where is that asked?
A. It is one of the Horizon Issues.
Q. I see.
A. So when you consider "as far as possible", cost benefit
shouldn't really be involved in that consideration. It
should be whatever is possible.
Q. Let me carry on with this line. What I understand you
to be saying here is that there is evidence that bugs
which had or may have had a non-transient effect on
branch accounts were sometimes not resolved on the
grounds that the costs to Fujitsu or the Post Office
outweighed the benefit to the subpostmasters. Is that
the claim you are making?
A. Yes, that's certainly what I have got from one of the --
there's certainly a document, I am sure I make
a reference to it, where there is a number of bugs or at
least one bug that's considered. The impact of that is
that it very rarely occurs and therefore it is not worth
investing the money in fixing it.
Q. Let's see how you do support it in your first report,
shall we? If we go to page {D2/1/97} at paragraph
5.161, you say:
"Whilst both Horizon and Horizon Online contain
a number of measures and controls designed to check
system integrity, these mechanisms have been shown to
have failed. This is a point agreed upon in the joint
statement. It has been identified that known
issues/bugs were often deferred and dealt with on a
cost/benefit basis."
Here you have actually increased the scope and, if
I may say so, ambition of the claim. You are now saying
it happened "often". Is that your considered view of
the documents you have seen?
A. Could I just turn back, please, to the paragraph before?
Q. Of course. {D2/1/97}
(Pause)
A. It might be unfair for me to say "often".
Q. I'm going to come to whether it is unfair, Mr Coyne, but
first of all I'm going to ask you to analyse the
evidence that you rely on in support of the claim.
Because it is fair to say you are not saying it happened
once or twice over 20 years, you are saying it said it
happened "often", and over a 20-year period "often"
would usually mean dozens if not, in the context of
Horizon, hundreds of times. Would you agree with that?
A. No, I would take "often" to be ten or 15 times.
Q. So ten or 15 times over 20 years, you think it is fair
to describe that as "often", without bearing in mind the
context of a system which has undertaken something like
49 billion transactions during that period. 10 or 15
times as opposed to 49 billion transactions?
A. But this doesn't relate to the number of transactions,
does it? It relates to changes to the system. I do not
think the two correlate.
Q. Is it your claim then that your understanding of the
documents you have read is that there have been 10 or 15
occasions when a decision has been made not to fix a bug
on a cost benefit basis. Is that your understanding of
the evidence you have seen?
A. There are certainly a number of them. It might be
incorrect of me to say "often". There are certainly
a number of them.
Q. Well, I'm going to suggest to you, just so you
understand where I'm coming from, Mr Coyne, that the
evidence gets nowhere near anything like that and that
if you didn't know it, you certainly ought to have done.
So you can expect some questions from me about that.
You repeat the claim in paragraph 5.199 at page
{D2/1/108} where you say:
"Whilst both Horizon and Horizon Online contain many
measures and controls for ensuring system integrity,
these mechanisms do/have failed. It has been identified
that known issues/bugs were often deferred and dealt
with on a cost/benefit basis."
So you repeat the word "often". This is a theme of
both your reports, isn't it, that this is something that
happened on a regular basis?
A. It is certainly something that has happened and I have
noted on a number of occasions.
Q. If we go to page {D2/1/98}, in support of the claim
that's made in 5.161 you give one piece of evidence and
that's a risk and compliance meeting minute of
18th September 2013.
A. Mm.
Q. Let's look at the minutes. The bundle reference is
{F/11/40}.
(Pause)
MR JUSTICE FRASER: Sorry, it is {F/1140/1}.
While the operator is doing that, I think this is
one of them that was subject to the review. But it
doesn't matter, this isn't a specific point on that.
But insofar as you review documents and produce versions
less redacted, can someone just make sure that the more
up to date version goes on the database rather than the
original one?
MR DE GARR ROBINSON: My Lord, yes. All the lesser redacted
versions I think are certainly in the trial bundles.
I stand to be corrected if that's not right. I don't
believe this is one of them --
MR JUSTICE FRASER: This might not be one of them.
MR DE GARR ROBINSON: -- in any event.
You will see, Mr Coyne, this is dated
18th September 2013, it is the Post Office Risk &
Compliance Committee.
A. Mm.
Q. And what you rely on is at page {F/1140/3}. So this is
what you are relying on in support of your contention
that bugs were fixed -- often fixed on a cost benefit
basis:
"Discussion:
"It was reported that following the recent
Ernst & Young external audit four risks been identified.
Three of the risks raised had been addressed however the
final risk, relating to the communication by Fujitsu of
changes made to the Horizon system, was still
outstanding.
"It was identified that it would cost over
£1 million to implement the mitigation being suggested
by the audit and that this was not proportionate to the
risk being managed."
So certainly an example of a cost benefit analysis
being applied.
"Decisions.
"The Committee agreed that the risk be accepted with
Dave Hulbert as the owner and Lesley Sewell being
ultimately responsible."
So that's the evidence you rely on for your
proposition.
A. That --
Q. And as you say in paragraph 5.161, Ernst & Young had
identified four risks. Three have been addressed but
the final would cost over £1 million to mitigate.
If we look at the risk itself, that is in
{F/1127/1}. This is a paper for the Risk & Compliance
Committee, and if we look at the background,
paragraph 2:
"2.1 The 2012/13 Ernst & Young IT audit found no
significant exceptions but did identify a small number
of improvement opportunities. Four high level
improvement opportunities were recorded. Three have
progressed and are either complete or in the process of
completing. For one, IT&C believe we have sufficient
process and mitigation in place to accept this risk.
This paper is to highlight this decision to the Risk &
Compliance Committee."
The next paragraph reads:
"2.2. The specific observation was with regard to
Change Management Monitoring control. The actual
observation read 'Management should make use of a system
generated list of changes in performing the monitoring
control to further enhance its effectiveness'.
"2.3 The risk being that changes may be made to the
system that are not approved and not found through
monitoring."
A. Yes.
Q. So what's being said here is there needs to be
a change -- or what's being suggested is that it would
be an improvement to the system if the change controls
applied within Post Office were made automatic so that
every time a change was being made to the system, this
would be automatically notified, electronically notified
to Post Office that it was happening. Do you see?
A. This is a very important aspect. So what this is saying
here is the current situation is that Fujitsu will make
a change to the Horizon system and it is not clear
whether Post Office will have either approved that
change or will be aware that that change has been made.
There is a weakness in the process that needs to be
addressed.
Now, on the basis --
Q. Wait, can I stop you there for a moment. Are you
suggesting Ernst & Young were saying there was
a weakness in the system that had to be addressed?
A. They are saying there needs to be a change to the -- a
change to the management monitoring control.
Q. Look back at paragraph 2.1, Mr Coyne:
"The 2012/13 Ernst & Young audit found no
significant exceptions but did identify a small number
of improvement opportunities."
Even looking at this document you can see that the
characterisation you have just adopted is simply not
right. These are improvement opportunities. These are
not problems that need to be addressed, are they?
A. From the text that's being said there about the changes
that need to be made then they are important.
Q. That's your judgment, is it?
A. Yes.
Q. It is not said anywhere that I have seen. Have you seen
it said somewhere in one of the documents?
A. I believe that there is more of an Excel spreadsheet
type layout to this document because this is the minutes
of the meeting that reviews the observations from
Ernst & Young. There is another Excel spreadsheet where
I think Ernst & Young provide their output.
Q. I see. So you are suggesting there is an Ernst & Young
document which doesn't suggest that this is
an improvement opportunity, it actually suggests it is
a real problem that needs to be addressed, is that your
contention?
A. Yes, I believe it sets out --
Q. Very good.
A. Sorry, I believe it sets out more context than what is
included here.
Q. When you wrote paragraph 5.161 you had read this
document, had you?
A. Yes.
Q. If we look at paragraph 3, I think you suggested before
that Post Office didn't know what changes were being
made and that's why it is so important. Is that your
logic?
A. Yes. What this is saying here is that there is a risk
that changes are made without being adequately informed
to the Post Office.
Q. We see the existing system described in paragraph 3:
"The Post Office Service Management team currently
monitor IT system changes on a monthly basis by cross
referencing known and approved changes against a list
produced by Fujitsu. E&Y observed that this could be
enhanced ..."
"Enhanced", not this is a serious problem that needs
to be remedied, they use the word "enhanced".
"... if the list was generated by the IT system
rather than by change records."
It is a relatively modest enhancement, isn't it,
wouldn't you agree?
A. I believe this is pointed out because there is the
possibility that the change records, which are a human
created document, may not have been completed adequately
and it would be better to have it flagged up that if the
system changes, an automated message goes to the
Post Office that the system has changed rather than rely
upon a paper system.
Q. We can all agree it would be better Mr Coyne.
A. Yes.
Q. The question is how serious the problem is that the
existing monitoring system is being applied.
If you look at the next paragraph:
"IT service management engaged with Fujitsu to
understand how this could be achieved and it was
concluded a very difficult and potentially expensive
approach to adopt this as all changes are recorded as
'events' within the IT system of which there are
multiple thousands per day with changes only being a
small percentage. The cost and difficulty in extracting
these specific change events on a regular basis would
outweigh the value in monitoring the change."
Do you see that?
A. Yes.
Q. One can see why someone might form that judgment, yes?
A. Yes, but if you break that down so that there are
multiple thousand a day, but the changes are only
a small percentage, so a small percentage of multiple
thousands is still quite a big number and that's changes
made to Horizon every day.
Q. Mr Coyne, you do seem to be deviating from the path that
I'm asking you questions about. We are not talking
about how many changes there were. We are simply
talking about how they are monitored and the existing
system is a human system for monitoring those changes
and what's being suggested is that it should become
an automatic system?
A. Yes.
Q. Then if we go over the page to {F/1127/2} the proposal
in paragraph 5 is:
"To continue with the existing process of monitoring
but to additionally raise this as a risk within IT&C and
to monitor any exceptions found through the existing
process. If exceptions are found then re-consider the
proposal from E&Y and assess the impact of the change
versus the benefit."
So a weighing up exercise is taking place and the
conclusion for the time being is: keep things as they
are but be aware of this potential improvement and
monitor in the meantime to see whether in fact it would
be a good idea to make it.
A. Yes.
Q. That's an entirely reasonable approach wouldn't you
agree?
A. Well, the approach is broadly saying let's carry on
doing what we do now and just look at it again in the
future.
Q. No. They are suggesting, first of all, carry on as we
are doing now which is a monitoring system. We have
a monitoring system; a suggestion is being made as to
how to improve it, but as well as carrying on with the
system, monitor the monitoring system to see if any
problems arise, if there are any exceptions and if there
are, then go back to the proposal of automatic
monitoring.
A. That is what it says.
Q. It is not possible for you sitting in your chair over
there to say that that was necessarily the wrong
decision, is it?
A. No, that's true.
Q. It is the kind of thing that a business would quite
sensibly want to do. They would like to assess whether
there is a problem or not before they spend such a vast
amount of money. You can't say that's intrinsically
unreasonable, can you?
A. No, it is not intrinsically unreasonable but it leaves
exposed a risk that could be dealt with but hasn't been
dealt with because of a cost benefit basis and that was
the observation that I made.
Q. But there is a much more important point, Mr Coyne,
which is that this has nothing to do with fixing bugs in
Horizon, does it?
A. Yes, it goes directly to fixing bugs in Horizon because
the changes that they are talking about making are
changes to Horizon.
Q. Mr Coyne, let me remind you of the sentence that you say
this document supports that's contained in
paragraph 5.161 of your expert report:
"It has been identified that known issues/bugs were
often deferred and dealt with on a cost benefit basis."
This proposal here and the decision that was made
from this proposal is not evidence of any known issue or
bug being deferred and dealt with in a cost benefit
basis, is it?
A. No. This isn't to do with the deferment of a bug, no.
MR JUSTICE FRASER: Mr De Garr Robinson, there is another
document that goes with this one and I'm slightly
concerned you might be putting some of your questions on
an incorrect basis. So just pause there one second.
If you could look at please -- and I don't want this
to go on the common screen -- {F/869/30}.
I don't want to set a hare running but I just feel
that if you look at {F/869/30} and then it is the third
column from the left and {F/869/31}, I think those are
the more detailed points that these conclusions you have
been taking the witness to go with. Now I might be
wrong.
MR DE GARR ROBINSON: I'm sorry, what am I supposed to be
looking at my Lord?
MR JUSTICE FRASER: The testing sample of 15 back end
changes, ten of which were in the counter and five of
which were manual. If you scroll down that column. Do
you have {F/869/30}?
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: If you start at that bullet point and
just read that column down to the bottom of that page
and then onto the next page leading to the conclusion
that begins "There is an increased risk...".
MR DE GARR ROBINSON: Based on a testing sample of 15 back
end changes?
MR JUSTICE FRASER: Yes, ten at the counter and 5 manual
changes. If you just read that column down to yourself
to the conclusion, "There is an increased risk...".
(Pause).
MR DE GARR ROBINSON: My Lord, yes. Is this the 2012/13
audit?
MR JUSTICE FRASER: No. I'll tell you what I'm going to do.
Mr Coyne could you go out for a minute please?
MR DE GARR ROBINSON: Is this the Ernst & Young management
letter of 2011?
MR JUSTICE FRASER: It is. Are you about to come onto it?
MR DE GARR ROBINSON: No, I'm not. This proposal~-- no I'm
not going into it.
(The witness withdraws)
MR JUSTICE FRASER: Hold on one second, Mr De Garr Robinson,
I am waiting for the witness to go out.
Now what would you like to say?
MR DE GARR ROBINSON: My Lord, this document relates to
a proposal for improvement that was made in the 2012/13
audit.
MR JUSTICE FRASER: I know.
MR DE GARR ROBINSON: My Lord I believe you are taking me to
the 2011 management letter.
MR JUSTICE FRASER: I know, but when the witness in part of
his answers to you says, "I think there is a document",
and he goes on to deal with it directly in his report
immediately after the passage you are asking him to, it
would help me hugely if you would make it clear with him
what the document is he says supports what he is saying
justifies his conclusion on the other document you are
putting to him.
MR DE GARR ROBINSON: I see.
MR JUSTICE FRASER: Do you see what I mean?
MR DE GARR ROBINSON: Yes. I must confess my focus was on
fixing bugs rather than his other answers.
MR JUSTICE FRASER: I entirely understand that. I'm just
faced with a witness who from time to time says: well,
there is a document, and it would be enormously helpful
to me if you could either, how shall I put it neutrally,
either get a tick in the box that that's a document or
a cross that there isn't such a document when you are
putting those questions to him.
MR DE GARR ROBINSON: I will try my Lord.
MR JUSTICE FRASER: I'm not saying --
MR DE GARR ROBINSON: Given the vast size of the bundles --
MR JUSTICE FRASER: I'm not saying you have to do it on
every single occasion, but on this particular occasion,
because the paragraph comes immediately afterwards, it
is much better to deal with it now than, for example,
store them all up until Friday evening at 4.30 when Mr
Green tries to re-examine him in ten minutes.
I'm attempting to explain to you why it is going to
be helpful to me.
MR DE GARR ROBINSON: Yes.
MR JUSTICE FRASER: Can we have the witness back in please.
(In the presence of the witness)
MR JUSTICE FRASER: Mr Coyne, don't read anything into that
at all. I just had to have an exchange about a
particular reference, that is all.
MR DE GARR ROBINSON: Mr Coyne, I have suggested to you that
this particular document has nothing to do with
deferring or not dealing with issues or bugs on a cost
benefit basis and I believe you have accepted that,
I hope you have. (Pause)
You have accepted that?
A. Yes, this particular document isn't about fixing bugs on
a cost benefit basis.
Q. You will have seen from the documents, these documents
that what was being considered by the Risk & Compliance
Committee was a proposal that was made in the Ernst &
Young audit of 2012/13 and we branched into that
question, although I must admit it wasn't a question
I was particularly interested in at that stage. But you
did say you thought you had seen a document. Is the
document you thought you had seen the one that's
referred to in 5.162 of your first report? That is at
{D2/1/98}. Might it be the Ernst & Young letter of
2011? Is that what you were thinking of?
A. Do you have that document?
Q. Yes. It is {F/869/1} that is the management letter. Is
that the document you were thinking of?
A. No, I think the way the document works is there is
a management letter and there is like a spreadsheet
version of the observations I believe.
Q. My learned friend is very kindly suggesting that it
might be at {F/1075.1/1}. Perhaps we can go to that.
MR JUSTICE FRASER: I think it is just a draft of the one
that I showed you.
MR DE GARR ROBINSON: This one is:
"IT component of management letter for the year
ended 31 March 2013".
Is that the document you had in mind?
A. Quite possibly. Can I just have a look at the --
Q. Perhaps we could scroll forward one page, first of all.
{F/1075.1/2}. There is an overview. Then there is
a series of detailed observations. Is that page 2?
There is a proposal about SAP payroll in paragraph 1
{F/1075.1/3}. In paragraph 2 a proposal about POLSAP
security parameters. If we go forward a page
{F/1075.1/4}. Paragraph 3, a proposal about POLSAP
again. Paragraph 4, "Change management monitoring
control".
A. Yes so that relates back --
Q. Is that what you were thinking of?
A. Yes. Sorry, can I just have a moment to read that to
remind myself. (Pause).
MR JUSTICE FRASER: If it doesn't jump out at you as the
document to which you were referring, please just say so
and Mr De Garr Robinson can move on.
A. This is certainly the document that the text in the
report refers to but there are documents that relate to
cost benefit analysis within I believe it is PEAKs.
MR DE GARR ROBINSON: That is entirely separate and I'm
going there, please don't think I'm going to leave you
out Mr Coyne. But when you said there was
a spreadsheet, is this the --
A. Yes, I think it is.
Q. I'm grateful. My Lord, I don't know if that assists
you?
MR JUSTICE FRASER: That's very helpful and I don't want to
knock you off your course or anything like that.
MR DE GARR ROBINSON: No, that's quite understandable.
So you have accepted that the evidence you cite for
the proposition "known issues" were often deferred and
dealt with on a cost benefit basis is no evidence for
that proposition at all. You have accepted that,
haven't you?
A. I accept that the reference that I have provided doesn't
go to a document that shows that but I have other
references that I will --
Q. You didn't have you -- say you think there is a PEAK and
we will come -- but you didn't have a PEAK in mind at
the time that you made your first report, did you?
A. Or I have not referenced it correctly when putting the
report together.
Q. Why is it you were able to give this example as evidence
of the claim that this often happened. How did it come
about? Did you not read the document --
A. No, there is more than one.
Q. Mr Coyne, you make a claim?
A. Yes.
Q. Then you immediately cite evidence in support of it?
A. Yes.
Q. And I'm asking you how it is you came to cite evidence
in support of it which is no evidence at all. Did you
not read the document?
A. I have certainly read the document. I believe it is
incorrectly referenced.
Q. What do you mean incorrectly referenced please, would
you explain that?
A. Yes. So I have had a note of the point of cost benefit
analysis and I have then searched for the appropriate
reference in my database of references and it looks like
I have referenced the wrong document, but it does
contain the words "cost benefit" and that's how the
mistake --
Q. Mr Coyne, isn't that exactly the sort of thing that you
accepted a few minutes ago that an expert should not do?
That it is incumbent upon an expert when citing
a document as evidence for a proposition to read the
document carefully to ensure that it is evidence for
that proposition?
A. Yes.
Q. Why did you not when going through your report read the
document carefully with a view to ensuring it was
evidence for the proposition?
A. With regard to this I have made a mistake in referencing
this document.
Q. Can I suggest to you that the mistake that you made is
that you were anxious to assert that fixes were deferred
and dealt with on a cost benefit basis and so what you
did was you found a document that talked about cost
benefit basis and you put it in?
A. I have found other documents but I have made a mistake,
I do accept that.
Q. Let's go forward to another document you cite.
I believe this is the only other document you cite in
your first report for this proposition. If we go to
paragraph 6.1 which is at page {D2/1/109}.
A. Sorry, I don't have that yet.
MR JUSTICE FRASER: Do you have your hard copy report in
front of you?
A. Yes, I do. Yes, I have that.
MR DE GARR ROBINSON: So is this evidence you have in mind?
Perhaps I could ask you to read paragraph 6.1 to 6.3.
(Pause).
Now in fairness to you you don't cite this document
as being in support of your claim that fixes were often
dealt with on a cost benefit basis, but was this
a document that you had in mind in supporting that
claim?
A. Yes.
Q. Well, let's look at that, shall we? The document is at
{F/1697/1}. This is called "Reconciliation and Incident
Management Joint Working Document". It is dated, the
current version, 18 March 2013.
If we could go forward to page {F/1697/9} of this
document to see its purpose. Perhaps I could ask you to
read section 1.1:
"What is a reconciliation?
"End to end reconciliation within HNG-X..."
That is Horizon Online, isn't it?
A. Yes.
Q. "... is the mechanism by which Post Office Ltd (POL) and
Post office Account (POA) establish which transactions
are complete and correct, and which are not. An
incomplete transaction is not necessarily a
Reconciliation error, but it might become one if it is
not completed in a timely manner. An incorrect
transaction is a Reconciliation error."
A. Yes.
Q. Over the page {F/1697/10}:
"Each and every reconciliation error is the result
of some system fault. That fault might, for example, be
a software bug fault (introduced through either design
or coding), a system crash, or a telephone line being
dug up. Such faults may affect transactions, thus it is
the job of Reconciliation Service to detect when and how
any transaction is affected by any system fault."
A. Yes.
Q. "A reported reconciliation error provides:
"A business impact in terms of an error report on
a transaction and;
"Evidence of a system fault that may need some
corrective action.
"It is acknowledged that not all system faults will
lead to corrective action as this is generally done on a
contractual and/or cost benefit basis."
A. Yes.
Q. So do you say that that is evidence for the proposition
that bug fixes in Horizon are dealt with or deferred on
a cost benefit basis?
A. Yes.
Q. What this document makes clear is that a system fault
has a very wide range of meanings, doesn't it?
A. Yes.
Q. On a fair reading it is a fault causing an error in the
systems by which transaction data are compared against
and reconciled with client transaction data?
A. Yes.
Q. It is a fault which it is the job of the reconciliation
service to investigate, to detect when and how the fault
was operated?
A. Yes.
Q. It is not a fault causing Horizon to have a false
transaction in the first place. That is the job of the
SSC to investigate, isn't it?
A. Yes.
Q. It is hardly surprising that the question whether
a system fault causing a reconciliation error is
corrected might involve a cost benefit analysis or
an analysis of who is responsible for what between the
client and the Post Office, is it?
A. I do not think the two go hand in hand. If you are
deciding whether to correct bugs on a cost -- sorry,
correct system faults, a subset of which is software
bugs or faults, depending on which is in the latest
version of the document, then you are deciding not to
fix those faults because it is either too expensive or
doesn't provide sufficient benefit.
Q. Mr Coyne, I'm suggesting to you that this document says
nothing about the approach adopted by the SSC to bugs in
Horizon causing discrepancies in branch accounts. Do
you not accept that?
A. Well, bugs in Horizon are fixed by Fujitsu.
Q. They are fixed by the SSC. They are not fixed by the
reconciliation service, would you accept that?
A. I don't believe that they are necessarily fixed by SSC.
I think they are identified by SSC.
Q. You are quite right, they are fixed by development when
the SSC passes it onto development.
A. Yes.
Q. But they are not fixed by the reconciliation service,
are they?
A. No they are not fixed by the reconciliation service but
the reconciliation service will report that there is
a problem; that will then be investigated and it will be
decided whether it is a software, bug or fault either
introduced through design or coding, whether it is
a system crash, whether it is a telephone line being dug
up. Somebody then will decide whether that system fault
should be fixed. But if it is a system fault that is
a software bug, it will be Fujitsu that fixes that even
though the fault might be found initially by the
reconciliation service.
Q. What I'm suggesting to you Mr Coyne is that you can't
reasonably infer that this passage of this working
document is talking about the SSC's and the fourth
tier's and the third tier's approach to the fixing of
bugs in Horizon?
A. I believe firmly it is. That's what it is saying.
Q. Very well. I suggest to you, Mr Coyne, that you
certainly can't infer that it was often decided by the
SSC or development not to fix bugs because the cost was
too great?
A. Well, what it actually says, this is generally done on
a contractual or cost benefit basis.
Q. If this were generally done there would be a significant
number of PEAKs recording that it had been done. It is
the kind of thing that would be recorded in a PEAK,
isn't it?
A. If the process is working properly, then it is likely
that there will be a PEAK identified for these, yes.
Q. If there were a significant number of PEAKs -- if this
were often done, there would be a significant number of
PEAKs evidencing the fact that it is done, correct?
A. There might only be one PEAK.
Q. Why might there only be one if it is often done
Mr Coyne?
A. Because if it becomes a problem that occurs repeated
times and there is a consideration of what the fix is,
Fujitsu won't raise tens or hundreds of PEAKs per day
all relating to the same fault. They will refer back to
the one single KEL that identifies the problem. That's
the purpose of PEAKs and KELs.
Q. If known issues are often deferred or dealt with on
a cost benefit basis, then there must be a significant
number of separate issues that are deferred or dealt
with on that basis; correct?
A. Yes.
Q. For each of those issues there will be at least one
PEAK, sometimes there will be more, considering that
aspect, yes?
A. There should be at least one PEAK or one KEL or probably
both.
Q. So if this is often done, there will often be known
issues in relation to which this is done so there will
often be PEAKs recording that this has been done;
correct?
A. No. If the fault is found again next week, Fujitsu go
back, they search for the reason for the fault, they
find that there is a KEL -- that is the purpose of the
KEL it is the knowledge base. So they will not then
raise another PEAK, they will see that it is going to be
fixed at some time in the future by that KEL.
Q. It may be you are not quite grasping the suggestion I'm
making to you. Are you suggesting that actually there's
only ever been one issue in relation to which this has
happened and that there's one KEL covering it happening
and therefore there will only be one PEAK covering it
happening, is that the logic of your position?
A. There is likely one KEL for every different
manifestation of the defect.
Q. Yes, so if it is being done often -- your claim is that
it is being done often -- then there will often be PEAKs
recording this decision being made. In fact there will
probably often be KELs as well, wouldn't there?
A. Yes but they don't duplicate a KEL every time a defect
happens.
Q. Mr Coyne, I'm struggling and it may be my fault. If
known issues are often dealt with on a cost benefit
basis, there must often be issues which are dealt with
on a cost benefit basis, correct?
A. Yes.
Q. And each of those issues will have their own KELs and
their own PEAKs, yes?
A. Yes.
Q. Perhaps I can put my point again. In circumstances
where we have a system in which Fujitsu often make
decisions to defer or not deal with known issues on
a cost benefit basis, there are going to be
a significant number of PEAKs and KELs recording that
being done, do you accept that?
A. I do but I do not think it is Fujitsu that is taking the
decision to do it on a cost benefit basis. If that is
the Post Office taking the decision then, yes, I agree
with that.
Q. Am I right in thinking that at the time you produced
this report you had not found any such PEAK because if
you had it would be in this report, wouldn't it?
A. I don't know whether I was aware of it at the time but
there are certainly PEAKs that talk about things being
deferred on a cost benefit basis, whether I had found it
at the time of this report or found it later I'm not
sure.
Q. I suggest to you, Mr Coyne, that you obviously hadn't
because if you had they would have been referred to in
this report, would they not?
A. Likely, yes.
Q. Right. So at the time you made this report is it likely
that you didn't know of any PEAKs in which this was
recorded, that your only evidence is the two documents
I have just taken you to?
A. This document is quite strong because it talks about the
corrective action generally being done on a cost
basis --
Q. Is it likely that when you made this statement the only
evidence you had to support it was the Risk and
Compliance committee meeting minutes and the document we
have just come to? I have had your commentary on
whether you think either of those documents is good
evidence. I'm simply asking you whether it is likely
that those were the only two pieces of evidence you had
at the time you made this report?
A. It is certainly possible that is the case, but I would
have to look at when the PEAKs arrived.
Q. You have just said there are PEAKs, plural, where this
has happened?
A. Yes.
Q. Now Mr Coyne I'm aware of one PEAK which might be said
to support a suggestion that there was one occasion on
which a cost benefit analysis was adopted and I'm going
to take you to it. It is referred to in your second
report. Are you aware of others that are not referred
to in your second report?
A. I believe there is more than one, yes.
Q. You clearly looked very hard for them. Would I be right
in inferring that that is the case? You looked very
hard for PEAKs to support this proposition, did you?
A. I don't believe there were any specific searches that
I conducted for that, no.
Q. So you didn't make any specific searches but you
stumbled over how many PEAKs to support the proposition?
A. A handful perhaps.
Q. I'm only aware of one, Mr Coyne. Might it be possible
for you to come to court tomorrow and tell me what the
others are?
A. Certainly, yes.
Q. Let's go to the one I'm aware of. Let's go to
{F/271/1}. This is a PEAK PCO120937. It is dated
20th May 2005. It is a customer call, at the top box of
the PEAK, details entered are:
"Summary: PM is stating that the system remmed out
some coin."
If we go down to the second line in the top box:
"PM is stating that the system remmed out some coin
its own and it produced no paper work, and they would.
Like to know if we can find out the Pouch number so they
can reverse it out."
Do you see that?
A. Yes.
Q. If we go over the page to {F/271/2} at 16th May, 12.01.
I won't read it out but you will see that Sarah English
is unable to recreate the problem. Do you see that?
A. At 12.01? Yes.
Q. Then if we go over the page to page {F/271/3} and this
time at the top of the page, 17th May, 13.55. This is
Gary Maxwell?
A. Yes.
Q. The last paragraph you will see he says:
"I have attempted to replicate this on our test
counter but could not abandon the session any other way
than by following the available menu options. In all
cases the session finished as expected without writing
any messages."
They can't recreate the problem, yes?
A. Yes.
Q. Often that's a reason for thinking that there may not be
a problem there. But often it may be that they are just
perplexed and one just doesn't know.
If one goes to {F/271/4} (Pause). 7.32 about three
boxes down, 13th June 2005. This is from Lionel Higman?
A. Higman I believe.
Q. Thank you.
"Is the frequency of occurrence for this problem
known? How severe are the problems caused? Is recovery
possible? Would it be sufficient to raise a KEL to
manage future occurrences more effectively?"
You see the question being raised there?
A. Yes.
Q. Then if you go down to 10.13. My note is terrible...
A. 10.33 might be relevant:
"Gary advises development have found a potential
bug. They want to know the scale of the problem as there
is potential risk in changing the code."
Q. Right. Thank you. For some reason I keep typing a 1
instead of a 3 in my notes. The concern being expressed
by development here is risk from making a change. Do
you see that?
A. Well, found a bug and then there is associated risk with
making the change. There are two elements.
Q. If we go back. I'm sorry I went too fast. If we go
back to page 3.
A. There was 10.13 on there that might be helpful as well.
Q. I'm sorry Mr Coyne, what were you saying?
A. If we go back to the page that we were just on at 10.13,
at the bottom.
MR JUSTICE FRASER: On 15th June?
A. 15th June, yes.
"First instance, check the PEAKs. The impact on the
office is that they will have an amount posted to the
suspense account ... equivalent to the remittance
amount. This will be evident until remedial action is
taken."
MR DE GARR ROBINSON: Stopping there. What does that mean?
A. So the suspense account is going to be likely in
balance. It is going to have an impact anyway.
MR DE GARR ROBINSON: It means that the postmaster is going
to know very well what happened, isn't it?
A. Well, the postmaster will have a suspense account that
doesn't balance. I do not think they will know what's
happened but they might know what the impact is.
Q. At any rate they will know there is a problem that needs
to be addressed, yes?
A. Did the postmaster raise the call on this one? Did they
ring up in the first instance reporting the fault?
Q. I believe the passage I read you -- yes, postmaster is
stating the system remmed out some coins, do you
remember?
A. Yes.
Q. The postmaster knew exactly what had happened. And he
says {F/271/5}:
"There is nothing the PM can do to correct this. The
sensible recovery option is for the NBSC to issue an
error notice for the remittance amount as in this case."
So that had happened in this case it would seem?
A. Yes, so that is a correction on the accounts.
Q. What we now call transaction corrections.
"The alternative would be for SSC to create the
necessary Riposte messages to cancel out the error
(currently being tested).
"A KEL has been created on the back of this call and
will be updated shortly.
"Given the frequency of the problem & the apparent
risk involved in introducing a code fix the KEL should
be adequate."
Do you see that? What's being presented there by
Gary Maxwell is that it is a very low frequency problem
and there's risk involved in introducing the code fix.
Do you know why there is a risk involved in
introducing a code fix?
A. There's inherent risk in any change to a system. So
introducing any fixes to code brings a risk with it.
Q. So up until now the debate is about risk of doing
a change compared with the benefit of doing the change,
bearing in mind it is such a rare problem. Do you see
that?
A. Yes.
Q. Then if we go down to 11.28 we see Mark Scardifield.
This is where the word "cost" gets mentioned:
"Weighing up the cost and risk of an attempted fix
against the fact that this has only been reported once,
I do not believe that we should make a code fix. If
further incidents of this problem are reported we can
review this decision. Gary has raised a KEL, so
returning for closure as "Published Known Error".
Do you see that?
A. Yes.
Q. So what we have is a debate which largely revolves
around the risk involved in making a fix in this part of
the system?
A. Yes.
Q. And what Mark Scardifield does is he brings in the word
"cost"?
A. Yes.
Q. Do you say that this PEAK is evidence for the
proposition that decisions about fixing problems were
often made on a cost benefit basis?
A. It certainly does indicate that.
Q. It indicates that what's happened is that the parties
are concerned primarily with risk and on that basis they
decide that they will leave the problem as it is. It
doesn't support the notion that decisions are often made
on a cost benefit basis, does it?
A. It is another reference to decisions being taken on
a cost benefit basis.
Q. It is one instance. It is a PEAK where the word "cost"
is referred to?
A. Yes and the other document that we talked about, things
are generally done on a cost benefit basis.
Q. Let's look at the KEL that was created for this?
A. Before we move on, just for context, can we just read
the top column there because it talks about what the
possible remedial actions are.
MR JUSTICE FRASER: Which page?
A. F/271 and it is the one above 10.14.
MR JUSTICE FRASER: If you look to the right, is it page 5?
A. Page 5 because it talks there that the alternative would
be for SSC to create the necessary Riposte message to
cancel out the error.
MR JUSTICE FRASER: You are going to have to give us both,
Mr De Garr Robinson and me, the date and the time please
of where you are reading.
A. Certainly, it is the very top box. It is above
15th June 2005.
MR JUSTICE FRASER: The one in the pale green at the very
top?
A. Yes. It starts with "... remittance amount as in this
case."
But of interest is the alternative remedial action
to fix the accounts is:
"... would be for SSC to create the necessary
Riposte messages to cancel out the error".
That is the insertion of messages in Riposte which
is relevant to other aspects that we are talking about.
MR DE GARR ROBINSON: Thank you. Can we now look at the KEL
that was created for this fix because what I want to
suggest to you, Mr Coyne, is that the focus of this
decision or the main reason for this decision was risk.
If we could look to {F/276/1} please. This is the
KEL that was created?
A. Yes.
Q. It says:
"The PM was attempting to do a remittance out ADC
for a bag of coins. After entering an invalid bar code
for the particular type of transaction the session was
aborted by pressing the 'home' key."
Then:
"Firstly, during the remittance out process pressing
'home' should not be an available option. The only way
of completing the session is to navigate the screen
prompts & cancel the session or return to Desktop and
bin the item on the sales stack.
"By aborting the session in this way the EPOSS
remittance items on the sales stack were incorrectly
committed. Normally these messages are only committed
once a successful barcode has been scanned/entered & at
the same time as the LFS remittance out message."
Then there is some computing gobbledygook:
"The PM will not be able to reverse the transaction
using existing reversal (because of the tran type) or
will they be able to use the pouch reversal function (as
no pouch id exists).
"The suspense account will be debited/credited by
the amount of the remittance. The office snapshot/cash
account report will continue to show the remittance
amount under unclaimed payments."
Their solution. It refers to Atos, but Atos wasn't
on the scene in 2005. So I apprehend that one big
word-processing change was made when Atos was involved
many years later but it says:
"Solution:
"Development have identified a possible bug in the
counter code. However, due to the frequency of the
problem & the risks involved in making the necessary
changes it has been decided that a code change will not
be made. Raise a call on EDSC. In the one case to date,
an error notice was issued to the office for the amount
of the remittance."
What we have here is a debate in a PEAK about risk
where there is a reference to cost and then we have
a KEL that has been produced as a result of that debate.
That KEL does not refer to cost, it just refers to risk.
Do you suggest, Mr Coyne, that this document, this
KEL and this PEAK together, constitute evidence that
deferring and dealing with known issues were often dealt
with on a cost benefit basis?
A. It is certainly another document that indicates that
cost was a consideration in whether to fix.
Q. Well this is the only document of which I'm aware, PEAK
or KEL, in which this point has been made. You say that
you are aware of others?
MR GREEN: My Lord, I have tried to let it run but I mean in
that actual section of his report at 5.166 on page 99 he
actually specifies the other PEAK.
MR JUSTICE FRASER: Give me the reference.
MR GREEN: It is 5.166.
MR JUSTICE FRASER: No, the bundle reference.
MR GREEN: Sorry, {D2/1/99} which is the next page of his
report in the same section.
MR DE GARR ROBINSON: This is his second report?
MR GREEN: No, in the first report where we were dealing
with the Ernst & Young stuff and you put to him only
documents and so forth, and in 5.166 he specifically
mentions a PEAK. It was put to him he hadn't looked at
any.
MR JUSTICE FRASER: Is this the one at {F/275.1/1}?
MR GREEN: It is my Lord, yes.
MR JUSTICE FRASER: Yes. Mr De Garr Robinson, I'm very
happy for you to look at that and resume with it in the
morning or do it now?
MR DE GARR ROBINSON: No, my Lord, I'm content to break
today.
MR JUSTICE FRASER: Okay. So Mr Green what was your point
on the way the question was put that it is the only
document?
MR GREEN: When you look back across the transcript --
MR JUSTICE FRASER: The transcript says --
MR GREEN: It is the only document I'm aware of.
MR JUSTICE FRASER:
"Question: This is the only document of which I am
aware, PEAK or KEL, in which this point has been made."
MR GREEN: And it has been going on for about 20 minutes.
And it is not really fair to Mr Coyne when one ought
to be aware of what he says in that very section of his
report on this point.
MR JUSTICE FRASER: Right, well --
MR DE GARR ROBINSON: My learned friend seems to be
suggesting that the document {F/275.1/1} is further
evidence of a decision to defer a fix on the basis of
a cost benefit analysis. Let's look at that, shall we,
can we look at {F/275.1/1} please. So this is dated
18th June 2005. The call logger is:
"Deleted user. ITU, SV&I."
I'm not sure what that means. POL have raised the
following incident:
"'A cash declaration was made in "Stock Balancing"
for the amount displayed on the Snapshot. When the Cash
Variance was checked afterwards a Gain of £45.05 was
displayed.
"The discrepancy was the cash value of the
transactions performed on the stock unit after rolling
into branch trading."
Then there is further information about it.
Could we perhaps go over the page {F/275.1/2}.
16th June at 15.59:
"I've attempted to recreate the situation which
occurred over a CA Rollover in this instance and the
injecting a transaction like the mails txns ...
Difference being that up here the trace has turned up
a transaction received ... and in the supplied audit log
nothing at all appears ..."
Then the next box says:
"After several days of attempting to recreate this
problem ... I have only seen one instance of this
problem."
Excuse me. (Pause)
Could we go back to the first page of this PEAK,
please {F/275.1/1}. You see the call type on the
left-hand side, Mr Coyne?
A. Yes.
Q. "Business Integration Testing Incidents/Defects."
This is the testing PEAK, isn't it?
A. Yes, I think that is right.
Q. Could you explain what a testing PEAK is?
A. It is a defect that has arisen on the system during
testing, so not in live operation.
Q. Does a testing PEAK like this, is it capable of
constituting evidence that known issues in the system
are dealt with on a cost benefit basis?
A. Certainly, if they are dealing with defects that are
found in testing on a cost benefit basis and therefore
don't fix the defect it will go out to live.
Q. Now, is it -- and it may be unfair to ask you this
question because you didn't raise it but now you have
seen this PEAK do you remember it?
A. I'm not sure I do. Can I look, please, at the end of
the ...
Q. I don't know how long it is.
MR JUSTICE FRASER: It is five pages.
MR DE GARR ROBINSON: Could we look at page 4, please.
I must confess I hadn't read 5.166 as being even
relevant to this point. {F/275.1/4}
A. It looks like it needs to go out to Escher to be fixed
rather than Fujitsu. It says at the bottom:
"This amounts to a declaration by POA that we are no
longer interested in obtaining a fix."
But it doesn't make a reference to a cost benefit.
Q. Thank you.
MR JUSTICE FRASER: Right. Is that a good --
MR DE GARR ROBINSON: I think we are at 4.30 pm.
MR JUSTICE FRASER: I think it probably is. Right.
A couple of things. Mr Coyne, you have some
homework, but generally when you are giving your
evidence tomorrow you have got in front of you, and
that's why I checked this afternoon, a hard copy of your
report?
A. Yes, I have.
MR JUSTICE FRASER: It is very tempting when
Mr de Garr Robinson is asking questions just to look at
the screen. It's probably a good idea also to flick to
your report but it is completely up to you whether you
do that or not.
And Mr Green, you might want to store points such as
that one up for re-examination, if you have any, because
you might find that upon mature reflection they are not
the points you thought they were when you jumped to your
feet, although I appreciate you have been restrained, or
you think you have been restrained today.
And Mr de Garr Robinson, 10.30 am tomorrow.
I assume you don't have any objection to the witness
looking on his knowledge database to do the homework
that you set him?
MR DE GARR ROBINSON: My Lord, of course not. Obviously he
won't be talking to anyone.
MR JUSTICE FRASER: No, I'm just about to remind him.
Mr Coyne, normally you won't have homework but it is
particularly important that you, not your assistant,
does that.
A. Absolutely, my Lord.
MR JUSTICE FRASER: If he finds anything what do you want
him to do in terms of bringing it with him or notifying
references? Are you happy to get that tomorrow morning
at 10.15?
MR DE GARR ROBINSON: My Lord, yes. I don't think I will be
going back to it immediately. It will be something to
be dealt with on another day.
MR JUSTICE FRASER: All right. In that case, Mr Coyne, if
you can just find them and bring -- I don't know whether
it is easy to print them off. If you do find anything
bring it with you and we will deal with that first thing
but you won't necessarily be asked questions about it
because Mr de Garr Robinson will want to have a look at
it. Is that clear?
A. It is indeed.
MR JUSTICE FRASER: Anything else? 10.30 am tomorrow.
Thank you very much.
(4.36 pm)
(The court adjourned until 10.30 am on
Wednesday, 5th June 2019)