Would you like to read 2423 words on the meaning of the word "robust"?
Now you can, below.
It was a big deal in the Post Office Horizon trial because the Post Office contended that Horizon is robust whereas the claimants thought it was only relatively robust. On such things trials are won and lost, so getting a working definition of robust together was one of the first things the judge did.
The segment below comprises paragraphs 36 - 56 of the Horizon Issues judgment.
Language scholars and engineers may enjoy it:
The meaning of “robustness”
Turning to the disagreement about Issue 3, given the parties disagreed about whether the Horizon system is (or was) “robust”, it is a fairly elementary step to consider the meaning of that term, and how it is being used by the parties. Context is important so far as the meaning of the word “robust” is concerned. If someone is in robust health, it usually is taken to mean that they are healthy, or even very healthy. A robust exchange of views can be a polite way of referring to an argument. Given the importance of the concept to the Horizon system, its prominence in the Post Office’s defence of the system, and its express inclusion (admittedly in inverted commas) in the Horizon Issues, I asked each side in the litigation during oral closing submissions for a reference from their pleadings or submissions for the meaning which they ascribed to the word. I referred to this as their benchmark definition. Robustness was referred to by both sides in the litigation in numerous places, but not always in the same precise terms, and clarity is to be welcomed.
The claimants answered this by reference to the remainder of Issue 3, namely “extremely unlikely to be the cause of shortfalls in branches” and explained that the claimants had found the word robust “difficult to define” other than by reference to this. This would mean therefore that it had no separate independent meaning other than as a summary of the longer second part of Issue 3. In other words, a robust system would be one that is extremely unlikely to be the cause of shortfalls in branches. The claimants also implicitly, if not expressly, criticised use of the term both by the Post Office in its pleadings and written submissions as being more aligned to public relations than as a performance standard.
The Post Office asked for some time to provide the reference that I requested. Given the meaning of “robust” is so central in the Post Office’s defence of the Horizon system, I granted the Post Office the time that was requested.
The Post Office subsequently, after the trial ended, submitted a short document entitled “the Post Office’s case on the meaning of robustness”. This was not what was intended when I sought a reference from the Post Office to their definition, and the document submitted went rather further and made wider ranging submissions. The document did state, so far as the meaning of the word is concerned, the following:
“In Post Office’s submission, the meaning of robustness is a matter for expert opinion. Robustness is a well-established concept in the IT industry and is the subject of academic study: see para. 361 of Post Office’s written closing”.
I do not consider that the meaning of words is a matter for expert opinion. The two experts in this case are IT experts, not experts in linguistics or the meaning of language. However, the meaning of robustness within the field of IT is, arguably, a matter upon which the experts’ opinions should be considered, not least because they were applying that term to their expert exercise. The Post Office also relied upon the 1st Experts’ Joint Statement which in respect of Issue 3 stated the following as agreed:
“There are different dimensions of robustness, such as robustness against hardware failure, software defects and user error. The robustness of the system also depends on the processes around it.
Robustness does not mean perfection; but that the consequences of imperfection must be managed appropriately. If the extent of imperfection is too high, this would be very difficult to do which would imply less robustness.
Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its lifetime. The level of robustness may have increased or decreased as the system was changed.
The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this indicates any lack of robustness.”
In the areas of disagreement in this Joint Statement, each expert provided the following. Mr Coyne stated (inter alia):
“For the purposes of addressing the robustness of Horizon, I have applied the following definition of robustness:
‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling.’ ”
In consideration of the likelihood of Horizon to be the cause of shortfalls in branches, Horizon is not determined to be robust in this regard because:
(a) it contained high levels of bugs, errors and defects as set out under Issue 1 above which created discrepancies in the branch accounts of Subpostmasters;
(b) it suffered failures of internal mechanisms which were intended to ensure integrity of data;
(c) the system did not enable such discrepancies to be detected, accurately identified and/or recorded either reliably, consistently or at all;
(d) the system did not reliably identify ‘Mis-keying’, which is inevitable in any system with user input, and did not reliably have in place functionality to restrict users from progressing a mis-key;
(e) it required numerous processes and workarounds to be in place to allow Fujitsu to modify data already recorded by Horizon, which would not be required in a “robust” system; and/or
(f) there were weaknesses and risks of errors and other sources of unreliability within Horizon.”
(italics present in original)
Dr Worden stated in the same Joint Statement:
“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The term 'robust' is not, as implied in para 3.1 of the outline, either ill-defined or a piece of IT public relations. Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?
In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr Coyne implying that Horizon fell short of its robustness objectives.
In my current preliminary opinion, Horizon is a highly robust system, and this has important implications for the other Horizon issues, notably issue 1.”
It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition, and expressly said it was not adequate. In any event, the meaning of any word – even “robust”, or “robustness” – ought to be capable of description by the parties themselves. Although on its face it did not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr Coyne’s text, in other words that part of the text that contained his definition of robustness (which was in italics in the 1st Joint Statement). Obviously if the parties (or their experts) could agree the definition to be applied so far as the Horizon System is concerned, that ought to be identified. I therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne, and if not, what its alternative definition was.
This led to a further document being received from the Post Office dated 18 July 2019. It referred to the passage in the 1st Joint Statement (which is quoted at [40] above) as “the agreed definition”. That rather overlooks that Mr Coyne identified the definition of robustness which he was applying, and Dr Worden expressly disagreed with this in the same Joint Statement under the heading “Areas of Disagreement”, and stated “the definition of ‘robust’ proposed above by Mr Coyne is not adequate, for the reasons given below”. It also overlooks that in the 3rd Joint Statement, paragraph 3.1 had an agreed entry which stated the following:
“Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved.
From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events.
This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”
(emphasis added)
Later in the same document of 18 July 2019 the submission was made by the Post Office that “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. It was also submitted that “As Post Office understands it, this is what Mr Coyne meant when in his comments in [the 1st Joint Statement] he defined robustness as “the ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
This was precisely the definition which Dr Worden, in his areas of disagreement on the 1st Joint Statement, described as “inadequate”. The end position therefore is as follows.
The claimants found “robustness” difficult to define in the abstract and tied it in with the other wording of Horizon Issue 3; a robust system would be “extremely unlikely to be the cause of shortfalls in branches”. That however is a consequence of how a robust system would operate, not a definition of what robustness means.
The Post Office defined it as follows: “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. The Post Office was also prepared to accept Mr Coyne’s italicised definition in the 1st Joint Statement, namely ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
Mr Coyne applied the definition he set out in italics in the 1st Joint Statement, quoted in the immediately preceding paragraph of this judgment and at [41] above.
Dr Worden’s definition was as follows:
“Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?”
The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of Dr Worden.
The Post Office made submissions in paragraph 3(b) of the written submissions dated 18 July 2019 on robustness that stated that Mr Coyne cannot have intended to exclude the effect of countermeasures when he considered the concept of robustness. I shall return to this topic when dealing with countermeasures. This is because some of the countermeasures considered by Dr Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which both parties agree are outside of the Horizon System).
I do however accept the Post Office’s submissions that there is not a great or material difference in the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’ submission that robustness is difficult to define. Dr Worden defined robustness by using what he termed as “the dimensions of robustness”. It is rather circular to describe the meaning of robustness as being “robust against” some particular risk. Although Mr Coyne provided his definition in the 1st Joint Statement, the statement by Dr Worden that this was “inadequate” may only have been aimed at the entirety of Mr Coyne’s entry in the areas of disagreement, as effectively accepted by the Post Office in their most recent written submissions on the subject. Whether that is an explanation of the lack of agreement in the Joint Statement, I also agree with the Post Office that Mr Coyne’s definition is not materially different to that used by Dr Worden.
Robustness is indeed an engineering concept. It means the ability of any system to withstand or overcome adverse conditions. A robust system is strong and effective in all or most conditions. The robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences; this is the same meaning as how robustness was described in the Post Office’s written submissions dated 18 July 19. Robustness does not mean perfection.
The exercise necessary above, to arrive at the definition of robustness in [54] above, is not judicial pedantry. Given the central importance of robustness to the disputes about the Horizon System, and the Horizon Issues, it is in my judgment essential. It is mildly surprising, given how central the assertion of robustness has been to the Post Office’s defence of the Horizon System, that Dr Worden’s interpretation of the term has been relied upon so heavily by the Post Office, given the term was used by the Post Office for some years prior to his involvement.
However, regardless of that passing observation, I find that both experts correctly understood what robustness in fact means, and applied the definition at [54] above in considering their expert evidence. I will return to the expert evidence in some detail later in the judgment, including in the Technical Appendix.
Now you can, below.
It was a big deal in the Post Office Horizon trial because the Post Office contended that Horizon is robust whereas the claimants thought it was only relatively robust. On such things trials are won and lost, so getting a working definition of robust together was one of the first things the judge did.
The segment below comprises paragraphs 36 - 56 of the Horizon Issues judgment.
Language scholars and engineers may enjoy it:
The meaning of “robustness”
Turning to the disagreement about Issue 3, given the parties disagreed about whether the Horizon system is (or was) “robust”, it is a fairly elementary step to consider the meaning of that term, and how it is being used by the parties. Context is important so far as the meaning of the word “robust” is concerned. If someone is in robust health, it usually is taken to mean that they are healthy, or even very healthy. A robust exchange of views can be a polite way of referring to an argument. Given the importance of the concept to the Horizon system, its prominence in the Post Office’s defence of the system, and its express inclusion (admittedly in inverted commas) in the Horizon Issues, I asked each side in the litigation during oral closing submissions for a reference from their pleadings or submissions for the meaning which they ascribed to the word. I referred to this as their benchmark definition. Robustness was referred to by both sides in the litigation in numerous places, but not always in the same precise terms, and clarity is to be welcomed.
The claimants answered this by reference to the remainder of Issue 3, namely “extremely unlikely to be the cause of shortfalls in branches” and explained that the claimants had found the word robust “difficult to define” other than by reference to this. This would mean therefore that it had no separate independent meaning other than as a summary of the longer second part of Issue 3. In other words, a robust system would be one that is extremely unlikely to be the cause of shortfalls in branches. The claimants also implicitly, if not expressly, criticised use of the term both by the Post Office in its pleadings and written submissions as being more aligned to public relations than as a performance standard.
The Post Office asked for some time to provide the reference that I requested. Given the meaning of “robust” is so central in the Post Office’s defence of the Horizon system, I granted the Post Office the time that was requested.
The Post Office subsequently, after the trial ended, submitted a short document entitled “the Post Office’s case on the meaning of robustness”. This was not what was intended when I sought a reference from the Post Office to their definition, and the document submitted went rather further and made wider ranging submissions. The document did state, so far as the meaning of the word is concerned, the following:
“In Post Office’s submission, the meaning of robustness is a matter for expert opinion. Robustness is a well-established concept in the IT industry and is the subject of academic study: see para. 361 of Post Office’s written closing”.
I do not consider that the meaning of words is a matter for expert opinion. The two experts in this case are IT experts, not experts in linguistics or the meaning of language. However, the meaning of robustness within the field of IT is, arguably, a matter upon which the experts’ opinions should be considered, not least because they were applying that term to their expert exercise. The Post Office also relied upon the 1st Experts’ Joint Statement which in respect of Issue 3 stated the following as agreed:
“There are different dimensions of robustness, such as robustness against hardware failure, software defects and user error. The robustness of the system also depends on the processes around it.
Robustness does not mean perfection; but that the consequences of imperfection must be managed appropriately. If the extent of imperfection is too high, this would be very difficult to do which would imply less robustness.
Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its lifetime. The level of robustness may have increased or decreased as the system was changed.
The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this indicates any lack of robustness.”
In the areas of disagreement in this Joint Statement, each expert provided the following. Mr Coyne stated (inter alia):
“For the purposes of addressing the robustness of Horizon, I have applied the following definition of robustness:
‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling.’ ”
In consideration of the likelihood of Horizon to be the cause of shortfalls in branches, Horizon is not determined to be robust in this regard because:
(a) it contained high levels of bugs, errors and defects as set out under Issue 1 above which created discrepancies in the branch accounts of Subpostmasters;
(b) it suffered failures of internal mechanisms which were intended to ensure integrity of data;
(c) the system did not enable such discrepancies to be detected, accurately identified and/or recorded either reliably, consistently or at all;
(d) the system did not reliably identify ‘Mis-keying’, which is inevitable in any system with user input, and did not reliably have in place functionality to restrict users from progressing a mis-key;
(e) it required numerous processes and workarounds to be in place to allow Fujitsu to modify data already recorded by Horizon, which would not be required in a “robust” system; and/or
(f) there were weaknesses and risks of errors and other sources of unreliability within Horizon.”
(italics present in original)
Dr Worden stated in the same Joint Statement:
“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The term 'robust' is not, as implied in para 3.1 of the outline, either ill-defined or a piece of IT public relations. Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?
In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr Coyne implying that Horizon fell short of its robustness objectives.
In my current preliminary opinion, Horizon is a highly robust system, and this has important implications for the other Horizon issues, notably issue 1.”
It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition, and expressly said it was not adequate. In any event, the meaning of any word – even “robust”, or “robustness” – ought to be capable of description by the parties themselves. Although on its face it did not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr Coyne’s text, in other words that part of the text that contained his definition of robustness (which was in italics in the 1st Joint Statement). Obviously if the parties (or their experts) could agree the definition to be applied so far as the Horizon System is concerned, that ought to be identified. I therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne, and if not, what its alternative definition was.
This led to a further document being received from the Post Office dated 18 July 2019. It referred to the passage in the 1st Joint Statement (which is quoted at [40] above) as “the agreed definition”. That rather overlooks that Mr Coyne identified the definition of robustness which he was applying, and Dr Worden expressly disagreed with this in the same Joint Statement under the heading “Areas of Disagreement”, and stated “the definition of ‘robust’ proposed above by Mr Coyne is not adequate, for the reasons given below”. It also overlooks that in the 3rd Joint Statement, paragraph 3.1 had an agreed entry which stated the following:
“Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved.
From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events.
This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”
(emphasis added)
Later in the same document of 18 July 2019 the submission was made by the Post Office that “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. It was also submitted that “As Post Office understands it, this is what Mr Coyne meant when in his comments in [the 1st Joint Statement] he defined robustness as “the ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
This was precisely the definition which Dr Worden, in his areas of disagreement on the 1st Joint Statement, described as “inadequate”. The end position therefore is as follows.
The claimants found “robustness” difficult to define in the abstract and tied it in with the other wording of Horizon Issue 3; a robust system would be “extremely unlikely to be the cause of shortfalls in branches”. That however is a consequence of how a robust system would operate, not a definition of what robustness means.
The Post Office defined it as follows: “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. The Post Office was also prepared to accept Mr Coyne’s italicised definition in the 1st Joint Statement, namely ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
Mr Coyne applied the definition he set out in italics in the 1st Joint Statement, quoted in the immediately preceding paragraph of this judgment and at [41] above.
Dr Worden’s definition was as follows:
“Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?”
The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of Dr Worden.
The Post Office made submissions in paragraph 3(b) of the written submissions dated 18 July 2019 on robustness that stated that Mr Coyne cannot have intended to exclude the effect of countermeasures when he considered the concept of robustness. I shall return to this topic when dealing with countermeasures. This is because some of the countermeasures considered by Dr Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which both parties agree are outside of the Horizon System).
I do however accept the Post Office’s submissions that there is not a great or material difference in the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’ submission that robustness is difficult to define. Dr Worden defined robustness by using what he termed as “the dimensions of robustness”. It is rather circular to describe the meaning of robustness as being “robust against” some particular risk. Although Mr Coyne provided his definition in the 1st Joint Statement, the statement by Dr Worden that this was “inadequate” may only have been aimed at the entirety of Mr Coyne’s entry in the areas of disagreement, as effectively accepted by the Post Office in their most recent written submissions on the subject. Whether that is an explanation of the lack of agreement in the Joint Statement, I also agree with the Post Office that Mr Coyne’s definition is not materially different to that used by Dr Worden.
Robustness is indeed an engineering concept. It means the ability of any system to withstand or overcome adverse conditions. A robust system is strong and effective in all or most conditions. The robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences; this is the same meaning as how robustness was described in the Post Office’s written submissions dated 18 July 19. Robustness does not mean perfection.
The exercise necessary above, to arrive at the definition of robustness in [54] above, is not judicial pedantry. Given the central importance of robustness to the disputes about the Horizon System, and the Horizon Issues, it is in my judgment essential. It is mildly surprising, given how central the assertion of robustness has been to the Post Office’s defence of the Horizon System, that Dr Worden’s interpretation of the term has been relied upon so heavily by the Post Office, given the term was used by the Post Office for some years prior to his involvement.
However, regardless of that passing observation, I find that both experts correctly understood what robustness in fact means, and applied the definition at [54] above in considering their expert evidence. I will return to the expert evidence in some detail later in the judgment, including in the Technical Appendix.
