Let’s be courageous, part 2

(… continued from Let’s be courageous, part 1, posted on 12 Aug 2016).

Under the subheading Rising costs in Section One of his Taxpayer’s Alliance briefing paper, Rich man’s toy: The case for scrapping HS2, Policy Analyst Harry Fairhead quotes data on the comparative per-kilometre costs for high speed rail projects, country by country, that has been extracted from an article in the Daily Telegraph. This newspaper article dubs HS2 “the most expensive high speed project in existence” and claims that the £42.6bn budget (excluding rolling stock costs), that was the appropriate figure when the article was written, “makes it more than ten times the cost per kilometre of some global counterparts” – in fact, it is sixteen times the lowest figure quoted in the article (for Turkey).

In my blog A bad deal at any price (posted 28 May 2016) I reported on some reasons for this cost disparity that had been put forward by the Acting Technical Director of HS2 Ltd, Giles Thomas in an interview with BBC Radio. The key element to this explanation appears to be that HS2 is required to transit some “very expensive” and densely populated parts of the UK. Now I can readily see that building a line across rural China is a totally different proposition from traversing the Chilterns, and might justify some of the six to one UK to China cost difference that the Telegraph’s figures display. However, I would have thought that high speed train constructors in Japan face many of the same problems as their UK counterparts, and perhaps even greater problems, and yet they manage to build their high speed railways for an eleventh of the cost of HS2. The newspaper also tells us that the per-kilometre cost of HS2 even manages to exceed the projected cost of the Californian high speed rail project by more than half as much again.

What is clear is that, for some reason or reasons, the UK is not able to construct high speed railways at the lower costs that apply everywhere else. Any review of the HS2 project should, therefore, take this inescapable fact into account, and judge whether the higher construction costs that apply in these islands mean that the taxpayer can ever get good value for money from high speed rail.

In his Section One, under the subheading Bold assumptions, Mr Fairhead makes some comments about optimism bias. I feel that these comments will benefit from some clarification, and attempt to provide this elucidation below.

“Historically”, according to the National Audit Office, “the majority of major projects in government have not delivered the anticipated benefits within original time and cost expectations” and there is “endemic over-optimism which characterises decisions to commit to projects and the subsequent management of them” (see footnote 1).

H M Treasury requires that “project appraisers” in government departments to “make adjustments of their estimates of capital and operating costs” by adding an “adjustment percentage” to them (see footnote 2). This addition is referred to as the “optimism bias adjustment”. The Treasury recommends values for optimism bias adjustment “that should be used in the absence of more robust evidence”: for “non-standard civil engineering” projects, like HS2, the recommendation is that the bias should be in the range 6% to 66% (see footnote 3). The Treasury Guidelines recommend that the upper bound value is used “as the starting value for calculating the optimism bias level”, but allows for the value to be reduced below this upper bound “according to the extent to which the contributory factors have been managed” (see footnote  4).

According to documentation published at the time of the ministerial decision to go ahead with HS2 Phase 1, this is the procedure that was adopted to determine the optimism bias adjustment for that part of the route, and it was determined that the optimism bias adjustment should be reduced from 66% to 34% for HS2 Phase 1, as reported in Mr Fairhead’s paper (see footnote 5): as far as I can determine, no detailed justification for making this reduction has been published.

H M Treasury appears to recognise the limitations of the data upon which its recommendations have been made and, “in the absence of a more specific evidence base”, encourages departments “to collect data to inform future estimates of optimism, and in the meantime use the best available data” (see footnote 6). In keeping with this advice, the Department for Transport (DfT) commissioned one of the world’s leading experts on megaproject management, Professor Bent Flyvbjerg, to investigate evidence of optimism bias levels in past transportation projects and make recommendations for appropriate adjustments to be employed in future project planning by the Department. It is from this work that Mr Fairhead obtains the 68% optimism bias adjustment recommendation that he quotes (see footnote 7).

Despite this departmental specific guidance being available to the planners of HS2, I can find no evidence that it has been employed in determining a suitable optimism bias adjustment for the project.

When considering the appropriate level of optimism bias adjustment to apply to a project an important factor is the level of risk that this adjustment will prove to be inadequate – in other words the risk that the total actual spend will exceed the project budget, including the optimism bias adjustment. For the HS2 contingency calculation, which includes optimism bias adjustment, this is turned around and the figure has been derived on the basis that there is a 95% confidence that the budget, including the contingency, will not be exceeded in practice. Accordingly, it seems appropriate also to consider the level of optimism bias adjustment that reflects a degree of confidence that it will be sufficient in 95% of cases.

Unfortunately the Treasury Guidelines make no reference to confidence levels, and Professor Flyvbjerg tells us that the data from which the guideline values have been derived, sourced from Mott MacDonald, has “no probability distribution or percentiles available” (see footnote 8). Professor Flyvbjerg’s recommendations do, however, stipulate confidence levels: he tells us that a 68% adjustment will give us 90% confidence and provides a graph that indicates that extending this to 95% (5% risk of cost overrun) requires increasing the optimism bias adjustment to around 80% (see footnote 9).

Another important factor when deciding the appropriate level of optimism bias adjustment is the stage that the project implementation has reached. Professor Flyvbjerg stipulates that the levels that he recommends “refer to, and should be applied to, estimated budgets at the time of decision to build” (see footnote 10). He also recommends that “if a project has moved beyond the approval stage to the stage of detailed design or construction, uplifts should normally be adjusted downwards” (see footnote 11). However, this expectation that uncertainty will reduce as the project matures appears to be challenged by the indications that I referred to in part 1 of this blog series that HS2 costs are rising.

What we can conclude from all this, I think, is that it is not easy to get the optimism bias adjustment right and that the calculation process used by HS2 Ltd/DfT appears to be shrouded in mystery. Yet very large adjustments to the HS2 budget, measured in £billions, are involved.

Clearly, any reviewer of the HS2 project should be satisfied that the optimism bias adjustments that have been included in the project budgets are set at an appropriate, and fully justifiable, level.

(To be continued …)

Footnotes:

  1. See paragraphs 1 and 2 of the report, Over-optimism in government projects, National Audit Office, December 2015.
  2. See paragraphs 2.1 and 3.1 of the paper, Green Book supplementary guidance: optimism bias, HM Treasury, April 2003.
  3. See paragraph 3.1 and Table 1 in Green Book supplementary guidance: optimism bias.
  4. See paragraphs 3.11 and 3.12 in Green Book supplementary guidance: optimism bias.
  5. See paragraphs 7.8 and 7.9 of the publication, The Economic Case for HS2: Value for Money Statement, Department for Transport, January 2012.
  6. See paragraph 1.2 in Green Book supplementary guidance: optimism bias.
  7. See Table 6 (entry for “Rail” and “90% percentile”) in the report Procedures for Dealing with Optimism Bias in Transport Planning Guidance Document, Flyvbjerg, B in association with COWI consultancy group, June 2004.
  8. See the footnote to Table 6 in Procedures for Dealing with Optimism Bias in Transport Planning Guidance Document.
  9. See Figure 9 in Procedures for Dealing with Optimism Bias in Transport Planning Guidance Document.
  10. See the first paragraph of Section 4.2 of Procedures for Dealing with Optimism Bias in Transport Planning Guidance Document.
  11. See the second paragraph of Section 4.3 of Procedures for Dealing with Optimism Bias in Transport Planning Guidance Document.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: