Networks and distributed systems

The RVSM Pre-Implementation Safety Case: Update

Peter B. Ladkin

I met with the RVSM Program Manager Joe Sultana, the Deputy Program Manager Chris Bouman, and the Safety Manager Bernd Tiemeyer at Eurocontrol in Brussels for about three hours on 18 November 2002. They indicated to be that they believed I had misunderstood the argumentation in the Safety Case, and the data from Eurocontrol's ACASA project on which they claimed it was based.

Eurocontrol's first point is as follows. I said in my published critique

Let the (estimated) risk with ACAS for each class T.i in the taxonomy be R(T.i). Let the relative frequency of occurrence of instances of class T.i be P(t.i). Then one calculates the overall risk as Σ(i in I) R(T.i).P(T.i) [10, Chapter 10], and one shows that this is less than TLS.

That kind of calculation is not performed in SC. Indeed, the only things said in SC about any of this are various reformulations of (1a) and (1b).

But those are the only two alternatives. So, to summarise, either

(a) risk without ACAS majorises risk with ACAS and one can use reasoning as in proposition (1b), or
(b) one must estimate Sigma(i) R(T.i).P(T.i).

(a) is not correct, as (3) asserts, and (b) is nowhere to be found.

Eurocontol claimed that they had indee performed such an analysis as in (b). A division into cases was contained in the Functional Hazard Assessment document [1], but this contains only considerations about two-aircraft interactions (as is apparent from the graphical scenarios) and not multi-aircraft interactions, such as the incident at Trasadingen to which I referred. Eurocontol claimed that a calculation of

Sum(i in I) Risk-with-ACAS(situation T.i).Probability(T.i) has been performed, by the ACASA project, and had been shown by that project to be less than Sum(i in I) Risk-without-ACAS(situation T.i).Probability(T.i)

Eurocontrol gave me the relevant ACASA report documents. The ACASA project considered the interactions of RVSM with ACAS [2]. They used radar data from a couple of days of operations, massaged them to distribute the aircraft according to RVSM separation instead of conventional separation, and investigated the results of the simulation. The Final Report [2] documents the frequencies of various events observed in the simulations. They observed no incidents which raised safety concerns. A calculation similar to

Sum(i in I) Risk-with-ACAS(situation T.i).Probability(T.i) was not apparent in any of the documentation; neither was their an indication that such a calculation had been performed.

I concluded that Eurocontrol's contention, that they had performed a probabilistic risk analysis calculation such as I had suggested, is incorrect.

2. The three incidents which I cited in my critique, the A340 balloon-climb incident on the North Atlantic track, the Swiss chain-reaction incident leading to an AIRPROX, and the South German midair collision over Lake Constance, are not identified or explicitly handled, or similar cases explicitly handled, in any of the documentation that has been cited to me. These cases raise issues, that I detailed, concerning the adequacy of the RVSM Safety Case, and these issues must be addressed in any satisfactory Safety Case.

3. If one wishes to base a conclusion concerning a likelihood of collision of one collision every 200 million to 400 million flight hours on statistical evidence, such as that provided by the ACASA project, it is a matter of straightforward Bayesian mathematics (as indicated, for example, in [3]) that one must have data corresponding approximately to that many flight hours. According to Eurocontrol's own statistics, they handle 8 million flights a year, expecting an increase to 10 million flights a year within the next few years. Assuming an average flight time per flight of 2 hours in Eurocontrol airspace, that would require some 10 years to 20 years of data to accumulate statistics with which one could attain a reasonable level of confidence in the estimate of collision likelihood which Eurocontrol wants to achieve. Eurocontrol has accumulated a few days of massaged data used in the ACASA project, and a couple of years of data since RVSM was introduced. With that data, they are unable to improve their level of confidence through statistical means in their desired estimate of risk of collision (the Target Level of Safety, TLS).

Since confidence in the TLS cannot be attained statistically on the current level of accumulated data, Eurocontrol cannot base their Safety Case on inference from this data. But other argumentation, that also takes into account the NAT incident and multi-aircraft conflicts triggered by ACAS, are not to be found in the Safety Case. The Safety Case remains flawed, as I claimed.

[1]Eurocontrol, RVSM Programme, Functional Hazard Assessment, 12 February 2001, Document RVSM697, PDF document available from www.ecacnav.com/rvsm/library.htm

[2] Eurocontrol, ACAS Programme, ACASA Project, Work Package 3, Final Report on ACAS/RVSM Interaction, September 2001, Document ACAS/ACASA/01-028, PDF document available from Eurocontrol ACAS/ACASA Project Home Page

[3] B. Littlewood and L. Strigini, ACAS Programme, Validation of Ultra-High Dependability for Software-Based Systems, Communications of the ACM 36(11):69-80, November 1993, available from Validation of Ultra-High (etc): Paper Access Page, City University, London