Originally, I was working on a piece about the ‘relevance’ of outcomes. Reading Ben Goldacre’s “statistics toilet book”, however, I came across the term Surrogate Outcomes. An interesting subject and so ‘relevance’ went on hold for a while. Surrogate Outcomes are when you measure something else than what you really are interested in because you have a hard time measuring one things and you know or assume that the other things is correlated to it. Based on your measurements you draw conclusions about the first thing. I had heard the term before, but this time it struck a note and I decided to look a bit deeper into the subject.
Surrogate Outcomes
In clinical trials, a Surrogate Outcome (or Surrogate Marker) is a measure of effect of a specific treatment that may correlate with a real clinical endpoint, but does not necessarily have a guaranteed relationship. Surrogate Outcomes are used when the primary endpoint is undesired (like for example death), or when the number of events is very small, thus making it impractical to conduct a clinical trial to gather a statistically significant number of endpoints.
One known example to illustrate Surrogate Outcomes is cholesterol. It is known that higher cholesterol levels increase the likelihood for heart disease, but the relationship is not linear. Many people with normal cholesterol levels develop a heart disease, while many others with high cholesterol do not. The primary endpoint in this case is ‘Death from heart disease’, but ‘cholesterol level’ is used as the Surrogate Outcome. A clinical trial may show that a particular drug is effective in reducing cholesterol, without showing directly that it prevents fatalities.
Because we have problems to measure Safety directly (huge problems, in fact), we use quite a lot of Surrogate Outcomes to measure our efforts. Somewhat confusingly, maybe, after reading the general explanation above: one of the commonly markers is the number of fatalities. Let’s look at a real life example to see how this works and what pitfalls we may encounter.
The Case
Last week, a news report on our intranet caught my attention. It was about the Road Safety PIN Award 2016 for Outstanding Progress in Road Safety, awarded to Norway by the European Transport Safety Council (ETSC). The article echoed the press release of the National Road Administration (Statens Vegvesen, SVV) that claimed that Norway was honoured with a European award for the safest roads in the world.
Let’s be generous and not dwell on the obvious mistake in the title of SVV’s press release, which is likely a slip of the pen of an over-enthusiastic PR-consultant, extrapolating Europe to Global scale. Let’s instead focus on other elements. Without having the possibility to go in-depth (road safety is a fairly complex, yet fascinating, subject that would require much more study), I’d like to address some important points.
How to Measure Safety?
Now this is a difficult question, and one for that to my knowledge no one has managed to find a satisfactory solution. This does not stop people pretending that they can, and so we find a wide selection of statistics everywhere. The most common way is of course counting the number of fatalities or injuries. Ignoring for the moment the dilemma whether one should measure something by its absence, I think that the biggest problem lies in the certainty with which some people conclude that absence of accidents (or negative outcomes) means that something is safe.
That is clearly a logical fallacy. It is a basic ground rule in safety (but one that is little understood and practiced by professionals, politicians and public alike) that while the occurrence of accidents can mean that there are problems with regard to safety; the absence of accidents does NOT mean that things are safe. You can achieve the same for example by pure chance, luck or under-reporting.
There is another major problem, namely the relative randomness of consequences like fatalities and injuries. Related to road safety, the difference between life and death, or between serious and light injuries may lie in things like what kind of traffic participant you are, the type of car, speed, the angle in which you are hit, and obviously the number of people involved in the accident. A much better indicator for safety than the number of outcomes would be the number of accidents (e.g. collisions). We cannot tell from the data, but maybe there are just as many accidents in Norway as in previous years and it’s just through some of the aforementioned factors that fewer people die?
It is hard for organisations like ETCS to get good data. The ETCS bases her report on numbers of fatalities (and serious injuries) that each country reports. They encounter very much all of the problems above, and some more. Even though I doubt that many fatal accidents will go unreported, some countries have shaky registrations and routines, some only register certain accidents, and definitions of what a serious injury is differ from country to country. Information about the number of accidents seems to be unavailable. I tried to find them for Norway, but I fear that this information is spread over many players, including the Police, National Road Administration, municipalities, insurance companies (even Statistics Norway does not have the info, I checked) and probably the minor things go entirely unreported.
Toying with Relatives
What I often find problematic, is how reports like this toy around with relative numbers. Even though I often prefer relative numbers to absolute numbers, they can also contribute to confusion or paint a more positive (or negative for that matter) picture than you would. Be wary when you are presented with a series of percentages that all serve to support a certain point of view. As Gerd Gigerenzer has taught us, we have always to ask “percentage of what”. And just check if you are suspicious.
I am not saying that the ETCS has done something obviously wrong, when they first write that there was a 44% decrease between 2010 and 2015 and then continue with “an impressive 20% drop in 2015 compared to 2014 levels”. It is just a bit redundant or unclear what they want to say. Most likely, they are trying to stress the good news. That message can be constructed in a number of ways, however. Look at the statistics and you can calculate that there was a 20% reduction in 2015 compared to 2012 levels as well.
As we saw, outcomes can be seemingly random and fickle. They can just as well go up again. The Norwegian minister of traffic acknowledges this. He is pleased about the praise from ETCS saying that “the best reductions were reached in Norway, where the number of road deaths decreased by 20%...”, but appears to be better briefed than the Council. “The number of accidents on Norwegian roads will vary due to randomness from year to year”, he said.
Indeed, the 20% reduction may sound like a lot, but from 2012 to 2013, there was an increase of over 40 fatalities (29%, just to confuse you further with percentages). Easy come, easy go. In situations where the variation can be so large, flinging around relative numbers compared to the previous year have little or no value and one should rather observe long-term trends. Still this practice is very common in many safety reports.
More Surrogate Markers
I do not know how the concept of leading and lagging indicators goes together with Surrogate Markers, but let’s just try and maybe start a discussion. If I am entirely missing the point, please point this out to me!
Judging from the use in clinical trials, Surrogate Markers tend to be leading indicators: lower cholesterol should lead to lower chance of heart disease. Fatalities, however, or accidents are clearly a Lagging Surrogate Marker for Road Safety. Apparently, the ETSC also uses Leading Surrogate Markers. Their press release states that “Declines in the level of police enforcement of traffic offences are contributing to Europe’s failure to cut the numbers dying in road collisions”, and continues “In a separate report on enforcement, ETSC found that, in over half the countries where data is available, the number of tickets issued over the last five years for use of a mobile phone while driving has reduced, suggesting lower levels of enforcement across Europe”.
In this case enforcement is seen as a Surrogate Marker for Safety (defined as ‘fewer fatalities’) because it is assumed that more enforcement leads to better compliance with traffic safety regulations leads to fewer accidents leads to fewer fatalities and better safety. This reasoning makes intuitive sense to many people, but is not without problems because things are not always that linear. More enforcement can also lead to a greater deal of keeping up appearances and after the control post is passed people speed up just an extra bit to make up for ‘lost time’.
There are other problems. The press release tells us: “Sweden, The Netherlands and Finland are among countries that have reported falls in speeding tickets issued”. I do recall that several years ago the Dutch Police force (and without any doubt Police in other countries too) had specific numerical goals for the number of speeding tickets per year. I do not know if they still have, but abolishing these ridiculous goals will probably lead to a different focus (most likely whatever new political goal they got). Speaking of focus, complaining about reduced traffic enforcement clearly ignores other priorities that might just be a bit more important for society right now - like terror attacks and dealing with the largest wave of fugitives in Europe since Atilla the Hun.
Besides, also speeding tickets are nothing but a Surrogate Marker, because it is assumed that is says something about speeding behaviour. Of course, there are other possible reasons for fewer speeding tickets, like better general compliance of speed limits (not that I serious believe that this is the case, but hypothetically speaking). Interestingly, also this one is a marker that tries to measure something (traffic safety, or rather compliance) by its absence…
Like it or not, unless someone comes up with a brilliant way to measure safety as the ‘real deal’, we will have to work with Surrogate Outcomes for the time being (and all of future). This is okay as long as we understand the limitations and communicate within those limitations.
Where Is the Systems View?
There is one more comment that I must make. The cry for more enforcement is basically a claim that the system is safe if it were not for those stupid and non-compliant people in it. The title and the message from the press releases mentioned above also reflect this view. The SVV claims that the award is about “the safest roads" (not so strange after all that is what they are all about) while they probably should have talked about the traffic system as a whole.
Because, are the roads really so safe? Intuitively I would say Danish roads are much safer than Norwegian ones. And compare the German Autobahns to motorways in Norway and you can count proper motorways almost on one hand (even though there is clearly improvement in recent years). Many Norwegian roads can be characterised as narrow, there are many tunnels (many of them also pretty narrow), the roads go through challenging landscapes, they are strongly affected by weather and seasons and do not underestimate the presence of rather large wild animals. Thinking of those, collisions with large animals like moose or reindeer rarely lead to fatal accidents (or serious injuries for people), but lead to major to damage and have a rather high potential. Although there are many of these accidents, I do not think they reflect in the ETCS numbers at all because they lack particular outcomes.
So, when talking safety on the roads, this might be in many cases thanks to the drivers (who adjust their speed to conditions and handle the constant variability in a rather good way) and not be despite of the drivers. Another important factor is certainly the improving quality of cars, and the ability of many Norwegians to afford them.
It would be fun to do a proper study of all these factors, and I would not be very surprised if the findings would echo many of the remarks that John Adams already made in his brilliant book almost 25 years ago. Adams also mentioned already an issue that was raised by SVV Director Guro Ranes: “We are concerned about the negative trend for seriously injured pedestrians and cyclists”. Adams questioned the fact that there was much attention protecting the best-protected traffic participants (car drivers). This only lead to riskier behaviour, while there was little attention for the weaker parties.
Safe, or Not Safe?
Having made all these critical remarks, it must be said that there appears to be a steady decline in fatalities since the 1960s. Also, the number of (light and serious) injuries also seems to have a steady decline. If you feel like it, you can download numbers from Statistics Norway and play with them to check for yourself (as I did).
I cannot say what has led to this trend, but there has been a lot of good and serious work on road and traffic safety, so one can assume that from a variety of measures at least some have had positive effects. Has the traffic system become safer? We have some circumstantial evidence pointing that way, but I would hold back hallelujah-stories, because there are still many hazards and also worrying new ones, like for example a growing number of East European trucks in doubtful state, with unfit tyres, etc.
As an aside, doing a bit of research on the web I found an interesting British take on the matter, commenting that they were the second safest, but were ‘punished’ for not making more progress from an already safe situation. Well, yet another argument to leave Europe…
A positive, yet critical, conclusion
One might wonder why some Safety People (among which yours truly) appear to be so grumpy. Can we not just be happy that there is a low number of fatal accidents? Should we not celebrate a low number of traffic fatalities? Yes, we should, because it is good news! However, should we also conclude that we are safest based on some outcome number? No! We just cannot tell without additional information! So please learn to be reluctant to draw quick and easy conclusions, even if it is flattering.
Road traffic is a complex system where safety is created by (or emerges from) a large number of factors and their relationships. Do not give in to over-simplification (especially illogical forms), and whenever you see positive numbers also ask for the Bad News and/or look for evidence that disproves a Good News hypothesis. Confirmation can be (too) easy. Trying to falsify may be harder, but it will make your findings more robust and valuable!
Postscript:
Not even a week went by, and one of the messages above was confirmed - fatality levels on Norwegian roads were much higher in the first half of 2016 than the same period the previous year. Sometimes we just hate it to be right...
Also published on Linkedin.
