SMS BASIC Scores are
Not Valid Predictors of Crash Frequency
Inam Iyoob, PhD
Director of
Engineering, Transplace
I am the Director of Engineering for Transplace and am a
data analyst and mathematical expert with a PhD in Engineering from the University
of Arkansas and a Masters in
Engineering degree from Oklahoma State
University. I have 12 years of work
experience with Transplace.
In advising shippers and brokers to use SMS methodology, the
Agency concludes: “Internal, external,
and independent (University of Michigan’s Transportation Research Institute)
evaluations have all shown that, of the six BASICs based on regulatory
compliance (the Crash Indicator BASIC is based on actual crashes), the Unsafe
Driving BASIC and the Fatigued Driving (HOS) BASIC have the strongest
relationships to future crash risk.”
In a separate study by Wells Fargo, the 200 largest
carriers, for which there is actually sufficient data, were measured. No perceptible correlation between safety and
SMS percentiles was noted in Unsafe Driving or in Fatigued Driving, the two
BASICs the Agency proclaims as most definitive.
The Wells Fargo Study concluded, “Quite simply, we found very little
relationship (i.e., not statistically significant) between Unsafe Driver or
Fatigued Driver scores and actual Accidents per Power Unit.”
Months after release of the Wells Fargo study, the Agency
attempted to re-substantiate the University
of Michigan and Volpe
National Transportation
Systems Center
studies in a paper devoted largely to touting the benefits of progressive
intervention entitled “Review of Wells Fargo Equity Research Report on
Compliance, Safety, Accountability” published March 16, 2012.
At the request of ASECTT, I have reviewed the FMCSA’s
defense of SMS methodology as a valid predictor of carrier safety.
In refutation of the Wells Fargo conclusion, the Agency has
submitted the two graphs shown below (Figures 1 and 2) arguing that the older
2009 Volpe National Transportation Study is more accurate than the Wells
Fargo’s study because it effectively measures 29 and 43 thousand carriers, not
just the largest 200.
Figure 1: FMCSA Regression of Averages – Unsafe Driving
Figure 2: FMCSA Regression of Averages – Fatigued Driving
An examination of the study
demonstrates that FMCSA’s data cannot be used to predict the crash performance
of individual carriers, even though the FMCSA claims SMS scores are correlated
to the average crash frequency of hundreds of carriers at each percentile
integral. Consumers of freight transportation do not select “average” carriers,
they select individual carriers and the Agency study offers no proof that SMS
methodology is a predictor of individual carrier safety performance at any
percentile level.
Based upon data obtained from the FMCSA’s own data bank, I
was asked to perform a detailed study of individual carrier percentile rankings
and crash frequency correlations.
That study resulted in the graphs shown in Figures 3 and 4. The study clearly shows that with respect to
individual carriers, percentile rankings of carriers both above and below the
arbitrary “monitoring thresholds” indicated with the are not
valid predictors of crash frequency. Regression analysis shows that SMS
percentile scores account for less than one percent of the variation in crash
frequency for each of these BASICs.
Figure 3: Unsafe Driving – Plot of 26,435 Carriers
Figure 4: Fatigued Driving – Plot of 35,933 Carriers
I can’t see any useful purpose in averaging the crash data
of hundreds of carriers in each of 100 different percentiles and then
calculating a regression of the average values. The purpose of regression
analysis is to explain variation. Averaging hundreds of carriers at each
percentile eliminates most of the variation in the data. It is not
statistically accurate to say the SMS methodology and BASIC percentile scores
are an accurate predictor of carrier safety predicated upon the crash data the
Agency uses to justify its conclusions.
Logically, unsafe driving and driver fatigue do impact
crashes. However, the way the SMS BASICs Unsafe
Driving and Fatigued Driving are
captured, calculated and interpreted by FMCSA does not show any correlation to
crashes. Hence usage of SMS data for carrier selection will unduly favor some
and penalize others, and thus should be avoided.
| 
As a small Trucking co. owner, I feel the SAFER System is setup like a shelf life; it’s only a matter of time before you hit the % and have high numbers. This is due to SAFER’s focus to find problems and not also true solutions.
ReplyDeleteOur drivers go through a class 1 inspection, get the CVSA stickers to put in the window and on the trailer, feeling really good they passed the big one, just to get a "warning" about a strap isn't tight enough or something. When the inspection hits SAFER it becomes a violation not a warning. There is no balance for passing an inspection only punishment for any small thing. (IE. strap not tight enough, chain 3” out of spacing, drivers signature isn’t legible (fatigued driver) and so on).
In order to balance the system, we should get points removed for a good inspection. Where is the balance? I don’t see it? Out of a Class 1 DOT inspection, 37 inspection points are evaluated. You pass 36 clean, you get a warning on the 37th and for this we get 7 bad points in SAFER and no recognition for the other 36. Where is the balance?
I was told by a FMCSA inspector “just getting good inspections, our numbers will go down”.
How can good inspections happen when it is their job to uncover the dust and find something wrong and when they don’t find something they just say “You’re free to go!” and you don’t receive a report.
DOT officers have said” I can find something and fail a Brand new truck.”
We need to be heard on this SAFER system, it doesn’t work. I have looked at some of the Large O/O companies and their numbers are getting high. Not because of real accidents but because of bad inspections.