The Pitfalls of Big Data

Aug. 1, 2017
Deriving arbitrary targets isn’t an effective way to measure performance.

Peter Drucker, the legendary business guru and consultant, is credited with coining the expression, “What gets measured gets improved.” In the collision repair world, we have dozens of things that we measure internally in hopes of effecting improvement. With the passage of time and the advent of big data, I’m sure that Drucker would agree that the next important question is, “What should we measure?” I’m afraid that we are already on the path of measuring the wrong things and deriving arbitrary targets from them. Let me give you some scenarios that support the need to take a step back and reconsider the outcomes and unintended consequences of measuring the wrong things.

Recently we did a couple of bumper repair clinics for about 50 field appraisers from a major insurance company. During one of the clinics, the regional claims manager told me that his group was among the worst when it came to bumper repair percentages on their estimates. That raised a red flag and sure enough, I received an email from a DRP coordinator that listed the month by month and year-to-date (YTD) bumper repair percentages of the seven shops to which this coordinator was assigned. The YTD repair percentage ranged from 28 percent to 56 percent among the seven shops and the average was about 44 percent.

Besides the raw data, the coordinator included a note stating, “As you know, we are focusing on bumper repairs. Let’s look for more opportunities to repair bumpers.” The shop with the highest percentage got a shout-out in the email and that was it. There was no mention of historical trends, no graphs, no data on market averages and certainly no mention of what was expected if one was to be considered “good” in this category.

Trust me when I tell you that many more insurance companies will grab this big data and begin to measure you with it. I’ve been in this industry long enough to know that a lot of shops are not going to challenge or question this data or try to get it put into context, and sadly, some shops will try to manipulate the data to meet an arbitrary expectation or to put a checkmark in a box on some DRP scorecard.

Using a proxy for cycle time is another example of measuring the wrong things and we’ve already seen the negative outcomes of using this imperfect proxy data. Although we have the capability of measuring cycle time as a keys-to-keys activity, somehow measuring cycle time based on length of rental is becoming more prevalent because of big data analytics. Once again, most shops don’t challenge the accuracy or validity of the rental cycle time data.

Things really deteriorate when rental cycle time isn’t measured on all rentals. There are some companies that only measure the length of rental for their insureds and not claimants. How this is statistically valid is beyond me, but I have seen some bad behaviors that come from it. What happens if an insured wants to drop off their car for repairs on a Thursday and get into a rental? Often they are told that’s not allowed by the insurance company and they need to bring in the vehicle on a day that will least impact the length of rental. Yet how is it reasonable that a claimant can come in any time without this sort of pushback because their length of rental is not being measured?

I’m also dubious of measurements that are really only beneficial to one entity. Take severity, for example. Why is severity measured? In part, it is measured so that insurance company underwriters and actuaries can price the policy premiums properly based on vehicle type, driver experience, etc. Somehow, the average of all severity across all claims is now being used as an arbitrary target for shops to achieve.

I would say that severity is not within the control of a body shop. We don‘t control the year, make and model of vehicles we repair. We don’t control where vehicles get hit, the speed at which the collision occurs or the number of airbags that deploy because of the number of occupants in the vehicle. We don’t control which parts get damaged nor do we control the price of the parts. Measuring average severity is the wrong measurement. It would be better to use estimate accuracy data. After all, if the insurance company appraiser or reinspector agrees with the cost of a shop’s estimate, then severity is what it is.

To be sure, there can be great value in measuring certain activities and results for the purpose of improving your business. I merely want to caution you to be wise when choosing the big data and statistics that you will study.