Nobody Understands Data: Tesla, the Times and the Twitterati
I am archiving older pieces I have written on other sites, making this the definitive home for all my work. Enjoy!
I once read an article titled You Are Boring which took to task people who, among other sins, "like to say ‘Science!’ in a weird, self-congratulatory way." I’ve met many such people. I’ve acquiesced to them, smiling and nodding as they extrapolate wildly from too little data to some all-encompassing theory about Northeastern Republicans or some other easy target. I once witnessed someone use the non-word "ecosphere," but he’s a friend of mine so I forgave him.
The relevance of the preceding will be made clear momentarily. Meanwhile, there’s a sizable public ignorance crisis (I exaggerate) afoot! A little over a week ago, a New York Times reporter attempted to drive a Tesla Model S electric sedan from Washington, DC to Millford, CT and back, using only Tesla’s Supercharger charging stations along the I-95 corridor. Things went sour when his vehicle lost significant charge and range overnight, leading to a disputed series of events that ended with his $101,000 loaner sitting on a tow truck flatbed. Tesla CEO Elon Musk responded with annotated graphs produced from the vehicle telemetry which Tesla says it records by default in all press events to protect itself from agenda-laden scribes that might besmirch its honor, and then accused the reporter, one John M. Broder, of a “fake” review and staging events.
(Google for more information. There have been additional volleys, with third parties from the Atlantic Wire to Wired weighing in.)
Update: The Times’ Public Editor has weighed in with what I think is a fair assessment, acknowledging that debate will continue for those who will never be convinced to change their minds.
This being the internet, lines were drawn, tempers flared and boring twerps came out yelling victory for “data!” (shades of “science!”)… except there is no data.
Tesla hasn’t released the actual vehicle data. Now, I’m not accusing Elon Musk of producing a fake response and massaging the data. No, actually, what I’m saying is that statistics require conditioning and data requires interpretation. It is easy to draw false correlations from masses of raw data points, especially when emotions run high. I’m saying that we don’t actually know what happened, and all this sound and fury is probably occluding a much more pedestrian state of affairs.
My biases, upfront: I think the Model S is an incredible car and electric vehicles are the future, but I’m bearish on batteries. I’ve never heard of Broder before, and I won’t be seeking out his writing after this. Elon Musk is the real-life Tony Stark, except better. Tesla, Space X and Solar City are a trifecta of potentially world-changing companies. (PayPal is meh, though.)
A random example: Broder claims he turned the temperature down and was “freezing.” Musk infers that Broder claimed to have done so at a specific mile count, then triumphantly crows about the much higher temperature set point recorded by the car at that time… conveniently ignoring the giant chasm of set point decline immediately after it. Further, commenters on multiple sites and services online seized on Broder’s claim of “freezing” despite the set point being a “comfortable” 64F… except that the vehicle set point is NOT the actual internal temperature but the target set for the heating system, and that in extreme cold weather a modest set point can leave extremities located by exterior body panels like the floor quite cold—put your hand by the window on a viciously cold day and note the “layer of cold air” there, despite the rest of the cabin being toasty. The cabin temperature sensor is typically placed out of direct sunlight so as not to cause solar heating to yield off readings, as an example of how environmental sensitivities are factored in.
Was Broder lying? Probably not. Was Musk fudging the data? Certainly not. This was a simple case of failing to condition the datum—set point is not ambient cabin temperature—and reaching a flawed conclusion as a consequence. (Does the Tesla record the ambient cabin temperature reading? Why didn’t Musk provide that if it does? Who knows.)
Or take another: Musk provides a graph that purports to show Broder driving “back and forth in a small, 100 space parking lot for about 5 minutes” and insinuates this was a deliberate attempt to run down the battery. Here we have insufficient data; the vehicle is fitted with GPS, so let’s correlate the timestamp on the speed log with position and see what the actual movement pattern is. Absent that, Broder’s response that he circled the lot once trying to find the Supercharger point is reasonable, given the length of highway rest exit approaches.
(The YouTube video of the Tesla Road Trip crew, a group of actual Model S owners replicating Broder’s trip under fairly similar conditions, driving up to the Supercharger doesn’t disprove Broder’s claim that it was unmarked and unlit. This was a much better researched and prepared group.)
The real meat of contention stems from the vehicle’s precipitous loss of charge overnight, falling from 90 miles of range to a mere 25. Here, Broder claims to have followed various instructions given to him by Tesla personnel, claims that Tesla has not actually refuted, except for Musk’s counterclaim that Broder “disconnected the charge cable when the range display stated 32 miles … expressly against the advice of Tesla personnel.” Which, really, is not that interesting, though it makes for a juicy journalistic integrity scandal, which is how many people are reacting to it.
The really valuable datum here is the charge loss overnight in extreme conditions. This is a significant finding, and one I imagine Tesla engineers are examining the car in detail to get to the bottom of. I want to know more about this stuff!
To be clear, I don’t place much value on Broder’s “review.” He stares right at his own data, published in his own article, and draws an erroneous conclusion: faced with a 32-mile range estimate and 61 miles to his destination, the vehicle travels 51 miles. To claim it “fell short” is asinine—it exceeded its stated range by 59%! FIFTY-NINE PERCENT!
Nobody who is actually interested in buying a Model S (with its 18-month or so waiting list) cares in the slightest what the New York Times has to say about the car. Actually, nobody who is interested in buying any car cares what the New York Times has to say about it. We’re being drawn into a pointless cacophony of name-calling, finger-pointing and side-taking, and the real story is being lost because, apparently, nobody understands data.
Tempest, teacup. He-said, he-said soap opera. Diametrically opposed media theater. Nobody will remember this in a few months. At least, that’s what the data suggests.