Harvard's Berkman Center Bungles Broadband

Professors at a leading research unit put suspect data into a bad model, fail to include crucial variables, and even manufacture the most central variable to deliver the hoped-for outcome.

Climate-gate? No, call it Berkman's broadband bungle.

In October, Harvard's Berkman Center for the Internet and Society delivered a report, commissioned by the Federal Communications Commission, comparing international broadband markets and policies. The report was to be a central component of the Administration's new national broadband Internet policy, arriving in February 2010.

The 231-page report was an ode to foreign broadband success and especially to the regulatory model of "open access," a euphemism for mandated sharing of network assets at government-set prices. Although U.S. Internet innovation is flourishing, the Berkman Center found the U.S. tragically lagging other nations in consumer broadband penetration, prices, and network speeds. In a perfect set-up for a dramatic re-regulation of U.S. communications networks, Berkman concluded that open access mandates have "a positive and significant effect" on broadband penetration and that the effect is "somewhat larger . . . and more robust than previously thought."

Just one problem. Actually many problems. The report botched its chief statistical model in half a dozen ways. It used loads of questionable data. It didn't account for the unique market structure of U.S. broadband. It reversed the arrow of time in its country case studies. It ignored the high-profile history of open access regulation in the U.S. It didn't conduct the literature review the FCC asked for. It excommunicated Switzerland.

The heart of Berkman's case - its regression model - used a small data set with observations from just two years, 2002 and 2005. Berkman ignored nine of the 12 major studies that showed either a negative relationship or no link between "open access" and broadband penetration. Instead, as the basis of its own regression model, it chose one of the three studies that did find a link. Never mind that this model by John de Ridder had already been discredited in a subsequent, more sophisticated analysis. (De Ridder himself acknowledged his model's shortcomings.)

But even the faulty de Ridder study didn't produce the eye-popping correlations Berkman was seeking. So Berkman simply replaced a variable. But not just any variable. To better fit its preconceived thesis, Berkman arbitrarily changed 17 of the 33 values of the crucial "unbundling" variable that measured the strength of a nation's "open access" regulation. A "6" becomes a "1," a "10" flips to a "0," a "4" to a "9." Yes, much better now. Regulation is so successful!

It then reran the same model with a simpler "yes, no" unbundling variable, instead of the previous 1-10 scale. In this case, it switched eight nations from "yes" to "no." It thus gave the U.S. a "no" despite the fact the U.S. had robust unbundling regulations from 1996 to 2005. It also assigned Germany a "no," although, as economists Robert Crandall, Everett Ehrlich, and Jeff Eisenach reported, Germany "had 2.5 million unbundled lines devoted to broadband out of a national total of 10.7 million broadband lines!" Apparently unable to find any remotely plausible way to reassign the value for Switzerland, they simply discarded it. Switzerland was a highly inconvenient case of a nation with no open access regulation but very high broadband penetration rates. So much for a country.

That wasn't all. Economist Robert Hahn asked why Berkman acknowledged in one section of the report that poverty rates and income were important factors in broadband take-up but then excluded these crucial variables from their econometric analysis.

As Crandall et al summarized in their scathing critique of Berkman's serial statistical miscues, "many of the regression results reported by the Berkman Study are the result of simply replacing the values for selected observations with data points that are more favorable to its conclusions." After correcting Berkman's model, the economists found "open access" regulation actually reduces broadband penetration. "Obviously," they concluded, "the Commission should not base its policies on analyses of what amounts to manufactured data."

Berkman's qualitative analysis was, if possible, just as misleading. It passed along faulty data on broadband speeds and prices. It asserted South Korea's broadband boom was due to open access regulation, but in fact most of South Korea's surge happened before it instituted any regulation. The study said Japanese broadband, likewise, is a winner because of regulation. But regulated DSL is declining fast even as facilities-based (unshared, proprietary) fiber-to-the-home is surging.

Berkman also enjoyed comparing broadband speeds of tiny European and Asian countries to the whole U.S. But if we examine individual American states - New York or Arizona, for example - we find many of them outrank most European nations and Europe as a whole. In fact, applying the same Speedtest.com data Berkman used, the U.S. as a whole outpaces Europe as a whole! Comparing small islands of excellence to much larger, more diverse populations or geographies is bound to skew your analysis.

The U.S. has some work to do delivering broadband to some rural areas and boosting adoption, especially among low-income Americans. And we need lots more wired and wireless bandwidth to keep up with demand and spur new innovation. But the U.S. broadband Internet sector overall has never been this healthy and vibrant. Consumer bandwidth is up 1,000% over the last five years. Miraculous mobile devices seem to rain from the sky. Web video is transforming entertainment and education.

The Berkman study was an attempt to throw the Internet into the Washington grinder along with health care, credit, clunkers, and carbon. But the Internet is a lonely slice of the U.S. economy that is manifestly working. Perhaps because it is relatively free from onerous federal regulation. The FCC is considering other harmful Internet regulations like "net neutrality," but at least on "open access" Berkman's botched report could help sink its own policy goal.