North America's fastest Internet backbone belongs to CompuServe, and the overall performance of the Net's fattest pipes improved dramatically over the holiday slowdown, according to the latest version of a controversial backbone survey that is expected to be released today.
The survey, called the Third Keynote/Boardwatch Index of Backbone Providers, was produced jointly by Keynote Systems and Boardwatch, a trade journal for Internet service providers.
"We contend that if you are shopping for backbone services for your Web site, this is a reasonable guide you can use in your shopping," said Gene Shklar of Keynote Systems.
But some network engineers sharply criticized the index, stating that it was based on inappropriate metrics and that it doesn't take into account factors such as hardware and software installed at either end of the pipes.
Described by its creators as an industry standard ranking, the index rates the time it takes to download a 10 KB chunk of data from various Web sites over 39 US and Canadian backbones. But critics say equating Web server responses with backbone performance is comparing apples and oranges.
"The problem with the Keynote/Boardwatch study is that it measures a metric that approximates the end-user Web surfing experience to each backbone's Web site and then represents this value as a representation of the backbone's overall speed," said Chris Layton, a senior network engineer with digitalNation Internet Services in an email.
Other networking experts were similarly critical of the survey.
"The Keynote studies are artificial benchmarks which have nothing to do with Internet performance as observed in the wild," said Michael Dillon of Memra Software, an Internet networking consultancy.
The controversy centers in part around the fact that webmasters pay varying attention to the degree to which they optimize their servers' HTTP response time, depending on their site's intended purpose. Such optimizing can have a strong impact on a survey that aims to measure performance of the pipes - not the boxes that sit on them. Further, the location of "peering" points, where networks exchange packets, can have a major impact on the rankings.
The fact that CompuServe, a content provider, has led the pack ahead of networking heavyweights such as MCI, which ranked second, UUNet, which ranked eighth, and GTE/BBN, which placed 11th, points to flaws in the methodology, critics allege.
"Keynote's study is based on the premise that your corporate Web server will be optimized to be as fast as possible," said Layton. "This assumption is typically valid for content providers and less likely to be valid for other companies," he said.
Acording to Keynote's Shklar, most Web sites surveyed were owned by the backbone provider being tested.
Layton said that Compuserve devotes a great deal of energy to optimizing its servers because that company provides content, and impatient content seekers demand an immediate response when they click on a link to get a story.
"Now Sprint, on the other hand, concentrates on selling leased lines," said Layton. "Their Web server is managed by their marketing group. It is somewhere outside of the core of their network.
"This is reasonable for [Sprint] because their corporate pages take a relatively low number of hits and are tangential to their core business goals."
But Shklar defended the findings, which are compiled with the help of his company's Perspective site performance measurement service. The measurements were recorded between 15 December 1997 and 14 January 1998 every 15 minutes, in 27 major metropolitan areas around the US.
"You can never do a test of backbone performance to the satisfaction of a network engineer," said Shklar. "They look at backbone performance differently than users and Web-site owners do," he said.
Responding to the criticism that some Web sites are speedier than others, Shklar said that all surveyed backbone providers are invited to supply Keynote with an IP address of the fastest site on their network.
But Dillon alleged that the wide publicity given to the Keynote/Boardwatch index has led some providers to rig their setups so they will look good on the tests - shortcuts that ultimately have little bearing on actual network performance delivered to customers.
Shklar also said that the measurements take geographical factors into account. But one source, a former network engineer at a backbone provider that is one of the 39 in the survey, said that when he was asked to furnish an IP address to Keynote for testing, he gave them an address that was mirrored, or duplicated, on both the east and west coasts.
"[Keynote/Boardwatch] are measuring the transit points, not the backbones," said the source, who spoke on condition on anonymity. If a smaller backbone provider suffers poor peering with one of the top three providers, their ranking in the index will suffer, said the source. Shklar agreed that peering was essential, and should be a determining factor in selecting a backbone.
According to the index, the top five performing backbone providers that webmasters should consider for their sites are owned by, in descending order, CompuServe, MCI, Digex, IBM, and AT&T.
But an informal Wired News survey of a number of major performance-critical Web sites including search engines, content, and commerce sites reveals that many of them use backbones that didn't place in the Keynote/Boardwatch top five.
A test using the Unix traceroute command revealed that many connect, in part, to some of the lower-ranking backbones on the survey (major commercial sites typically use multiple backbones). Time Inc.'s Pathfinder sites use MCI, which ranked second in the survey. However, Amazon.com uses UUNET, which ranked eighth, CNN and AltaVista use BBN, which ranked 11th, Yahoo uses GlobalCenter, which ranked 19th, and Excite uses Genuity, which ranked near the bottom of the list, at 35th.
