net.wars

The digital universe is big. Really big. You just can't believe how mind-bogglingly big... Oh, never mind.

There's nothing like a good the-sky-is-falling scenario to get a one-day conference awake, and today at the LSE was no exception.

"It's a catastrophe waiting to happen," said Leslie Willcocks, the head of the Information Systems and Innovation Group at the LSE, putting up a chart. What it showed: the typical data center's use of energy and processing power. Only 1.5 percent of the total energy usage powers processing; 80 percent of CPU is idle. Well. They weren't built to be efficient. They were built to be reliable.

But Willcocks wasn't gearing up to save the planet. Instead, his point was that all this wastage reflects a fetish for connectedness: "The assumption is you have to have reliable information on tap at all times." (Cue Humphrey Appleby: "I need to know everything. How else can I judge whether I need to know it?") Technology design, he argued, is being driven by the explosion in data. The US's 28 million servers today represent 2.5 percent of the US's electricity needs; in 2010 that will be 43 million. This massively inefficient use of energy is trying to fix what he called a far bigger problem: the "data explosion". And, concurrently, the inability to manage same.

In 2007, John Gantz, chief research officer at IDC, said, for the first time in human history the amount of information being created was larger than the amount of storage available. That sounds alarming at first, like the moment you contemplate the mortgage you're thinking of taking out to buy a house and realize that it is larger than the sum of all your financial assets. At second glance, the situation isn't quite so bad.

For one thing, a lot of information is transient. We aren't required to keep a copy of every TV signal - otherwise, imagine the number of copies we'd add every Christmas just for rebroadcasts of It's a Wonderful Life. But once you've added in the impact of regulatory compliance and legal requirements, along with good IT practice, consider the digital footprint of a single email message with a 1Mb attachment. By the time it's done being backed up, sent to four recipients, backed up, and sent to tape at both sending and receiving organizations it's consuming over 51.5Mb of storage.

And things are only going to get exponentially worse between now and 2011. The digital universe will grow by an order of magnitude in five years, from about 177EB in 2006 to 1,773EB in 2011. More than 90 percent of it is unstructured information. Even more alarming for businesses is that while individual consumers account for about 70 percent of the information created, enterprises have responsibility or liability for about 85 percent of it. Think Google buying YouTube and taking on its copyright liability, or NASA's problem with its astronauts' email.

"The information bomb has already happened," said Gantz. "I'm just describing the shape of the mushroom."
To be sure, video amps up the data flows. But it's not the most important issue. Take, for example, the electronification of the NHS. Discarding paper in favor of electronics saves one kind of space - there's a hospital in Bangkok that claims to have been able to open a whole new pediatric wing in the space saved by digitizing its radiography department - but consumes another. All those electronic patient records will have to be stored, backed up and stored and backed up again in each new location they're sent to. Say it all over again with MP3s, electronic patient records, digital radio, VOIP, games, telematics, toys...

No wonder we're all so tired.

And the problem the NHS is solving with barcoding - that people cannot find what they already have - is not so easily solved with information.

Azeem Azhar, seven months away from a job as head of innovation at Reuters, said that one thing he'd learned was that every good idea he had - had already been had by someone else in the organization at some point. As social networks enable people to focus less on documents than on expertise, he suggested, we may finally find a way around that problem.

The great thing about a conference like this is that for every solution someone can find a problem. The British Library, for example, is full of people who ought to know what to keep; that's what librarians do. But the British Library has its roots in an era when it could arrogantly assume it had the resources to keep everything. Ha. Though you sympathized with the trouble they have explaining stuff when an audience member asked why, given that the British Library has made digital copies, it should bother to keep the original, physical Magna Carta.

That question indicates a kind of data madness; the information we derive from studying the physical Magna Carta can't all be digitized. If looking at the digital simulacrum evokes wonder, it's precisely because we know that it is an image - a digital shadow - of the real thing. If the real thing ceases to exist, the shadow grows less meaningful.

Wendy M. Grossman's Web site has an extensive archive of her books, articles, and music, and an archive of all the earlier columns in this series. Readers are welcome to post here, at net.wars home, at her personal blog, or by email to netwars@skeptic.demon.co.uk (but please turn off HTML).

Posted by Wendy M. Grossman at 11:34 PM | Permalink | Comments (0) | TrackBacks (0)

Back in November 2005 the CEO of AT&T, Ed Whitacre, told Business Week that he was tired of big Internet sites like Google and Yahoo! using "my pipes" "for free". With those words he launched the issue of network neutrality onto the front pages and into the public consciousness. At the time, it seemed like what one of my editors used to grandly dismiss as an "American issue". (One such issue, it's entertaining to remember now, was spam. That was in 1997.) The only company dominant enough and possessed of sufficient infrastructure to impose carriage charges on content providers in the UK was BT - and if BT had tried anything like that Ofcom would - probably - have stomped all over it.

But what starts in America usually winds up here a few years later, and this week, the CEO of Virgin Media, Neil Berkett, threatened that video providers who don't pay for faster service may find their traffic being delivered in slow "bus lanes". Network neutrality, he said, was "a load of bollocks".

His PR people recanted - er, clarified a day or two later. We find it hard to see how a comment as direct as "a load of bollocks" could be taken out of context. However. Let's say he was briefly possessed by the spirt of Whitacre, who most certainly meant what he said.

The recharacterization of Berkett's comments: the company isn't really going to deliberately slow down YouTube and the BBC's iPlayer. Instead, it "could offer content providers deals to upgrade their provisioning." I thought this sounded like the wheeze where you're not charged more for using a credit card, you're given a discount for paying cash. But no: what they say they have in mind is direct peering, in which no money changes hands, which they admit could be viewed as a "non-neutral" solution.

But, says Keith Mitchell, a fellow member of the Open Rights Group advisory board, "They are in for a swift education in the way the global transit/peering market works if they try this." Virgin seems huge in the context of the UK, where its ownership of the former ntl/Telewest combine gives it a lock on the consumer cable market - but in the overall scheme of things it's "a very small fish in the pond compared to the Tier 1 transit providers, and the idea that they can buck this model single-handedly is laughable."

Worse, he says, "If Virgin attempts to cost recover for interconnects off content providers on anything other than a sender-keeps-all/non-settlement basis, they'll quickly find themselves in competition with the transit providers, whose significantly larger economies of scale put them in a position to provide a rather cheaper path from the content providers."

What fun. In other words, if you're, say, the BBC, and you're faced with paying extra in some form to get your content out to the Net you'd choose to pay the big trucking company with access to all the best and fastest roads and the international infrastructure rather than the man-with-a-van who roams your local neighborhood.

ISPs versus the iPlayer seems likely to run and run. It's clear, for example, that streaming is growing at a hefty clip. Obviously, within the UK the iPlayer is the biggest single contributor to this; viewers are watching a million programs a week online, sopping up 3 to 5 percent of all Internet traffic in Britain.

We've seen exactly this sort of argument before: file-sharing (music, not video!), online gaming, binary Usenet newsgroups. Why (ancient creaking voice) I remember when the big threat was the advent of the graphical Web, which nearly did kill the Net (/ancient creaking voice). The difference this time is that there is a single organization with nice, deep, taxpayer-funded pockets to dig into. Unlike the voracious spider that was Usenet, the centipede that is file-sharing, or the millipedes who were putting up Web sites, YouTube and the BBC make up an easily manageable number of easily distinguished targets for a protection racket. At the same time, the consolidation of the consumer broadband market from hundreds of dial-up providers into a few very large broadband providers means competition is increasingly mythical.

But the iPlayer is only one small piece of the puzzle. Over the next few years we're going to see many more organizations offering streaming video across the Net. For example, a few weeks ago I signed up for an annual pass for the streaming TV service for the nine biggest men's tennis tournaments of the year. The economics make sense: $70 a year versus £20 a month for Sky Sports - and I have no interest in any of Sky's other offerings - or pay nothing and "watch" really terrible low-resolution video over a free Chinese player offering rebroadcasts of uncertain legality.

The real problem, as several industry insiders have said to me lately, is pricing. "You have a product," said one incredulously, "that people want more and more of, and you can't make any money selling it?" When companies like O2 are offering broadband for £7.50 a month as a loss-leading add-on to mobile phone connections, consumers don't see why they should pay any more than that. Jerky streaming might be just the motivator to fix that.

Wendy M. Grossman's Web site has an extensive archive of her books, articles, and music, and an archive of all the earlier columns in this series. Readers are welcome to post here, at net.wars home, at her personal blog, or by email to netwars@skeptic.demon.co.uk (but please turn off HTML).

Posted by Wendy M. Grossman at 2:32 PM | Permalink | Comments (0) | TrackBacks (0)

Some years back when I was writing about the data protection directive, Simon Davies, director of Privacy International, predicted a trade war between the US and Europe over privacy laws. It didn't happen, or at least it hasn't happened yet.

The key element to this prediction was the rule in the EU's data protection laws that prohibited sending data on for processing to countries whose legal regimes aren't as protective as those of the EU. Of course, since then we've seen the EU sell out on supplying airline passenger data to the US. Even so, this week the Article 29 Data Protection Working Party made recommendations about how search engines save and process personal data that could drive another wedge between the US and Europe.

The Article 29 group is one of those arcane EU phenomena that you probably don't know much about unless you're a privacy advocate or paid to find out. The short version: it's a sort of think tank of data protection commissioners from all over Europe. The UK's Information Commissioner, Richard Thomas, is a member, as are his equivalents in countries from France to Lithuania.

The Working Party (as it calls itself) advises and recommends policies based on the data protection principles enshrined in the EU Data Protection Directive. It cannot make law, but both its advice to the European Commission and the Commission's action (or lack thereof) are publicly reported. It's arguable that in a country like the UK, where the Information Commissioner operates with few legal teeth to bite with, the existence of such a group may help strengthen the Commissioner's hand.

(Few legal teeth, at least in respect of government activities: the Information Commissioner has issued an opinion about Phorm indicating that the service must be opt-in only. As Phorm and the ISPs involved are private companies, if they persisted with a service that contravened data protection law, the Information Commissioner could issue legal sanctions. But while the Information Commissioner can, for example, rule that for an ISP to retain users' traffic data for seven years is disproportionate, if the government passes a law saying the ISP must do so then within the UK's legal system the Information Commissioner can do nothing about it. Similarly, the Information Commissioner can say, as he has, that he is "concerned" about the extent of the information the government proposes to collect and keep on every British resident, but he can't actually stop the system from being built.)

The group's key recommendation: search engines should not keep personally identifiable search histories for longer than six months, and it specifically includes search engines whose headquarters are based outside the EU. The group does not say which search engines it studied, but it was reported to be studying Google as long ago as last May. The report doesn't look at requirements to keep traffic data under the Data Retention Directive, as it does not apply to search engines.

Google's shortening the life of its cookies and anonymizing its search history logs after 18 months turns out to have a significance I didn't appreciate when, at the time, I dismissed it as insultingly trivial (which it was): it showed the Article 29 working group that the company doesn't really need to keep all that data for so long. In

One of the key items the Article 29 group had to decide in writing its report on data protection issues related to search engines (PDF) is this: are IP addresses personal information? It sounds like one of those bits of medieval sophistry, like asking how many angels can dance on the head of a pin. In the dial-up days, it might not have mattered, at least in Britain, where local phone charges forced limited usage, so users were assigned a different IP address every time they logged in. But in the world of broadband, where even the supposedly dynamic IP addresses issued by cable suppliers may remain with a single subscriber for years on end. Being able to track your IP address's activities is increasingly like being able to track your library card, your credit card, and your mobile phone all at the same time. Fortunately, the average ISP doesn't have the time to be that interested in most of its users.

The fact is that any single piece of information that identifies your activities over a long period and can be mapped to your real-life identity has to be considered personal information or the data protection laws make no sense. The libertarian view, of course, would be that there are other search engines. You do not actually have to use Google, Gmail, or even YouTube. But if all search engines adopted Google's habits the choice would be more apparent than real. Time was when the US was the world's policeman. With respect to data, it seems that the EU has taken on this role. It will be interesting to see whether this decision has any impact on Google's business model and practices. If it does, that trade war could finally be upon us. If not, then Google was building up a vast data store just because we can.

Wendy M. Grossman's Web site has an extensive archive of her books, articles, and music, and an archive of all the earlier columns in this series. Readers are welcome to post here, at net.wars home, at her personal blog, or by email to netwars@skeptic.demon.co.uk (but please turn off HTML).

Posted by Wendy M. Grossman at 3:03 PM | Permalink | Comments (0) | TrackBacks (0)

The first time I saw James Randi he was hauling a load of fresh chicken guts out of a guy's stomach.

Of course, in my eagerness to make it sound like a good story I've jazzed that up a bit. The chicken guts were real and the guy's stomach was real (he was an innocent audience member who'd been recruited for the purpose of demonstration), but the pull-outage was clever sleight-of-hand. The year was 1982 and the occasion was a lecture demonstration at Cornell University. The point was demonstrating how "psychic surgeons" achieve their effects.

The next time I'll see James Randi is on April 19, when he's giving a talk at Conway Hall, in London. I don't think chicken guts will be involved, though a number of other prominent skeptics will also be speaking and you just never know.

It was Randi's ability to demonstrate plausible explanations for the apparently inexplicable that blew me away on that particular day. A lot of people like to claim that skeptics are closed-minded, but in fact it seems to me that the key to skepticism is tolerance of uncertainty and patience. A skeptic sitting in an empty house and hearing inexplicable creaking thinks, "I wonder what that is." A believer thinks, "Must be a ghost." Randi never claimed to be able to explain everything, but he went a long way toward showing me that things that friends thought must be inexplicable might still have natural explanations if you had the patience to wait to find out what they were and the right kind of mind to. A lie goes round the world while the truth is still putting its boots on; it takes seconds to claim something's paranormal but years of research to find out the truth.

One of the sad things about science these days is that so many disciplines require so much expensive equipment and funding that it's hard for an amateur to make much of a contribution. There are, to be sure, exceptions: some friends on Crete were successful in finding the nests of griffin vultures and did a lot of work keeping count, and anyone can look for fossils and hope to fill in a gap in the record. But few can afford their own radio telescope, particle collider, or climate modelling supercomputer. Randi showed that amateurs with a particular bent - a knowledge of stage magic and deception - were more effective at assessing paranormal claims than many scientists.

None of this would qualify Randi as a subject for net.wars except that recently he's been the subject of Usenet spam. Most people who do not participate in Usenet are under the impression that all newsgroups drowned under email levels of spam long ago. But in fact until the last month, when the Chinese apparently discovered Usenet, spam levels have been negligible for quite a few years now. Once Web boards, blogs, and social networks got going Usenet became even more of a minority pastime than it was in its heyday. Spamming Usenet doesn't cost much, but why bother when the audience is relatively tiny?

But people who want to boast that they've bested James Randi apparently want to lump themselves in with ads for cheap knockoffs of Nike shoes, Breitling watches, and Prada handbags. And so a version of this message began popping up randomly. It is, of course, all over the Net by now, and there's not a lot anyone can do other than debunk it and hope someone notices.
To deal with the most trivial bit, the bit that asks if James Randi is "even a real name". Well, it's not the name Randi was born with, although it's a modification of his first and middle names. But he's been using it consistently for something over 50 years, and it is his legal name. So it's real enough for all intents and purposes.

The million-dollar challenge was a relative newcomer that had its origins in a similar $10,000 challenge that Randi had going for more than 30 years. The increased money made the challenge a much juicier story, of course. But as this rational game theoryish analysis of the challenge makes clear, the challenge was only ever likely to attract the deluded. As I understand it, the mailbag got ridiculous in both size and content. There's plenty of evidence for that; the apparent basis of the claim that Randi was beaten is impenetrable. It is true, though, that until the beginning of this year the challenge rules stated that the prize would continue to be offered until it was awarded, including after Randi's death. Now, it ends March 6, 2010. (Get your claim in now!)

The end of the challenge is the end of an era for skeptics. For years, if any paranormal claimant was particularly insistent that he could dowse for oil or read minds we could say, "If you're so psychic, why ain't you taking Randi's challenge?" Now, my god - we're going to have to think of new stuff to say.

Meantime, come watch Randi in person and find out about the kinds of tests he's been doing all these years.

Wendy M. Grossman's Web site has an extensive archive of her books, articles, and music, and an archive of all the earlier columns in this series. Readers are welcome to post here, at net.wars home, at her personal blog, or by email to netwars@skeptic.demon.co.uk (but please turn off HTML).

Posted by Wendy M. Grossman at 1:05 PM | Permalink | Comments (0) | TrackBacks (0)