Testing times
Many years ago, Ray Hyman, a psychology professor at the University of Oregon and one of the 26 founders of the Committee for Skeptical Inquiry, dabbled in reading palms. There are, as I've heard him say in describing the experience, benefits to picking this particular line in psychic claims because hands give you all sorts of helpful clues. First, there's the Sherlock Holmes act of noting calluses and other physical indicators of profession, class, and health and matrimonial state. Even more helpfully, when people like what you're telling them they tend to push their hands toward you; when they don't, they pull away, helpfully guiding what line to take. However he did it, people were generally very positive about the accuracy of Hyman's readings.
A lot of people would pat themselves on the back and call themselves great palm readers. Hyman, who may have been a born skeptic, took a different tack: he started telling people the *opposite* of what he thought he saw in their palms. And he still got the same enthusiastic responses. What this told him, and tells us, is that the assessments had nothing to do with the readings and everything to do with the people's level of belief in palm readers and the personality Hyman projected. His was an absolutely valid experiment to conduct in the name of science.
So to this week, when the news emerged that OKCupid deliberately ran three experiments on its users. The first involved a seven-hour window in which the service removed the pictures from a blind dating app it had available at the time. In the second the service hid either pictures or text to establish how much people relied on pictures in choosing dates. In the third the service told pairs of users the computer said were poorly matched that they were exceptionally well-matched and vice-versa. This last study is like what Hyman did: it tests the algorithm to see if it has any validity or whether its apparent success is all down to users' desire to believe it works.
Founder Christian Rudder's conclusion: "OKCupid definitely works, but that's not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth." In which case, what he's actually proved is that OKCupid's algorithms are about as good as chance. There's a logical reason: users all want the service to work.
OKCupid's experiments seem to me legitimate in a way that Facebook's experiments with manipulating newsfeeds were not. Even though most people's Facebook friends lists include plenty of marginal acquaintances, manipulating the newsfeed interferes with real emotions and pre-existing relationships. By contrast, OKCupid was testing the effectiveness of its service, and followed up by sending users the real compatibility scores. The New York Times reports that some users were modestly dismayed, but even before the test had low expectations of such sites. Realistically, that's the point: surely no one believes that testing OKCupid's algorithms has destroyed their one true path out of loneliness.
Articles like the one written by Milo Yiannopoulos at Business Insider miss the point. He suggests that the FTC might view the test as "unfair and deceptive behavior" and paints a sad picture of the desperately lonely person who trusts the compatibility scores, takes a chance, goes on a date, and has a terrible time. Yes, that's lost time. But even OKCupid's best match doesn't guarantee anything different. In fact, what the test proved is that their compatibility scores provide very little useful guidance to which correspondents might actually be worth the trouble to meet. It's not often you find a company willing to expose the threadbare nature of its own business model.
More helpfully, at ThinkProgress, Lauren C. Williams notes the trusting way people submit data to dating and other sites without recognizing that the companies' goals are quite different from their own. The incompatibility of motives is more obvious in the Facebook case: users want better but protected contact with the people who matter in their lives, while Facebook wants to mine their social graphs to sell advertising. OKCupid, now owned by Match.com, also mines user data to sell advertising, but pays users back with potential relationship matches - not limited to dates. What the test should tell users is that they're not getting what they think they're paying for; few sites admit that.
Having said all that defending OKCupid's tests, caveats remain. We don't know how many tests the company did, what the stated goals were, or whether they selectively published the results (though granted, these are not the results you'd expect them to cherry-pick). If you're going to experiment, you might as well do it right: publishing the information needed for others to independently replicate the results is the way to go, even though that would mean also publishing the algorithms it uses to match people.
But if users are going to leave the service over this, it shouldn't be because of the experiment itself. It should be because what the experiment shows is that the service's ballyhooed matching algorithm is nonsense. So you're paying with your data for a crapshoot. Why is that worth paying for?
Wendy M. Grossman is the 2013 winner of the Enigma Award. Her Web site has an extensive archive of her books, articles, and music, and an archive of earlier columns in this series. Stories about the border wars between cyberspace and real life are posted occasionally during the week at the net.wars Pinboard - or follow on Twitter.