Beyond Big Data: Q&A with Alistair Croll

You write extensively about big data (and wrote a book on web monitoring). How do you think data is changing the way we understand businesses?

There’s a book by Eric Ries called The Lean Startup. The whole principle of lean is build, measure then learn.

The lean model says identify the thing that’s most uncertain, and then build in experiments just enough to verify or repudiate that uncertainty and then iterate. And the way you do that measuring part is data.

So if you learn to iterate quickly and you collect data and learn from it and build those lessons into the next iteration faster than the competition, you will win.

This is true of any industry, from press to start-ups to big companies, and this data-driven intel is absolutely exploding because of the sheer volume of new sources of data we’ve created.

There are people out there who see this vast amount of data and their eyes glaze over. So how important is the presentation of that data in a form and medium that people can digest and interact with?

It’s absolutely essential. Julie Steele edited a book on visualization and does a great job of explaining the importance of something being accessible.

There’s a company called Narrative Science that will take data and turn it into sentences. They take all the math that’s captured in little league baseball games and basically write newspaper articles that look like they were written by a human. And the uptake is huge because parents want to see an article written about their kid.

When it comes to filtering and analyzing this data, how much of a role will technology and machines have as opposed to, you know, humans?

In Philip K. Dick’s short story, Minority Report, precogs detect crimes before they happen using their psychic powers. Image via Wikipedia.

The human race is very good at coming up with systems to deal with problems. We haven’t got the speed to evolve a bigger brain at the rate that technology is changing so we had to build ourselves a prosthetic brain, and it’s a smartphone.

In a dozen years it will be unthinkable not to have one. When you are born you will have a smartphone that is part of you. It will follow you through your life. You will upgrade it, but it will have all of your contacts, it will have your life, logging stuff like your blood pressure and medications. It will know where you travelled, it will have a history of all your transactions. And none of this is science fiction.

The real moral issue around this is that we use the data to form predictions. And just because you think I’m behaving like a terrorist or a pornographer or a thief doesn’t mean I am. But if you act as if I am, you have effectively persecuted me for thought crime. We don’t need precogs for it, we have predictive analytics.

What you’re describing is not an ideal scenario, is it?

No. I just wrote a piece for O’Reilly last month called “Big data is our generation’s civil rights issue” and one of the best responses came from Anders Sandberg, who writes for the ethics blog at Oxford University.

He said that nobody sees the offer they didn’t get. So if only the white people got an offer for 20 percent off, that’s the same as giving a tax to minorities. But nobody’s aware of the offer they didn’t get because it’s not persecution, it’s just, “I didn’t get that offer.”

The banks used to do this thing called redlining in the 1950s and ‘60s where they would draw a border around a part of the town with all the minorities in it that they didn’t want to lend money to and they would say “no” to loans with those zip codes on them.

Just because we’ve figured out a way to personalize the redlining doesn’t make it any less unfair.

A map of Philadelphia showing redlining; the practice of refusing people from certain districts loans, based on racial stereotyping. Image via Wikipedia.

Now that we’re in futurist territory, what kinds of jobs or roles do you see arising to help navigate these issues?

American scholar Cathy Davidson says that over 60 percent of the jobs our kids will have don’t exist yet.

The first thing is data science. One of the best things I think society could do right now is to stop teaching calculus and start teach statistics. I mean you took calculus and trigonometry in school, but when’s the last time anyone’s used calculus?

When was the last time you used probabilities and statistics? Probably five times on the way here, right? Bayesian math, opinion polls, chances of rain… Yet we spend a very short time thinking about probability and statistics and lots of time learning about calculus.

So in a world where we are dealing with information most of the time, I would say statisticians, data analysts and data architects are very big positions. Also, privacy officers, and machine-assisted curators and programmers. I think the biggest thing is teaching machines how to do stuff.

So does that mean the future is hopeless for us liberal arts and humanities types?

“The reality is that when a machine tells you what you want, a hipster’s going to tell you what you should have instead.” Image via Flickr by Bruno Santos.

No, I think the future is very good for liberal arts and humanities types because when you get far enough up in any field it’s always philosophy. I mean, we just talked about a bunch of different issues, from the rights of the individual to thought crime; these are philosophy issues. These are issues of taste.

And if you look at the most successful e-commerce sites right now, they’re the ones that are curated, they are the ones full of hipsters telling you which fixie bike you need or which ironic moustache T-shirt you should wear.

The reality is that when a machine tells you what you want, a hipster’s going to tell you what you should have instead. There’s naturally going to be this tension between individuality and prediction.

You have been involved in the launch of a number of conferences, including Bitnorth, the International Startup Festival and CloudConnect. How important are events in this age when we can actually connect so easily using technology?

There’s a great book by Edward Tenner called Why Things Bite Back about the unintended consequences of technology. He says, for example, that with computers we expected the paperless office, but in fact what the computer did was turn everyone into their own desktop publisher and we use more paper than ever.

I think the same is true of conferences. We saw the emergence of video conferences, audio conferences, email and all these other remote messaging technologies. What that’s done is reduce the cost of distribution so the traditional media industries like print and publishing are threatened, but it has also reduced the amount of physical interaction we get.

If you are going to go and do networking, increasingly it’s about the lobby conference and the events are becoming more and more structured around that interaction.