Jaguar Feel Wimbledon, From Jaguar UK

Jaguar Feel Wimbledon, From Jaguar UK

In 2015, “empathic media,” automated technologies used to understand people’s emotions, took two steps forward. Both were in the UK.

The first was at Wimbledon. In partnership with the storied tennis tournament, three tech and media companies—Maido, Mindshare and Lightwave—launched a campaign called “Feel Wimbledon.” Using sensor-equipped wristbands, they tracked the moods and emotions of the Wimbledon crowd by monitoring the heart rate, localized volume, motion and skin temperature of 20 fans. This data was used to create ‘living ads’ for Jaguar by visualizing the audience’s fluctuating emotions.

The second took place along London’s Oxford Street, where M&C Saatchi (along with Clear Channel and Posterscope) tested advertising billboards with hidden Microsoft Kinect cameras that read viewers’ emotions and reacted according to whether a person’s facial expression was happy, sad or neutral. Wisely, the test adverts featured a fictitious coffee brand named Bahio. The really innovative part was that the ad improved itself until it received positive reactions from its audience.

There is nothing ethically troubling about the particulars of Jaguar and fictional Bahio campaign, but what they portend raises questions about marketing and ambient intelligence. The campaigns are early users of soft biometrics and artificial intelligence, technologies that are sure to see an upswing in advertising use and raise thorny ethical questions.

Soft biometrics differs from hard versions—which confirm identity by iris texture analysis, fingerprinting, facial scanning, and voice and signature testing—because the they are not used to identify people, but rather to enhance interaction between people and machines. For advertising, this involves tracing the body’s behaviour, inferring moods and emotional states, and using this information to target advertising. The artificial intelligence part comes in as soft biometric feedback from viewers is used to improve ads by tailoring them to the viewer.

M&C Saatchi London came up with an algorithm that recognizes the attention generated by an ad. Video via M&C Saatchi London.

The opportunity for wider use of soft biometric data will rise in tandem with both personal sensors, like mobiles and wearables, and commercial ambient sensors, like those in the Bahio billboards. If and when wearables become embedded in everyday life, emotionally sensitive empathic media will grant advertisers greater insight into our emotions through how we speak to our mobile devices, more granular facial recognition and emotional insights derived from our heart rates, respiration patterns and how our skin responds to stimuli.

In the earlier cases, consent is unproblematic. Wimbledon’s tennis fans knew what they were signing up for and the M&C Saatchi campaign was cautious in its ambitions with no data being retained. However, outdoor media owners such as Clear Channel are both obviously interested in collecting data about how campaigns are performing on their ad spaces, and working out ways to monetize data from tomorrow’s smart cities. The virtual and the real will soon meet through two-way interaction between mobile and wearable devices and urban environments, raising the necessity to better understand how emotion, behaviour and consumer histories are used. We might also be mindful that this data is of interest to surveillance agencies.

Related: The Public Money and Private Tech Behind Connected Cities

Interviewed at Cannes this year for BBC’s Click, Sir Martin Sorrell, CEO of WPP1 , was asked whether WPP would want to reach people with a marketing message knowing their emotive state.

“Yes,” he responded “one would like to do that but within the grounds of people knowing exactly what they’re getting into so

Sir Martin Sorrell,CEO, WPP Plc, United Kingdom by Sebastian Derungs

Sir Martin Sorrell,CEO, WPP Plc, United Kingdom by Sebastian Derungs

demystifying the process, simplifying the process, making people understand what they’re letting themselves in for… that’s really important.”

This is a laudable position, but how will it work in practice and how can it be ensured that advertisers stick to this moral highground? As it stands, this is a call for a consent mechanism of sorts. Although opt-in is not mentioned, the comments imply that without informed agreement WPP would not pursue this line of advertising. We’ll see.

The twist with all of this is that empathic media does not require personal information. Online, consent is typically required for the processing of personal data. What is at stake today is not personal data, but aggregated data about bodily reactions to ads and nearby events (a tennis match for example). In years to come this will surely become a hot regulatory issue, but as it stands, data protection and privacy concerns are based on the principle of identification, not intimacy.

Related: Ask Permission!: Navigating the Labyrinth of Data and Marketing

Let us take at face value for now the claims that data will be aggregated, and that any personally identifiable information derived from mobiles and wearables will not be sold or passed on from the point of initial data collection. Let us also temporarily assume that aggregated emotional data cannot be merged with other data sets so to re-identify people. If all of this is possible: what precisely is the ethical problem with the use of soft biometrics in advertising, if there is one?

After all, the creepy line isn’t a fixed thing. Social media is a prime example, as we willingly share, in public, insights into our lives that were inconceivable 20 years ago. Further, as informational privacy and personally identifiable information aren’t the problem, what possible concerns does this leave us with?

"First 4SQ mayor Perk," Louis Oliveira, flicker

“First 4SQ mayor Perk,” Louis Oliveira, flicker

At this stage, the ethics are unclear, but this is the future of empathy-based advertising:

  • Public spaces will be used to mine aggregated data about citizens’ emotions;
  • The relationship between our bodies and environment will be altered;
  • Our publically mediated emotions will in part fuel our media systems;
  • And emotional insights will generate a more fluid, dynamic and reactive and emotionally aware public environment.

The problem as I see it is temptation and history. The opportunity to link biometric information from opt-in or check-in apps such as Facebook and Foursquare—and thereafter devices, histories, locational data, friend lists and contacts—with other ads and retail environments is going to be enormously tempting. The history of behavioral advertising and online programmatic approaches tells us that the capacity to use emotional data in tandem with behavioral histories will be irresistible.

As Sir Martin Sorrell suggests, more discussion is required so that people and regulators understand what they’re getting themselves into. Somewhat ironically, we might take our cue for ethical standards from one of Adland’s favorites, Bill Bernbach of DDB. Famously he says “All of us who professionally use the mass media are the shapers of society. We can vulgarize that society. We can brutalize it. Or we can help lift it onto a higher level.”

My question for today’s marketing luminaries is this: Can and will the use of empathic media in advertising take us towards a higher level?

  1. Spafax is wholly owned by WPP.