What did we learn from predicting song popularity?
A recent data science competition focused on whether one could predict the success of a Taylor Swift song from less than half a second of audio. One sample sounded like:
The objective was to predict whether that clip (and some 20k others) came from a popular song, or not. As mentioned in a previous post this is a case where there aren't descriptive elements that "make sense" - no genre, gender, demographics, contact history, or anything that marketers typically rely on to help them understand what is going on. Just 3,300 numbers - for the audio among you, down-sampled to 11k.
The winning models got to above 99% accuracy which sounds a bit to good to be true. While technically correct there are some interesting lessons to be learned.
First, by rethinking the problem as a segmentation problem rather than individual estimates the results got a lot better. That is, by grouping clips based on their similarity the accuracy improved - this is no different than targeting audiences as opposed to specific individuals.
Second, simple models tended to work as well as complex. In this case, accuracy mattered so effort was put on improving that as much as possible. But there are times when good enough is, well, good enough. It turns out that a simple model of "how similar is this clip to its nearest neighbor" worked very well. With this challenge, that makes sense, pop songs are similar across their 3-4 minutes. After the competition I tried some really, really simple (and possibly stupid ideas) and did better than my official submission. Don't over think it.
Third, we needed to take a step back and align technique with problem. The same data and the same logic resulted in different results based simply on the approach taken. Always use two or more methods.
A big thanx to Devin Diderickson for posting his approach and thought process. (He finished second, I finished sixth).
Saturday, March 21, 2015
Marketing Analytics Requires Judgment
What advantage do marketers have over machines?
There are lots of cool examples in the use of math, models and machines to predict, classify and recommend. Some can be found on econsultany's site. Within the domain of marketing analytics there is the application of machine learning whereby we are simply trying to discern cues about our business from the data. It works, we can predict content you might like on NetFlix or recognize a motorcycle from a car (most of the time). But it is hard to do; NetFlix offered a million dollar prize if someone could improve on their approach by 10%.
Why is analytics hard?
Computers can't see.
The more complex the problem and/or approach, the more time we spend on working out how not to generate spurious results. Something as simple as correlation needs human oversight. Here is one of my favorites from a great collection of spurious findings.
It appears that Nicholas Cage movies and drowning in swimming pools go hand-in-hand.
Then there's the one about the decrease in honey bee colonies tracking with the rise in juvenile arrests for marijuana. Really?
Computers lack what seasoned marketers bring to the table - judgement. At present a lot of the work in the "big data" space is focused on how to best capture human knowledge in the analytics. In image recognition for instance how do we find that there are in fact cats on YouTube if we don't there are cats to begin with?
In short, we are trying to find ways to learn what we don't know we need to learn. This unsupervised learning can be a lot like kindergarten recess where we need an adult on the playground to keep us safe.
Thinking and judgement will never go out of style.
Always ask: Does it make sense?
There are lots of cool examples in the use of math, models and machines to predict, classify and recommend. Some can be found on econsultany's site. Within the domain of marketing analytics there is the application of machine learning whereby we are simply trying to discern cues about our business from the data. It works, we can predict content you might like on NetFlix or recognize a motorcycle from a car (most of the time). But it is hard to do; NetFlix offered a million dollar prize if someone could improve on their approach by 10%.
Why is analytics hard?
Computers can't see.
The more complex the problem and/or approach, the more time we spend on working out how not to generate spurious results. Something as simple as correlation needs human oversight. Here is one of my favorites from a great collection of spurious findings.
It appears that Nicholas Cage movies and drowning in swimming pools go hand-in-hand.
Computers lack what seasoned marketers bring to the table - judgement. At present a lot of the work in the "big data" space is focused on how to best capture human knowledge in the analytics. In image recognition for instance how do we find that there are in fact cats on YouTube if we don't there are cats to begin with?
In short, we are trying to find ways to learn what we don't know we need to learn. This unsupervised learning can be a lot like kindergarten recess where we need an adult on the playground to keep us safe.
Thinking and judgement will never go out of style.
Always ask: Does it make sense?
Tuesday, March 10, 2015
Linking Instagram, Pinterest and Houzz to e-commerce
What products do my pictures suggest?
This is a continuation of an idea that arose from a conversation in a bar around the question: "How does content work?"
Recommendation engines typically work with data from one site. We've all seen examples of "people also bought..." on web sites. But an interesting idea would be to link my curated content (Instagram, Pinterest, or Houzz) to the shopping experience of an e-commerce site.
To illustrate the idea, here's a widget from Houzz narrowed to bedrooms:
Assuming for a second that is my idea book, I would appreciate the shopping site recommending the second bed below rather than the first.
To make this a reality requires a couple of things. First, we need access to both sets of pictures - this is a perfect use case for a widget. The benefit seems obvious: "Log on to your x account and we'll do the heavy lifting of sorting thru a couple of hundred thousand items for you."
Second, we need to identify what the pictures contain or portray. This argues for auto-tagging of images to classify and assign attributes. It is unlikely that traditional work-flow processes can handle this task because consistency across two businesses is required. For this to work we need the same method of classifying two sets of pictures (mine and theirs) to a common set of standards. The good news is there are web services for doing this on the fly. Although, if I were an e-commerce site or travel site I'd pre-process my pictures as part of the editorial/content management process.
Third, we need to define something that allows us to rank the pictures. And that means we need a measure of similarity. In market basket analysis it is easy - count the times each pair of products appears in the shopping cart and pick the most common. In this case though, all we have is a list of tags for each picture. In this case there are approaches to similarity ranging from a count of the shared tags among the whole set to computing how many steps are required to convert one set to the other.
The web services I looked at both give a 'confidence' score about the tags. This numeric value can be used to filter tags (clean out those we're not sure of) or be used to weight the tags. This gives the solution a new dimension and also helps to remove judgment from the picture.
Next up. Working to get my own idea book and curated content classified and do some more experimentation.
This is a continuation of an idea that arose from a conversation in a bar around the question: "How does content work?"
Recommendation engines typically work with data from one site. We've all seen examples of "people also bought..." on web sites. But an interesting idea would be to link my curated content (Instagram, Pinterest, or Houzz) to the shopping experience of an e-commerce site.
To illustrate the idea, here's a widget from Houzz narrowed to bedrooms:
Assuming for a second that is my idea book, I would appreciate the shopping site recommending the second bed below rather than the first.
To make this a reality requires a couple of things. First, we need access to both sets of pictures - this is a perfect use case for a widget. The benefit seems obvious: "Log on to your x account and we'll do the heavy lifting of sorting thru a couple of hundred thousand items for you."
Second, we need to identify what the pictures contain or portray. This argues for auto-tagging of images to classify and assign attributes. It is unlikely that traditional work-flow processes can handle this task because consistency across two businesses is required. For this to work we need the same method of classifying two sets of pictures (mine and theirs) to a common set of standards. The good news is there are web services for doing this on the fly. Although, if I were an e-commerce site or travel site I'd pre-process my pictures as part of the editorial/content management process.
Third, we need to define something that allows us to rank the pictures. And that means we need a measure of similarity. In market basket analysis it is easy - count the times each pair of products appears in the shopping cart and pick the most common. In this case though, all we have is a list of tags for each picture. In this case there are approaches to similarity ranging from a count of the shared tags among the whole set to computing how many steps are required to convert one set to the other.
The web services I looked at both give a 'confidence' score about the tags. This numeric value can be used to filter tags (clean out those we're not sure of) or be used to weight the tags. This gives the solution a new dimension and also helps to remove judgment from the picture.
Next up. Working to get my own idea book and curated content classified and do some more experimentation.
Monday, March 09, 2015
Marketing Measurement: Lessons from Physics
How should we measure our efforts?
There is nothing new about the idea of measuring marketing. What is changing however is out ability to do so. Yet, we're not there yet.
Metrology, or the science of measurement, requires three things:
In marketing, Ashu Garg of Foundation Capital recently wrote a nice white paper on the "Decade of the CMO" in which he articulated the seven most important metrics in marketing.
There is nothing new about the idea of measuring marketing. What is changing however is out ability to do so. Yet, we're not there yet.
Metrology, or the science of measurement, requires three things:
- An internationally accepted unit of measure.
- How to realize those units in practice, i.e just what is a meter?
- Chains of traceability between a measurement and a standard
- Time - the second used to be a fragment of a celestial cycle and now is 9B+ spins of an atom; we've moved from personal time zones (there were thousands in the US before the transcontinental train) to highly precise coordination. What good would Snapchat be without a common sense of expiry?
- Distance - starting out as the length of the pharaoh's arm and moving to 1 ten-millionth of the distance between the North Pole and the Equator it is now how far light travels is a very short period of time. Imagine an inaccurate GPS.
- Mass - the weight of pure water, at sea level and at freezing point has been the reference point, this is the last measurement to based on an artifact of something else rather than a fundamental law of physics. Media impressions have been often described in "tonnage" to reflect reach and frequency.
- Moles - measures how much stuff is involved without worrying about mass or weight. It makes conversions easier to understand and handle. For instance, two parts hydrogen + one part oxygen equals one part water. Gross/Target Rating Points are one way we try to standardize across channels.
- Light - possibly the easiest to grasp and yet most peculiar, light is responsible for what we see and is defined in waves. The challenge is that our eye adapts to light creating two types of measurements - energy and "in the eye of beholder." What we observe in marketing is altered by both our observations and our biases.
- Heat - is defined by how fast something moves with absolute zero being the absence of movement. Temperature is simply a measure of activity. Word of mouth and influence would be analogous to positive heat. Poor customer service would be negative due to the friction it creates.
- Electricity - is all about the flow of stuff from one place to another thru time and space (lightning strikes for example). As evidence of the use of fundamentals, heat can be measured by electricity - the oven probe does it with a thermocouple. The customer journey and path-to-purchase deal with the flow of people with lots of insulators, leaky funnels and conductors.
In marketing, Ashu Garg of Foundation Capital recently wrote a nice white paper on the "Decade of the CMO" in which he articulated the seven most important metrics in marketing.
- Marketing ROI
- Customer Experience
- Conversion Rate/New Customers
- Overall Sales
- Marketing-influenced Sales
- Revenue-per Customer
- Social Media Metrics
While all good, it would seem we are still in the middle ages of marketing measurement. There are no precise or accepted ways to define many of those metrics on unassailable reference points. Just what is an "experience" or an "influence"? Even concepts like "sales" can be fuzzy concepts to nail down within one company let alone across companies.
Another takeaway from the series is: figuring this out isn't easy or quick. It often takes standing on the shoulders of giants to make progress so expect this to evolve over time. In the short term, we run the risk of measuring what we can, not what we need. It will take dedication from those who think differently to crack this nut.
Which leaves us with a fundamental question:
Another takeaway from the series is: figuring this out isn't easy or quick. It often takes standing on the shoulders of giants to make progress so expect this to evolve over time. In the short term, we run the risk of measuring what we can, not what we need. It will take dedication from those who think differently to crack this nut.
Which leaves us with a fundamental question:
What are the building blocks of consumer choice?
Subscribe to:
Posts (Atom)
Posts
