Linking Instagram, Pinterest and Houzz to e-commerce

What products do my pictures suggest?

This is a continuation of an idea that arose from a conversation in a bar around the question: "How does content work?"

Recommendation engines typically work with data from one site. We've all seen examples of "people also bought..." on web sites. But an interesting idea would be to link my curated content (Instagram, Pinterest, or Houzz) to the shopping experience of an e-commerce site.

To illustrate the idea, here's a widget from Houzz narrowed to bedrooms:

Assuming for a second that is my idea book, I would appreciate the shopping site recommending the second bed below rather than the first.

To make this a reality requires a couple of things. First, we need access to both sets of pictures - this is a perfect use case for a widget. The benefit seems obvious: "Log on to your x account and we'll do the heavy lifting of sorting thru a couple of hundred thousand items for you."

Second, we need to identify what the pictures contain or portray. This argues for auto-tagging of images to classify and assign attributes. It is unlikely that traditional work-flow processes can handle this task because consistency across two businesses is required. For this to work we need the same method of classifying two sets of pictures (mine and theirs) to a common set of standards. The good news is there are web services for doing this on the fly. Although, if I were an e-commerce site or travel site I'd pre-process my pictures as part of the editorial/content management process.

Third, we need to define something that allows us to rank the pictures. And that means we need a measure of similarity. In market basket analysis it is easy - count the times each pair of products appears in the shopping cart and pick the most common. In this case though, all we have is a list of tags for each picture. In this case there are approaches to similarity ranging from a count of the shared tags among the whole set to computing how many steps are required to convert one set to the other.

The web services I looked at both give a 'confidence' score about the tags. This numeric value can be used to filter tags (clean out those we're not sure of) or be used to weight the tags. This gives the solution a new dimension and also helps to remove judgment from the picture.  

Next up. Working to get my own idea book and curated content classified and do some more experimentation.