Beware the lure of crowdsourced data

sasha uritsky

Jun 18, 2014

Crowdsourced data can often be inconsistent, messy or downright wrong

We all like something for nothing, that’s why open source software is so popular. (It’s also why the Pirate Bay exists). But sometimes things that seem too good to be true are just that.

Repustate is in the text analytics game which means we needs lots and lots of data to model certain characteristics of written text. We need common words, grammar constructs, human-annotated corpora of text etc. to make our various language models work as quickly and as well as they do.

We recently embarked on the next phase of our text analytics adventure: semantic analysis. Semantic analysis the process of taking arbitrary text and assigning meaning to the individual, relevant components. For example, being able to identify “apple” as a fruit in the sentence “I went apple picking yesterday” but to identify “Apple’ the company when saying “I can’t wait for the new Apple product announcement” (note: even though I used title case for the latter example, casing should not matter)

Interested in reading more? Check out our other blogs:

MAKING AI MAINSTREAM

We are experiencing a strong demand for conversational AI solutions. It is coming from every corner of the B2C market. It is growing by the day.

Conversational AI is becoming increasingly popular among the consumer facing business community. It is easy to see why - AI offers sales and customer service scalability and therefore is critical for the long-term success of a business.

Conversational AI solutions such as chatbots, voice bots, and virtual assistants provide much needed speed and efficiency, in an age where the rapid advancement of technology makes them virtually the only sustainable customer service solution.

Bu there is a catch - AI is complicated. Mainstream businesses do not have in house AI expertise. And it is not part of their business model to develop such expertise.

Today’s market offer several good conversational AI solutions, such as IBM Watson or Google DialogFlow. However, getting a business value out of them requires the very AI expertise that mainstream companies do not possess.

So what can be done?

Any AI solution should follow these three steps in order for the mainstream business community to fully benefit from it:

Conversational AI should come as a service,
The service should be available in natural language,
The service should be fully personalized.

In the next several posts we will explore how the AI industry, including nmodes, is moving towards achieving these goals.

Abundance of Information Often is a Liability

A massive change has occurred in the world during the last ten to twenty years. Until recently and throughout the history of mankind information was hard to access. Obtaining and sharing information was either a laborious process or impossible, and the underlying assumption was that information can never be enough.

Today, of course, we have the opposite picture. Not only information is easily available, it keeps pouring in from a growing number of sources, and we continuously find ourselves in situations when there is more information than we want or able to process.

A major task we, as species, are facing is therefore how to reduce or filter out relevant information. It is, to repeat, in direct opposition to the task we’ve been accustomed to during all previous centuries, which was how to obtain information.

Since this change took place only recently, within a lifetime of one generation, we didn’t have time to develop efficient set of procedures to address the new problem. But the work has started and will only accelerate with time.