Beware the lure of crowdsourced data

sasha uritsky

Jun 18, 2014

Crowdsourced data can often be inconsistent, messy or downright wrong

We all like something for nothing, that’s why open source software is so popular. (It’s also why the Pirate Bay exists). But sometimes things that seem too good to be true are just that.

Repustate is in the text analytics game which means we needs lots and lots of data to model certain characteristics of written text. We need common words, grammar constructs, human-annotated corpora of text etc. to make our various language models work as quickly and as well as they do.

We recently embarked on the next phase of our text analytics adventure: semantic analysis. Semantic analysis the process of taking arbitrary text and assigning meaning to the individual, relevant components. For example, being able to identify “apple” as a fruit in the sentence “I went apple picking yesterday” but to identify “Apple’ the company when saying “I can’t wait for the new Apple product announcement” (note: even though I used title case for the latter example, casing should not matter)

Interested in reading more? Check out our other blogs:

nmodes Technology - Overview

nmodes ability to accurately deliver relevant messages and conversations to businesses is based on its ability to understand these messages and conversations. Once a system understands a sentence or text, it can easily perform a necessary action, i.e. bring a sentence about buying a car to the car dealership, or a complaint about purchased furniture to the customer service department of the furniture company.

Understanding sentences is called semantics. nmodes has developed a strong semantic technology that stand out in a number of ways.

Here is how nmodes technology is different:

1. Low computational power. We don’t use methods and algorithms deployed by almost everyone else in this space. The algorithms we are using allow us to achieve high level of accuracy while significantly reducing the computational power. Most accurate semantic systems, e.g. Google’s, or IBM’s, rely on supercomputers. By comparison our computational requirements are modest to the extreme, yet we successfully compete with these powerhouses in terms accuracy and quality of results.

2. Private data sources. We work extensively with Twitter and other social networks, yet at the same time we process enterprise data. Working with private data sources means system should know details specific only to this particular data source. For example, when if a system handles web self-service solution for online electronics store it learns the names, prices, and other details of all products available at this store.

3. User driven solution. Our system learns from user’s input. Which makes it extremely flexible and as granular as needed. It supports both generic topics, for example car purchasing, and conversations concentrating on specific type of car, or a model.

What Is Conversational AI

Conversational Artificial Intelligence solutions can communicate with people in their natural languages. The interactions happen via speech or text – our most common forms of interaction.

The most popular example of a conversational AI solution is chatbot.

The chatbot popularity began in 2016 with Facebook’s announcement of a developer-friendly platform to build chatbots on Facebook messenger. Soon, chatbots became the buzz of the technological community and spread across various industries. As a next step, toolkits that helped build a bot in five minutes grew popular, companies raced to the market with new bot announcements and the world woke up to a new chatbot-based reality.

A well developed conversational AI chatbot is able to interact on a near-human level. If we think about it, most companies’ customer service and sales centers deal with a core of 6-12 repeating issues. conversational AI software allows companies to develop an intelligent response channel that can cover the most common customer interactions.

Another advantage in using Conversational AI is in the marketing and branding domain. Chatbots allow the companies to stay on their message without veering off course . With AI, the scripts are all written and approved in house. Even when the AI system learns, when the appropriate training techniques are implemented, the system will adhere to the required profrssional verbiage.