Beware the lure of crowdsourced data

sasha uritsky

Jun 18, 2014

Crowdsourced data can often be inconsistent, messy or downright wrong

We all like something for nothing, that’s why open source software is so popular. (It’s also why the Pirate Bay exists). But sometimes things that seem too good to be true are just that.

Repustate is in the text analytics game which means we needs lots and lots of data to model certain characteristics of written text. We need common words, grammar constructs, human-annotated corpora of text etc. to make our various language models work as quickly and as well as they do.

We recently embarked on the next phase of our text analytics adventure: semantic analysis. Semantic analysis the process of taking arbitrary text and assigning meaning to the individual, relevant components. For example, being able to identify “apple” as a fruit in the sentence “I went apple picking yesterday” but to identify “Apple’ the company when saying “I can’t wait for the new Apple product announcement” (note: even though I used title case for the latter example, casing should not matter)

Interested in reading more? Check out our other blogs:

CHATBOT PLATFORMS. How to choose the right one?

Chatbot platforms are essential tools if you need to build and run a chatbot.
There are many available on the market, big and small, popular and not so much.

Here are some useful thoughts that should help you navigate the complex world of chatbots and conversational AI solutions.

All chatbot platforms can be split into two categories: those that let you create chatbots without any programming, and those that require programming. Now, the idea that you don’t need to possess technical knowledge to build a chatbot seems appealing but the reality is not so rosy. In fact, I have yet to see a professional chatbot created without coding.
Chatbots rely on sophisticated algorithms and advanced knowledge of linguistics. These technologies are so complex that at the moment there are no plug-and-play solutions available. The companies like Chatfuel, Manychat, Flow XO and many others are attempting to fill that void and offer chatbot platforms that are simple in use. The best way to make the chatbot creation simpler is by dropping the need to code them. However this simplicity comes at a price: chatbots made without coding are limited, rigid and in general, primitive.
So to summarize: if you want to impress your girlfriend use Chatfuel. If you need a professional chatbot that delivers on your business goals and provides customer satisfaction use advanced chatbot platforms with programming capabilities.

One of the main, if not the main, tasks of the chatbot platforms is to connect your chatbot to the user interfaces. There are many ways for your chatbot to interface with the world: on Facebook messenger, on the website, on the mobile app, via SMS, on Twitter , on Skype, on Slack, on Telegram, and more. A good chatbot platform should seamlessly connect the chatbot to most of these channels. Chatbot platforms do not make your chatbot smarter. For this you need AI Engines (brief disucssion on AI Engines: http://nmodes.com/entry/2018/03/29/what-are-ai-engines-and-how-choose-one/).

For best results create your chatbot on a chatbot platform, then connect it to AI engine.

One of the top chatbot platforms on the market is Microsoft Bot Framework. It is robust, powerful, with a wide variety of useful functionality built-in. Another good chatbot platform is DialogFlow. DialogFlow has a slightly different architecture in the sense that it is a chatbot platform and an AI Engine all in one interface.

Chatbot platforms can be used to create conversation flow for your chatbot. There are several schools of thought here: some prefer to delegate conversation flow to AI engines. Chatfuel and other tools with the emphasis on simplicity (build your chatbot in minutes, no coding necessary) offer easy graphical interfaces for conversation flow creation. And there is always a reliable option to create conversation flow in an old-fashioned way, programmatically.

Which option to choose? Depends on your chatbot requirements and the business needs the chatbot is expected to address.And if you have questions feel free to ask: http://http://nmodes.com/contact-us/

Pros and cons of automation

Automation drives forward the economy. It allows businesses to scale and service large groups of customers. Automation first appeared in traditional industries, such as cotton production in England in 18th century or car conveyors in the US in early 20th century. The automation replaced physical labor.

With the invention of computers automated systems began to replace intellectual labour such as math calculations. Most of the software applications we use today can be described as automation. Online payments processing, online tickets purchasing, tax returns software, computer games, search engines, and endless other programs are all examples of software automation system.

As a next step we are now aiming at automating human decision making processing and high-level intellectual activities, historically considered to be sole domain of humans.

One interesting aspect of automation is lesser quality of service compared to manual service.

This is to be expected. If we gain in quantity we lose in quality.The gain in quantity is what automation is about - it allows to reach out to a large number of customers. Manual product or service can reach out to individuals only. The price we pay for the ability to deliver product or provide service en masse is the drop in quality.

Sometimes automation is an obvious choice. This is when the gain, the scalability, hugely outweighs the costs, lower quality. Search engine is a popular successful example. In other cases, the advantage in not so obvious. Online travel booking offers fast service without leaving the comforts of the home, but it does not often deliver the best option, such as finding the cheapest flight, and therefore many people still use ‘manual’ travel agents.