Language Processing

To be able to communicate vocally with Cortana, she needs to understand what we’re saying. In our situation that involves us using a service called LUIS (luis.ai), which is described as:

A machine learning-based service to build natural language understanding into apps, bots, and IoT devices.

https://azure.microsoft.com/en-us/services/cognitive-services/language-understanding-intelligent-service/

LUIS allows you to create an application that examines your request, tries to define your intent, and associated entities, and then tell you what it believes is the correct response. To do this, you start by creating some intents, for example: HomeAutomation.TurnOn, or 
HomeAutomation.TurnOff.

You then create a list of utterances for these intents, which are sentences you would expect someone to speak to perform the task, such as: “Turn the kitchen light on”. Once you have a number of utterances (usually a minimum of 5), you can then train your application to find commonalities and patterns in the utterances which cause it to be associated with the given intent. If you’ve ever done any Machine-Learning before, you’ll understand what’s going on here. If you haven’t, don’t worry, it’s not essential knowledge to getting this all working.

You can now test your application if you want, to make sure it can differentiate your intent from the input. This is where you provide a sentence not in your list of utterances and see if your application correctly provides the right intent.

Intents are examined and given a score (0 to 1 in decimal format), which is the applications confidence in knowing what you are saying. The intent with the highest score is then declared to be the correct intent.

Example:

{ text: 'turn off kitchen light',
intents:
{ HomeAutomation_TurnOff: { score: 0.772598863 },
HomeAutomation_Control: { score: 0.0294161271 },
HomeAutomation_Scene: { score: 0.01918456 },
HomeAutomation_Status: { score: 0.01667956 },
HomeAutomation_TurnOn: { score: 0.0116767986 },
None: { score: 0.00246443041 },
HomeAutomation_GetLocation: { score: 0.00227342732 } },

As you can see, in our example of turning off the kitchen light, the TurnOff intent was clearly the winner.

Well, that all great, but what is it we’re trying to control? These are known as entities, and they are the context objects in your request. These are added in your LUIS app in a similar way as your intents, but with some additional properties, such as type: Simple, List, Composite, personName, DateTime, and many more. For lists, you can also add synonyms to your values, so for example: an entity named Event of type List, may have an item value of “Get Up” which has a synonym of “Wake Up”. This helps the language processor know that when you say something like “Turn on the coffee maker when I wake up“, it’s the same as “Turn on the coffee maker when I get up“. 

Once you have defined your entities, your app will try to put them into context with your utterances automatically, although sometimes you may need to help it if it picks the wrong entity.

The culmination of this, along with training the app at various stages as you add new elements of speech, results in something like this:

As you can see, my request to “Turn off kitchen light” resulted in my app recognizing a device, an operation, and a room. From there I could pass that data to a HomeAssistant script which could then pick the correct action and perform it.

The testing for this is done in the Bot Framework Emulator, available for all platforms, which we will get to later.

So, for this first step I highly advise not to bother creating a bot and not writing any code yet. Create a LUIS app and add some basic intents and entities and build something that understands what you want to do by asking it in a textual way.

One thought on “Language Processing”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.