How to Build an AI-based Recommendations Service in Azure and Microsoft Cognitive Services
Overview
We’ve all seen Amazon’s clever ‘Customers who bought this item also bought…’ feature on their website. Amazon use it to upsell and to expose parts of their catalogue that customers might not know about. It’s clear that Amazon spent a great-deal of time designing and implementing their solution but it is now within reach of small organisations too thanks to services like the one we’ll show here.
How it Works
You might be able to imagine for yourself how this kind of recommendations engine (or collaborative filter, to give it its technical name) works. In Amazon’s case, they upload vast quantities of sales data to an AI they’ve built and then, when you’re browsing their products, they use the AI to see what other products people bought who also bought the product you’re looking at. The site then shows this list as scrollable table on the product page. This all happens instantly in real-time.
In our example, we too will upload raw sales data, train the AI and then create a web service we can use in our website. This is done using the Microsoft Cognitive Services Recommendations solutions template running in Microsoft Azure.
Getting Started
You’ll need a working Azure account. If you have one already you can use that, otherwise, you can get a free Azure account here. Please sign-in to your Azure account now.
The Recommendations Solution Template
Go here to find the template we’ll be using. Here’s what it looks like at the time of writing…
Deploying the Solution
It’s important to understand that almost everything you do in Azure costs money and this is true of the resources created for this solution. Please give careful consideration to this as we move through the deployment steps.
Click the ‘Deploy’ button to start the process.
This is step one of the Cortana Intelligence Deployment wizard…
Complete the form to your requirements, here we’ve given the deployment a name (this will be used to create an Azure Resource Group), chosen our subscription and a location. Hit Create.
This page tells us what needs to be created for the solution to run, an App Service account, a Storage Account and Application Insights to monitor how it’s doing…
Click Next to move along.
This page asks which kind of App Service you want to create. Feel free to review their different specifications as this will affect the performance and functionality of the service.
For our purposes a Basic 1 account is fine. Click Next.
The next page is about replication in the Storage Account. You can either choose Local and Geo replication.
For our purposes, LRS (local replication) is fine. Click Next.
The next page is about Application Insights. It’s this that allows you to see the performance and usage of your new recommendations service.
Choose you closest region and click Next.
The template will now deploy the solution into your Azure account, it takes a couple of minutes and then you’ll get an Email and a new page with some important information on it…
All of those URLs and keys can be found within the resources created by the wizard but this is a handy consolidated view of all the important information so you might want to copy-and-paste it into a document for safe keeping.
Our New Recommendations Service Resource Group
Go to your Azure portal and look at your resource groups. You should see a new one named from the first field on the first page of the deployment wizard, in our case, recsrvc1. Here’s what you’ll find inside…
These are a set of pretty standard resources, an App Service Plan and its associated App Service, Application Insights and a Storage Account.
Loading Some Data
The wizard has deployed all of the back-end code into the Azure resources for you, so let’s take a look. From the final step of the deployment wizard, mark the Admin Key and copy it to your clipboard. Then click on the Recommendations UI link. You should see something like this…
Paste the Admin API Key into the field and click Login.
You’ll now see an empty recommendations solution with no model…
Train Your First Model
Click on Train New Model and let’s see what we need to do…
Hmm, looks a bit complicated doesn’t it but fear not, here’s what we need to do.
- Prepare your sales data
- Prepare your Storage Account
- Upload your sales data
- Fill-out the form
- Click ‘Train’
Prepare Your Sales Data
This page contains everything you need to know about this solution including the source code and some examples. The Getting Started Guide is pretty easy to follow. However, we’re interested in getting our sales data into the correct format, this document talks about that. Microsoft refer to the data as Usage Events, simply, it’s a big list of what you sold, who to and when.
The Usage file is a comma separated value (CSV) file in this format…
<UserID>,<ItemID>,<Time>
So an example might look like this…
456,RedBucketSmall,2018-01-09T16:05:26
456,BlueBucketSmall,2018-01-09T16:05:26
998,RedBucketLarge,2018-01-09T16:35:24
Here you can see that customer ID 456 bought a small red bucket and a small blue bucket at the same time (presumably on the same order), and then, shortly after, customer ID 998 bought a large red bucket.
This trickiest part of this exercise is to get your sales data into this format. It’s just these three items that you need; who, what and when.
Naturally, if you can get your raw data into Excel, that’s a great place to work with it and get it into the correct format. Getting the time into the correct format can be tricky, however, you can select your time column, right-click and choose Format Cells then ‘Custom’ and enter ‘dd/mm/yyyyThh:mm:ss’ into the Type field, this will format the time into the required format.
Once you’ve got your sales data into the correct format, you need to export it as a CSV to your local machine.
Preparing your Storage Account
Now that we’ve got our data in the correct format, we need to prepare the Azure Storage Account to hold the data. A convenient tool for this is the Azure Storage Explorer, this is a local application that allows you to interact with your Storage Account directly. Download and install Storage Explorer from here.
Open Storage Explorer, add your Azure Account and sign in.
The interface is much like with Windows File Explorer, you’ll see you Azure accounts on the left, navigate to your subscription, then the new Storage Account and finally the Blob Containers. You can see that that deployment wizard has already created some containers (rather like Windows folders) for the internal parts of the AI models, etc.
Right-click on ‘Blob Containers’, choose ‘Create Blob Container’ and call it ‘sales-data’. Then click on the new ‘sales-data’ container, click on the ‘New Folder’ icon in the toolbar and create a folder called ‘usage’.
Upload Your Sales Data
You now need to upload your sales data CSV file into that ‘usage’ container. Choose ‘Upload’ from the toolbar and select that you want to upload some files.
Use the three dots to find the sales data file you created and upload it. Our file is called UsageExport.csv…
Fill-out the Model Training Form
Now that we have our usage data in-place, let’s return to the New Model and complete it as best we can. Here’s what the completed form should look like; we’ve used ‘sales-data’ as the container name and ‘usage’ as the sub-folder. (Interestingly, the model will import all the files it finds in that folder but we just have one for now.)
I usually change, ‘Co-occurrence Unit’ to ‘Timestamp’ and set the ‘Decay Period’ to 60 days. You can view all the details for these options using the ‘Model Parameters Documentation’ link.
Click ‘Train’
Time to train the model, click ‘Train’. The model will now be created and trained using your data. It doesn’t take long for our 40,000 row data file but that will vary based on your dataset. Here’s how it looks when it’s finished…
Let’s Try it Out!
Click ‘Score’ to open a page that let’s you play with your newly created recommendations service. Try entering one your popular product IDs into the field and click ‘Get Recommendations’. Product ID 7 is popular for us, here’s what we see…
The results field shows a sorted list of products bought by other customers who also bought product ID 7. The Score value represents how confident the AI is about its decision.
If you enter a list of product IDs, like you would if the customer had items in their website shopping cart, you receive a set of results based upon all their items…
See how the list updates and the confidence score changes?
Your Website
In the real-world you’d need this to be integrated into your own e-commerce website and for that you’d use your web developers. The newly-trained model can be accessed using a REST-based API. It’s beyond the scope of this document but we can help with that integration and with the routine sales data updates that are needed to keep the model relevant.
Extra Stuff
What you’ve seen here is actually just the tip of the iceberg, this recommendation service has many customisable parameters and advanced options:-
- You can, for instance, upload quoted items, clicked/browsed items, (as well as sold items) into the model but have them ‘weigh’ less in the AI’s decision making.
- You can customise the results set for each specific customer based on their previous purchases.
- You can artificially add weight to particular/new items to ‘promote’ them in the recommendations.
Overview
Adding a recommendations engine to your e-commerce website (or customer Emails*) could work very nicely for you. Amazon are tight-lipped about how well it works for them, saying only, “Our mission is to delight our customers by allowing them to serendipitously discover great products” but I think it’s clear to see that recommendations are critical to their e-commerce experience.
*-Don’t forget to use the recommendations engine to customise your newsletters for each of your customers. Amazon do this all the time too!
Next Step
We would be delighted to help you design, implement and maintain a recommendation service like this for you, please feel free to contact us or reach-out to me directly if you have any questions.
Author
Jason Timmins – Technical Director – MBM Ltd – 01902 324494 – jason@mbmltd.co.uk
