decorative yellow lines on background

However, personally, it felt like a novelty in the midst of an era of existential crisis for internet technologies.

Fast forward to today, and we can see that the proliferation of smartphones, messaging platforms with open APIs (Twilio, Messenger, Slack, WhatsApp etc), and advancements in artificial intelligence have graduated chatbots from internet novelty to business necessity.

With the potential to reach more customers in more places, chatbots offer the potential to greatly improve how customers interact with businesses. Research from 2016 found that users spent 85% of their time in as little as 5 mobile apps each month. Recently Facebook announced that its Messenger platform has surpassed 1.3 billion monthly users,  displaying steady user growth and showing its huge reach. Despite the opportunity, there are many hurdles to creating a successful chatbot, most notably insight into the optimal chatbot experience.

As companies such as KLM, eBay, and TD Ameritrade bring their digital experiences to platforms like Messenger, it’s crucial to understand what works, what doesn’t and how users will engage with your bot early on in your development cycle.

Enter Optimizely Full Stack. Optimizely Full Stack SDKs allow product development teams to experiment and roll out features across all channels, including chatbots. To get a better understanding of how companies can truly experiment everywhere, my colleague Martijn Beijk and I collaborated to build our very own Facebook Messenger bot with Optimizely.

Building Our Messenger Bot

For our bot we emulated a recommendation feature, similar to eBay’s Messenger bot. For our experiment, we wanted to know whether it’s better to recommend one product or multiple products at a time.

graphical user interface, text, application

We built the bot in NodeJS and Express, in part because Node’s concurrency works well for applications handling large amounts of requests, and ease of setup. Alternatively, we could’ve easily accomplished the same functionality using another language/framework such as Python + Flask, or Ruby + Sinatra.

First we created an Optimizely Full Stack project, and implemented the Optimizely Node SDK into our Node Express app.

graphical user interface, website

Next we configured the Optimizely SDK in our app to use a helper function to grab the Optimizely datafile and initialize the Optimizely client:

const OPTLY = require('optimizely-server-sdk'), defaultErrorHandler = require('optimizely-server-sdk/lib/plugins/error_handler'), defaultLogger = require('optimizely-server-sdk/lib/plugins/logger'), rp = require('request-promise'); module.exports.initializeClient = (url) => { let options = {uri: url, json: true}; // Grabs datafile from CDN let initializePromise = new Promise((resolve, reject)=>{ rp(options).then((datafile) => { console.log('Initializing Optimizely Client with Datafile: ', datafile); let optlyClient = OPTLY.createInstance({ datafile: datafile, errorHandler: defaultErrorHandler, logger: defaultLogger.createLogger() }); resolve(optlyClient); }); }); return initializePromise; };

For a production use case, you could also grab the datafile via Optimizely’s REST API and leverage Optimizely Webhooks to keep the SDK in sync with changes you make in the Optimizely interface.

Instantiating Optimizely in our Chatbot App

With the helper function to initialize Optimizely setup, we could then easily instantiate the Optimizely Full Stack client when our server starts so that we can begin using the SDK to run experiments.

let OPTLY_URL = `${process.env.PROJECT_ID}.json`, optimizelyClientOb; optimizely.initializeClient(OPTLY_URL).then((client)=>{ console.log('Intialize promise resolve'); optimizelyClient = client; });

Once that was completed we built out the endpoint to accept incoming events from Facebook – for example when a user sends our bot a message, or takes an action from within Messenger.

// Post endpoint to receive requests from Facebook'/webhook/', (req, res) => { messaging_events = req.body.entry[0].messaging for (i = 0; i < messaging_events.length; i++) { event = req.body.entry[0].messaging[i]; // Check to see if the event was a message from if (event.message && event.message.text) { text = event.message.text console.log(‘Heres\’s what the user said ’, text); continue } // Check to see if the event was a postback for a Facebook event - console.log(‘Hey this was a postback!’) continue } } res.sendStatus(204); });

Facebook Helper Functions

We also created some helper functions to handle sending a message back to Facebook and the user.

// Sending a Messenger template message function sendGenericMessage(sender, data) { messageData = data; request({ url: '', qs: {access_token:token}, method: 'POST', json: { recipient: {id:sender}, message: messageData, } }, function(error, response, body) { if (error) { console.log('Error sending messages: ', error) } else if (response.body.error) { console.log('Error: ', response.body.error) } }) }

Creating the experiment in Optimizely

Now that we’ve finished setting up our bot, we can build our experiment. For this simple use case we only need two variations for testing the single product vs. multiple product recommendation.  We’ll also want to define our metrics for this experiment. In this case we simply tracked if a user clicked to purchase an item, and the associated revenue generated. However, since users can also click through the bot links to our website.  

If we want to, we  can also track events that occur outside of Messenger such as new account creations, products viewed, or purchases on your website or in your mobile app. We can easily do this using the Full Stack JavaScript SDK to track these events. The key thing that we will want to do is share the same datafile between our NodeJS server code and JavaScript client code so that we can be in sync.

graphical user interface, application

Implementing the Experiment in Node

Finally, we built out the new functionality that we wanted to test. Here we added a condition to check if the user was asking for a recommendation as well as the logic for how to recommend products using hard coded product data.

if (event.message && event.message.text) { text = event.message.text if (text.toLowerCase().includes('recommend')) { // Generate random user ID var userId = uuid(); // Creates an array of the products from sample products/responses var categories = Object.keys(recs.products); var randomNumber = Math.floor(Math.random() * categories.length); // Select a random category to respond with var category = categories[randomNumber]; // Uses global activate function defined on line 9 var optimizelyVariation = optimizelyClient.activate('product_recs', userId); // Gets response from sample response json given category + variation var responseData = recs.products[category][optimizelyVariation]; var products = responseData.attachment.payload.elements; // Adds generated user ID to products for tracking if user clicks to buy optimizelyVariation === "single_project" ? helpers.addUserId(userId, products) : helpers.addUserIdSingleProd(userId, products); // Sends chatbot response sendGenericMessage(sender, responseData); continue } if (event.postback) { // Fired when user clicks "Buy Now" var payloadResponse = JSON.parse(event.postback.payload); // Grabs userID from payload var userId = payloadResponse.user; // Builds receipt var receipt = helpers.buildReceipt(payloadResponse); // Track event to Optimizely optlyClient.track('purchased', userId, {}, {revenue:receipt.attachment.payload.summary.total_cost * 100}); // Send receipt sendGenericMessage(sender, receipt); continue }

We set up a rule to parse incoming requests from Facebook to detect the word “recommend”. If anyone sent our bot a message looking for recommendations, we’d then activate them into our experiment, and either return one product or three different products.

Messenger bot in action!


Just like that we created two different variations for how our chatbot should respond with recommendations, and were able to track the effectiveness of each variation in terms of revenue and conversions.

This is just one simplified example of how Optimizely Full Stack can be implemented and applied to emerging channels like chatbots when it comes to experimentation. Product teams building chatbots can also use Full Stack to improve the customer experience with more advanced conversation algorithms, better timing of message sends, and even when to allow live agents to take control from the bot.  Experimenting in chatbots with Full Stack opens up a world of possibilities for teams to build better experiences and drive more customer loyalty.

Additional Links: