Alto receives thousands of messages from patients every day, from questions about cost or delivery instructions to concerns about side effects. A care team member responds manually to each one in an important but time-consuming process that keeps patients waiting until their message reaches the top of the queue.

To get patients the help they need sooner — and to give our care team additional bandwidth for other important aspects of prescription fulfillment — we explored using a chatbot for some of this work. The first automation candidate? Requests to update medication quantity, which we chose for three reasons:

• Quantity change requests have a clear, simple resolution — the medication quantity is changed, and insurance is re-billed if necessary.

• The requests typically include all of the information necessary for automation, such as the medication name and desired quantity.

• We get these requests frequently enough to make automation impactful, but their volume is also small enough to be a good pilot.

Choosing Dialogflow CX

There are many options for conversation flow frameworks. We chose these four as our starting point since they would not require a new business associate agreement (BAA).

• Amazon Lex

• Dialogflow ES

• Dialogflow CX

• Custom, in-house framework

Our investigation of these frameworks included three key questions:

• What is the estimated cost based on our message volume?

• What is the level of customization and control? Can it handle complex conversation flows as needed?

• What is the developer experience like? How are the conversation flows defined? What are the steps for modifying or creating versions of a flow?

We ultimately decided to move forward with Dialogflow CX. While it was the most expensive option per message, it also provided the following benefits:

• Strong conversation customization, with Pages for tracking conversational state, reusable Intents, and configurable Routes within and between flows

• Native support for Versions and Environments, which allows us to easily create, deploy, and potentially roll back versions, as well as manage different versions across environments

• Web UI editor for rapid prototyping and validation during development

Additionally, the agents and flows can be managed in Terraform, which we already use at Alto.

Adding in OpenAI

We learned that while Dialogflow CX is great at defining and managing conversations, it’s less optimal for recognizing the intent of user messages or extracting complex entities such as medication names. To improve our chatbot’s recall and accuracy, we chose to pre-process messages using OpenAI models and prompts.

Using a training set of hand-labeled user messages, our data science and analytics team fine-tuned an OpenAI model, which we use to categorize an incoming request. We then map the category to a static message that we know Dialogflow will be able to recognize. Using a known, static message in place of the patient's original content reduces the likelihood of Dialogflow falling back on less predictable behavior.

We also prepend a system prompt to extract entities from the patient’s message. This prompt, augmented by the patient’s list of medications, instructs OpenAI’s chat engine on how to identify the medication name, requested quantity, and unit of measurement. Once our model identifies these values, we initiate a Dialogflow session with the values as parameters. For example, the following message would result in the associated entities.

Message: I would like to change my Metformin delivery to 15 tablets.
Result: {"medication_name":  "Metformin", "new_quantity":  15, "unit_of_measurement":  "tablet"}

Building the conversation flow

After fine-tuning our OpenAI model to handle message classification and entity extraction, we turned to the Dialogflow agent and conversation flow. We prototyped the agent first in the Dialogflow UI and later defined it in Terraform to facilitate reviewing changes and tracking versions in Git.

Each step in the conversation corresponds to a Page in Dialogflow. Transitions between pages used Intents to match responses like “yes” or “no” and to check conditions such as "Was the prescription identifier recognized and confirmed?” There were also pages to allow the user to exit the conversation and route their messages to our care team. For example, if the incorrect prescription was identified or the desired quantity was incorrect, the patient would have an easy way to exit the automation.

In our agent, each page also used a Webhook as part of its Fulfillment for the following purposes:

• Tracking conversation progress in our analytics system, which allows us to verify implementation quality and identify any areas for potential improvements

• Performing complex actions on the prescription, such as verifying the quantity of medication remaining on the prescription and applying the quantity updates

• Formatting custom response messages to the patient. A prescription that is out of refills may need a different response than one that can’t be updated due to legal regulations. By making those decisions and formatting the message in our own systems, we have better control of the wording and result.

Webhook implementation

Our webhooks are blind to the overall state of the conversation in Dialogflow. They rely only on the value of conversation parameters (or lack thereof) to perform any required actions, format messages, or trigger failure cases. These parameters are a mix of values extracted by OpenAI and from the patient’s messages.

At Alto, we use Ruby on Rails for the majority of our server-side code. While Google provides a convenient Ruby Gem for interacting with Dialogflow’s APIs, Dialogflow deeply nests parameters in the body of the webhook request. To make the request body more suitable to Rails controller actions, we overrode the standard behavior for an ActionController’s parameter handling:

      sig { override.returns(ActionController::Parameters) }
      def params
        parameters = super
        session_info = JSON.parse(request.body.read)['sessionInfo']
        request_params = session_info['parameters'] || {}
        parameters.merge!(request_params)
        parameters
      end

Because our webhooks rely on SessionInfo parameters alone, our controller actions can safely ignore the rest of the webhook request. With this override applied to a base controller for all webhooks, webhook implementers are free to use params as they normally would, without having to worry about the specifics of the request structure.

Additionally, we define and enforce a standard response structure using our existing API tooling and Sorbet. Before messaging a patient, we transform this response structure into the required WebhookResponse format, which eliminates the need to worry about Dialogflow’s specifics.

Controlling the conversation

There are two common limitations of chatbots in the wild:

1. They ask for information that has already been provided

2. They don’t understand what was just said to them

These issues are understably confusing or frustrating for users. In a situation as critical as medication delivery, we can’t afford to take these risks. To reduce the potential for misunderstanding, we constrain the responses offered to a patient whenever possible. For example, if a webhook has determined that we need to ask a yes or no question in order to proceed, it can declare the possible responses as part of the payload in its WebhookResponse. 

{
  "fulfillmentResponse": {
    "messages": [{"text":  {"text": ["Do you like ice cream?"]}}],
  },
  "payload":  {
    "suggestions":  [
      {
        "message": "Yes",
      },
      {
        "message": "No",
      }
    ]
  }
}

When we eventually receive the final result that should be sent to the patient in a DetectIntentResponse, we can look at the webhook payloads within the QueryResult structure and extract any values. If there are suggestions, the UI will restrict the user to buttons for sending them, instead of allowing a free response. The above example is trivial, but for complex cases like identifying a specific prescription among several with overlapping values, constraining the response set ensures that the patient response is something Dialogflow can properly handle.

Looking ahead

The first iteration of our chatbot has already helped patients achieve their desired results in under a minute, without any manual intervention. Previously, completing a patient’s quantity change request could take up to 45 minutes, often with multiple messages sent back and forth. While not every patient message can or should be automated by a chatbot, we are continuing to expand functionality for those that can. This allows our care team to spend more time on complex cases while offering many patients near-instantaneous help.