Your partner in navigating the ins and outs of prescriptions — at no extra cost.

Alto Pharmacy Technician

Alto Pharmacist

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies based on team member feedback and a rigorous assessment of our company’s culture and benefits.

Join us in delivering our industry-leading pharmacy care to millions of people across the country. As a member of the team, you’ll play an integral role in improving our customers’ quality of life and redefining what’s possible in pharmacy. 

Be Part of a Great Place to Work — Join Our Team

Hear first-hand from members of the Alto technology teams.

Read the latest Alto blog posts covering healthcare, family health, managing diseases, and other Alto highlights.

Blog | Alto Pharmacy

Redux is great for managing client state: it allows data to be synchronized across the whole app, reduces prop drilling, and makes it easier to understand when state changes occur.

Server state is a different story. Sure, it’s nice to have data available everywhere in our app, which is why we tend to fetch data from the server and dump it into our Redux store. But because our client application doesn't own that server data, it’s hard to know when it has become stale. And what if multiple components trigger calls to fetch the same data? How do you dedupe requests? How do you decide when to refetch?

Why We Switched to React Query for Managing Server State

Redux is great for managing client state: it allows data to be synchronized across the whole app, reduces prop drilling, and makes it easier to understand when state changes occur.

Server state is a different story. Sure, it’s nice to have data available everywhere in our app, which is why we tend to fetch data from the server and dump it into our Redux store. But because our client application doesn't own that server data, it’s hard to know when it has become stale. And what if multiple components trigger calls to fetch the same data? How do you dedupe requests? How do you decide when to refetch?

Below, we’ll dive into how we’re using React Query at Alto to solve these problems and achieve separation of client and server state.

The Old Way: Fetch Server Data and Store It in State

Historically, we would fetch data and set the response in our React component state.

fetchUser(params) {
 this.setState({ isLoading: true });
 
 return getUser(params)
   .then(({ data }) => {
     this.setState({
       isLoading: false,
       user: data.user,
     });
   })
   .catch(() => {
     toaster.toast({ title: 'Something went wrong...' });
     this.setState({ isLoading: false });
   });
}

To make that data accessible from other components, we’d set that data in the global Redux store. This works fine, except that you have to manage your own loading state, the logic for when and when not to make the request, updating the store when mutating data on the server, and knowing when data is stale.

The React Query Way: Fetch and Cache Server Data

With React Query, fetching data and using that data is more declarative.

const { data, error, isLoading } = useQuery(
 // This is our Query Key, which is unique per query so that React
 // Query can manage the cache for us.
 ['user', { userId }],
 
 // This is our Query Function that actually does the data fetching.
 // We can use anything we want: built-in fetch, axios, etc.
 fetchUser({ userId }),
 
 // This query will not execute until the userId exists.
 { enabled: !!userId },
);

Here, we define our query for a user with a Query Key and a Query Function. The Query Key is unique per query, so React Query can manage our cache for us, and if any part of the Query Key changes, the query will automatically fetch new data. In this example, if the user ID changes, the query will fetch new data for that specific user and store that data in the cache.

Naming is hard, but caching is harder. React Query gives us caching of server data out of the box with cache invalidation and request deduping. If we use this same query with the same Query Key in another component, it'll check the cache: if the data is already there, React Query will simply return it, avoiding extra network requests!

Additionally, useQuery runs wherever you use it — no more fetching data when a component mounts. You just use the query and display the data when it’s ready. The cherry on top? The query returns status updates by default, so we don’t need to store isLoading in component state all the time.

Three Optimizations We Get With React Query

When we rebuilt Alto Connect, our web app for healthcare providers, we found impactful benefits in using React Query to help us manage server state instead of using Redux. Alto Connect provides instant access to patient and prescription information and enables messaging with the pharmacy. Because healthcare providers need to quickly switch between patients and clinics, caching data is important to provide a delightfully fast experience. Apart from the cache, what other optimizations do we get by using React Query? Below are a few that helped us.

Once we’ve queried for some data, it gets entered into our cache, so we can utilize that cached data when using the same query in a different component or when fetching new data for a stale query in the background.

React Query is configured with aggressive but sane defaults and considers cached data as stale. Stale queries will always query for new data when the following criteria are met:

The query is optionally configured with a refetch interval

The fetching for new data when the browser window or tab is refocused is especially useful because the healthcare providers using Alto Connect often switch between multiple browser tabs. If there’s an update to a task, status, or message while they’re checking a patient’s medical chart, it will be immediately reflected in the UI when they come back to Alto Connect.

We also set a refetch interval for our queries involving messages. In the old app, we used WebSockets to enable real-time messaging. Although it’s possible to integrate WebSockets with React Query, we opted for a polling solution instead. Because our app isn’t a real-time messaging application, and we don’t need messages to show up immediately, we set a refetch interval on our messages query to poll for new messages every 30 seconds.

const { data, error, isLoading } = useQuery(
 ['user', { userId }, 'messages'],
 fetchMessages({ userId }),
 {
   // This query is disabled from automatically running if we don’t have a userId
   enabled: !!userId,
 
   // Refetch every 30 seconds
   refetchInterval: 30000,
 
   // Continue to refetch while the tab/window is in the background
   refetchIntervalInBackground: true,
 },
);

This is where the enabled option, which disables a query when it evaluates to false, is particularly powerful. We can pause polling for messages if we route to another page and we know that we won’t make any unnecessary requests if there isn’t a valid userId. We don’t have to worry about closing any socket connections or manually clearing an interval.

Most web apps have some sort of navigation menu, and if you already measure or understand your user behavior, you can prefetch data before users even navigate to a different page. Custom hooks makes it easy to expose a function to prefetch data:

export const usePrefetchUsers = () => {
 const queryClient = useQueryClient();
 const prefetchUsers = async () => {
   await queryClient.prefetchQuery(
     ['users'],
     fetchUsers,
   );
 };
 
 return { prefetchUsers };
};

We prefetch queries when users mouseover or hover over a navigation item so that when they navigate to the next page, the data is either actively being fetched or is already in the cache waiting.

We’ve been talking a lot about queries, but what about mutating data? Mutations, in contrast to queries, are imperative. While queries run automatically when we use them, we need to fire a mutation method in order to run mutations. And in order for our user experience to feel snappy, we can optimistically update the cache and then invalidate it after successfully executing the mutation. useMutation gives us some nice side-effect handlers to deal with every stage of the mutation lifecycle. Now when the user sends a message, that new message is immediately reflected in the UI and we keep the data in sync by invalidating the cache and refetching the messages.

With Redux, we needed to build out solutions to each challenge of managing server state. React Query, on the other hand, gives us those tools right out of the box. We no longer need to worry about when to invalidate or fetch new data, making the user experience more predictable — and the developer experience more enjoyable.

Last summer, Moringa Fellows Irma Figueroa and Adrian Mendez joined Alto as the company’s second batch of engineering interns. Here, they share their learnings, accomplishments, and advice for future Alto interns.

Two Engineering Interns Weigh In: What to Expect From Your Alto Internship

The Moringa tree flourishes almost anywhere it’s planted. It’s hardy, fast-growing, and resistant to drought — an apt metaphor for the two interns who joined us from the Moringa Fellowship program last summer.

“Before Alto, I had never actually contributed to a large codebase,” admits Irma Figueroa, an intern on the Supply Chain Team. By the end of their three-month internships, however, Irma and fellow intern Adrian Mendez had shipped impactful features for Alto’s internal pharmacy management system and customer-facing app that are still in production today.

Below, we chat with Irma and Adrian to learn what it was like interning at Alto, which projects they worked on, and what advice they’d give to future interns. 

Adrian, who interned on the Patients Team before returning to join as a full-time software engineer last fall, was surprised by the freedom he had to choose what he worked on during his internship: “Early on, my manager gave me a rundown of the projects the team was juggling and the option to choose which one interested me the most,” he says. “I was able to decide the kind of experience I was going to have.” 

Adrian adds that meetings with his mentor evolved over the course of his internship, as he became more familiar with the codebase: “From the start, we had weekly meetings to talk about whatever was on my mind. Those meetings started off as general advice and questions and turned into more time to pair or run things by each other.”

Because both Adrian and his mentor were working on the same, long-term project, some of those meetings were also dedicated to project work. This included “sitting with [my mentor] while creating the project outline, pairing on larger tasks, and syncing up on progress.” 

Which projects did you work on during your internship? 

On the Supply Chain Team, Irma was tasked with a unique inventory challenge: Technicians at our production pharmacy locations use handheld scanners to scan the barcodes on newly-received inventory. Our technicians often need to scan hundreds of inventory items at a time — making it all too easy to lose track of which items have been successfully scanned onto which shelves. 

“I created a reusable counter component, displayed in the header of the scanner app, that would represent the number of items scanned into a shelf.” Irma says. The best part? Getting to test the handheld scanner herself. “I really enjoyed using the scanner for testing purposes. It was rewarding to see my code in action and definitely super fun!” 

On the Patients Team, Adrian worked on a project to add a homescreen to the customer-facing native app. Previously, the app dropped users into a list of all their active medications, which made it difficult to know where to look or what to do next. The new homescreen allows users to schedule the prescriptions most relevant to them and to view or edit any upcoming deliveries. 

“The first step was to create new additions to the design system — essentially taking the designs and extracting system-wide components. After that, the fun part was taking all those components and putting them together like building blocks!” Adrian says.

It was especially rewarding, he adds, to have been part of the project from start to finish. “I joined right as the design phase started, and I was there mostly every step of the way. For how short my time here was, having helped create this landmark project is really cool to think about.” 

What advice would you give to future Alto interns?

Irma and Adrian agree: Don’t be afraid to ask for help. “You'll often feel challenged and possibly intimidated, but be patient with yourself and remember you have an entire team of experienced engineers ready to help,” Irma says. 

“Everyone at Alto is more than willing to help you out, so don't be afraid to ask questions! You're on the same ship as everyone else, so your success equals everyone’s success,” Adrian adds. “Just remember that your work got you here, and they wouldn't have invited you if they thought you couldn't handle it. They made an investment in your ability to grow, so stay curious.” 

Interested in interning at Alto? Check out our careers page here.

With more engineers contributing to Alto’s codebase than ever before, we needed to establish better boundaries: which team owns what? Here’s how we developed a tagging-based solution to code ownership and drove and measured adoption.

How We Established Better Boundaries and Clearer Code Ownership at Alto

Some problems are nice to have. Over the course of 2020 the engineering team at Alto saw explosive growth, expanding from about fifteen to more than seventy individual contributors. Growth at this pace required us to rethink our team structures and priorities more than once. This growth felt amazing: the new capacity enabled us to make progress on our backlogs like never before.

But still, there were challenges. We saw a steady rise in merge conflicts as more people contributed to the same code base. Build times increased as more and more code was contributed. A big influx of new perspectives and talent also meant new gaps in historical context and responsibility. New teams took big bites out of their new priorities — great! But what about the old ones? Who was going to fix that bug? Who needs to review this PR? Who is accountable for this performance problem?

Who, in short, owned what? By the end of 2020, it was clear to us that a good, practical, working answer to this question would be the critical first step to continuing smooth, rapid, distributed iteration on our product.

Up to this point, the large majority of product development at Alto had happened in the context of a Ruby on Rails monolith, which — over a half decade of intense development — had grown pretty large. We’d stuck more or less to the default Rails convention of organizing code by function rather than feature, which meant that our code was distributed across a few mostly flat and very broad subdirectories such as '/models' and '/controllers', making it difficult to learn much about the structure and ownership of our business domains by merely looking at how the code was laid out. We knew that this was not going to work for us over the long term.

Our long term vision started to coalesce around a notion of “better boundaries.” First, we would modularize the monolith along the lines suggested by Shopify’s packwerk utility. Then, we would carve some of those modules out into separate engines that could be built and tested separately. Finally, we’d deploy and operate some of those engines independently as their own services. Clear ownership was fundamental to this plan: on either side of an effective boundary, you need to know not just what it is but also who owns it. We lacked that clarity.

The RFC (Request for Comments) we ultimately adopted opened with four questions:

How might we produce a domain model and do something useful at the same time?

How will we avoid imposing (the wrong) prescriptive model from the top down?

How will we keep it up to date and accurate?

We considered jumping straight into moving files around into a more modular structure. Ultimately we decided against this approach as a first step for a number of reasons. The benefit of moving files around is mostly abstract — it’s not immediately clear what to do with a better module structure. It’s relatively expensive to move files around in a Rails app — doing so typically requires you to rename the contents (and update all its callers), which amounts to a lot of busy work, and it’s extra expensive to be wrong. And the in-between state is rough: which example should I follow in deciding where to put my new file? First you are obligated to create a map of the future state, and then to keep that map in sync with reality as you go.

Instead, we opted for a lightweight solution that would allow us to map our domains and their owners in the code itself — helping to ensure it’s up to date and accurate — without having to restructure it at the same time. We’d do this by tagging code in code comments, and then building tools to leverage these tags in various ways to increase visibility and enforce ownership. This has the nice effect of producing a new domain map incrementally from the bottom up, rather than prescriptively from the top down.

We called this the '@owners' tag, and it looks like this:

# @owners { team: platform, domain: orders }
class Wunderbar::OrdersController < Wunderbar::WunderbarController
  def show
		...
  end

  # @owners { team: inventory }
  def out_of_stock
    ...
  end
end

We based our tooling around extending the YARD documentation tool to recognize this as a custom tag. You can teach YARD to recognize a custom tag by passing a command line option like '--tag owners:"Owners"', which by default will cause the tag value to appear in generated documentation under the header “Owners”. The Ruby API for declaring a tag programmatically is shown in the snippet below as well.

We wanted the tag to be interpreted hierarchically to handle cases where ownership has diverged within the same class or module, such that we might want to tag one or two methods differently from the rest, as well as to ease the burden when all of the classes / modules / methods under a namespace belong to the same team and domain, since you can just tag the shared parent. We set an initial goal around getting 100% of our methods covered by at least a “team” value, at which point we would require all code to be covered in our build process.

To get a basic feel for how to work with these tags, make sure the 'yard' gem is available, tag a couple of your files, and then try running the code below in your project. It will print out the tag associated with each tagged code block.

require 'yard'
# Define the custom tag.
YARD::Tags::Library.define_tag('Owners', :owners)

# Parse your codebase. You might want to pass a more specific path here.
YARD.parse('.')

# Load all the method, class and module code blocks into memory
code_blocks = YARD::Registry.all(:method, :class, :module)

# Print the tag associated with each tagged block.
code_blocks.each do |code_block|
  tag = code_block.tag('owners')
  # automatically tag nested classes, modules and methods.
  while tag.nil? && code_block.namespace != nil && [:class, :module].include?(code_block.namespace.type)
    code_block = code_block.namespace
    tag = code_block.tag('owners')
  end
  print "Code Block: #{code_block}, tag: #{tag.text}, line: #{tag.line}" if tag
end

We ultimately developed a handful of CLI tools based on this technique that allow us to produce reports measuring the percentage of code covered by team and domain at different levels, among other things:

bin/yardowners --help
Usage: bin/yardowners [options]
        --methods                    view by methods
        --files                      view by files
        --teams                      teams report
        --domains                    domains repot
        --team=TEAM                  filter by team
        --domain=DOMAIN              filter by domain
        --codeowners                 update codeowners
        --lint                       lint owners tags
        --require-domains            lint for presence of domain in  
owners tags
    -h, --help                       display this help and exit

We use this CLI to add checks to our CI process around coverage / correctness, as well as to report progress toward our goals of 100% coverage.

To get the tagging process started across the organization, we dumped all of our class and module names (thanks again, YARD!) into a big spreadsheet, and then asked each team to get together and spend a bit of time putting their names next to the ones they owned. We adopted the attitude that it was better to have the wrong value than no value, since it would be easy to adjust later.

We covered a good 80% of the code using this spreadsheet by the time each team had taken an initial pass. From there, we filled the blanks in with “TODO”, and then wrote a little script to automatically insert the tags into the code (again based on the YARD metadata, which conveniently gives you the line numbers for the top of each class / module declaration).

At that point, we enabled a linter requiring 100% team owner coverage on all new PRs. We set a new goal to eliminate TODOs, and chipped away at it over the course of a month or so. We then repeated this process for the “domain” values, driving that to 100% as well. We found that the spreadsheet-based divide-and-conquer approach allowed us to cover a lot of ground very quickly.

Once we were able to parse our '@owners' tags, the first and most important thing we did with them was to automatically generate our CODEOWNERS file. We enforce that each tagged team maps to a team name in github and check that CODEOWNERS is up to date during the build process. This new process introduced some friction, as teams started getting tagged every time a file they were tagged on changed. This turned out to be great, because it sparked a number of very productive negotiations around ownership and scope that got hashed out in code review. We keep our CODEOWNERS file synced with Sentry, too, so that exceptions raised by files you tagged as yours get assigned to your team automatically. We also track and report on the performance of individual endpoints by team according to how they are tagged.

The new owners tag hasn’t fixed all our problems by any means, but it has proved fundamental to our ability to operate as a growing set of increasingly independent teams with well specified responsibilities. In subsequent posts, we’ll talk more about how we’re using our new domain map and a suite of other cool tools to build better boundaries within and ultimately escape from our monolith.

Interested in learning more about engineering at Alto? Follow us on LinkedIn.

To automate a manual, error-prone pharmacy workflow, we turned to event streaming. Here’s what we learned from one of the first Kafka use cases at Alto.

How We Used Kafka to Build an Event-Driven Automation Pipeline and Scale Our Pharmacy

Imagine this: You're at the doctor’s office, and your doctor just wrote you a new prescription. Minutes later, before you even check out with the receptionist, you already have a notification from Alto Pharmacy on your phone that your prescription is ready. All you have to do is pick the date and time that works best for your free courier delivery.

This is the future we’re building at Alto: a future where a prescription can be transcribed, billed, and verified by a pharmacist in five minutes, flat. To get there without dramatically increasing Alto’s head count, we need to automate much of the process, starting with the first step: transcribing, or intaking, a prescription.

As a pharmacy, we receive prescriptions in many different forms. Some are sent electronically, in a structured format, but others are handwritten, phoned in, or even faxed. To represent all of these forms in a single table, we previously required every incoming prescription to be transcribed by an Alto team member, who would answer questions like:

Whom is this prescription for? Is this customer already in our system?

Where did this prescription come from? Who wrote it?

What is the relevant prescription information (medication to dispense, quantity, days supply, directions, etc.)?

This manual process is both time-consuming and prone to human error, which, if uncaught, could impact our ability to process the prescription downstream. Furthermore, at our busiest times, it could take up to two hours for a team member to get through the pending intake queue — unacceptable if we hope to have prescriptions ready to schedule before people even leave their doctor’s office. 

Automating intake required close collaboration between three teams:

Data Science, who built the decision-making algorithms 

Data Engineering, who served up the results of those algorithms

Product Engineering, who used those results to automatically intake prescriptions 

To allow the teams to operate in parallel without toe-stepping — and in the language most suitable for each (the Data Science and Data teams prefer Python, while product engineers work mainly in Ruby) — we separated the code into two environments: 

1. Data Prod Interface (DPI), where the code for generating automation results lives

2. Wunderbar (our proprietary pharmacy operations platform), where the code for requesting and using those automation results lives

The key question: to establish robust communication between these two environments, should we use an API-based approach, or an event-driven one? Each has its pros and cons. 

Legacy approach everyone at Alto is familiar with

Existing enforcement of strong types and data

Difficult to scale with increased data & requests

Requires ongoing authorization / authentication

Infrastructure relevant to other business areas

Automatic retries, reprocessing, and queueing

Our existing infrastructure currently utilizes a hefty API-based structure, so there was a natural tendency to go down this path, but we knew that beginning to migrate services towards an event-based architecture with Kafka at its heart would help us in the long term.

Kafka is a framework implementation developed by LinkedIn, now open-source with the Apache Software Foundation. It aims to provide a unified, high-throughput, and low-latency platform for handling real-time data feeds. 

Alto uses Amazon Web Services as our cloud provider, and luckily, they offer an out-of-the-box managed solution for Kafka, MSK. To configure Kafka with our system, our infrastructure team worked with AWS to set up managed instances to comply with our business requirements. Our Infrastructure team will share more details about this work in an upcoming blog post. 

Kafka offers many benefits. The asynchronous nature of the message system allows for retries, reprocessing, and queueing as well as easy recovery from outages. In addition, the infrastructure work done for setting up the managed cluster would be a one-off workstream. When complete, we’d be able to re-use the same setup and configurations across various business needs to streamline other projects that leverage Kafka. 

Setting up for success: defining a strict contract

One element we had to get right from the start was the data contract. As we send messages between DPI and Wunderbar, we need to ensure messages are produced with the correct schema, so that downstream consumers know what to expect.

Protocol Buffers, or Protos, is an open-source, cross-platform data format used to serialize structured data. Developed by Google, Protos allow you to define how you structure your data once and then use generated code to easily write and read the structured data in a variety of different languages. 

At Alto, we’ve been using Protos for the past few years to speed up development with auto-generated code and ensure data continuity between our Ruby on Rails backend and JavaScript / React / React Native frontend. Protos may seem antithetical to JavaScript and Ruby since they’re both dynamically typed languages, but in practice, they have saved us countless hours of debugging and back-and-forth Slack messages. 

For this project, we turned again to Protos to define the schema for messages produced by DPI and Wunderbar. We defined IntakeAutomationRequest and IntakeAutomationResult messages in Go and generated the types (Sorbet, Flow, etc.), which both DPI and Wunderbar now refer to when producing or consuming messages.

Because Wunderbar is already configured to run on AWS and houses our existing intake logic, it was easy enough to send a prescription to DPI for automation. Where we would normally create a task for an operations team member to manually transcribe the prescription, we instead produce a message to our “request automation” Kafka topic using the ruby-kafka library.

Our consumer, instantiated in an executable that runs on a dedicated worker, consumes messages from a “receive automation” Kafka topic. First, the consumer verifies that a given message corresponds to a prescription we previously sent to DPI. Then, it persists any automation results included in that message. If the automation results are sufficient, we skip intake altogether. Otherwise, we create an intake task to be handled by an operations team member.

Because this project was among our early use cases for Kafka, we wanted to be prepared for the worst-case scenario: we produce a message requesting automation results, and for whatever reason—maybe DPI is down—we don’t hear back. For us, we defined a reasonable time frame as fifteen minutes, in order to avoid building up a large backlog of work that would have to be offloaded back onto our operations team.
To this end, we store metadata about the automation lifecycle in an intake_automation_results table, which includes timestamps representing when we sent a message to DPI and when we received a response back. A cron job runs every minute to check that table for rows where received_at is blank, and sent_at was more than fifteen minutes ago. If this occurs, we flag the prescription to be manually intaken by an operations team member.

The diagram below illustrates our intake automation architecture.

Intake automation is now live at Alto. We’re seeing prescriptions fly through in just a few minutes — during which time our automation creates a user profile if one doesn’t already exist, links the new prescription to that profile, and finds and bills the user’s insurance; a pharmacist reviews the prescription for clinical accuracy; and the user receives a text message to schedule their delivery. This single flow, prior to intake automation, once took up to four hours to complete!

Going from zero to one is quite a feat, and it wouldn’t have been possible without an event-based approach. Now that intake automation has laid the groundwork, we know that Kafka will only continue to provide exponential value to Alto as we grow and migrate more of our services to this event-based architecture. 

Each type of user has a different product or set of products we’ve built for them. From within each product, we collect, transmit, process, and store a tremendous amount of sensitive information. However, our biggest area of security, privacy, and compliance risk is our internal pharmacy management tool called Wunderbar that is used by our operations (ops) users every day to process prescriptions. Securing access to Wunderbar is mission-critical to protect our data and our business, and, of course, we need our super-secure implementation to be super-smooth to avoid slowing down our hard-working ops users.

Developing Robust Authentication Using Rails, Omniauth, and Okta

Alto Pharmacy aims to deliver a better pharmacy experience for everyone who needs medication. To achieve that, we need to enable three different types of users:

Individual customers who need prescriptions

Healthcare providers: the nurses, doctors, physicians assistants, and other medical professionals who prescribe medications

Operations users: the Altoids who intake prescriptions, bill insurance, fill prescriptions, and more

We wanted to be able to change our Wunderbar session lifetimes without hampering either our security efforts or the usability of our product. Moreover, we wanted session lifetimes of other apps to be able to change appropriately with business risk. Before starting this project, we’d implemented a Google OAuth 2.0 authorization flow that allowed ops users to log into Wunderbar via Google and take advantage of transparent session refresh for a limited amount of time.

However, while Google Workspaces was (and is) great for our email, documents, and calendars, it isn’t as configurable or extensible as a proper SSO solution. One painful side effect of this limitation was that, in order to ensure our refresh sessions were limited to twelve hours in Wunderbar, any user of the Alto Gmail drive also had to log back in every twelve hours just to see email — even though there’s not nearly the same level of sensitive data in email.

Plus, our IT team had already picked the tool of the future: Okta. And the goal was to enable SSO, or Single Sign On: if we still used Google for SSO, it wouldn’t really be SSO!

It was clear we needed to move to Okta for authenticating in Wunderbar, too.

Okta is a big company with many different authentication mechanisms. For us, the big question was really: do we want to authenticate via SAML or Open ID Connect (OIDC)? There are a lot of great resources around comparing these two standards and what they’re good at; here’s a good chart comparing and contrasting parts of these two authentication flows:

From this Medium post, which is a great primer on the differences

For us, the most attractive thing about SAML is that it can be used to provision user accounts from your upstream IDP; for example, Okta could be configured to push username, email, group, role, etc., information to our platforms with which we could automatically provision user accounts.

In the end, we decided the OIDC implementation would provide for a very similar user experience to that which our operations users were already accustomed to, with the same capability to perform transparent token refresh.

This transition was pretty easy because we were moving from Google OAuth 2.0 flow to Okta’s OIDC flow. OAuth 2.0 and OIDC are very similar; just know that OAuth 2.0 is technically an authorization framework (once you have a user, what can they access) vs an authentication framework (actually identifying and validating the user). However, there were some odd things about our previous implementation, and I only had one person to blame for those choices: myself. It’s great to get the opportunity to re-implement our authentication system — in a fast moving startup, you rarely get the chance to revisit problems so soon!

The biggest issue at hand with our previous Google flow was the way in which I’d chosen to re-authenticate users whose Wunderbar sessions had expired. In this case, OAuth provides a handy mechanism: a refresh token. Here’s how access tokens and refresh tokens are exchanged from the client and the authentication server in OAuth 2.0 flows:

At the time, we had session lifetimes of 60 minutes when accessing Wunderbar. These cookies also lasted 60 minutes. We faced a problem: if somebody walked away from their laptop for 61 minutes after authenticating, when they came back, how could we make it so they didn’t have to go back through the Google OAuth 2.0-initiated login flow from the beginning?

My solution at the time persisted a secure cookie with the refresh token value client-side with a longer expiration length. Then, if the user didn’t have a valid access token, we’d simply present the refresh token back to the server and get a new access token all over again. This worked well, but exposing secrets to the client always presents a risk of compromise. Refresh tokens are longer-lived, so they're of even higher value to protect!

Okta’s documentation made it very easy to stand up a simple authentication server and set up a local development server. Okta has a great blog post guide with tools we were already using (Rails and Omniauth, specifically) to get up and running. Beyond that, our biggest challenge was ensuring that we had a smooth rollout.

There was an issue in how we’d test and use this flow in our staging environments, however. At the time we were working on this (early 2020), there were two issues:

Our sandbox environment URLs were non-deterministic and could be varied depending upon how an engineer user named their cluster.

Okta apps did not support wildcard redirects in their oauth redirect URLs. This controls where, upon hitting Okta’s servers, they should redirect your payload to handle the meat of authentication.

For a time, our clever infrastructure engineering team managed around these issues with the use of an intermediary keycloak server. However, Okta now supports wildcard redirects that we’re successfully using today:

The biggest functional change to our authentication flow was in our refresh flow. The biggest challenge in landing this change was in how we deployed it to end users in production.

We didn’t want to expose the refresh token because it presented too much risk. What do we do instead? Something a little bit risky, but that we preferred after threat modeling our scenarios. Security engineering is often about making tradeoffs given a number of constraints. Today, our session lifetimes are down to fifteen minutes, and our refresh lifetime for an active session (e.g. somebody making at least one request within this window) is 60 minutes. If somebody comes back after their access token expires, that’s fine; we’ve persisted the encrypted refresh token to the database, and we can use attributes from the expired but still cryptographically valid token in order to look that refresh token up.

This was crucial for Okta especially because Okta refresh tokens are valid for 100 days by default!

Disruptive change is hard. We didn’t want to disrupt the workday of our hard-working ops users who are caring for our many customers and provider partners. The biggest intent here was to keep a safety blanket fallback to the old authentication flow until we were quite confident in its functionality. This broke down into a multi-phase approach:

1. Dark-launch the new Okta flow hidden behind a URL and with a small allow-list of users (mostly just the Security team) assigned to the new application in Okta.

2. Ask for volunteers! These volunteers would go about their normal work day in a variety of functional areas (customer care, pharmacy, fulfillment, engineering) using Okta instead of Google.

3. Change the default login flow to use Okta, but keep Google working just fine as a secondary action hidden behind a URL.

This allowed us to always ensure that we didn’t disrupt peoples’ days too much!

We achieved our goals and now, nearly two years later, we’re successfully using Okta’s OIDC flow within our core pharmacy management app. What did we learn in the process?

Not every API is necessarily fully implemented in every client. At the time of implementation, Okta’s Ruby client wasn’t fully supported. In order to support our refresh flow, we needed to upstream a small change into the client library, which required a little investigation.

Exposing secrets involves risk! In our case there wasn’t a glaring need to expose refresh tokens client-side at all.

Slow rollouts are great, especially with disruptive changes! A big thank-you to our team members who were willing to be part of the experiment group last year when this rolled out.

Interested in learning more about engineering at Alto? Follow us on LinkedIn and check out our open positions.

Last year, we set out to tackle this technical debt by splitting the User model into separate concerns and refactoring the entire system into this new paradigm.

Tackling Technical Debt and Improving Security: How Alto Untangled an Overloaded User Model

At Alto, we have three types of users: customers, healthcare providers (nurses, doctors, etc), and operations users who run the pharmacy. For each type of user, we maintain a separate application (though they share a monolithic backend for now): our customer app; our Alto Connect app for providers; and our internal pharmacy management tool called Wunderbar, which powers the pharmacy experience from prescription intake to fulfillment.

Early on, we made the pragmatic choice to represent both customers and operations users with a single User model, distinguishing them with a “roles” attribute. This required repetitive checks that, over time, slowed down our engineering teams, presented substantial privacy risk, and posed difficult scaling challenges as we evolved our monolith into different microservices. 

The challenges of relying on a single User model for different types of users

The notion of a User is core to our concept of identity from which authentication and authorization proceeds. From a security perspective, it’s difficult to build an extensible and clean authentication and authorization system when you’re overloading one model for two very different populations of users. From a privacy perspective, this muddied customer records (which are subject to more restrictions because they’re healthcare records) with workforce records, hamstringing many analytics efforts.

Authentication: Alto employees who were also Alto customers were required to use their work emails for their customer accounts. Not only was this restriction frustrating for Alto employees, but it also posed a security risk: employees should only be able to access Wunderbar via corporate SSO, not their personal account passwords. 

Authorization: Engineering new authorization capabilities required intricate knowledge of the idiosyncrasies of our user structure. This both slowed down development and created tremendous risk that any mistake could easily escalate user permissions.

Privacy: Because they existed in the same table, both customer records and Alto operations user records were classified as Protected Health Information, or PHI, which severely limited what we were able to do with them.

Attribution: Our robust audit logs were significantly less useful when a user ID in the transaction log might refer to either an ops user or a customer.

State management: Some updates pertaining to admin identity lived on the customer model, some lived on the employee model, and others lived in other various models.

Architecture: Tight coupling would have prevented us from decoupling our monolithic system into smaller services.

Our backend is primarily a monorepo Rails application that shares a tremendous amount of functionality between our customer, provider, and ops user-specific domains. Generally we use JWTs to secure customer and ops user sessions. These JWTs are issued only once valid credentials have been present.

Ops users are required to log in via Okta SSO, enforcing both valid username/password as well as a two-factor authentication (2FA) prompt. We use Devise and Omniauth to wire up the OIDC flow with our upstream identity provider, and we use CanCan to expand a set of roles per-user into a set of capabilities that restrict what resources they can access/modify.

Ops users primarily work in our internal pharmacy management app. However, the state of these users was shared between several different tables: Users, which otherwise refer to customers, and Employees, which share some ops user-specific state tied to Users.

Large change sets of code being deployed at once are harder to code review, harder to unwind if something goes wrong, and are more likely to be more disruptive. For this project, we knew that such a large architectural change had to be broken up into many more-digestible pieces if we were to be successful. The plan:

1. Create a new dedicated model for our ops users

# == Schema Information
#
# Table name: wunderbar_users
#
#  id                        :bigint(8)        not null, primary key
#  user_id                   :bigint(8)        indexed
#  email                     :string           not null, indexed
#  first_name                :string           not null
#  last_name                 :string           not null
#  token_last_invalidated_at :datetime
#  roles_mask                :integer          default(0), not null
#  title                     :string           not null
#  team                      :string           not null, indexed
#  is_active                 :boolean          default(TRUE), not null
#  slack_user_id             :string
#  pod                       :string

This was the easy part: create the database structure for a properly isolated admin user type, which we called a WunderbarUser.

We now had our admin user state split between Employees and Users, with a new table that engineers and analysts wanted to use that was not yet populated, and we had to make progress while allowing new hires and new customers alike to onboard to our platform.

2. Synchronize state between the old tables and new table

Next, we created a WunderbarUser for any User who was, or ever had been, a Wunderbar user. This required some historical digging —we had a lot of people who were onboarded, and even some offboarded, before we even had the intermediate Employee record!

Two major choices at this stage really helped the project succeed. First, we ensured that the WunderbarUser records all had IDs matching the corresponding User records. This is because a lot of our legacy audit records (that we’re required to preserve and keep useful as a Covered Entity under HIPAA) referenced an `admin_user_id` column populated with the ID of the User. If we just created all new entries for WunderbarUsers, these old logs would require at least one level of indirection (e.g. querying back to User) to make useful.

     # Create with the same ID as the user that we're creating this from.
     wunderbar_user_attributes = {
       id: user.id,
       user_id: user.id,
       email: user.email,
       roles_mask: user.roles_mask,
       date_of_birth: user.date_of_birth,
       token_last_invalidated_at: user.token_last_invalidated_at,
       first_name: user.first_name,
       last_name: user.last_name,
       preferred_first_name: user.preferred_first_name,
       is_active: false,
     }

Second, we created a new WunderbarUser any time we created a new Employee record (e.g. when we onboarded a new employee) and synchronized any changes between Users, Employees, and WunderbarUsers tables. As a side note: we knew we’d have to protect this syncing logic to ensure that we only copied each change once — otherwise, our callbacks would fire forever!

3. Migrate functionality slowly to use the new ops user model

We migrated logic to reference WunderbarUsers where previously we’d referenced Users and/or Employees. Authentication, authorization, audit logs, and pharmacist licenses were all updated here — albeit painstakingly!

We then migrated all of the foreign keys our various Postgres database tables held to either the Users or Employees tables to instead reference the WunderbarUsers. This ended up being around a hundred foreign keys, but a constraint-only migration that fortunately didn’t involve any data changes.

In short: our goal was to do this change slowly and steadily enough that nobody would notice until, one day, we could announce to the team, “Hey, we did a big security-pertinent project, and now you can change your customer email to be anything.”

Authentication: While we use the same JWT format for each, our customer and operations user authentication systems are now totally separated.

Authorization: With separate user types, we have separate capabilities and domain-specific authorization checks for each area, enforced in code.

Experience: Onboarding as a new Altoid (somebody who works here; come join us!) is totally separate from the customer experience, and you no longer need to link your accounts together.

Privacy: Our ops users who choose to be customers enjoy the full privacy afforded to our customers without tying their account to their work identity in any way.

Attribution: Legacy audit logs are still index-able and query-able with the same IDs, just in a slightly different table.

State management: Customer state lives on the User table, and ops user state lives on the new WunderbarUsers table. Our callbacks on each have been dramatically pared down, and our intermediary Employee table is entirely gone, along with a number of other now-ancillary tables.

Architecture: We have a clear path to federate RBAC checks without shipping our entire customer database between services.

Bonus: Performance and code readability generally improved by removing complex joins and join tables and isolating concerns between the customer and ops user domains.

Another Bonus: Significant refactorings and increased test coverage along the way while we cleaned up some very old, very stale code.

Large architectural changes of legacy systems with heavy use are rarely 100% smooth. Here are a few learnings to keep in mind the next time you tackle a hefty piece of technical debt. 

1. Always validate your assumptions (e.g. the data your constraints are against) when foreign keys are concerned. Multiple times we assumed, “Hey, we’ve migrated a bunch of really big tables that are really old (insurance claims, prescriptions, and more), so there’s no way we could have another admin user that we haven’t created a commensurate WunderbarUser for, right?” Wrong! Your data can always have inconsistencies, but a simple SQL query (e.g. validate that all values of the column you’re migrating the foreign key to have a matching WunderbarUser) can save a lot of time and headaches.

2. When migrating foreign key constraints on tables with active reads in Postgres, ensure you break it into two steps. Migrate your foreign key with `{ validate: false }`

 def up
   remove_foreign_key :inventory_orders, to_table: :users, column: :created_by_id
   add_foreign_key :inventory_orders, :wunderbar_users, column: :created_by_id, validate: false
 
   validate_constraint
 end

Validate the named constraint of your foreign key separately

Blog

Why We Switched to React Query for Managing Server State

Two Engineering Interns Weigh In: What to Expect From Your Alto Internship

How We Established Better Boundaries and Clearer Code Ownership at Alto

How We Used Kafka to Build an Event-Driven Automation Pipeline and Scale Our Pharmacy

Developing Robust Authentication Using Rails, Omniauth, and Okta

Tackling Technical Debt and Improving Security: How Alto Untangled an Overloaded User Model

Considerations and Approaches for Implementing Biometric Login: Part 2

Considerations and Approaches for Implementing Biometric Login: Part 1

How Alto Reduced Time to Resolution for Production Incidents

Celebrating American Pharmacist Month At Alto

Celebrating National Coming Out Day at Alto

Introducing Christopher Tidball, Director of Product Management - Supply Chain & Fulfillment

Switch to a better pharmacy.