Shop Alto Essentials to add pharmacy products to any prescription delivery!

Your partner in navigating the ins and outs of prescriptions — at no extra cost.

Alto Pharmacy Technician

Alto Pharmacist

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies based on team member feedback and a rigorous assessment of our company’s culture and benefits.

Join us in delivering our industry-leading pharmacy care to millions of people across the country. As a member of the team, you’ll play an integral role in improving our customers’ quality of life and redefining what’s possible in pharmacy. 

Be Part of a Great Place to Work — Join Our Team

Hear first-hand from members of the Alto technology teams.

Read the latest Alto blog posts covering healthcare, family health, managing diseases, and other Alto highlights.

Blog | Alto Pharmacy

When Jake Moffatt says he loves building things, he means it — sometimes literally. A construction project manager turned engineer, Jake has spent the past decade building software at mission-driven organizations. Today, as Engineering Manager of Alto Assistant, he leads the team that’s bringing wizard-like support to Alto’s customer-facing app.

Get to know Jake Moffatt, Engineering Manager on the Alto Assistant Team

When Jake Moffatt says he loves building things, he means it — sometimes literally. A construction project manager turned engineer, Jake has spent the past decade building software at mission-driven organizations. Today, as Engineering Manager of Alto Assistant, he leads the team that’s bringing wizard-like support to Alto’s customer-facing app.
Below, Jake chats with fellow engineering manager Mike Riess and recruiter Braden Hanlon about the work ahead for Alto Assistant.

I’d love to learn more about your background to get us started. How did you end up in software engineering?

When I was twelve, I started learning to code in QBASIC because I wanted to make video games. Once I wrote my first program, I was hooked! I loved computers and the creative power they gave to anyone willing to invest the time and energy to learn. Since then, I’ve built games, personal tools and projects, desktop software, and a number of apps and services — I love building things. My career has gone back and forth between construction and software engineering, but it’s been firmly in the software industry for the last ten years or so.

My favorite place to be in a company is working on something new that has a big impact or addresses an important problem. In a previous role, I built an app that helped people find roommates in a city where housing is prohibitively expensive. At that same company, I led the team responsible for creating a new vertical that eventually split off into its own company.

The goal of Alto Assistant is to help customers help themselves before reaching out to support. Our first project is introducing a wizard-like support “assistant” (hence the team name!) that guides the customer through a series of questions to narrow down the area of support. We’ve already identified a few tasks the assistant can help customers complete on their own, including making changes to upcoming deliveries or scheduling new orders.

In cases where we can’t help them just yet, our new tool generates data to help us to learn exactly what customers need help with. It’s a top priority to bake analytics into every new feature we build; even though we have years of experience supporting our customers — and we know intuitively which problems they frequently experience — we don’t have conclusive data to draw upon just yet.

We plan to start with a couple of frequent use cases and measure how often customers reach out to us for support. Then, we’ll chip away at the problem until we’ve seen a substantial improvement in the user experience. From there, we’ll move on to the next biggest problem and repeat the cycle. There’s a reason “Chip away it” is an Alto value: It’s the playbook that we’ve used to automate other critical parts of the pharmacy value stream like intake, billing, and fulfillment.

What opportunities are you most excited about?

Support drives much of the cost of every shipment today, so this is a high-impact area for our business. A year from now, it would be a stellar outcome if the Assistant team could reduce the number of issues that customers need support on by half.

We’re also thinking about the user experience beyond the app. We plan to improve the tooling that our care specialists use to provide support as well. We’ve already discussed some promising ideas like creating the concept of a conversation in support communications rather than an infinite stream of messages and building out enriched communications. We also intend to develop a custom tool that can expedite a customer’s prescription when they need their medication ASAP.

The possibilities are many, and it will take a team of determined, passionate, empathetic people to solve these challenging problems. I’m super excited to see what the Assistant team will do together.

Any lessons learned from previous experience that you’re applying to your role at Alto?

There’s always the temptation to tackle everything at once with new business initiatives. My experience leading new verticals at Apartment List and Snapdocs taught me the importance of being laser-focused on one or two key metrics instead of trying to build out a fully-realized product from the beginning. That’s exactly what we’re doing on the Alto Assistant Team with our clear focus on reducing communications per shipment — how often customers reach out for help because they could not find what they needed in our app.

It’s also important to recognize that we won’t have all the answers right out of the gate! I like to think of product development as a three-step sequence: learning, growth, and maintenance. Alto Assistant is so young that we’re very much still in the learning phase. We maintain a tight iteration loop with everything we ship: launch, measure the impact, and use the results to iterate and refine our hypotheses about what our users want. Our final product could be anything from a customer support wizard to an AI chatbot to a ticketing system — we have to rely on our key metric of communications per shipment, experimentation, and user feedback to shape the direction.

What got you excited to join Alto? What would you pass on to an engineer thinking about applying to the company?

I was impressed by the Altoids I met during the interview process. I knew I would be working with people from whom I could learn a lot, like Elnaz, who started as an individual contributor and went on to become Director of Engineering and, now, Alto’s first Head of Strategy & Operations. I was also impressed by the quality of the interview process itself — a good sign that the company was well-organized internally. Finally, the pitch for the role was extremely clear, with already-established key metrics that would directly contribute to the company’s bottom line and help Alto become more profitable.

For engineers, I think Alto is a great place for both technical challenges and career growth. We’re still small enough that every individual gets exposure to a broad portfolio of high-impact projects. It’s also a great environment for learning about how a business works — the Product & Technology organization does a great job of sharing information and presenting problems in a way that gets to the root business problem we’re trying to solve. There’s a clear line between any project a team is working on and a foundational business metric. Most importantly, we’re a company that makes a truly positive impact on our users’ lives.

Interested in learning more about engineering at Alto? Follow us on LinkedIn.

We’re excited to share that Alicia Boler Davis will be Alto’s next CEO. She will start on September 1, 2022. This is a big moment for Alto, so we wanted to share how we got here.

Introducing Alto’s New CEO - Alicia Boler Davis

We’re excited to share that Alicia Boler Davis will be Alto’s next CEO. She will start on September 1, 2022.

This is a big moment for Alto, so we wanted to share how we got here.

Alto has come a long way since we started out in 2015 by acquiring AG Pharmacy in San Francisco. Those first few months revealed just how much opportunity there was to build a better pharmacy experience for patients by rebuilding the entire end-to-end technology stack, combining it with a more efficient operational process, and backing it all with a talented and caring team. And with that insight we started forging ahead with bright eyes and relentless optimism toward our ultimate mission: fulfill medicine’s true purpose - to improve quality of life - for everyone who needs it.

Like any transformational startup, the journey has not been without its challenges. We’ve been consistently humbled by the complexity of the pharmacy industry and the challenges inherent in transforming how it works. Yet we’ve also been consistently amazed by how much opportunity there really is to make a meaningful difference in the lives of patients, and how receptive people are to a better solution. And through persistence and grit, combined with an increasingly talented and diverse team, we have overcome all the challenges that we have faced so far. We have served over 300,000 patients, with a 90 Net Promoter Score, are unit profitable, and are expected to pass $1 billion in annualized revenue in the next few months. We’re making faster progress than at any point in the company’s history. 

So if things are going so well, why are we hiring a new CEO? The opportunity ahead of us is huge, and along with that the challenges are getting more complex. Despite how far we’ve come, we are still only 0.2% of the total pharmacy industry market share. Capturing this massive opportunity will require leveling up our already world-class team with an even greater level of bold, rigorous, caring, experienced leadership.

As we got to know Alicia over the past few months, we realized that we had found the perfect partner to lead us through the next phase of Alto’s growth. Alicia was most recently the SVP of Global Customer Fulfillment at Amazon, and before that the EVP of Global Manufacturing at General Motors. Alicia grew up in Detroit, and got her first job as an engineering intern at GM. Over the next 25 years she worked her way up to running a plant, then running multiple plants, then running all of quality and customer experience, and finally running all of global manufacturing. She was the first Black woman to accomplish each of these things at GM. In 2019 she joined Amazon, and was quickly promoted to the prestigious S-team, which is the exclusive inner circle of the most senior leaders at Amazon. When she was barely a year into the job, the Covid-19 pandemic hit, and over the next two years she oversaw Amazon nearly doubling its fulfillment network capacity to keep up with surging demand, while managing a team of over 800,000 people. Alicia built a reputation for successfully navigating the toughest challenges while driving unparalleled growth by bringing out the best in her teams.

From the day we started, we have always tried to build a team that is balanced between the rigor and operational excellence needed to execute in this complex industry, along with the heart and care that we saw was missing from the industry. These are not mutually exclusive attributes, but they are extremely rare to find in the same person. In Alicia, more than any other leader we’ve ever met at her level, we’ve found someone who can bring that balance that we seek. She is one of the world’s top executives in operations, and is a rigorous, disciplined operator. And at the same time, she leads with real heart and care for the people that she serves and that she works with.

With Alicia coming on board as CEO, Matt and I will continue to be involved full-time, both in supporting Alicia and with our own portfolios of projects to lead. We’re more committed than ever to Alto, and more excited than ever about the real opportunity we have in front of us. And we are so early in this journey that it is clear there is no shortage of work to do and problems to solve. Matt and I have never been attached to any particular role or title, and have each worn many different hats over the years. We are humble about what we are good at and what we are not, and just want to do what is best for our team in order to achieve our mission. In the short term we will focus on ramping Alicia up, and after that we will work more on what we do best: building our long term strategy, leveraging our entrepreneurial spirit to spin up new business lines, defining and evolving our culture, and sharing the Alto vision.

We are just getting started, and there is still so much opportunity to fix this broken industry, and truly help people live happier and healthier lives. We can’t wait to see what we will do together in this next phase of the Alto journey.

Receiving feedback from your manager or coworkers can be scary. Giving them feedback can be even scarier. It’s why many companies lack a culture of regular feedback and rely instead on a stress-inducing biannual performance review cycle.

How We’re Making It Easier (and More Fun!) to Give Teammates Feedback

One of Alto’s company values is “communicate with candor.” To make sure we live by this value, we recently set out to make peer feedback more frequent, honest, and fun. Here’s how it went.

Earlier this year, Alto’s patient care team hosted its first Speedback session, inspired by a Thoughtworks idea of the same name.

Set aside a dedicated hour for team members to give feedback to one another.

Group participants into pairs and have multiple rounds. Each round lasts four minutes, giving both participants two minutes to speak.

Change pairs after each round, ensuring that each participant meets once with every other. 

Each round begins with highlighting what the feedback recipient is excelling at, then noting opportunities for improvement.

By creating a fun format for discussions and encouraging constructive criticism, we hoped to take the fear out of feedback and give every participant valuable and actionable insights.

Because Speedback rounds are, well, speedy, it was important that the team came prepared.

A week before the session, every team member identified specific areas they wanted to improve in, from communicating more effectively to keeping coworkers motivated. These were then posted in Lattice, an HR tool we use at Alto, for everyone else to see.

Over the next week, team members reviewed one another’s feedback requests and brainstormed suggestions to share on Speedback day. If a team member wanted tips for writing cleaner code, other Speedback participants could look through past code review sessions for examples.

Since Alto is a remote-first company, the team also had to plan for a virtual Speedback session. We used Zoom breakout rooms, which include helpful features such as countdown timers for the rounds.

After some initial finicking with Zoom, the team quickly got into a groove. Each participant received valuable feedback from every other, which they used to grow over the next few months.

Feedback was also cross-functional. Engineers received feedback from product managers, and product managers received feedback from engineers. Managers received feedback from their reports and vice versa.

“With the feedback I received, I realized that each person responds best to slightly different communication styles,” says Patient Care Engineering Manager Nick Helvig. “It’s helped me better understand and collaborate with the entire team.”

The team’s first Speedback session occurred two months before Alto’s annual performance review cycle. This gave participants valuable information to use for improvement, as well as documentation of their strengths, weaknesses, and growth.

As one team member put it, “What I appreciate most about Speedback is the opportunity to get super clear on where both you and your teammates are looking to improve. It’s also a great way to track your progress and see how far you’ve come in the past two months, the past year.”

Patient Care has since made Speedback a ritual every two months.

To make sure there is useful follow-up to prior feedback, the team has adjusted the timing of rounds from four minutes to six. The first two minutes are now dedicated to revisiting items from the last session, allowing participants to reflect on their progress as well as remaining opportunities for improvement.

Curious to give Speedback a try? Start by putting three blocks of time on your team’s calendar: the Speedback session, a reminder to request feedback for yourself, and a reminder to prepare feedback for others.

Ultimately, Speedback does a lot more than make feedback less scary — it integrates regular, honest discussions into a company’s culture. We hope you’ll find it as helpful as we have!

Redux is great for managing client state: it allows data to be synchronized across the whole app, reduces prop drilling, and makes it easier to understand when state changes occur.

Server state is a different story. Sure, it’s nice to have data available everywhere in our app, which is why we tend to fetch data from the server and dump it into our Redux store. But because our client application doesn't own that server data, it’s hard to know when it has become stale. And what if multiple components trigger calls to fetch the same data? How do you dedupe requests? How do you decide when to refetch?

Why We Switched to React Query for Managing Server State

Redux is great for managing client state: it allows data to be synchronized across the whole app, reduces prop drilling, and makes it easier to understand when state changes occur.

Server state is a different story. Sure, it’s nice to have data available everywhere in our app, which is why we tend to fetch data from the server and dump it into our Redux store. But because our client application doesn't own that server data, it’s hard to know when it has become stale. And what if multiple components trigger calls to fetch the same data? How do you dedupe requests? How do you decide when to refetch?

Below, we’ll dive into how we’re using React Query at Alto to solve these problems and achieve separation of client and server state.

The Old Way: Fetch Server Data and Store It in State

Historically, we would fetch data and set the response in our React component state.

fetchUser(params) {
 this.setState({ isLoading: true });
 
 return getUser(params)
   .then(({ data }) => {
     this.setState({
       isLoading: false,
       user: data.user,
     });
   })
   .catch(() => {
     toaster.toast({ title: 'Something went wrong...' });
     this.setState({ isLoading: false });
   });
}

To make that data accessible from other components, we’d set that data in the global Redux store. This works fine, except that you have to manage your own loading state, the logic for when and when not to make the request, updating the store when mutating data on the server, and knowing when data is stale.

The React Query Way: Fetch and Cache Server Data

With React Query, fetching data and using that data is more declarative.

const { data, error, isLoading } = useQuery(
 // This is our Query Key, which is unique per query so that React
 // Query can manage the cache for us.
 ['user', { userId }],
 
 // This is our Query Function that actually does the data fetching.
 // We can use anything we want: built-in fetch, axios, etc.
 fetchUser({ userId }),
 
 // This query will not execute until the userId exists.
 { enabled: !!userId },
);

Here, we define our query for a user with a Query Key and a Query Function. The Query Key is unique per query, so React Query can manage our cache for us, and if any part of the Query Key changes, the query will automatically fetch new data. In this example, if the user ID changes, the query will fetch new data for that specific user and store that data in the cache.

Naming is hard, but caching is harder. React Query gives us caching of server data out of the box with cache invalidation and request deduping. If we use this same query with the same Query Key in another component, it'll check the cache: if the data is already there, React Query will simply return it, avoiding extra network requests!

Additionally, useQuery runs wherever you use it — no more fetching data when a component mounts. You just use the query and display the data when it’s ready. The cherry on top? The query returns status updates by default, so we don’t need to store isLoading in component state all the time.

Three Optimizations We Get With React Query

When we rebuilt Alto Connect, our web app for healthcare providers, we found impactful benefits in using React Query to help us manage server state instead of using Redux. Alto Connect provides instant access to patient and prescription information and enables messaging with the pharmacy. Because healthcare providers need to quickly switch between patients and clinics, caching data is important to provide a delightfully fast experience. Apart from the cache, what other optimizations do we get by using React Query? Below are a few that helped us.

Once we’ve queried for some data, it gets entered into our cache, so we can utilize that cached data when using the same query in a different component or when fetching new data for a stale query in the background.

React Query is configured with aggressive but sane defaults and considers cached data as stale. Stale queries will always query for new data when the following criteria are met:

The query is optionally configured with a refetch interval

The fetching for new data when the browser window or tab is refocused is especially useful because the healthcare providers using Alto Connect often switch between multiple browser tabs. If there’s an update to a task, status, or message while they’re checking a patient’s medical chart, it will be immediately reflected in the UI when they come back to Alto Connect.

We also set a refetch interval for our queries involving messages. In the old app, we used WebSockets to enable real-time messaging. Although it’s possible to integrate WebSockets with React Query, we opted for a polling solution instead. Because our app isn’t a real-time messaging application, and we don’t need messages to show up immediately, we set a refetch interval on our messages query to poll for new messages every 30 seconds.

const { data, error, isLoading } = useQuery(
 ['user', { userId }, 'messages'],
 fetchMessages({ userId }),
 {
   // This query is disabled from automatically running if we don’t have a userId
   enabled: !!userId,
 
   // Refetch every 30 seconds
   refetchInterval: 30000,
 
   // Continue to refetch while the tab/window is in the background
   refetchIntervalInBackground: true,
 },
);

This is where the enabled option, which disables a query when it evaluates to false, is particularly powerful. We can pause polling for messages if we route to another page and we know that we won’t make any unnecessary requests if there isn’t a valid userId. We don’t have to worry about closing any socket connections or manually clearing an interval.

Most web apps have some sort of navigation menu, and if you already measure or understand your user behavior, you can prefetch data before users even navigate to a different page. Custom hooks makes it easy to expose a function to prefetch data:

export const usePrefetchUsers = () => {
 const queryClient = useQueryClient();
 const prefetchUsers = async () => {
   await queryClient.prefetchQuery(
     ['users'],
     fetchUsers,
   );
 };
 
 return { prefetchUsers };
};

We prefetch queries when users mouseover or hover over a navigation item so that when they navigate to the next page, the data is either actively being fetched or is already in the cache waiting.

We’ve been talking a lot about queries, but what about mutating data? Mutations, in contrast to queries, are imperative. While queries run automatically when we use them, we need to fire a mutation method in order to run mutations. And in order for our user experience to feel snappy, we can optimistically update the cache and then invalidate it after successfully executing the mutation. useMutation gives us some nice side-effect handlers to deal with every stage of the mutation lifecycle. Now when the user sends a message, that new message is immediately reflected in the UI and we keep the data in sync by invalidating the cache and refetching the messages.

With Redux, we needed to build out solutions to each challenge of managing server state. React Query, on the other hand, gives us those tools right out of the box. We no longer need to worry about when to invalidate or fetch new data, making the user experience more predictable — and the developer experience more enjoyable.

Last summer, Moringa Fellows Irma Figueroa and Adrian Mendez joined Alto as the company’s second batch of engineering interns. Here, they share their learnings, accomplishments, and advice for future Alto interns.

Two Engineering Interns Weigh In: What to Expect From Your Alto Internship

The Moringa tree flourishes almost anywhere it’s planted. It’s hardy, fast-growing, and resistant to drought — an apt metaphor for the two interns who joined us from the Moringa Fellowship program last summer.

“Before Alto, I had never actually contributed to a large codebase,” admits Irma Figueroa, an intern on the Supply Chain Team. By the end of their three-month internships, however, Irma and fellow intern Adrian Mendez had shipped impactful features for Alto’s internal pharmacy management system and customer-facing app that are still in production today.

Below, we chat with Irma and Adrian to learn what it was like interning at Alto, which projects they worked on, and what advice they’d give to future interns. 

Adrian, who interned on the Patients Team before returning to join as a full-time software engineer last fall, was surprised by the freedom he had to choose what he worked on during his internship: “Early on, my manager gave me a rundown of the projects the team was juggling and the option to choose which one interested me the most,” he says. “I was able to decide the kind of experience I was going to have.” 

Adrian adds that meetings with his mentor evolved over the course of his internship, as he became more familiar with the codebase: “From the start, we had weekly meetings to talk about whatever was on my mind. Those meetings started off as general advice and questions and turned into more time to pair or run things by each other.”

Because both Adrian and his mentor were working on the same, long-term project, some of those meetings were also dedicated to project work. This included “sitting with [my mentor] while creating the project outline, pairing on larger tasks, and syncing up on progress.” 

Which projects did you work on during your internship? 

On the Supply Chain Team, Irma was tasked with a unique inventory challenge: Technicians at our production pharmacy locations use handheld scanners to scan the barcodes on newly-received inventory. Our technicians often need to scan hundreds of inventory items at a time — making it all too easy to lose track of which items have been successfully scanned onto which shelves. 

“I created a reusable counter component, displayed in the header of the scanner app, that would represent the number of items scanned into a shelf.” Irma says. The best part? Getting to test the handheld scanner herself. “I really enjoyed using the scanner for testing purposes. It was rewarding to see my code in action and definitely super fun!” 

On the Patients Team, Adrian worked on a project to add a homescreen to the customer-facing native app. Previously, the app dropped users into a list of all their active medications, which made it difficult to know where to look or what to do next. The new homescreen allows users to schedule the prescriptions most relevant to them and to view or edit any upcoming deliveries. 

“The first step was to create new additions to the design system — essentially taking the designs and extracting system-wide components. After that, the fun part was taking all those components and putting them together like building blocks!” Adrian says.

It was especially rewarding, he adds, to have been part of the project from start to finish. “I joined right as the design phase started, and I was there mostly every step of the way. For how short my time here was, having helped create this landmark project is really cool to think about.” 

What advice would you give to future Alto interns?

Irma and Adrian agree: Don’t be afraid to ask for help. “You'll often feel challenged and possibly intimidated, but be patient with yourself and remember you have an entire team of experienced engineers ready to help,” Irma says. 

“Everyone at Alto is more than willing to help you out, so don't be afraid to ask questions! You're on the same ship as everyone else, so your success equals everyone’s success,” Adrian adds. “Just remember that your work got you here, and they wouldn't have invited you if they thought you couldn't handle it. They made an investment in your ability to grow, so stay curious.” 

Interested in interning at Alto? Check out our careers page here.

With more engineers contributing to Alto’s codebase than ever before, we needed to establish better boundaries: which team owns what? Here’s how we developed a tagging-based solution to code ownership and drove and measured adoption.

How We Established Better Boundaries and Clearer Code Ownership at Alto

Some problems are nice to have. Over the course of 2020 the engineering team at Alto saw explosive growth, expanding from about fifteen to more than seventy individual contributors. Growth at this pace required us to rethink our team structures and priorities more than once. This growth felt amazing: the new capacity enabled us to make progress on our backlogs like never before.

But still, there were challenges. We saw a steady rise in merge conflicts as more people contributed to the same code base. Build times increased as more and more code was contributed. A big influx of new perspectives and talent also meant new gaps in historical context and responsibility. New teams took big bites out of their new priorities — great! But what about the old ones? Who was going to fix that bug? Who needs to review this PR? Who is accountable for this performance problem?

Who, in short, owned what? By the end of 2020, it was clear to us that a good, practical, working answer to this question would be the critical first step to continuing smooth, rapid, distributed iteration on our product.

Up to this point, the large majority of product development at Alto had happened in the context of a Ruby on Rails monolith, which — over a half decade of intense development — had grown pretty large. We’d stuck more or less to the default Rails convention of organizing code by function rather than feature, which meant that our code was distributed across a few mostly flat and very broad subdirectories such as '/models' and '/controllers', making it difficult to learn much about the structure and ownership of our business domains by merely looking at how the code was laid out. We knew that this was not going to work for us over the long term.

Our long term vision started to coalesce around a notion of “better boundaries.” First, we would modularize the monolith along the lines suggested by Shopify’s packwerk utility. Then, we would carve some of those modules out into separate engines that could be built and tested separately. Finally, we’d deploy and operate some of those engines independently as their own services. Clear ownership was fundamental to this plan: on either side of an effective boundary, you need to know not just what it is but also who owns it. We lacked that clarity.

The RFC (Request for Comments) we ultimately adopted opened with four questions:

How might we produce a domain model and do something useful at the same time?

How will we avoid imposing (the wrong) prescriptive model from the top down?

How will we keep it up to date and accurate?

We considered jumping straight into moving files around into a more modular structure. Ultimately we decided against this approach as a first step for a number of reasons. The benefit of moving files around is mostly abstract — it’s not immediately clear what to do with a better module structure. It’s relatively expensive to move files around in a Rails app — doing so typically requires you to rename the contents (and update all its callers), which amounts to a lot of busy work, and it’s extra expensive to be wrong. And the in-between state is rough: which example should I follow in deciding where to put my new file? First you are obligated to create a map of the future state, and then to keep that map in sync with reality as you go.

Instead, we opted for a lightweight solution that would allow us to map our domains and their owners in the code itself — helping to ensure it’s up to date and accurate — without having to restructure it at the same time. We’d do this by tagging code in code comments, and then building tools to leverage these tags in various ways to increase visibility and enforce ownership. This has the nice effect of producing a new domain map incrementally from the bottom up, rather than prescriptively from the top down.

We called this the '@owners' tag, and it looks like this:

# @owners { team: platform, domain: orders }
class Wunderbar::OrdersController < Wunderbar::WunderbarController
  def show
		...
  end

  # @owners { team: inventory }
  def out_of_stock
    ...
  end
end

We based our tooling around extending the YARD documentation tool to recognize this as a custom tag. You can teach YARD to recognize a custom tag by passing a command line option like '--tag owners:"Owners"', which by default will cause the tag value to appear in generated documentation under the header “Owners”. The Ruby API for declaring a tag programmatically is shown in the snippet below as well.

We wanted the tag to be interpreted hierarchically to handle cases where ownership has diverged within the same class or module, such that we might want to tag one or two methods differently from the rest, as well as to ease the burden when all of the classes / modules / methods under a namespace belong to the same team and domain, since you can just tag the shared parent. We set an initial goal around getting 100% of our methods covered by at least a “team” value, at which point we would require all code to be covered in our build process.

To get a basic feel for how to work with these tags, make sure the 'yard' gem is available, tag a couple of your files, and then try running the code below in your project. It will print out the tag associated with each tagged code block.

require 'yard'
# Define the custom tag.
YARD::Tags::Library.define_tag('Owners', :owners)

# Parse your codebase. You might want to pass a more specific path here.
YARD.parse('.')

# Load all the method, class and module code blocks into memory
code_blocks = YARD::Registry.all(:method, :class, :module)

# Print the tag associated with each tagged block.
code_blocks.each do |code_block|
  tag = code_block.tag('owners')
  # automatically tag nested classes, modules and methods.
  while tag.nil? && code_block.namespace != nil && [:class, :module].include?(code_block.namespace.type)
    code_block = code_block.namespace
    tag = code_block.tag('owners')
  end
  print "Code Block: #{code_block}, tag: #{tag.text}, line: #{tag.line}" if tag
end

We ultimately developed a handful of CLI tools based on this technique that allow us to produce reports measuring the percentage of code covered by team and domain at different levels, among other things:

bin/yardowners --help
Usage: bin/yardowners [options]
        --methods                    view by methods
        --files                      view by files
        --teams                      teams report
        --domains                    domains repot
        --team=TEAM                  filter by team
        --domain=DOMAIN              filter by domain
        --codeowners                 update codeowners
        --lint                       lint owners tags
        --require-domains            lint for presence of domain in  
owners tags
    -h, --help                       display this help and exit

We use this CLI to add checks to our CI process around coverage / correctness, as well as to report progress toward our goals of 100% coverage.

To get the tagging process started across the organization, we dumped all of our class and module names (thanks again, YARD!) into a big spreadsheet, and then asked each team to get together and spend a bit of time putting their names next to the ones they owned. We adopted the attitude that it was better to have the wrong value than no value, since it would be easy to adjust later.

We covered a good 80% of the code using this spreadsheet by the time each team had taken an initial pass. From there, we filled the blanks in with “TODO”, and then wrote a little script to automatically insert the tags into the code (again based on the YARD metadata, which conveniently gives you the line numbers for the top of each class / module declaration).

At that point, we enabled a linter requiring 100% team owner coverage on all new PRs. We set a new goal to eliminate TODOs, and chipped away at it over the course of a month or so. We then repeated this process for the “domain” values, driving that to 100% as well. We found that the spreadsheet-based divide-and-conquer approach allowed us to cover a lot of ground very quickly.

Once we were able to parse our '@owners' tags, the first and most important thing we did with them was to automatically generate our CODEOWNERS file. We enforce that each tagged team maps to a team name in github and check that CODEOWNERS is up to date during the build process. This new process introduced some friction, as teams started getting tagged every time a file they were tagged on changed. This turned out to be great, because it sparked a number of very productive negotiations around ownership and scope that got hashed out in code review. We keep our CODEOWNERS file synced with Sentry, too, so that exceptions raised by files you tagged as yours get assigned to your team automatically. We also track and report on the performance of individual endpoints by team according to how they are tagged.

The new owners tag hasn’t fixed all our problems by any means, but it has proved fundamental to our ability to operate as a growing set of increasingly independent teams with well specified responsibilities. In subsequent posts, we’ll talk more about how we’re using our new domain map and a suite of other cool tools to build better boundaries within and ultimately escape from our monolith.

To automate a manual, error-prone pharmacy workflow, we turned to event streaming. Here’s what we learned from one of the first Kafka use cases at Alto.

How We Used Kafka to Build an Event-Driven Automation Pipeline and Scale Our Pharmacy

Imagine this: You're at the doctor’s office, and your doctor just wrote you a new prescription. Minutes later, before you even check out with the receptionist, you already have a notification from Alto Pharmacy on your phone that your prescription is ready. All you have to do is pick the date and time that works best for your free courier delivery.

This is the future we’re building at Alto: a future where a prescription can be transcribed, billed, and verified by a pharmacist in five minutes, flat. To get there without dramatically increasing Alto’s head count, we need to automate much of the process, starting with the first step: transcribing, or intaking, a prescription.

As a pharmacy, we receive prescriptions in many different forms. Some are sent electronically, in a structured format, but others are handwritten, phoned in, or even faxed. To represent all of these forms in a single table, we previously required every incoming prescription to be transcribed by an Alto team member, who would answer questions like:

Whom is this prescription for? Is this customer already in our system?

Where did this prescription come from? Who wrote it?

What is the relevant prescription information (medication to dispense, quantity, days supply, directions, etc.)?

This manual process is both time-consuming and prone to human error, which, if uncaught, could impact our ability to process the prescription downstream. Furthermore, at our busiest times, it could take up to two hours for a team member to get through the pending intake queue — unacceptable if we hope to have prescriptions ready to schedule before people even leave their doctor’s office. 

Automating intake required close collaboration between three teams:

Data Science, who built the decision-making algorithms 

Data Engineering, who served up the results of those algorithms

Product Engineering, who used those results to automatically intake prescriptions 

To allow the teams to operate in parallel without toe-stepping — and in the language most suitable for each (the Data Science and Data teams prefer Python, while product engineers work mainly in Ruby) — we separated the code into two environments: 

1. Data Prod Interface (DPI), where the code for generating automation results lives

2. Wunderbar (our proprietary pharmacy operations platform), where the code for requesting and using those automation results lives

The key question: to establish robust communication between these two environments, should we use an API-based approach, or an event-driven one? Each has its pros and cons. 

Legacy approach everyone at Alto is familiar with

Existing enforcement of strong types and data

Difficult to scale with increased data & requests

Requires ongoing authorization / authentication

Infrastructure relevant to other business areas

Automatic retries, reprocessing, and queueing

Our existing infrastructure currently utilizes a hefty API-based structure, so there was a natural tendency to go down this path, but we knew that beginning to migrate services towards an event-based architecture with Kafka at its heart would help us in the long term.

Kafka is a framework implementation developed by LinkedIn, now open-source with the Apache Software Foundation. It aims to provide a unified, high-throughput, and low-latency platform for handling real-time data feeds. 

Alto uses Amazon Web Services as our cloud provider, and luckily, they offer an out-of-the-box managed solution for Kafka, MSK. To configure Kafka with our system, our infrastructure team worked with AWS to set up managed instances to comply with our business requirements. Our Infrastructure team will share more details about this work in an upcoming blog post. 

Kafka offers many benefits. The asynchronous nature of the message system allows for retries, reprocessing, and queueing as well as easy recovery from outages. In addition, the infrastructure work done for setting up the managed cluster would be a one-off workstream. When complete, we’d be able to re-use the same setup and configurations across various business needs to streamline other projects that leverage Kafka. 

Setting up for success: defining a strict contract

One element we had to get right from the start was the data contract. As we send messages between DPI and Wunderbar, we need to ensure messages are produced with the correct schema, so that downstream consumers know what to expect.

Protocol Buffers, or Protos, is an open-source, cross-platform data format used to serialize structured data. Developed by Google, Protos allow you to define how you structure your data once and then use generated code to easily write and read the structured data in a variety of different languages. 

At Alto, we’ve been using Protos for the past few years to speed up development with auto-generated code and ensure data continuity between our Ruby on Rails backend and JavaScript / React / React Native frontend. Protos may seem antithetical to JavaScript and Ruby since they’re both dynamically typed languages, but in practice, they have saved us countless hours of debugging and back-and-forth Slack messages. 

For this project, we turned again to Protos to define the schema for messages produced by DPI and Wunderbar. We defined IntakeAutomationRequest and IntakeAutomationResult messages in Go and generated the types (Sorbet, Flow, etc.), which both DPI and Wunderbar now refer to when producing or consuming messages.

Because Wunderbar is already configured to run on AWS and houses our existing intake logic, it was easy enough to send a prescription to DPI for automation. Where we would normally create a task for an operations team member to manually transcribe the prescription, we instead produce a message to our “request automation” Kafka topic using the ruby-kafka library.

Our consumer, instantiated in an executable that runs on a dedicated worker, consumes messages from a “receive automation” Kafka topic. First, the consumer verifies that a given message corresponds to a prescription we previously sent to DPI. Then, it persists any automation results included in that message. If the automation results are sufficient, we skip intake altogether. Otherwise, we create an intake task to be handled by an operations team member.

Because this project was among our early use cases for Kafka, we wanted to be prepared for the worst-case scenario: we produce a message requesting automation results, and for whatever reason—maybe DPI is down—we don’t hear back. For us, we defined a reasonable time frame as fifteen minutes, in order to avoid building up a large backlog of work that would have to be offloaded back onto our operations team.
To this end, we store metadata about the automation lifecycle in an intake_automation_results table, which includes timestamps representing when we sent a message to DPI and when we received a response back. A cron job runs every minute to check that table for rows where received_at is blank, and sent_at was more than fifteen minutes ago. If this occurs, we flag the prescription to be manually intaken by an operations team member.

The diagram below illustrates our intake automation architecture.

Intake automation is now live at Alto. We’re seeing prescriptions fly through in just a few minutes — during which time our automation creates a user profile if one doesn’t already exist, links the new prescription to that profile, and finds and bills the user’s insurance; a pharmacist reviews the prescription for clinical accuracy; and the user receives a text message to schedule their delivery. This single flow, prior to intake automation, once took up to four hours to complete!

Going from zero to one is quite a feat, and it wouldn’t have been possible without an event-based approach. Now that intake automation has laid the groundwork, we know that Kafka will only continue to provide exponential value to Alto as we grow and migrate more of our services to this event-based architecture. 

Each type of user has a different product or set of products we’ve built for them. From within each product, we collect, transmit, process, and store a tremendous amount of sensitive information. However, our biggest area of security, privacy, and compliance risk is our internal pharmacy management tool called Wunderbar that is used by our operations (ops) users every day to process prescriptions. Securing access to Wunderbar is mission-critical to protect our data and our business, and, of course, we need our super-secure implementation to be super-smooth to avoid slowing down our hard-working ops users.

Developing Robust Authentication Using Rails, Omniauth, and Okta

Alto Pharmacy aims to deliver a better pharmacy experience for everyone who needs medication. To achieve that, we need to enable three different types of users:

Individual customers who need prescriptions

Healthcare providers: the nurses, doctors, physicians assistants, and other medical professionals who prescribe medications

Operations users: the Altoids who intake prescriptions, bill insurance, fill prescriptions, and more

We wanted to be able to change our Wunderbar session lifetimes without hampering either our security efforts or the usability of our product. Moreover, we wanted session lifetimes of other apps to be able to change appropriately with business risk. Before starting this project, we’d implemented a Google OAuth 2.0 authorization flow that allowed ops users to log into Wunderbar via Google and take advantage of transparent session refresh for a limited amount of time.

However, while Google Workspaces was (and is) great for our email, documents, and calendars, it isn’t as configurable or extensible as a proper SSO solution. One painful side effect of this limitation was that, in order to ensure our refresh sessions were limited to twelve hours in Wunderbar, any user of the Alto Gmail drive also had to log back in every twelve hours just to see email — even though there’s not nearly the same level of sensitive data in email.

Plus, our IT team had already picked the tool of the future: Okta. And the goal was to enable SSO, or Single Sign On: if we still used Google for SSO, it wouldn’t really be SSO!

It was clear we needed to move to Okta for authenticating in Wunderbar, too.

Okta is a big company with many different authentication mechanisms. For us, the big question was really: do we want to authenticate via SAML or Open ID Connect (OIDC)? There are a lot of great resources around comparing these two standards and what they’re good at; here’s a good chart comparing and contrasting parts of these two authentication flows:

From this Medium post, which is a great primer on the differences

For us, the most attractive thing about SAML is that it can be used to provision user accounts from your upstream IDP; for example, Okta could be configured to push username, email, group, role, etc., information to our platforms with which we could automatically provision user accounts.

In the end, we decided the OIDC implementation would provide for a very similar user experience to that which our operations users were already accustomed to, with the same capability to perform transparent token refresh.

This transition was pretty easy because we were moving from Google OAuth 2.0 flow to Okta’s OIDC flow. OAuth 2.0 and OIDC are very similar; just know that OAuth 2.0 is technically an authorization framework (once you have a user, what can they access) vs an authentication framework (actually identifying and validating the user). However, there were some odd things about our previous implementation, and I only had one person to blame for those choices: myself. It’s great to get the opportunity to re-implement our authentication system — in a fast moving startup, you rarely get the chance to revisit problems so soon!

The biggest issue at hand with our previous Google flow was the way in which I’d chosen to re-authenticate users whose Wunderbar sessions had expired. In this case, OAuth provides a handy mechanism: a refresh token. Here’s how access tokens and refresh tokens are exchanged from the client and the authentication server in OAuth 2.0 flows:

At the time, we had session lifetimes of 60 minutes when accessing Wunderbar. These cookies also lasted 60 minutes. We faced a problem: if somebody walked away from their laptop for 61 minutes after authenticating, when they came back, how could we make it so they didn’t have to go back through the Google OAuth 2.0-initiated login flow from the beginning?

My solution at the time persisted a secure cookie with the refresh token value client-side with a longer expiration length. Then, if the user didn’t have a valid access token, we’d simply present the refresh token back to the server and get a new access token all over again. This worked well, but exposing secrets to the client always presents a risk of compromise. Refresh tokens are longer-lived, so they're of even higher value to protect!

Okta’s documentation made it very easy to stand up a simple authentication server and set up a local development server. Okta has a great blog post guide with tools we were already using (Rails and Omniauth, specifically) to get up and running. Beyond that, our biggest challenge was ensuring that we had a smooth rollout.

There was an issue in how we’d test and use this flow in our staging environments, however. At the time we were working on this (early 2020), there were two issues:

Our sandbox environment URLs were non-deterministic and could be varied depending upon how an engineer user named their cluster.

Okta apps did not support wildcard redirects in their oauth redirect URLs. This controls where, upon hitting Okta’s servers, they should redirect your payload to handle the meat of authentication.

For a time, our clever infrastructure engineering team managed around these issues with the use of an intermediary keycloak server. However, Okta now supports wildcard redirects that we’re successfully using today:

The biggest functional change to our authentication flow was in our refresh flow. The biggest challenge in landing this change was in how we deployed it to end users in production.

We didn’t want to expose the refresh token because it presented too much risk. What do we do instead? Something a little bit risky, but that we preferred after threat modeling our scenarios. Security engineering is often about making tradeoffs given a number of constraints. Today, our session lifetimes are down to fifteen minutes, and our refresh lifetime for an active session (e.g. somebody making at least one request within this window) is 60 minutes. If somebody comes back after their access token expires, that’s fine; we’ve persisted the encrypted refresh token to the database, and we can use attributes from the expired but still cryptographically valid token in order to look that refresh token up.

Blog

Get to know Jake Moffatt, Engineering Manager on the Alto Assistant Team

Introducing Alto’s New CEO - Alicia Boler Davis

How We’re Making It Easier (and More Fun!) to Give Teammates Feedback

Why We Switched to React Query for Managing Server State

Two Engineering Interns Weigh In: What to Expect From Your Alto Internship

How We Established Better Boundaries and Clearer Code Ownership at Alto

How We Used Kafka to Build an Event-Driven Automation Pipeline and Scale Our Pharmacy

Developing Robust Authentication Using Rails, Omniauth, and Okta

Tackling Technical Debt and Improving Security: How Alto Untangled an Overloaded User Model

Considerations and Approaches for Implementing Biometric Login: Part 2

Considerations and Approaches for Implementing Biometric Login: Part 1

How Alto Reduced Time to Resolution for Production Incidents

Switch to a better pharmacy.