Your partner in navigating the ins and outs of prescriptions — at no extra cost.

Alto Pharmacy Technician

Alto Pharmacist

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies based on team member feedback and a rigorous assessment of our company’s culture and benefits.

Join us in delivering our industry-leading pharmacy care to millions of people across the country. As a member of the team, you’ll play an integral role in improving our customers’ quality of life and redefining what’s possible in pharmacy. 

Be Part of a Great Place to Work — Join Our Team

Hear first-hand from members of the Alto technology teams.

Read the latest Alto blog posts covering healthcare, family health, managing diseases, and other Alto highlights.

Blog | Alto Pharmacy

Redux is great for managing client state: it allows data to be synchronized across the whole app, reduces prop drilling, and makes it easier to understand when state changes occur.

Server state is a different story. Sure, it’s nice to have data available everywhere in our app, which is why we tend to fetch data from the server and dump it into our Redux store. But because our client application doesn't own that server data, it’s hard to know when it has become stale. And what if multiple components trigger calls to fetch the same data? How do you dedupe requests? How do you decide when to refetch?

Why We Switched to React Query for Managing Server State

Redux is great for managing client state: it allows data to be synchronized across the whole app, reduces prop drilling, and makes it easier to understand when state changes occur.

Server state is a different story. Sure, it’s nice to have data available everywhere in our app, which is why we tend to fetch data from the server and dump it into our Redux store. But because our client application doesn't own that server data, it’s hard to know when it has become stale. And what if multiple components trigger calls to fetch the same data? How do you dedupe requests? How do you decide when to refetch?

Below, we’ll dive into how we’re using React Query at Alto to solve these problems and achieve separation of client and server state.

The Old Way: Fetch Server Data and Store It in State

Historically, we would fetch data and set the response in our React component state.

fetchUser(params) {
 this.setState({ isLoading: true });
 
 return getUser(params)
   .then(({ data }) => {
     this.setState({
       isLoading: false,
       user: data.user,
     });
   })
   .catch(() => {
     toaster.toast({ title: 'Something went wrong...' });
     this.setState({ isLoading: false });
   });
}

To make that data accessible from other components, we’d set that data in the global Redux store. This works fine, except that you have to manage your own loading state, the logic for when and when not to make the request, updating the store when mutating data on the server, and knowing when data is stale.

The React Query Way: Fetch and Cache Server Data

With React Query, fetching data and using that data is more declarative.

const { data, error, isLoading } = useQuery(
 // This is our Query Key, which is unique per query so that React
 // Query can manage the cache for us.
 ['user', { userId }],
 
 // This is our Query Function that actually does the data fetching.
 // We can use anything we want: built-in fetch, axios, etc.
 fetchUser({ userId }),
 
 // This query will not execute until the userId exists.
 { enabled: !!userId },
);

Here, we define our query for a user with a Query Key and a Query Function. The Query Key is unique per query, so React Query can manage our cache for us, and if any part of the Query Key changes, the query will automatically fetch new data. In this example, if the user ID changes, the query will fetch new data for that specific user and store that data in the cache.

Naming is hard, but caching is harder. React Query gives us caching of server data out of the box with cache invalidation and request deduping. If we use this same query with the same Query Key in another component, it'll check the cache: if the data is already there, React Query will simply return it, avoiding extra network requests!

Additionally, useQuery runs wherever you use it — no more fetching data when a component mounts. You just use the query and display the data when it’s ready. The cherry on top? The query returns status updates by default, so we don’t need to store isLoading in component state all the time.

Three Optimizations We Get With React Query

When we rebuilt Alto Connect, our web app for healthcare providers, we found impactful benefits in using React Query to help us manage server state instead of using Redux. Alto Connect provides instant access to patient and prescription information and enables messaging with the pharmacy. Because healthcare providers need to quickly switch between patients and clinics, caching data is important to provide a delightfully fast experience. Apart from the cache, what other optimizations do we get by using React Query? Below are a few that helped us.

Once we’ve queried for some data, it gets entered into our cache, so we can utilize that cached data when using the same query in a different component or when fetching new data for a stale query in the background.

React Query is configured with aggressive but sane defaults and considers cached data as stale. Stale queries will always query for new data when the following criteria are met:

The query is optionally configured with a refetch interval

The fetching for new data when the browser window or tab is refocused is especially useful because the healthcare providers using Alto Connect often switch between multiple browser tabs. If there’s an update to a task, status, or message while they’re checking a patient’s medical chart, it will be immediately reflected in the UI when they come back to Alto Connect.

We also set a refetch interval for our queries involving messages. In the old app, we used WebSockets to enable real-time messaging. Although it’s possible to integrate WebSockets with React Query, we opted for a polling solution instead. Because our app isn’t a real-time messaging application, and we don’t need messages to show up immediately, we set a refetch interval on our messages query to poll for new messages every 30 seconds.

const { data, error, isLoading } = useQuery(
 ['user', { userId }, 'messages'],
 fetchMessages({ userId }),
 {
   // This query is disabled from automatically running if we don’t have a userId
   enabled: !!userId,
 
   // Refetch every 30 seconds
   refetchInterval: 30000,
 
   // Continue to refetch while the tab/window is in the background
   refetchIntervalInBackground: true,
 },
);

This is where the enabled option, which disables a query when it evaluates to false, is particularly powerful. We can pause polling for messages if we route to another page and we know that we won’t make any unnecessary requests if there isn’t a valid userId. We don’t have to worry about closing any socket connections or manually clearing an interval.

Most web apps have some sort of navigation menu, and if you already measure or understand your user behavior, you can prefetch data before users even navigate to a different page. Custom hooks makes it easy to expose a function to prefetch data:

export const usePrefetchUsers = () => {
 const queryClient = useQueryClient();
 const prefetchUsers = async () => {
   await queryClient.prefetchQuery(
     ['users'],
     fetchUsers,
   );
 };
 
 return { prefetchUsers };
};

We prefetch queries when users mouseover or hover over a navigation item so that when they navigate to the next page, the data is either actively being fetched or is already in the cache waiting.

We’ve been talking a lot about queries, but what about mutating data? Mutations, in contrast to queries, are imperative. While queries run automatically when we use them, we need to fire a mutation method in order to run mutations. And in order for our user experience to feel snappy, we can optimistically update the cache and then invalidate it after successfully executing the mutation. useMutation gives us some nice side-effect handlers to deal with every stage of the mutation lifecycle. Now when the user sends a message, that new message is immediately reflected in the UI and we keep the data in sync by invalidating the cache and refetching the messages.

With Redux, we needed to build out solutions to each challenge of managing server state. React Query, on the other hand, gives us those tools right out of the box. We no longer need to worry about when to invalidate or fetch new data, making the user experience more predictable — and the developer experience more enjoyable.

Last year, we set out to tackle this technical debt by splitting the User model into separate concerns and refactoring the entire system into this new paradigm.

Tackling Technical Debt and Improving Security: How Alto Untangled an Overloaded User Model

At Alto, we have three types of users: customers, healthcare providers (nurses, doctors, etc), and operations users who run the pharmacy. For each type of user, we maintain a separate application (though they share a monolithic backend for now): our customer app; our Alto Connect app for providers; and our internal pharmacy management tool called Wunderbar, which powers the pharmacy experience from prescription intake to fulfillment.

Early on, we made the pragmatic choice to represent both customers and operations users with a single User model, distinguishing them with a “roles” attribute. This required repetitive checks that, over time, slowed down our engineering teams, presented substantial privacy risk, and posed difficult scaling challenges as we evolved our monolith into different microservices. 

The challenges of relying on a single User model for different types of users

The notion of a User is core to our concept of identity from which authentication and authorization proceeds. From a security perspective, it’s difficult to build an extensible and clean authentication and authorization system when you’re overloading one model for two very different populations of users. From a privacy perspective, this muddied customer records (which are subject to more restrictions because they’re healthcare records) with workforce records, hamstringing many analytics efforts.

Authentication: Alto employees who were also Alto customers were required to use their work emails for their customer accounts. Not only was this restriction frustrating for Alto employees, but it also posed a security risk: employees should only be able to access Wunderbar via corporate SSO, not their personal account passwords. 

Authorization: Engineering new authorization capabilities required intricate knowledge of the idiosyncrasies of our user structure. This both slowed down development and created tremendous risk that any mistake could easily escalate user permissions.

Privacy: Because they existed in the same table, both customer records and Alto operations user records were classified as Protected Health Information, or PHI, which severely limited what we were able to do with them.

Attribution: Our robust audit logs were significantly less useful when a user ID in the transaction log might refer to either an ops user or a customer.

State management: Some updates pertaining to admin identity lived on the customer model, some lived on the employee model, and others lived in other various models.

Architecture: Tight coupling would have prevented us from decoupling our monolithic system into smaller services.

Our backend is primarily a monorepo Rails application that shares a tremendous amount of functionality between our customer, provider, and ops user-specific domains. Generally we use JWTs to secure customer and ops user sessions. These JWTs are issued only once valid credentials have been present.

Ops users are required to log in via Okta SSO, enforcing both valid username/password as well as a two-factor authentication (2FA) prompt. We use Devise and Omniauth to wire up the OIDC flow with our upstream identity provider, and we use CanCan to expand a set of roles per-user into a set of capabilities that restrict what resources they can access/modify.

Ops users primarily work in our internal pharmacy management app. However, the state of these users was shared between several different tables: Users, which otherwise refer to customers, and Employees, which share some ops user-specific state tied to Users.

Large change sets of code being deployed at once are harder to code review, harder to unwind if something goes wrong, and are more likely to be more disruptive. For this project, we knew that such a large architectural change had to be broken up into many more-digestible pieces if we were to be successful. The plan:

1. Create a new dedicated model for our ops users

# == Schema Information
#
# Table name: wunderbar_users
#
#  id                        :bigint(8)        not null, primary key
#  user_id                   :bigint(8)        indexed
#  email                     :string           not null, indexed
#  first_name                :string           not null
#  last_name                 :string           not null
#  token_last_invalidated_at :datetime
#  roles_mask                :integer          default(0), not null
#  title                     :string           not null
#  team                      :string           not null, indexed
#  is_active                 :boolean          default(TRUE), not null
#  slack_user_id             :string
#  pod                       :string

This was the easy part: create the database structure for a properly isolated admin user type, which we called a WunderbarUser.

We now had our admin user state split between Employees and Users, with a new table that engineers and analysts wanted to use that was not yet populated, and we had to make progress while allowing new hires and new customers alike to onboard to our platform.

2. Synchronize state between the old tables and new table

Next, we created a WunderbarUser for any User who was, or ever had been, a Wunderbar user. This required some historical digging —we had a lot of people who were onboarded, and even some offboarded, before we even had the intermediate Employee record!

Two major choices at this stage really helped the project succeed. First, we ensured that the WunderbarUser records all had IDs matching the corresponding User records. This is because a lot of our legacy audit records (that we’re required to preserve and keep useful as a Covered Entity under HIPAA) referenced an `admin_user_id` column populated with the ID of the User. If we just created all new entries for WunderbarUsers, these old logs would require at least one level of indirection (e.g. querying back to User) to make useful.

     # Create with the same ID as the user that we're creating this from.
     wunderbar_user_attributes = {
       id: user.id,
       user_id: user.id,
       email: user.email,
       roles_mask: user.roles_mask,
       date_of_birth: user.date_of_birth,
       token_last_invalidated_at: user.token_last_invalidated_at,
       first_name: user.first_name,
       last_name: user.last_name,
       preferred_first_name: user.preferred_first_name,
       is_active: false,
     }

Second, we created a new WunderbarUser any time we created a new Employee record (e.g. when we onboarded a new employee) and synchronized any changes between Users, Employees, and WunderbarUsers tables. As a side note: we knew we’d have to protect this syncing logic to ensure that we only copied each change once — otherwise, our callbacks would fire forever!

3. Migrate functionality slowly to use the new ops user model

We migrated logic to reference WunderbarUsers where previously we’d referenced Users and/or Employees. Authentication, authorization, audit logs, and pharmacist licenses were all updated here — albeit painstakingly!

We then migrated all of the foreign keys our various Postgres database tables held to either the Users or Employees tables to instead reference the WunderbarUsers. This ended up being around a hundred foreign keys, but a constraint-only migration that fortunately didn’t involve any data changes.

In short: our goal was to do this change slowly and steadily enough that nobody would notice until, one day, we could announce to the team, “Hey, we did a big security-pertinent project, and now you can change your customer email to be anything.”

Authentication: While we use the same JWT format for each, our customer and operations user authentication systems are now totally separated.

Authorization: With separate user types, we have separate capabilities and domain-specific authorization checks for each area, enforced in code.

Experience: Onboarding as a new Altoid (somebody who works here; come join us!) is totally separate from the customer experience, and you no longer need to link your accounts together.

Privacy: Our ops users who choose to be customers enjoy the full privacy afforded to our customers without tying their account to their work identity in any way.

Attribution: Legacy audit logs are still index-able and query-able with the same IDs, just in a slightly different table.

State management: Customer state lives on the User table, and ops user state lives on the new WunderbarUsers table. Our callbacks on each have been dramatically pared down, and our intermediary Employee table is entirely gone, along with a number of other now-ancillary tables.

Architecture: We have a clear path to federate RBAC checks without shipping our entire customer database between services.

Bonus: Performance and code readability generally improved by removing complex joins and join tables and isolating concerns between the customer and ops user domains.

Another Bonus: Significant refactorings and increased test coverage along the way while we cleaned up some very old, very stale code.

Large architectural changes of legacy systems with heavy use are rarely 100% smooth. Here are a few learnings to keep in mind the next time you tackle a hefty piece of technical debt. 

1. Always validate your assumptions (e.g. the data your constraints are against) when foreign keys are concerned. Multiple times we assumed, “Hey, we’ve migrated a bunch of really big tables that are really old (insurance claims, prescriptions, and more), so there’s no way we could have another admin user that we haven’t created a commensurate WunderbarUser for, right?” Wrong! Your data can always have inconsistencies, but a simple SQL query (e.g. validate that all values of the column you’re migrating the foreign key to have a matching WunderbarUser) can save a lot of time and headaches.

2. When migrating foreign key constraints on tables with active reads in Postgres, ensure you break it into two steps. Migrate your foreign key with `{ validate: false }`

 def up
   remove_foreign_key :inventory_orders, to_table: :users, column: :created_by_id
   add_foreign_key :inventory_orders, :wunderbar_users, column: :created_by_id, validate: false
 
   validate_constraint
 end

Validate the named constraint of your foreign key separately

 def validate_constraint
   validate_constraint_statement = 'ALTER TABLE "inventory_orders" VALIDATE CONSTRAINT "fk_rails_abcd1234";'
   ActiveRecord::Base.connection.execute(validate_constraint_statement)
 end

3. When working with legacy tables, be mindful of legacy systems. Our Billing service used a very outdated golang ORM that panicked if a column specified by the `struct` in source no longer existed; this caused a late-night outage that made auto billing requests back up the next day.

4. Scoping projects is hard! This was a larger project than our team had ever scoped and delivered before. Once all the dust settled, this project took several months longer than our initial scoping estimates.This project was worth doing, but better estimates allow us to prioritize better. If we’d had a great scope of this immense project upfront, would we have chosen to do it first? Also, this was a tremendously difficult project to separate into material stages; all the benefit was really backloaded.

In the end, this was a very satisfying and important project to complete. We wouldn’t have been able to accomplish it without a lot of help from our colleagues on the Platform team, the Security team, and across many other functions atAlto. Very large projects that involve a lot of moving pieces and impact other teams are always scary — but we can’t let that fear prevent us from making the changes we know we need to do in order to be successful long-term.

The Engineering Team is hiring! Learn more about open positions here.

In our previous post, we discussed how we designed our biometric authentication system with both privacy and a better user experience in mind. In this post, we'll show you exactly how we did it.

Considerations and Approaches for Implementing Biometric Login: Part 2

In Part 1, we discussed how we designed our biometric authentication system with both privacy and a better user experience in mind. In this post, we'll show you exactly how we did it.

To build a new authentication mechanism, we opted to use the react-native-biometrics library. This library interfaces with the device's biometrics, generates a keypair, and stores the private key within the device's keystore. When logging in with biometrics, the library creates a signature using the stored private key, and the signature is then sent to the server for validation. (Note: this library is now unmaintained. At the time of writing, this implementation can instead be accomplished with the following maintained libraries: expo-local-authentication to access device biometrics, keypair to generate a keypair, react-native-keychain to access the device keystore.)

Check if biometrics are supported and configured by the user for this device.

const { available } = await ReactNativeBiometrics.isSensorAvailable();

If yes, prompt the user for a biometric scan.

const { success } = await ReactNativeBiometrics.simplePrompt({ promptMessage: 'Confirm ID' });

If the scan succeeds, generate a keypair. The react-native-biometrics library's createKeys method will automatically save the private key to the device's keystore, and return the public key counterpart. Send the public key to your server.

const { publicKey } = ReactNativeBiometrics.createKeys('Confirm biometrics');

// Save the public key to your database
post('/save_biometric_key', { key: publicKey, device_id: 'xyz' })

When the user tries to login with biometrics, scan their biometrics. This time, if the scan succeeds, create a signature using the stored private key. The react-native-biometrics library's createSignature method handles all of this.

// Prompts for scan
// On scan success, accesses private key and uses it to generate a signature, which is then returned
const signatureResponse = ReactNativeBiometrics.createSignature({
  promptMessage: 'Face or fingerprint ID for Alto Pharmacy',
  payload: BIOMETRIC_SIGNATURE_PAYLOAD,
});

Send the signature to the server to validate against the public key. If validation succeeds, the user has successfully authenticated with biometrics and can proceed into the app.

// Call your server API to validate the signature against the public key that you've stored in the database
const validationSuccess = await post(
  `/validate_biometric_signature`,
  {
    signature: signatureResponse.signature,
    payload: BIOMETRIC_SIGNATURE_PAYLOAD,
		device_id: 'xyz',
  },
);

if (validationSuccess) {
  // Login Success
} else {
  // Login failure
}

import ReactNativeBiometrics from 'react-native-biometrics';

const setUpBiometricLogin = async () => {
  // Check if biometrics are supported and configured on this device
  const { available } = await ReactNativeBiometrics.isSensorAvailable();

  if (available) {
    // Prompt user to scan biometrics
    const { success } = await ReactNativeBiometrics.simplePrompt({ promptMessage: 'Confirm ID' });
      
    if (success) {
      // If scan is successful, generate a keypair, then save the public key to the server
      const { publicKey } = await ReactNativeBiometrics.createKeys('Confirm biometrics');

      post('/save_biometric_key', { key: publicKey, device_id: 'xyz' })
    }
  }
}

const loginWithBiometrics = async () => {
  // Scan user's biometrics. On scan success, generate a signature
  const signatureResponse = await ReactNativeBiometrics.createSignature({
    promptMessage: 'Face or fingerprint ID for Alto Pharmacy',
    payload: BIOMETRIC_SIGNATURE_PAYLOAD,
  });

  // Validate the signature against the public key saved on the server
  const validationSuccess = await post(
    `/validate_biometric_signature`, 
    {
      signature: signatureResponse.signature,
      payload: BIOMETRIC_SIGNATURE_PAYLOAD,
      device_id: 'xyz',
    },
  );

  if (validationSuccess) {
    // Login Success
  } else {
    // Login failure
  }
}

1. build a way to store a public key associated with a device

2. write a method to validate a signature against a public key

3. expose endpoints to save a public key, and validate a signature

Once biometric authentication is built, there are additional decisions to make. The right decisions for your app will depend on your goals; the approach we took here at Alto may not align with what is best for your app.

How should biometric sessions interact with our existing email and password sessions?

We opted to treat email and password sessions as longer-lived primary sessions, with biometrics as a shorter-lived secondary session on top of that primary session. This means that a user must first login with email and password in order to use biometrics on subsequent app opens. If the user logs out for any reason, they will need to log in again with email and password before they can use biometrics again.

How often should we prompt the user for biometrics vs. email and password?

We opted to prompt the user for biometrics far more frequently than email and password, but made this configurable in case we want to adjust.

We use tokens and cookies for session management.

There are plenty of edge cases to consider as you're developing biometric capabilities for your app, including: what should happen when the biometric scan fails, or when the signature validation fails? What should happen if a user disables biometric permissions from their device? What should happen when two users share a device? What should happen if 'risky' activity is flagged for this account? How can we disable biometric login if needed?

Face and fingerprint ID login has clear benefits for both the end-user and the application itself, but it comes with a host of implementation decisions. If you're considering building biometric login for your app, I hope this article gives you clarity on what biometrics is, the different ways you can integrate it into your app, and the tradeoffs and edge cases to consider.

At Alto, the Patients Engineering team worked closely with our Security Engineering and Product teams to align on our goals and build the best solution for our needs. Our Security team offered deep expertise on the authentication strategies to adopt and avoid, and our Product team kept us tethered to building the best experience for our users.

Working as a team is one of the best parts about developing at Alto. If you want to meet the awesome team I'm talking about, come join us - we're hiring!

Alto Connect is the industry’s first mobile app for clinical teams to track prescriptions and message with the pharmacy on-the-go.

Introducing Alto Connect

Big news from the team here at Alto: today we launched Alto Connect, the industry’s first mobile app for clinical teams to track prescriptions and message with the pharmacy on-the-go! You’ll also see the new Alto Connect brand on our desktop portal, formerly known as AltoMD: new name, same great pharmacy services.

The Alto Connect mobile app is the next evolution of Alto’s custom-built provider communications center, formerly known as AltoMD. Alto was the first pharmacy in the United States to offer a prescriber-focused pharmacy communication platform when the original desktop-based AltoMD product launched in March 2016. Since then, more than 500 doctors offices have integrated Alto Connect into their daily workflows and prescription management. 

Doctors offices who work with Alto Pharmacy can now access Alto Connect across all of their devices, providing greater flexibility and convenience for monitoring prescriptions on busy days spent juggling patient consultations and other business needs. Instead of spending hours on the phone or fighting with the fax machine, Alto Connect provides instant access to prescription information and pharmacy messaging, and Alto Connect users can move seamlessly between the two platforms throughout their workday.

With Alto Connect, clinical staff can view all current prescriptions at Alto Pharmacy, including delivery status, out-of-pocket costs and refills. You'll never have to wonder if a patient picked up their prescription or stayed on track with their refills between visits - our system tells you exactly where every prescription stands for every patient.

Have a question for our pharmacy team? Curious about insurance coverage, delivery times, or other aspects of our service? Our team is here to help! You can send us a general message or ask a question specific to one of your patient's prescriptions. 

Alto Connect's push and email notifications alert you when a prescription needs additional information to move forward — most commonly, a clinical clarification or additional data needed for a prior authorization. We'll only alert you when we're certain we need your team's input, so you can stay focused on your patients.

Using AltoMD on your computer today? Nothing changes other than the name! Ready to try Alto Connect mobile for yourself? Scan the QR code above to download the app - your AltoMD login works on the mobile app. 

Need help setting up an Alto Connect account or want training for your staff? Reach out to providersuccess@alto.com.

When engineering incidents happen, typical deployment pipelines are just too slow—especially if customers’ medications are on the line. Here’s the speedier rollback process we use at Alto Pharmacy to ship fixes in 3 minutes, flat.

How Alto Reduced Time to Resolution for Production Incidents

Production incidents caused by newly-merged code happen to us all. When they do, most engineering organizations discuss the root cause and how to prevent similar incidents in the future. At Alto, we evaluate two additional metrics:

Time to detection: How quickly did we know something was wrong?

Time to resolution: How quickly did we restore service?

Time to resolution is especially critical for us as a pharmacy, where a long-running incident could block a user from getting their medication on time.

Over 30 times a day, an engineer at Alto hits the merge button, deploying a new build to production. Our well-automated CI/CD pipeline, built by our Infrastructure team, handles the 3-step deployment process:

1. Build: install Ruby & JavaScript dependencies

The average runtime for our standard deployment pipeline is 30 minutes—reasonable for a CI/CD pipeline, but too long when shipping a fix for a production incident.

We introduced a “hotfix” workflow as a first pass at an expedited deployment pipeline for incident response. A hotfix is a commit that introduces new code to address the incident by either removing or reverting the buggy code that introduced the issue in the first place or by shipping new code that fixes the initial issue. 

The hotfix deployment pipeline reduces incident time to resolution by triggering a specific CI job that removes the test step from our standard pipeline, reducing the workflow to just the Build and Deploy steps.

Implementing a hotfix pipeline on top of our standard pipeline in CircleCI was relatively simple—in our `.circleci/config.yml`, any part of our CI workflow that should be skipped in case of a hotfix runs a job called “early_exit_if_hotfix” that will skip the current step if the last commit message of the code being built includes the keyword “[hotfix]”.

early_exit_if_hotfix:
    steps:
      - run:
          name: Early exit if hotfix
          command: |
            git_commit_message=$(git log --format=%B -n 1)
            if [[ $git_commit_message =~ "Merge pull request #" ]]; then
              echo "This is a merge commit. Checking if this is a hotfix deploy"
              git_commit_sha=$(git log --format=%H -n 1)
              git_commit_message=$(git log --format=%B $git_commit_sha^-)
            fi
            if [[ $git_commit_message =~ "[hotfix]" ]]; then
              echo "exiting early for hotfix deploy"
              circleci-agent step halt
            fi
            echo "not a hotfix deploy, continuing"

run_ruby_tests:
  steps:
early_exit_if_hotfix
run_tests

The hotfix workflow reduces time to ship a resolution, once we identify the fix, from 30 minutes to ~18 minutes. It’s faster than our standard pipeline but still requires considerable hands-on work from the engineer performing the hotfix.

To further speed up deployment and reduce manual work, we introduced a workflow that simply rolls back to a previously-built healthy deployment image. In this workflow, the entire pipeline is reduced to just the Deploy step.

To invoke a rollback, an engineer commits a message with the keyword “[rollback]”, followed by the SHA of the commit to rollback to. Similar to the hotfix approach, any build step that should be skipped during a rollback checks the commit message for the “[rollback]” keyword before running:

early_exit_if_rollback:
    steps:
      - run:
          name: Early exit if rollback
          command: |
            git_commit_message=$(git log --format=%B -n 1)
            if [[ $git_commit_message =~ "Merge pull request #" ]]; then
              echo "This is a merge commit. Checking for rollback sha..."
              git_commit_sha=$(git log --format=%H -n 1)
              git_commit_message=$(git log --format=%B $git_commit_sha^-)
            fi
            if [[ $git_commit_message =~ "[rollback]" ]]; then
              echo "exiting early for rollback deploy"
              circleci-agent step halt
            fi
            echo "not a rollback deploy, continuing"

run_ruby_tests:
  steps:
early_exit_if_hotfix
early_exit_if_rollback
run_tests

The rollback workflow looks for a build matching the specified SHA and posts a confirmation message for the engineer, then deploys that build to production. The rollback approach dropped our time to resolution even further, from 30 minutes to ~3 minutes.

There are three edge cases that we’ve also baked into our rollback workflow: potentially stale images, database migrations, and blocking new merges and deployments.

Our script has a special block warning the engineer if a target image is older than an hour. The older the image, the riskier the rollback, as the delta between the image and the latest commit on main is larger.

time_rollback=$(git show -s --format=%ct ${rollback_revision})
time_now=$(date '+%s')
if [ $(($time_now - $time_rollback)) -gt 3600 ]; then
    message="${message}\n\n:warning:The rollback commit is more than an hour old."
    message="${message}\nThe [hotfix workflow](https://www.notion.so/alto/Deploying-a-Hotfix-Rolling-Forward-bb8e9cea05b44c67a64e23ae652653b6) might be a better candidate for this deployment."
fi

We also want to warn engineers if any database migrations have occurred between the current image and the target by checking the diff of the db/migrate directory. Rolling back application code without applying changes to the database can result in the database and application code becoming out of sync, resulting in further problems. Our script will warn engineers if a migration has been detected between the current and target image and provide instructions on how to rollback migrations if required.

Because a rollback deploys an image that’s older than our latest build, we also need to block continuous deployment until we’ve restored the main branch to a safe state — typically by deploying a hotfix as soon as we can identify an appropriate fix. We accomplish this using a Github issue with a specific tag that, when open, triggers a job causing any active CI job to fail. After the incident is resolved, the incident manager closes the issue, which will retrigger the blocked jobs and resume the normal build and deploy workflow.

The expedited rollback process has reduced our overall time to ship a resolution for incidents down to a speedy 3 minutes—a valuable improvement for our internal operations users, as well as for our customers and healthcare providers who rely on Alto’s service to get prescriptions filled. Branching these incident response workflows off of our existing standard deploy pipeline keeps the maintenance cost low for our infrastructure team, and robust automation reduces the risk of manual error for our product engineers when they need to invoke these workflows.

We’re excited to welcome Christopher Tidball to the Alto team as Director of Product Management, Supply Chain & Fulfillment. Learn more about his passions both in and outside of work and what drew him to Alto.

Introducing Christopher Tidball, Director of Product Management - Supply Chain & Fulfillment

We’re excited to welcome Christopher Tidball to the Alto team as Director of Product Management, Supply Chain & Fulfillment. Most recently, Christopher led a large team of product managers, engineers and program managers at Amazon to power the company’s most complex fulfillment operations. 

As Alto continues to expand our geographic footprint and customer base, Christopher’s nearly 20 years of industry experience will support us in scaling our systems, so that we can continue to deliver the same high standards of care and service to even more communities.

We recently sat down with Christopher to learn more about his passions both in and outside of work, what drew him to Alto, and the operational opportunities he’s most excited about. 

Welcome to Alto! What inspired you to join the team?

Before a recruiter ever reached out to me about a role at Alto, my wife and I were very satisfied customers. We had been on a long journey of trying to start a family, and that ultimately led us to a fertility clinic that recommended Alto. We had a great experience from day one: the app was easy to use, the same-day deliveries from couriers were always on time, and Alto’s pharmacists walked us through the various medications in our delivery and answered all of our questions. 

Alto’s focus on the customer experience was immediately clear, and I made a mental note that this was the type of company I’d like to work for. Fast forward a few months later, a recruiter reached out and introduced me to the leadership team whose passion and experience sold me on leaving Amazon to come here.

As the Director of Product Management - Supply Chain and Fulfillment, I'll focus on the technology investments that can help us continue to grow and offer exceptional service to our customers. “Supply chain” covers a lot of ground: procurement, inventory, pharmacy operations, delivery, and more. There are tons of opportunities to automate processes to make it easier for Altoids to run the business, especially as we grow and expand. 

What areas of the business are you most excited about?

The operations side has always been exciting to me—it's where the magic happens. At the end of the day, if we don't have sustainable, efficient ways to fulfill the promises we make to our customers, we won't survive. 

Alto has a great business model, and many happy customers to show for it, but there’s always room to continue to innovate and improve the delivery experience. It's one of the few in-person interactions we have with the customer and we want to make it exceptional.

Tell us about life outside of Alto. Are you doing anything interesting at the moment?

I'm a pilot in training — I'm about 75% of the way to getting my single-engine license. I live in Seattle so the aerial views of the city, Puget Sound, and the Cascade Mountains are absolutely breathtaking. Flying gives you a sense of freedom that reminds me of when I first got my driver's license.

You can find more information about Christopher or connect with him directly on LinkedIn. If you have any questions about Alto, or are interested in joining our team, please visit alto.com/careers.

The Next Evolution of Our Company Values

A lot has changed at Alto since Matt and I first set out to transform the pharmacy experience back in 2015. (For one thing, we no longer work out of a storage closet.) In the past two years alone, we’ve quadrupled our team to more than 1,000 Altoids and opened new locations in Denver, Las Vegas, Seattle, New York City, Houston, Dallas, San Jose and San Diego.

Our mission — improving the quality of life for everyone who needs medicine — will never change, but a team of 1,000 needs a different recipe for success than one of 100. And as our team continues to grow, along with our geographic footprint, we’ve revisited our core values so that we can move forward with intention and clarity during this next phase of Alto’s journey.

Here are the values that guide our work today.

This was at the top of our founding values, and it’s still here. In fact, our resolve to put patients first has only gotten stronger — not because it’s what is best for the business, but because it is our only reason for being in business. Each member of our team strives to build a world where healthcare companies truly prioritize the needs of their patients, and we’ll do everything in our power to make that a reality.

My personal favorite of all our new values, dancing in the storm is about embracing the bumps in the road — and there are many when you’re taking on challenges within healthcare — and recognizing that you can always choose joy. We strive to help our teammates do the same by infusing a little bit of levity and laughter into the workday. It only takes one person to start a dance party!

Humility is not thinking less of yourself, but thinking of yourself less. Nor is it the absence of strong opinions, but the willingness to learn and change our minds. Sometimes our ego takes over — that’s only human. But we try to be mindful of it, and willing to let go of our ego and instead put our mission, customers, and teammates ahead of ourselves.

We view ourselves as creators of our circumstances, responsible for our actions and our own well-being. When we don’t like an outcome, we don’t project blame onto others or ourselves, but rather take radical responsibility for how we created the situation. More than any other value, this is an opportunity for deep internal reflection, and by practicing it consistently, we can maximize our own individual growth.

We say what is true for us in the moment, even when it is uncomfortable. We support our teammates to do the same by recognizing the validity of their experience and not taking it personally. Leading with candor isn’t always easy, but it feels much better to live authentically and honestly and to work with others who do as well. Candor doesn't have to be angry or hurtful - it can be kind and binding, compassionate and respectful.

It’s easy to become paralyzed by fear, overwhelmed by all of the possibilities in a complex situation. So unless a decision involves customer or employee safety, we don’t wait for perfect information. We trust our intuition and common sense and simply forge ahead, knowing that mistakes are in fact one of the best ways to learn.

There are no shortcuts or silver bullets in an industry as complex as healthcare. The way to achieve transformational change is through continuous improvement over a long period of time. When we’re overwhelmed by the magnitude of what we’re trying to accomplish, we remember that what each of us does adds up — one step at a time, one customer at a time — and that with small but steady progress we can accomplish great feats.

As a company grows, issues no longer affect a few customers, but a few thousand. It’s tempting to only look at trends and aggregates, but we know this is a trap. Greater distance from individual examples creates more opportunity for misunderstanding, bias, and incorrect assumptions. So we look beyond surface-level data and dive deep into the details. Only then can we fully understand an issue to its core.

Values themselves are just words, and the words themselves are not that important. What’s important is how we live them day in and day out, and the culture they support. In the months ahead, we’ll continue to integrate these values into the fabric of our company processes, our products, our customers’ experiences, and our work lives. And this work is not something just a few leaders or HR team members do - every Altoid has a hand in bringing these values to life and co-creating our culture.

Build Tomorrow’s Pharmacy with Us — Join Our Team

If you are passionate about creating a great company culture, join us in delivering our industry-leading pharmacy care to millions of people across the country. As a member of the team, you’ll play an integral role in improving our customers’ quality of life and redefining what’s possible in pharmacy.

View open roles in design, engineering, IT, sales, patient care, and more.

The Providers Team at Alto is charged with building the tools and service components necessary to deliver an exceptional pharmacy experience for healthcare providers. Read on to learn more.

Get to Know the Team That Has Doctors Calling Alto “The Pharmacy of The Future”

Editor's Note: The post was updated on November 8, 2021 to reflect the rebrand of AltoMD to Alto Connect.

Alto Pharmacy’s mission is to fulfill medicine’s true purpose — to improve quality of life — for everyone who needs it. The healthcare professionals who prescribe medications to Alto and the clinical teams that support them are critical partners in making our mission successful. To this end, the Providers Team at Alto is charged with building the tools and service components necessary to deliver an exceptional pharmacy experience for healthcare providers.

Every day, clinical teams -  including prescribers and other clinical staff like nurses, practice coordinators, and billing specialists - spend hours tracking down insurance information, coordinating approvals and appeals, discussing options with their patients, and playing phone tag with pharmacies. The Providers Team enables these healthcare providers to spend more time on patient care and less time on back office tasks. 

Alto’s provider communications platform, Alto Connect, and related services, including insurance benefits coordination and prior authorization management, remove clinical blockers that traditionally slow down or prevent an individual’s access to treatment. Practices that share Alto with a majority of their patients can save three or more hours per day on pharmacy and insurance administration tasks, with less time spent waiting on hold, fewer faxes and improved employee satisfaction (not to mention happier patients!). 

Anu Kashyap, a Product Manager on the team says, “I joined Alto to make a contribution to people’s health using technology, and the Providers Team is working on specific, painful healthcare problems that are critical to solve. These challenges have been incredibly fulfilling to work on.”

How does the Providers Team play into the overall growth of Alto Pharmacy?

The Providers Team is fairly new, as they were originally part of a larger Growth team within Alto. Kjersten Elias, Engineering Manager, explains: “Alto's growth model is driven by conversations about pharmacy choice in the doctor’s office, and in the fall of 2020, we dedicated an engineering team to optimizing our product offerings and improving the experience for the healthcare providers having those conversations.”

Anu adds, “The key for our business’ continued growth is having healthcare providers trust that we are the best pharmacy for their patients and leverage us as their pharmacy of choice. This type of growth channel is much more powerful than anything direct to the consumer. We can only do this however, if our providers have an excellent experience with us, and we’ve only scratched the surface here.” 

Being part of a newer team has distinct advantages. Software Engineer Dereck Quock notes, “We have unique opportunities to modernize how we build apps and define our infrastructure and technical needs. With the freedom to utilize cutting edge tech and new frameworks, there is so much room for innovation.”

What is the culture of the Providers Team like?

The engineers each raved about a small-but-important recurring calendar invite each Friday: Team Time. This protected time is a space to catch up, banter about anything other than work, and, if folks are feeling brave, face off with Anu, former Teen Tournament champion, in a rousing game of Jeopardy.

It's not hard to see why Kjersten and her team light up when they talk about team culture. Amy Chen, a Software Engineer on the team, shared that  “Kjersten is an incredible manager. Everyone is empowered to ask for and share feedback, fine-tune our processes, and check in with each other.”

“When I joined, I could tell right away this team really lifted each other up and had each other’s backs,” said Dereck. “It’s one of the greatest teams I’ve worked on.”

The Providers Team is hiring! Learn more about open positions here.

Get an inside look at the patients team at Alto Pharmacy to learn what it's like to work on Alto's customer-facing apps, which projects are on the horizon, and who makes the best barbecue short ribs.

Get to Know the Team That Builds Alto's Customer-Facing Apps

Imagine an app that allows you to buy things and have them delivered to you the same day, for free. Now take that app and put it in a highly regulated space like pharmacy. You face the challenge not only of building an e-commerce platform from scratch, but also of navigating medication pricing, federal and state regulations, and patient safety—and that's just the way the Patients Team likes it.

"It's a really exciting and fun problem to solve," says Mike Riess, the team's Engineering Manager. Software Engineer Angelique Nehmzow agrees: "There's a really strong product and design aspect, like more traditional startups, but then everything gets a little bit more interesting because it's healthcare."

Below, we chat with the team to learn what it's like to work on Alto's customer-facing apps, which projects are on the horizon, and who makes the best barbecue short ribs.

What does the Patients Team do at Alto? 

Ask anyone on the team why they joined, and chances are, they'll tell you they wanted to be on the product team closest to Alto's customers: "We really try to own the customer experience," says Peter Zeng, the latest Software Engineer to join the team. "If a customer is coming to the web app, or they're coming to our mobile app, we want to make sure that they're able to onboard with us as easily and conveniently as possible."

Blog

Why We Switched to React Query for Managing Server State

Tackling Technical Debt and Improving Security: How Alto Untangled an Overloaded User Model

Considerations and Approaches for Implementing Biometric Login: Part 2

Introducing Alto Connect

How Alto Reduced Time to Resolution for Production Incidents

Introducing Christopher Tidball, Director of Product Management - Supply Chain & Fulfillment

The Next Evolution of Our Company Values

Get to Know the Team That Has Doctors Calling Alto “The Pharmacy of The Future”

Get to Know the Team That Builds Alto's Customer-Facing Apps

How to Make the Most of the Alto Pharmacy App

Béquet Confections: The Story Behind Alto Pharmacy’s Caramel

Alto Pharmacy Announced as One of Forbes Best Startup Employers

Switch to a better pharmacy.