Your partner in navigating the ins and outs of prescriptions — at no extra cost.

Alto Pharmacy Technician

Alto Pharmacist

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community ofGreat Place to Work-Certified™ companies based on team member feedback and a rigorous assessment of our company’s culture and benefits.

Join us in delivering our industry-leading pharmacy care to millions of people across the country. As a member of the team, you’ll play an integral role in improving our customers’ quality of life and redefining what’s possible in pharmacy.

Be Part of a Great Place to Work — Join Our Team

Hear first-hand from members of the Alto technology teams.

Read the latest Alto blog posts covering healthcare, family health, managing diseases, and other Alto highlights.

Blog | Alto Pharmacy

The Providers Team at Alto is charged with building the tools and service components necessary to deliver an exceptional pharmacy experience for healthcare providers. Read on to learn more.

Get to Know the Team That Has Doctors Calling Alto “The Pharmacy of The Future”

Alto Pharmacy’s mission is to fulfill medicine’s true purpose — to improve quality of life — for everyone who needs it. The healthcare professionals who prescribe medications to Alto and the clinical teams that support them are critical partners in making our mission successful. To this end, the Providers Team at Alto is charged with building the tools and service components necessary to deliver an exceptional pharmacy experience for healthcare providers.

Every day, clinical teams -  including prescribers and other clinical staff like nurses, practice coordinators, and billing specialists - spend hours tracking down insurance information, coordinating approvals and appeals, discussing options with their patients, and playing phone tag with pharmacies. The Providers Team enables these healthcare providers to spend more time on patient care and less time on back office tasks. 

Alto’s provider communications platform, AltoMD, and related services, including insurance benefits coordination and prior authorization management, remove clinical blockers that traditionally slow down or prevent an individual’s access to treatment. Practices that share Alto with a majority of their patients can save three or more hours per day on pharmacy and insurance administration tasks, with less time spent waiting on hold, fewer faxes and improved employee satisfaction (not to mention happier patients!). 

Anu Kashyap, a Product Manager on the team says, “I joined Alto to make a contribution to people’s health using technology, and the Providers Team is working on specific, painful healthcare problems that are critical to solve. These challenges have been incredibly fulfilling to work on.”

How does the Providers Team play into the overall growth of Alto Pharmacy?

The Providers Team is fairly new, as they were originally part of a larger Growth team within Alto. Kjersten Elias, Engineering Manager, explains: “Alto's growth model is driven by conversations about pharmacy choice in the doctor’s office, and in the fall of 2020, we dedicated an engineering team to optimizing our product offerings and improving the experience for the healthcare providers having those conversations.”

Anu adds, “The key for our business’ continued growth is having healthcare providers trust that we are the best pharmacy for their patients and leverage us as their pharmacy of choice. This type of growth channel is much more powerful than anything direct to the consumer. We can only do this however, if our providers have an excellent experience with us, and we’ve only scratched the surface here.” 

Being part of a newer team has distinct advantages. Software Engineer Dereck Quock notes, “We have unique opportunities to modernize how we build apps and define our infrastructure and technical needs. With the freedom to utilize cutting edge tech and new frameworks, there is so much room for innovation.”

What is the culture of the Providers Team like?

The engineers each raved about a small-but-important recurring calendar invite each Friday: Team Time. This protected time is a space to catch up, banter about anything other than work, and, if folks are feeling brave, face off with Anu, former Teen Tournament champion, in a rousing game of Jeopardy.

It's not hard to see why Kjersten and her team light up when they talk about team culture. Amy Chen, a Software Engineer on the team, shared that  “Kjersten is an incredible manager. Everyone is empowered to ask for and share feedback, fine-tune our processes, and check in with each other.”

“When I joined, I could tell right away this team really lifted each other up and had each other’s backs,” said Dereck. “It’s one of the greatest teams I’ve worked on.”

The Providers Team is hiring! Learn more about open positions here.

Get an inside look at the patients team at Alto Pharmacy to learn what it's like to work on Alto's customer-facing apps, which projects are on the horizon, and who makes the best barbecue short ribs.

Get to Know the Team That Builds Alto's Customer-Facing Apps

Imagine an app that allows you to buy things and have them delivered to you the same day, for free. Now take that app and put it in a highly regulated space like pharmacy. You face the challenge not only of building an e-commerce platform from scratch, but also of navigating medication pricing, federal and state regulations, and patient safety—and that's just the way the Patients Team likes it.

"It's a really exciting and fun problem to solve," says Mike Riess, the team's Engineering Manager. Software Engineer Angelique Nehmzow agrees: "There's a really strong product and design aspect, like more traditional startups, but then everything gets a little bit more interesting because it's healthcare."

Below, we chat with the team to learn what it's like to work on Alto's customer-facing apps, which projects are on the horizon, and who makes the best barbecue short ribs.

What does the Patients Team do at Alto? 

Ask anyone on the team why they joined, and chances are, they'll tell you they wanted to be on the product team closest to Alto's customers: "We really try to own the customer experience," says Peter Zeng, the latest Software Engineer to join the team. "If a customer is coming to the web app, or they're coming to our mobile app, we want to make sure that they're able to onboard with us as easily and conveniently as possible."

The team is also responsible for the customer experience beyond onboarding, from the moment a customer schedules a medication for delivery to the time that it shows up on their doorstep. "We already offer great services like free, same-day delivery and cost savings. The Patients Team empowers customers to make the most of our offering," Nehmzow explains. "You can see what prescriptions you have, check on the status of your order, and schedule automatic refills. In a more traditional pharmacy, you don't get that kind of visibility. You have to call them up or even wait in line in person. At Alto, it's a different experience."

What was it like in the team's early days?

Today, there are eleven engineers on the team working to build that better end-to-end experience—but it wasn't always that way. When the team first formed in 2019, it consisted of just three members: Riess, Brian McRae, and Andy Lau, all of whom are still on the team.

"The first priority for us was just stabilizing the foundations," Riess says of the team's early days. Every other week, the three team members, along with Alto co-founders Matt Gamache-Asselin and Jamie Karraker, would meet to audit the app's core flows. "For the first three months we probably didn't make it through some flow at least 50% of the time," Riess recalls. "There would be a ton of bugs in the onboarding experience, things that weren't working with the flows or navigation or deep linking."

Among the team's first projects was a redesign of the checkout flow. "There was a lot of work to be done to streamline the checkout flow, which is pretty much the main function of the app," says McRae, who initially joined the team as a software engineer and later transitioned into product management. "We had to make it easier to add a payment, add your addresses, everything you need to do to schedule your prescription for delivery."

"The growth of the team, in terms of both size and stability of the platform, has allowed us to swing at more interesting bets for the company," Riess says. In particular, the Patients Team has turned their attention toward experiences personalized for specific medical conditions.

"Different customers have different needs. A customer who wishes to start a family will have a different experience than a customer who needs to order their routine heart medications that they've been taking for the last eight years,” McRae says. “The challenge for us is how we serve all these customers well through the app."

"I think that's where Alto can be such a differentiator," Riess adds. "A world-class pharmacy experience is about so much more than just dispensing medications. We want to build tools for customers that help them get the most out of their medications—everything from simple videos teaching them how to perform injections safely at home to the ability to synchronize deliveries for multiple prescriptions for managing complex health conditions."

With these new projects on the horizon, engineers have ample opportunity to focus on the areas that excite them most. "We're full-stack and mobile and web," Nehmzow says. "Part of the team focuses more on experimentation, iterating quickly, and using data to make decisions. Part of the team focuses more on platform work, tooling, and improving the developer experience. There are a lot of areas to explore and contribute to."

One spring night, the Patients Team gathered in a back yard in San Francisco’s Mission District for short ribs and an impressive spread of side dishes. "Kudos to Robbi, who's the grill master. I just provide the grill!" Nehmzow says, laughing. While the team barbecue was the first time some members had met in person, Zeng notes that the team has been supportive of one another even while working remotely during the pandemic.

"The first real project that I was assigned to was working with the communications system. Before writing my first piece of code, I spent two weeks exploring what already exists and understanding the different parts of the system," Zeng says. "I've been really impressed with how understanding my teammates have been. They make the space for you to learn all the different pieces of code we own."

That culture of support is by design, reflected in the six principles—complete with matching emojis—at the top of the #patients-team Slack channel: "Empower each other," "be an owner," "find the path," "learn by doing," "embrace complexity," and "foster trust." 

While these principles echo Alto's company-wide values, they are unique to the Patients Team. "For a long time, we were the only external-facing product team at Alto," Riess says. "These principles were intended to be this motivation around how we win. It's a combination of how we treat each other, how we reason about decision making, and how we approach solving a problem that's never existed before."

As the team looks toward the future, those principles have never been more relevant: "We have a goal and an area we want to focus on, but in terms of which projects we're going to do, it's not a top-down thing," Nehmzow says. "It's very much, 'We're going to figure out how to solve this problem together, even when it's difficult.' That's my favorite part about the team."

The Patients Team is hiring! Learn more about open positions here.

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies.

Alto Pharmacy Named Great Place to Work®

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies based on team member feedback and a rigorous assessment of our company’s culture and benefits. This important recognition is a testament to the dedication of the entire Alto team and the vision of our founders to create a healthcare company truly anchored on the needs and experiences of the customer.

80% of our team says that Alto is a great place of work, compared to just 59% of employees at  other companies that participated in this year’s assessment. Here are other key takeaways from our team’s’ responses:

93% of our team members say that you are made to feel welcome when you join Alto.

88% say that people care about each other here.

86% say that they are able to take time off from work when needed.

Every company has its own unique culture, but there are two things that all great workplaces share: trust in company leadership and a strong sense of pride in the work. For many Alto team members, being part of a mission-driven company is one of the best parts of working here. 

Our mission — to improve the quality of life for everyone who needs medicine — guides us in every business decision, every day. From fostering an inclusive culture that invests in talent from the same communities as our patients, to building a pharmacy model that meets individuals where they are in their health journey, everything we do at Alto is anchored on delivering meaningful change. 

When asked what makes Alto a great place to work, our team members emphasized our guiding company values — including our commitment to putting patients first and leading with empathy — and getting to spend each workday with colleagues who are also inspired and motivated by those values.

“Our bold, patient-centered mission creates a powerful level of empathy across the entire team,” shared Rob Anderson, Alto’s Head of Talent . “The people who work here are one of the best parts of coming into work, and one of the top reasons our workplace is so rewarding.”

And taking care of our team members is just as important as ensuring we leave no patient behind. In response to the past year’s unprecedented challenges, we introduced Alto Recharge Week, which includes expanded time off for all team members. This reset period allows team members to unplug and recharge, and it’s also a small show of  appreciation for how they supported our patients and healthcare providers throughout these challenging times.  

It’s an exciting moment to be part of Alto’s mission. Since our founding in 2015, we’ve been focused on delivering a totally new model of pharmacy care. Our community-based, courier-first pharmacy model simplifies the entire prescription process and is tailored to each patient’s unique needs, leading to better  outcomes, healthier lives, and happier customers. And with every year, Alto’s impact continues to grow:

More than 36 million individuals now live within Alto’s free same-day courier service zones. 

In the past twelve months, we opened new locations in New York City, Houston, Dallas, and San Jose and expanded our coverage areas in Seattle, Las Vegas, and Denver. 

We've expanded our specialized resources for patients living with diabetes and managing heart and lung conditions .

We’ve accomplished so much over the past six years, and we’re still just getting started.

“I don’t know a single person who hasn’t needed something from a pharmacy at some point, and there’s still so many people across the country who have yet to experience our unique community pharmacy experience,” Alto’s Recruiting Manager Lauren Binet said. “That potential for impact is incredibly motivating.”

As our geographic footprint has grown, so has our team. In the past two years alone, we’ve quadruped our team to more than 1,000 Altoids!

View open roles in design, engineering, IT, sales, patient care, and more.

About Great Place to Work Certification™

Great Place to Work® Certification™ is the most definitive “employer-of-choice” recognition that companies aspire to achieve. It is the only recognition based entirely on what employees report about their workplace experience – specifically, how consistently they experience a high-trust workplace. Great Place to Work Certification is recognized worldwide by employees and employers alike and is the global benchmark for identifying and recognizing outstanding employee experience. Every year, more than 10,000 companies across 60 countries apply to get Great Place to Work-Certified.

All third party trademarks (including logos and icons) referenced by Alto Pharmacy remain the property of their respective owners. Unless specifically identified as such, Alto’s use of third party trademarks does not indicate any relationship, sponsorship or endorsement between Alto and the owners of these trademarks.

In recognition of Women’s History Month, we highlight four women at Alto shaping the future of pharmacy. These leaders share their advice for women in the workplace, their unique super powers, and what they appreciate most about Alto's culture.

Celebrating Women at Alto Pharmacy

Liz Kelly, National Sales Manager Fertility 

I'm responsible for hiring, coaching and supporting the needs of the national fertility sales team. I also work closely with fertility business and operations teams to deliver on our clinic and customer promises.

Before joining Alto, I was finishing up a sabbatical where I spent my time volunteering on the board of a non-profit called ServeHAITI. Prior to that, I was the West Coast Area Business Director at EMD Serono and worked with Matt and Jaime [Alto’s co-founders] to secure a contract to distribute fertility meds when they were still working out of AG Pharmacy!

What do you appreciate the most about Alto's culture? 

Hands down my favorite part of being on the Alto team is that the patient is truly at the center of every decision we make. I value that we make business decisions with our mission in mind.

What do you like doing outside of work?  

Of late, my favorite pastime is heading to the dog beach with my rescue, Bodhi, who has been the best medicine to ease some of the COVID-19 isolation I experienced over the past year. Before the pandemic, I made sure to travel to at least one new international spot a year. I love the experience of getting exposed to all things about another culture: cuisine, culture, belief systems, language.

I like to think that my super power is seeing the greatness in others, sometimes before they see it in themselves.

I aspire to live by the motto "be the change you wish to see in the world." I encourage women to speak up for what they believe is right, even if nobody else agrees. That includes planning for your own career path and making it a priority to develop your skill sets so that you are prepared to climb the ladder that serves your own greatness.

I graduated from pharmacy school in May of 1979 and started working at AG Pharmacy after passing my boards later that year. I bought AG in 1998 when the previous owner wanted to retire - he had been with AG since it first opened in the early 60's.

My family moved to San Francisco from Hong Kong when I was five years old. Our story is one similar to those of many immigrant families who came to America looking for a better life in search of the American dream: own a home, have a car, raise children safely and give them the opportunity of a good education and a chance to have a better life. My parents raised their children here, emphasizing family, hard work and education. They lived their American dream, and I have them to thank for all that I am today.

What’s your favorite thing to do outside of work? 

I love to travel. There are so many unique places to visit in this world. With the different cultures, the food, and the amazing sceneries/landscapes, there are endless things to see and experience! 

The thing I appreciate most about Alto's culture is that everyone works so hard together to bring about the best pharmacy experience for all our customers. You can feel the energy and the motivation of all the Altoids. I cannot emphasize how amazing it is to be a part of this team and to watch Alto grow and evolve into the pharmacy of the future and the pharmacy of our dreams!

What’s your favorite memory thus far at Alto? 

In the very early days, when Alto was still Scriptdash Pharmacy, a customer needed to get her trigger shot [as part of a fertility protocol] within the next few hours. It was close to rush hour, and we had no couriers available to deliver to the South Bay. Our CEO Matt came to the rescue on his motorcycle, and off he went to make the delivery. It was such a wonderful and warm feeling to watch and cheer him on together. That is truly what Alto love is all about!

I manage our amazing IT Support team, helping all Altoids get the technology support they need, when they need it. 

For the better part of the last decade I have been managing IT Support teams. From Facebook and Twitter to Lyft and Uber, I’ve seen it all (or at least most of it!).

Our incredible dedication to our mission. In my short time here, I truly see us putting patients first, and it’s moving to see. 

What do you like doing outside of work? 

I’m a big traveler, but since COVID-19 has put a damper on that, I enjoy flying around Colorado with my partner in our vintage Beechcraft Bonanza. I also love hopping on my motorcycle and just exploring! 

I’m in love with Central and Eastern Europe. Some of my favorite places are Prague, Vienna and Budapest. 

Elnaz Moshfeghian, Senior Engineering Manager

I’m originally from Iran but moved a lot growing up, including Oklahoma, Australia, Qatar, Kuwait, and Chicago. I have been in San Francisco for the last six years. I stumbled across software engineering in college and found a passion for building products and teams. I joined Alto as the third engineer and this is now the largest company I’ve ever worked at. 

I lead a group of 20 engineers across three teams. Two teams are focused on getting prescriptions ready to schedule quickly and accurately, and one is focused on enabling faster, higher quality, and more enjoyable product development across the broader product and technology organization. My work involves a lot of collaboration, creativity, transparency, and flexibility. 

I was an early software engineer at a small startup; before that I was a hybrid software engineer and product manager at a design and development studio. 

I like how passionate everyone is about what they do. 

I’m at my best managing a crisis or finding a path through conflicting strong opinions. 

I want to start with some advice for our allies: get more curious about the experience of the women you work with and learn about ways you can support and amplify the work of the women around you. Diversity, equality and inclusion is a team sport and, even if we win a game or two, it’s about the continuous pursuit of excellence. 

For my fellow women at Alto, I wish I knew earlier to focus more on improving my craft and impact so I can get a seat at the table, rather than trying to juggle advocacy and my own work. Once you’re at the table, you can really advocate for hiring, growing, and retaining women. 

To learn more about the work of our six employee resource groups (ERG’s) focused on building community - including Umoja (Black), Auténtico (LatinX/Hispanic), Women (WOAA), Proud (LGBTQIA+), Asian/Pacific Islander, and Working Parents - read our 2020 Diversity, Equality and Inclusion Report. 

Are you passionate about shaping the future of pharmacy? Join us as we deliver our industry-leading pharmacy care to millions of Americans. 

At Alto, we’re obsessed with our mission to improve quality of life for everyone who needs the help of medicine. Our mission guides us in all we do and every decision we make. 

Our mission and diversity work are tied together with one word from our mission -- everyone. Too often, only those with significant resources access the best and benefit from innovation in healthcare. We see it everyday in the data. Too many among us continue to struggle with solvable challenges, in our own wellbeing and in the larger healthcare system. These statements and environments have existed in the workplace as well, influencing the products, services, and policies that steer how the organization works for all of us.  Our work is to make Alto different.  Our work is to build and sustain an inclusive culture.  Our work is to attract, develop, reward and retain talent from all the communities we serve as patients and build a product that supports everyone. 

Today marks another step in our diversity, equity, and inclusion journey as we release our first annual diversity report. We share it because this work is central to our mission and in doing so, we use transparency to hold ourselves accountable, as well as hopefully play a part in raising expectations and norms throughout the healthcare and technology industries.

In 2020, across the company we saw a 13ppt increase in women and 7ppt increase in underrepresented groups (Black, Hispanic/LatinX and Native professionals).

As for women in leadership, we saw increases of 3ppts in Women Managers and 1ppt in Product, Engineering and Design driven by both hiring and promotion. At the leadership level, 32% of leaders identify as women.

In 2020, we saw 10ppt increase across our underrepresented managers to 13% Black, 16% Hispanic/LatinX, 2% Native/Indigenious.  In the leadership group, we saw an 8ppt increase to 15% Black and 2% Hispanic/LatinX. 

In 2020, we also established our Alto Advisory Board and our first two anchor members, who will help guide Alto towards our mission, advise on key decisions, and support strategic initiatives. We’ve formed the Advisory Board to support our leadership as we navigate the regulatory landscape, launch new features to modernize the pharmacy experience, and adhere to the highest standards of operational excellence and patient care.  

Dr. Hildreth is the President and Chief Executive Officer of Meharry Medical College, one of the nation’s oldest and largest private historically Black academic health sciences centers. He received his Bachelor’s degree from Harvard University, his Ph.D. in immunology from Oxford University as a Rhodes Scholar, and his medical degree from Johns Hopkins School of Medicine. He was the first Black American in the 125-year history of Johns Hopkins School of Medicine to earn full professorship with tenure and is internationally known for his influential HIV research.

Jennifer Kent served as the Director for the California Department of Health Care Services, where she led the second largest public healthcare system in the nation, with an annual budget of over $100B and serving over 13 million Californians. She oversaw the administration of 24 managed care plans as well as 56 county behavioral health plans. She has also held leadership roles in the Health & Human Services Agency, Governor's Office and has experience in the private sector as a lobbyist, consultant, and association executive. She has more than 15 years of health administration and policy experience in California.

Alto created our DEI strategy with our team aligned to four pillars – Hire, Include, Develop, and Invest.  Each pillar's actions tie to our people and our business. This means the return on investment is clear and goals are aligned to business outcomes.

Medical Health Care Premiums and Part Time Health Care Benefits in SF: 

In order to increase access and affordability to critical health-care services, we asked our broker to offer a special mid-year open enrollment 2020, reducing premiums for most plans by 5%. We reduced out-of-pocket maximums and speciality office visits by an average of 37%. We reduced co-insurance as well as costs for labs, x-Rays, and emergency room visits for our Cigna plan by 50%. Finally, prescriptions copays were reduced by 25% for Brand medications and 75% for Generic prescriptions. This benefits the employee with direct access to care and a better employee experience as compared to direct contribution to the city fund.

We increased our paid parental leave across all US locations to 8 weeks (up from 6 in 2020) for all parents requesting leave as of January 1st.  Throughout 2020 parents and other caregivers were impacted significantly as schools, daycare, and caregivers were closed, isolated, or unable to rely on pre-covid support. We offered laptops to parents who couldn’t afford them for each child to participate in remote learning.  We offered expanded benefits above and beyond FFCRA. Finally, we offered paid sessions for remote learning, tutoring, and experiences for parents and caregivers.

Finally, in December we launched Modern Health with 6 paid coach/therapy sessions and access to a digital wellbeing library and resources to all Altoids.

We launched six ERGs in 2020. Formalizing the work of our original Diversity Committee into six groups: Umoja (Black), Auténtico (LatinX/Hispanic), Women, Proud (LGBTQIA+), Asian/Pacific Islander, and Working Parents. These groups are focused on building community with both those who identify and allies. They support our education, development and investment in our communities. Each has been given a budget, an executive sponsor, and access to other company resources.

We recognized as a leadership team that we needed to educate ourselves. We were not going to rely on those from underrepresented communities to educate us. We embarked on four three-hour sessions focused on learning about diversity and inclusive language, naming our privilege, and starting the work each of us wanted to do. It was only after these sessions, with clear self-awareness and awakening; that began the work of empowering other leaders -- managers and other Altoids on this journey. There is more work to be done, as Neil Frye, Chief People Officer, mentioned in his blog post in late 2020 -- this is not a sprint, nor is it a marathon; it is life-long work.

Focus on Latinx Senior Manager to VP talent; with targeted outreach, activation and networking for opportunistic and open hiring

Expand employee resource group events throughout the year supporting inclusive environment, educating and investing in our communities

Get to full participation in DEI Workshops and launch additional sessions like What’s My Privilege and Moving to Action across the company

Explore new partnerships with Community & Higher Ed Institutions (i.e. Historically Black Colleges and Universities, Hispanic Serving Institutions, etc.)

2020 Diversity, Equity & Inclusion Report

Justin Moy, Senior Software Engineer at Alto, explains how the engineering team developed a staging environment strategy that could keep up with our growing organization.

Creating Sandboxes: How to Make It Easy for Your Engineering and Product Teams

When we formed our Infrastructure team in 2020, our charter centered on scale, reliability, and the developer experience. Our first priority? Developing a staging environment strategy that could keep up with our growing organization.

Initially, we had a set of command-line tools for staging environments that could be run from an engineer’s workstation. However, this approach required significant manual effort to create and manage an environment which led to inconsistencies like outdated data and lack of full application coverage and integration. The required workstation dependencies and maintenance added to the overhead. These tools were also different from the ones used to create a production environment, causing drift between the two.

To address these issues, we focused on improving our tooling to create consistent and reproducible results for both our staging and production environments. Here are the five steps we took to get there:

Luckily, we were already here. Our deployment process involved creating and deploying Docker containers and using a Procfile to run specific parts of our applications (think separating out web and workers). Packaging our applications as immutable Docker images allowed us to ship consistent and portable code. These images are run as containers which can be executed on separate hosts and scaled independently of each other.

We wanted to increase our observability, mature our internal monitoring, and maintain our infrastructure in a more consistent and reproducible manner. Kubernetes, and its automated container orchestration (deployment, scaling, and management), allowed us to accomplish all of the above goals.

It was easy for us to get started in Kubernetes thanks to Kind, which allowed us to run clusters locally. This increased development speed and reduced the friction of learning a new external provider.

Helm makes it easy to manage packaged applications in Kubernetes. Some community-provided applications include: NGINX for load balancing and reverse proxying HTTP and other traffic; Prometheus and Cortex for metrics, monitoring, alerting, and dashboarding, Loki for logs; and Sentry for exceptions.

In order to ease the integration with other infrastructure components, enable rapid deployment, and, most importantly, provide us with the consistency and reproducibility we wanted, we created and deployed Helm charts for our applications.

From here, we could stamp out our applications for developer use in sandbox environments: complete end-to-end environments, separate from production, that enable engineers to rapidly develop and test their code. And if an engineer is working on multiple features, they can spin up multiple sandbox environments and manage their applications and data independently of each other. Victory!

We had already created pipelines in Concourse to automatically deploy and upgrade our infrastructure Helm charts. To eliminate the manual toil of spinning up new environments and to reduce the friction around sandbox adoption for our engineers, we extended these automated pipelines by building a Git-driven continuous deployment pipeline. When an engineer created a commit to a repo defining the state of a sandbox, our Concourse job would apply these changes.

At this point, we could deploy environments that resembled production but were separate from production and other sandbox environments. They connected to the rest of our observability stack, including metrics, logs, traces, and exceptions.

As we prepared to scale to the rest of the product and engineering organization, we focused on the following ideas: further reducing manual toil, removing barriers to entry, reducing cost, and optimizing the development workflow. The diagram below shows our sandbox architecture.

In Kubernetes, an operator acts on custom resources in order to automate a task or set of tasks. The custom resource allows us to create a well-defined entity and extend configurability options to users. The operator allows us to customize individual sandbox environments based on changes to the custom resource.

We utilized Kubebuilder to create our operator and CRD (custom resource definition). Kubebuilder was a great starting point and provides tools to ease rapid development of a new operator.

Using this pattern, we could easily perform CRUD operations (create, read, update, delete) on a custom resource using standard tools like kubectl, and the operator would handle updating our deployments in Kubernetes. These updates could include: updates to application configuration, deploying a new Docker container, injecting different secrets, etc. This flexibility allows for testing of many different scenarios like: developing a new feature, reproducing a bug, upgrading frameworks, and security and load testing.

Increase adoption by removing barriers to entry (admin web UI)

We built a web UI so users would be able to deploy sandboxes intuitively and easily without extensive knowledge of the underlying tools. The web UI interacts with our custom resource and lets the operator handle updates. It allows our teammates—both engineers and non-engineers—to easily spin up their own environments for development, demo, or training purposes without modifying production data.

In addition, we show helpful information about a sandbox to a user. This includes links to each of our applications; pre-filtered logs, metrics, tracing, and exceptions; and commands to connect to specific applications to ease debugging.

Because the sandbox environments are integrated with our monitoring stack, we have the full metrics and logging that we expect from our applications. We created an autoscaler service that performs queries against this information and scales our sandbox environments up and down. We’re able to decrease our costs by over 90% when these environments are not in use and are able to quickly scale them back up when they are being used.

Optimize the development workflow (image updater)

An additional feature we built into our sandbox environment is the ability to follow a feature branch in our applications. When an engineer pushes a commit to a pull request in Github, a corresponding sandbox will automatically update to this commit. This reduces several manual steps in the development workflow and allows engineers to test their code in near-to real time.

Sandbox environments have made an impact on the engineering organization and beyond.

Engineers can quickly desk check new code in a production-like environment during code reviews.

Shared sandboxes reduce the blast radius of bug bashes.

Non-engineers can spin up sandboxes to demo or provide training on features without impacting production data via the Admin UI. This has proven particularly useful for Product Managers and Designers on the team to be able to play with product features and better understand how they work in order to suggest improvements.

Teams have built use case scenarios into their applications. This allows easy reproducibility of critical paths and functionality.

With these five steps—Containerize, Kubernetes-ize, Helm-ify, Automate, Optimize—we’ve laid the foundation for deploying all of our environments in an automated and consistent manner. In addition to applying many of our learnings to improve our production environment, we successfully built core infrastructure that our engineers, product managers, and designers use each month to test their changes and reduce production downtime.

Interested in learning more about engineering at Alto? Follow us on LinkedIn. 

In this post, we'll share how we identified the problem of failing cypress seeds, what the culprit was, and three ways to make seeding test data more robust.

How to Debug Slow and Failing Cypress Seeds

End-to-end test failing before it even begins? Here’s how we timed, debugged, and sped up our seed data.

At Alto, we use Cypress to test our Rails application end-to-end. As our application grows, so does our end-to-end test suite — along with the seed data it requires. This led us recently to a perplexing problem:  when run in CircleCI, our end-to-end tests would mysteriously time out during the seed step, causing them to fail and block deploys.

Seeds can be painful to debug. They’re typically set up through a single request and, depending on how thorough the tests are, call upon most of the models involved in the application. If a model throws, say, a validation error because of bad seed data, you'll see it in your Cypress logs. If a model hangs, the request often doesn't show up in the logs at all. There's no status code, no error message—all you know is that somewhere, somehow, some process is failing to finish.

In this post, I’ll share how we identified the problem, what the culprit was, and three ways to make seeding test data more robust.

To Identify the Problem, Focus on Your Logs

We started by comparing logs from our failing CircleCI runs with those from local test runs. Turns out, our CircleCI logs in failed runs stopped after a specific log message. 

For example, when our test runs succeeded, the logs looked something like this:

...
LOG: creating model
WARN: foo with id=123 not found
WARN: foo with id=456 not found
WARN: bar with id=789 not found
ERROR: Invalid parameter to function `foobar`
LOG: model created
... rest of logs

But when they failed, they instead looked like this:

. . .
LOG: creating model
WARN: foo with id=123 not found
WARN: foo with id=456 not found
[end of logs]

This let us know where to focus: after the function that looks for foo, but before or during the one that looks for bar. Of course, this only works if you have enough log statements scattered around in the first place. I’d recommend logging the time it takes to create each model in seeds, which has the added benefit of telling you where to start if your seeds are becoming annoyingly slow. 

At the top of our file in db/seeds, we declare the following function to add timings to our CI logs:

@last_time = Time.now
def log_time(label)
    Rails.logger.info("Time [#{label}]: #{(Time.now - @last_time)}")
    @last_time = Time.now
end

We then call the function after each create statement in our seeds:

Product.create!({...params})
log_time(“Product Create”)

This prints logs in the following format:

...log metadata message=Time [Product Create]: 12345
...log metadata message=Time [User Create]: 1102
...log metadata message=Time [Address Create]: 105
...

These logs will help you figure out problematic model create calls, but each model may have a lot going on in its callbacks (before_save, before_commit, after_save, etc). You can follow a similar approach to time those callbacks:

# in the model file
Class Model < ApplicationRecord
	
...

before_save :foo

...

def foo()
   start_time = Time.now
   [ ... things that happen inside the function ]
   Rails.logger.info("Time [foo]: #{(Time.now - start_time)}")
   [ function return statement, if applicable ]
end

It’s important to note that the most time-consuming statements in your seeds often cause the most trouble, but that’s not always the case. If your seeds are hanging due to external API calls, it’s entirely possible for those calls to usually take a short time to complete. If you have hanging tests, I would recommend logging test runs in CircleCI, since you may not be able to find the problem locally. 

Through timing our seeds, we narrowed down our search to five of our most time-intensive models. We were then able to time the individual callbacks within those models. One callback  in particular—containing a call to an external database—raised the most suspicion. It took up most of the time it took to create the model, didn’t return any information needed in our test, and was not mocked or stubbed out. 

Requests to this test database would usually work, however, since we call on it multiple times on every CI run, with concurrent CI runs, these requests would sometimes take a long time to complete, leading to a test timeout. 

In our case, since we were making calls to a hosted database in our tests, we changed our test environment to create and refer to an isolated instance of the database. While this was the best approach for our specific issue, we also explored some other ways of stubbing out API calls and skipping callbacks from your seeds file, which we describe below. Use these approaches with caution, as the more you stub, the less true-to-life your tests will be. 

Ideally, you would have already stubbed out third-party API calls in your code using environment variables. An example approach:	

def do_the_thing()
    if Rails.env.production?
    	[ ...do the thing ]
    else
	[ ...do some fake thing if needed ]
       Rails.logger.info("We theoretically would have done the thing.")
end

However, you can also use Typhoeus to mock third-party api calls in seeds, similar to how you might in a test. Imagine that your Product model makes a call to "https://slack.com/api/foo”. We can then mock that call out as follows:

mock_response = Typhoeus::Response.new({code:’500’, body: {
    “ok”: false,
    “error”: ‘not_authed’
}.to_json})
Typhoeus.stub("https://slack.com/api/foo”).and_return(mock_response)
Product.create!({...params})

Again, I would advise you to stub check for environment variables in your code rather than stub out individual API calls in your seeds and tests. But if for some reason you need to do the latter, it’s possible!

Similarly, you might want to stub out a function you’ve written if it takes a long time to run and doesn’t affect the outcome of your tests. This can’t be stubbed out in the ways we mentioned above since (1) you might need it  for — and hopefully are testing it in  — other tests and (2) it doesn’t have a url. minitest has a tool for this. Imagine you’re trying to stub out some functions from a Photograph model.

require ‘minitest/autorun’

Photograph.stub :fetch_product_photo_url, ‘’ do
   Product.create!({...params})
end

This will return an empty string when Photograph.fetch_product_photo_url is called within the Product model’s callbacks. You can also replace the empty string with anything else you might want the function to return.

Last but not least, it’s possible to skip entire callbacks and validation. I wouldn’t recommend skipping all of them, as that can cause models in your tests to stray very far from how your models actually work, but you may find it useful to skip specific callbacks.

While all the methods that I’ve suggested above are possible and can be useful, remember that stubbing and skipping from your seeds may just be band-aids for issues you have elsewhere. In our case, we could have stubbed out a model function or created an isolated database for testing. While both methods would have been functionally the same, doing the latter allows us to have to worry less about that database in the future—if, for example, we write more functions that depend on it, or update the mocked functions to do more than they currently do. Keeping tests in line with how the product works also keeps it more in line with the actual user experience: if it’s taking a long time to create a model in your seeds, it probably takes a long time for your users, too. 

Since adding timing logs and creating an isolated database to our end-to-end tests, we have a regular source of truth for the efficiency of our models, are able to develop more smoothly with fewer spontaneous test failures, and have more confidence in our ability to quickly pinpoint and resolve seed timeouts in the future. 

Last year, we built a feature to show patients all of their medication pricing options, including when paying cash might be cheaper. Here are two surprising trends we've discovered since.

Two Engineers Crunch the Numbers: When Is It Cheaper to Pay Cash for Your Medication?

Tom Cahill and Ashley Qian, Sr. Software Engineers

In 2019, we launched a streamlined flow in our patient app to handle a surprising scenario: when the out-of-pocket price for a medication is less than the insurance copay.

Since launching our price selection feature last year, we've saved thousands of patients money by bringing this disparity to their attention. On average, patients save 30% off their total copay when their out-of-pocket price for a medication is cheaper than the insurance copay. The average savings per prescription is $23.30. 

While several factors impact how drug pricing works, here are two trends we’ve seen in the data so far.

These Two Therapeutic Classes Are Most Likely to Be Cheaper Out of Pocket

A therapeutic class is a method of categorization that groups medications based on the conditions that they are intended to treat. Here's what we found when we aggregated prescriptions by therapeutic class:

Across the 394,367 prescriptions we examined, roughly 0.5% had lower out-of-pocket costs. Comparing different therapeutic classes, anti-parasitic drugs (example: permethrin) appear most likely to have a cheaper out-of-pocket price by a fairly wide margin at 2.94% of cases. Cardiac/cardiovascular drugs (example: rosuvastatin calcium aka Crestor) and blood medications (example: tranexamic acid) stand out as well with over 1% of cases.

Looking at the results in terms of median percentage savings, a few classes stand out as having a fairly high disparity between prices: anti-histamines, contraceptives, anti-neoplastics and anti-infectives. 

Brand Medications Are Less Likely to Be Cheaper Out of Pocket

Blog

Get to Know the Team That Has Doctors Calling Alto “The Pharmacy of The Future”

Get to Know the Team That Builds Alto's Customer-Facing Apps

Alto Pharmacy Named Great Place to Work®

Celebrating Women at Alto Pharmacy

2020 Diversity, Equity & Inclusion Report

Creating Sandboxes: How to Make It Easy for Your Engineering and Product Teams

How to Debug Slow and Failing Cypress Seeds

Two Engineers Crunch the Numbers: When Is It Cheaper to Pay Cash for Your Medication?

Diversity At Alto: Building an Inclusive Healthcare Company and Culture

Get to Know Alto’s VP of Engineering, Mike McClay

How (and Why) We’re Standardizing Our Commit Messages

Airbnb’s Former Head of Business Technology Joins Alto

Switch to a better pharmacy.