by Justin Turner
February 6, 2020
The less work a customer has to do to order Curbside or Delivery the better.
Getting the groceries you need should be as simple as filling your cart, pushing a button, and waiting for a cheerful H-E-B Partner (what we call our employees) to hand them to you at the right place and the right time.
But here’s the thing—less work for them nearly always means more work for us.
To successfully complete a Curbside or Delivery order at H-E-B, our custom-built fulfillment ecosystem must orchestrate a whole host of complex events to ensure our eStore employees can get the right order to the right customer at the right time.
Over the last two years, we completely reimagined what each part of that ecosystem should look like, altering the processes, teams, and technology H-E-B Digital Fulfillment utilizes to meet the ever-growing needs of an ever-changing business.
When we began piloting our Curbside fulfillment system we used a modest setup: an iOS mobile application running on a java server application with a Bluetooth scanner. When the first few stores opened, our team’s main goal was to meet our business objectives as quickly as possible. We used Scrum methodology and raw engineering talent to rapidly deliver business value, but as we grew and expanded beyond that initial set up, we unknowingly made technical decisions that solved our issues well at the time, but laid a complex foundation that would be extremely difficult to change.
As the business continued to transform, we layered on new functionality. Soon we found that the risk associated with deployments was beginning to slow us down and that we were running into major scaling challenges. Last year, we knew we had to make some real foundational changes. We had both technical and business-minded goals:
Increase the speed of delivery while minimizing the risk associated with code changes
Increase uptime and resilience
Increase replaceability of functionality as our business pivots
Be able to observe and truly understand what is going on in our systems
Allow our business to make decisions that are powered by technology, not defined by it
With these goals in mind, we determined the best path forward was a microservices architecture—breaking our complex needs into smaller applications.
To get us there, we used event storming to define the services’ bulkheads and then took the plunge and completely rewrote them. We knew that turning a mission-critical monolith into individual services would be a daunting task with serious risk, but the payoffs were critical to our evolving business. The approach was deliberate.
To make this massive change work, we had to quickly build competencies in technologies and processes that were new to H-E-B. We added the role of a Site Reliability Engineer (SRE) to our teams to help us operate Kubernetes, the tech that orchestrates our microservices. More importantly, the SREs increased our observability and understanding of our applications. Their work defined our user journeys and service level objectives to ensure high uptime and meaningful alerting. We introduced the concept of chaos engineering to actively introduce failure and test the resilience to failures that are sure to occur going forward.
To succeed with microservices, we also had to change our team culture and processes. We broke into several squads that own services within a bulkhead and dedicated those squads to fully building and operating their services.
All these changes meant we also needed to adjust how we measured success, which led to us adopting four key metrics: lead time, deployment frequency, mean time to restore, and change fail percentage. With new goals as our guide, we can drive towards becoming a high performing team that practices continuous delivery.
We are nearing the end of our overhaul efforts, and roughly 50 percent of our services are in production with the remainder on-track for completion early in 2020. Soon, H-E-B Digital Fulfillment systems will be fully cloud-native and organized into bulkheads that will ensure high availability. This new structure will support us as we continue to head into uncharted territory, guaranteeing that we can pivot as needed and provide the best possible applications to the store employees and delivery drivers who provide H-E-B’s amazing service to our customers.
Working in Digital Fulfillment is deeply rewarding—we regularly interact with the H-E-B Partners that serve our customers and get to see the impact of our work daily. It’s thrilling to be able to move quickly and own our destiny in a role that serves as a major pillar of our business.
If you’re ready to feel this particular combination of rewarding and challenging, we’re looking for several roles on our team.
Justin Turner is a Senior Software Engineering Manager for Curbside and Delivery Fulfillment. You can connect with him on Linkedin.
We keep our head in the clouds—but we keep some of our data in our own, on-prem data center. Tour the tech hub that powers all of our systems.
How do you tackle the gender gap in hiring? In 2017, we started our Women in Tech group to share stories, experiences, and connections. Here’s what we’ve learned.