How our engineers built and rebuilt everything all at oncePath 2

How our engineers built and rebuilt everything all at once

by Eithan Zilkha

December 9, 2019

In the arc of internet history, the combination of grocery (especially fresh foods) and e-commerce is relatively new. H-E-B was an early entrant, and we focused our technology brainpower on improving our brick-and-mortar store infrastructure and internal tooling for operational efficiency. 

But, we under-invested in customer experience, and needed to rebalance our efforts. We decided to quickly shift our 114 year-old gears to focus on creating consumer-facing web and mobile apps that are on the level of—and leading—the highly competitive grocery landscape.  

The bar was high: our newly appointed Chief Digital Officer, Jag Bath, tasked us with evolving a world-class engineering organization to develop consumer products that rivaled those of nationwide retail titans. Quickly, we needed to build new platforms to support Curbside and Delivery, and handle the massive operations work needed to enable stores to support this rapidly growing business—while expanding our engineering team to build those very platforms. We needed to build the best car in the world, while driving it. Thankfully, the ways to make all of those systems succeed is similar. 

Start with the past

Before we began, it was critical for us new Partners to spend time examining and understanding the internal workings of H-E-B and the specific challenges involved in building a new mobile application. Moving fast in the engineering world often leads companies to incur a fair amount of technical debt. Both our back-end and front-end systems had been built on older technology in order to ship something for our customers quickly (and to give the teams time to learn and improve). Additionally, our native applications had already been through a couple of iterations of redesign. The first major challenge was to gain a thorough understanding of these systems, and their history. Then, we needed to examine where to take them in the future—while also weighing what was needed in the short-term—to deliver a mobile application to meet the changing demands of our customers.

We've always hired exceptional engineering talent at H-E-B, including before this phase of H-E-B Digital was ever formed. Their knowledge was crucial. They were more familiar with the initial code and the back-end systems—and with the thought that went into these pieces. We leaned on their insights into the background of the application development lifecycle and redesigns, and asked them to explain the reasoning behind the technology choices and business logic in the systems. 

Culture counts

As we looked to staff this team around the current base, we made sure to focus on culture as much as finding incredibly talented engineers. One or two "bad seeds" can have a detrimental effect on team morale, efficiency, and other intangibles, and this negative effect can spread rapidly. We also know the churn of those bad seeds can be high—but often not before the damage is done. 

Philosophically, we want team members that are highly collaborative and who are seeking to make an impact and contribute to our fun, high-energy, and productive environment (a culture H-E-B spent over a century building). Skills wise, we want self-motivated folks with experience building apps from ideation to delivery, who have worked on large teams, and are passionate about technology and what they're building. 

Build on the best

A poor foundation—whether talent or technology—can corrode everything moving forward. A strong talent base is so valuable for several reasons. A strong team can establish more rigorous screening and interviewing processes, so we can continue hiring well as we scale (plus an initial talent base provides better referrals). Once more members are in place, that "strong at the start" group provides mentorship as we scale down to more junior engineers (providing career growth for all). Most of all, we have higher confidence that the building blocks of our technology will be built right. 

For our mobile apps, we started out by building everything with dependency injection and unit testing in mind, and using ReactiveX where we felt it would most speed up our development in our client API layer. We spent more time building foundational custom design components upfront so that the UI code built around them would be solid. As we hired more engineers, we continued building more components on top of a solid foundation and adding unit tests along the way.

Manage Mindfully

Strong management at the onset was also critical. Good people want to work for good managers, and together they establish a good culture. Managers have the perspective to analyze all the moving parts of H-E-B and determine the proper short and long term steps for our teams to succeed (where do we start, what should we build first, what type of team do we need to accomplish our goals, etc). 

We have plans, but we know they can and should shift based on learnings and guidance from talented engineers. Establishing strong practices and tweaking according to these learnings are key to success. 

Adjust as needed

As mentioned, we continually have to find the right balance between identifying useful parts of the business to building on top of those versus what we need to completely replace in order to succeed in the long term. 

For example, H-E-B already had a commerce server in place, but it was stateful. While that’s not ideal for a highly scalable mobile application, we could not afford to spend over two years to rebuild this technology. Instead we worked with teams across H-E-B to get all the parts working together. We looked at every API interaction on a sequence diagram and identified everything that was broken and assigned the various teams at H-E-B to correct them so the mobile applications actually work on top of these APIs. At the same time, we have a plan in place to also rebuild the most critical and troublesome parts of the system and deploy them incrementally.

It's been an incredibly challenging and fulfilling journey to build a mobile application that will shoot to the top of those reasons to love H-E-B lists and hopefully become an integral part of its amazing customers' lives (just like H-E-B already is) while also growing a team our engineers adore just as much. The journey is ongoing and we can't wait to see where we go next.

Eithan Zilkha is Director of Mobile Engineering. You can connect with him on LinkedIn.

All we’re missing is you.