“There’s no team-building exercise more effective than being in a foxhole together.”
The word “foxhole” may conjure images of soldiers shielding themselves from enemy gunfire, but this phrase takes on a different meaning for Cameron Hawkins. The engineering manager for System1 was describing his team huddled not in a hole in the ground, but behind computer screens, facing one of the greatest challenges they’ve encountered recently: a major tech consolidation effort.
This task was complex for a variety of reasons. From a technology standpoint, it required patience and close attention to detail. It also involved bringing together team members with conflicting ideas.
Yet in the end, this experience proved to be highly beneficial, revealing that often the trickiest aspect of engineering cannot be found in a line of code. “This project would have failed spectacularly if we did not learn how to work together and understand each other’s perspectives,” Hawkins said.
Teamwork and a spirit of continuous learning are essential for tackling complex issues in engineering — something engineers from System1, BlackLine and Ruggable know well.
Because of this, the three pros interviewed below approach technical challenges not simply as roadblocks to overcome — but as opportunities to evolve.
System1’s responsive acquisition marketing platform enables the organization to grow its portfolio of brands and drive customer engagement.
What’s the biggest technical challenge you’ve faced recently in your work?
Last year, we undertook a major tech consolidation effort. We had two systems that were similar enough to be considered redundant, so we decided to combine them for efficiency. This was complex for a number of both technical and interpersonal reasons.
Technically, both systems had existed for years. While we were duplicating features to the new system, we had to replicate all of the subtle nuances of the first system. We had to do that in a way that was compatible with the existing features of the new system. This is an inherently difficult process and begs questions, such as, “How do you weigh the clear advantages of completely rewriting a feature with the benefit of hindsight against the risk of changing the functionality that has been relied on for years?” It required patience and close attention to detail to do this well.
This process was also complex from an interpersonal standpoint. When consolidating systems, one often needs to consolidate teams. Senior engineers often have strong opinions about the proper way to do things, and when those ideas clash, it can be hard to mitigate the tension. We needed to learn how to listen to each other, work together, compromise and trust each other’s expertise.
How did you and your team overcome this challenge, and what were some of the specific technologies you used?
We leveraged multiple Amazon Web Services technologies, such as Simple Queue Service and Simple Notification Service, Lambda and DynamoDB in addition to other tools, including Redis, to replicate the features to the new stack. However, the most important tech we used was Slack and Google Meet, which helped us learn how to communicate more effectively and respect each other. We were up against a wall of business stakeholders waiting to reap the benefits of such a large undertaking, and we delivered.
Often the hardest element of engineering is human.”
How did this technical challenge help you and your peers grow as engineers?
This was a complex task. It was a healthy reminder that often the hardest element of engineering is human. It can be difficult to work hard on something for so long and essentially have to help kill it by taking its best parts and replicating them somewhere else. Likewise, it can be difficult to see how another team did something without understanding why they made the decisions they did and trust that it will make sense in time. Yet all of these things are important and critical for the success of this kind of project.
Ruggable manufactures a wide range of machine-washable rugs for both indoor and outdoor use.
What’s the biggest technical challenge you’ve faced recently in your work?
Recently, we were trying to give employees in our manufacturing facilities more control over which inventory items they choose to fulfill while ensuring that older and higher-priority orders aren’t overlooked. Previously, we presented these employees with a single item in a specific location for them to work on before showing them another item. If we had multiple employees, they would be shown different items and would be assigned to that specific item until they picked it, marked it as missing or were signed out due to inactivity.
Our new approach involved showing users all the available items that could be picked at a particular facility, allowing them to choose any item from any location to fulfill an order. The goal was to make it so that, regardless of which item was picked, we would use that to fulfill the next higher-priority order for the corresponding stock-keeping unit.
When you have a single individual choosing items, the solution is pretty simple: Look at all of the orders for a particular SKU, and assign it to the highest-priority order. The problem becomes more challenging when you have multiple people picking the same SKU at the same time. In this case, the accuracy and timeliness of fulfillment are important considerations. It was also crucial to ensure employees didn’t see items that were no longer relevant, so we wanted to find a way to remove items that were picked by other employees.
How did you and your team overcome this challenge, and what were some of the specific technologies you used?
We needed to find a way to centralize the processing of all these requests to pick an item, yet we also didn’t want to compromise on throughput of our application by limiting ourselves to a single process.
FIFO queues in AWS SQS became one of the keys to solving this problem. In a traditional SQS queue, the messages are pulled in order, but if you have multiple instances of a lambda consuming messages from this queue, there is no guarantee that the first item from the queue will finish processing before the second item does. On the other hand, FIFO queues have a group ID field, in which anything with the same group ID is guaranteed to be processed in order. Messages from another group ID may be processed at the same time, allowing you to process as many items in parallel as you have defined group IDs.
While a group ID can be anything, we chose to make it descriptive of the product variant as well as its location. This allowed us to maximize the number of operations that could be completed simultaneously while ensuring that any orders for the same item from the same facility wouldn’t get assigned to the same order, unless a customer ordered more than one item.
We leveraged subscriptions in GraphQL to ensure our employees had an updated list of items at all times. Once the lambda function had been completed, it pushed a message to Redis that would notify the front end that an item had been picked successfully, allowing us to print out a label and remove the line from all operators’ screens. In other applications, we’ve had the front end continuously poll the back end since we didn’t have events pushed to the front end.
HOW THE SOLUTION WORKS
From a high level, Ooi and his team developed the following process to solve their technical challenge:
- The front end would call a GraphQL mutation to pick an item from inventory and then subscribe to updates through a GraphQL subscription.
- That mutation would perform some validations and publish a message to a first in, first out AWS Simple Queue Service.
- A lambda would be triggered by the SQS message that would look up the next highest priority item and assign it to the chosen item. A successful choice would result in a message being published to a Redis Pub/Sub.
- The front end would be notified of the updated item through its subscription, remove the item from the list of choosable items for all operators, and print out a label.
How did this technical challenge help you and your peers grow as engineers?
I’ve largely avoided using asynchronous teamwork in cases when an employee would have to wait on the outcome of that operation. It’s hard to say exactly how long a task would take to complete, if at all. Making employees wait for that isn’t a great experience, yet in this case, it was a necessary evil that ended up working out well. We did introduce some latency into the request by taking this approach, but it’s not more than a few hundred milliseconds and isn’t something our employees would notice, considering they’re also physically moving our products. I’d consider using this approach again in the future if the circumstances called for it.
Prior to this challenge, I lacked experience developing against Redis. Subscriptions had been a feature that we had hoped to leverage in GraphQL for a long time, but because of our GraphQL setup, it hadn’t been an option before. Luckily for us, a method to utilize it came out just prior to this project, so we were able to take advantage of it.
This was also a really great opportunity to teach some of the engineers about different technologies, the risks and benefits of the architecture and ways we could handle this type of challenge in the future. The architecture and sequence diagrams for this project are two elements I’ve discussed most frequently with my team throughout my time at the company so far.
BlackLine helps midsize and large organizations automate and control their accounting processes.
What’s the biggest technical challenge you’ve faced recently in your work?
My team works on a React-based component library, which other teams in the company use to build their sites. Recently, we upgraded some of the public JavaScript libraries our own component library depends on. Those dependencies required other things to be upgraded, which created a snowball effect.
Everything appeared to work fine at first. Our tests passed, and our demo site still worked. So we had no idea that the bundled output we publish to our internal NPM site would throw a JS error when our internal customers installed and used this new version. The problem was particularly tricky, because the error surfaced once it was installed in other applications.
How did you and your team overcome this challenge, and what were some of the specific technologies you used?
We overcame this challenge by hunting through changelogs of upgraded dependencies and looking for odd React imports in our code. After enlisting the help of more engineers, brainstorming ideas and exhausting all possibilities, I started removing huge portions of our Babel and rollup configs until I saw the weird import disappear. Then, I was able to narrow it down to the library that was supposed to be ignored by Babel. I learned that someone had included an import directly to a file of a distributed bundle, which did not match our regex that excluded that library from Babel.
If something is taking too long to solve on your own, don’t feel too proud or ashamed to ask for help.”
How did this technical challenge help you grow as an engineer?
This challenge improved my bundler game, specifically my understanding of rollup, Webpack and Babel. I learned a lot by digging through recent documents and releases for these tools. The biggest takeaway for me stems from a lesson I learned long ago: If something is taking too long to solve on your own, don’t feel too proud or ashamed to ask for help.
