When tech companies begin to scale, success can quickly become a hurdle to overcome. While it might seem like a good problem on the surface, tech teams have to rise to the occasion and implement systems that can handle a more robust workload.
Built In LA spoke to two local CTOs about the challenges their teams face, the qualities of an ideal software engineer and tech trends they’re watching as 2018 rolls forward.
A tech leader and a former startup co-founder, Andy Shin brought with him a wealth of experience when he joined the team at the Culver City-based GOAT in late 2016. Since then, the business has raised $85 million in funding, made its first acquisition and rolled out a handful of new features to its e-commerce platform. It’s on the way to becoming the most popular sneaker marketplace. Meanwhile, Shin’s team completely revamped the platform’s architecture to ensure it could handle the absurd number of requests it receives around the holiday season.
What technologies does your team use?
We’re breaking away from the monolithic Ruby on Rails architecture on which the product was originally built. To handle the organizational, feature and performance requirements, we’ve moved to an event-driven microservice pattern using AWS Lambdas and Kinesis for the event pipeline. Our back-end services are written in a combination of Ruby, Go, Node and Python — whichever language fits best for the needs of both the problem and the team.
On the front-end, our web applications are all single-page apps written in React with Redux for state management and GraphQL for data fetching. Our iOS app is written in Swift using a Data, Context, Interaction (DCI) paradigm and an Immutable Model layer inspired by Pinterest's Plank. Our Android app is written in Kotlin and Java using a Model-View-Intent (MVI) pattern and RxJava. Finally, from a data storage perspective, we use AWS Aurora, Redis, Elasticsearch, Redshift, Memcache and more.
Describe one major technological hurdle your team had to overcome and how they did so.
Each year, we experience tremendous traffic and growth starting around Black Friday and continuing through the holiday season. It’s always been taxing on our old Ruby on Rails back-end (the same back-end we’ve had since launch), but in Q4 of last year, even our best practices were ineffective in handling an order volume of over 500,000 requests per minute through our mobile and web APIs.
We overcame this by re-architecting our Ruby on Rails into smaller, event-driven microservices, which allowed us to essentially distribute our read and write load across several databases. This also enabled us to reorganize our engineering teams into KPI-driven “pods”, which are cross-functional delivery teams that are fully capable of making all changes to the system. Now, they're able to take on big initiatives and make changes holistically.
What are some qualities you look for in potential hires?
Technology is involved in every part of the business, so engineers here can make small changes with big results. Accordingly, we look for candidates who are passionate, curious and motivated to be responsible for building something great. With our recent merger with sneaker consignment pioneer, Flight Club, we’re also looking for talent with experience or interest in retail technology and creating innovative retail 2.0 experiences.
How do you set up your employees for success?
I work to empower my team to be autonomous, entrepreneurial and provide them with the tools they need to make a real impact. We strive to create an environment in which people don’t feel like they have to start their own company, because they’re building a company with us. Whether you want to delight customers with a cool mobile app, build out automation in our warehouses or support your teammates in the tools they use every day, you can — and you can make a huge difference doing so.
What are some industry trends you are keeping an eye on for the rest of 2018?
Retail technology: Today's online shoppers have high expectations when it comes to quickly and easily receiving goods and services, which is evident in the proliferation of services provided by Amazon, Postmates, Uber and Lyft. Even brick and mortar retailers have excelled in offering personal experiences in ways that can’t be replicated by an online store. Because every customer has a powerful computing device in their pocket, retail locations can leverage these devices to improve a shopping or browsing experience through the use of augmented reality, location data and data science. Retail stores that create unique, in-store-only experiences that leverage technology will win.
Data science: We leverage data science and machine learning to perform vital functions of our business — shoe authentication, personalization, fraudulent payment detection and logistics optimization. We use a combination of traditional, collaborative filtering with the latest deep learning feature representations for buyer recommendations. We've seen a 20 percent increase in buyer conversion because of this technology.
Serverless technology: Our new technology will lend itself to a more modular, scalable and maintainable ecosystem. Using deployment templates modeled after the serverless framework and AWS Cloudformation with Lambda, releasing code becomes ridiculously easy and allows our developers to focus on what matters: building it.
When it comes to facilitating hundreds of millions of dollars worth of payments each year, reliability can never be a concern for LA-based Verifi, especially as the sophistication of cyber attacks rises. Recently, CTO Chris DeRossi and his team underwent a lengthy process to renovate the Verifi architecture so that not even a complete data center failure could interrupt the communication between the company’s other data facilities.
What technologies does your team use?
Today, all of our back-end development is done in Java with the Spring framework, while our front-ends are built on Angular 4. Public and private APIs all follow RESTful semantics for scalable, stateless and highly available services. Currently, asynchronous processing and streaming are based on Kafka, but as we complete our migration to AWS, we’re beginning to look at Kinesis. We’re also starting to plan for containerized deployment with AWS Fargate and Amazon ECS.
Describe a major technical hurdle your team had to overcome. How did they do it?
Handling the data related to payments and transactions for some of the largest banks and merchants in the world requires our systems to be highly reliable. We needed a multi-data center architecture that would ensure service availability and data integrity, even in the event of a full data center failure or the loss of connectivity between our data centers. Database replication latency, network partitioning and potential transaction conflicts are some of the difficult challenges that have to be accounted for.
There was no simple, easy solution. Instead, the team broke the problem down into smaller pieces and invested the time to develop the right approach in each case. Adopting idempotency was viable for many — but not all — of the use cases. We found it important to understand when the business context didn’t justify extreme measures; there were simply some situations where an alert that triggered manual intervention was the best answer.
Throughout this process, one key thing we learned was the importance of real-time visibility. The more we invested in instrumentation, the faster we could identify broken-edge cases as well as successful processing by multiple active data centers.
Curiosity, intelligence, self-motivation and a passion for what they do are the top qualities I value in our employees."
What are some qualities you look for in potential hires?
Curiosity, intelligence, self-motivation and a passion for what they do are the top qualities I value in our employees. With these characteristics, it doesn’t matter as much what programming languages you already know, or which technologies you’ve used in previous jobs. I know I’ve found someone I’m probably going to want on the team when they are giddy to tell me about a project they finished in their own time that helped them learn something new.
As a CTO, who is someone you look to for inspiration? Why?
When Bill Gates was the CEO of Microsoft in the ’80s and ’90s, he was notorious for holding product reviews where he’d challenge some of his brightest engineers with tough technical questions. At the same time, his corporate leadership propelled Microsoft into one of the most important and valuable companies in the world. I believe it’s important for CTOs and technical leaders, in general, to maintain their technical skills while expanding their scope to other areas of the business.
How do you set up your employees for success?
We take employee development very seriously at Verifi. It starts with complete, thoughtful job descriptions for every level, which lets individuals know exactly what’s expected of them in their current position while also establishing what is required to advance to the next level. Project estimates and commitments are made at the team level rather than being decided by management and handed down. We also have formal programs for recognition and career advancements. As a testament to our employee focus, our department of 75 people had zero attrition for all of 2017.
What are some industry trends you are keeping an eye on in 2018?
The trend to utilize AI and machine learning in everything is certainly worth monitoring. As tools get better and skilled practitioners become more widely available, opportunities to innovate within existing disciplines will continue to expand.
The potential benefits of an interconnected world coupled with the low-cost of including WiFi in even the smallest devices are likely to impact every aspect of our lives. The explosion of the Internet of Things is fueling another trend, which is a backlash to the regulation of privacy and security concerns. The EU’s General Data Protection Regulation (GDPR) could be the first of many changes brought about in an attempt to protect individual privacy.
Finally, the recognition of blockchain technology as more than just the magic behind cryptocurrencies could pave the way for improvements in many areas, from supply chain management to voting.
