Senior Data Engineer

 Abacus Insights is a mission-driven, growth-stage technology company focused on transforming the healthcare industry, ultimately creating a more personalized patient experience, improving health outcomes, and lowering the overall cost of healthcare. Abacus Insights provides a flexible, efficient, and secure platform that organizes and exchanges healthcare data from various sources and formats, allowing our customers to uncover differentiated insights that address their client's needs. In addition, our employees know that they play an active role in keeping our customers' data safe and are responsible for ensuring that our comprehensive policies and practices are met. 

With our deep expertise in cloud-enabled technologies and knowledge of the healthcare industry, we have built an innovative data integration and management platform that allows healthcare payers access to historically siloed and inaccessible data. Through our platform, these healthcare organizations can ingest and manage all the data they need to transform their business by supporting their analytical, operational, and financial needs. 

Since our founding in 2017, Abacus has built a highly successful SaaS business, raising $100 Million from leading VC firms with deep expertise in the healthcare and technology industries. We are solving problems of massive scale and complexity in a sector ready for disruption. We're growing quickly and would love for you to be a part of it! 

About the Role:  

We are seeking an accomplished Sr. Data Engineer to join our dynamic and rapidly expanding Tech Ops division. With significant projected growth, we invite you to contribute to our success. In this role, you will collaborate with customers, data vendors, and various internal teams to integrate data into a state-of-the-art data platform. This platform is constructed on highly scalable, flexible, and resilient cloud architectures. As a trusted advocate for our clients, you will play a key role in facilitating the successful adoption of Abacus’s core data management solutions. Your expertise will ensure seamless data ingestion and management, leveraging cutting-edge technologies and best practices in data engineering. 

You will: 

• Work with clients & implementation team and understand the data distribution requirements
• Develop data pipelines integrating with Databricks and Snowflake, ensuring data accuracy and integrity.
• Lead the architecture and implementation of solutions for health plan clients, optimizing cloud-based technologies.
• Collaborate with cross-functional teams and clients to translate requirements into technical solutions.  
• Implement robust data security measures and ensure compliance with healthcare regulations like HIPAA.  
• Provide technical leadership, mentoring junior engineers and staying updated on industry trends.
• Focus on optimizing data processes for efficiency, regularly reviewing architectures, and creating detailed documentation.  
• Conduct training for team members and clients to enhance understanding and utilization of data solutions. 

 
What we're looking for: 

• Bachelor’s Degree in computer science, computer engineering or any related IT field 
• 5+ years of data engineering experience
• Basic US Healthcare Knowledge (e.g. Medical Claims, Member/Eligibility etc.)
• Experience communicating with both technical and non-technical stakeholders
• Proficient with Python and SQL
• Build and manage efficient ETL pipelines using Databricks workflows or other orchestration framework
• Familiarity with both structured and semi-structured data and ingesting and processing this data using pyspark
• Fundamental AWS services (S3, SQS) or similar services for other clouds
• Terraform and GitLab CICD
• Query tuning and performance optimization in SQL and/or SparkSQL
• Familiarity with data warehousing (snowflake or similar) 

Bonus Points: 

• Experience working for a cloud-based data services provider to large healthcare clients.
• DBT
• Kafka

Equal Opportunity Employer 

As a mission-led technology company helping to drive better healthcare outcomes, Abacus Insights believes that the best innovation and value we can bring to our customers comes from diverse ideas, thoughts, experiences, and perspectives. Therefore, we dedicate resources to building diverse teams and providing equal employment opportunities to all applicants. Abacus prohibits discrimination and harassment regarding race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.