Lead architecture, development, and optimization of a cloud-native data platform. Design high-volume batch and streaming pipelines, optimize data processing across AWS, Snowflake, Redshift, and Athena, and implement Iceberg-based data management. Establish security, privacy, observability, and DataOps practices while mentoring engineers, guiding technical direction, collaborating with data science teams, and delivering scalable petabyte-level data infrastructure.
Work Location: United States
Work Mode : Remote
Pay Range :$85K-$140K /Yr Base + Annual Bonus
The posted range is the hiring range for this role — a subset of the broader range available to employees over time — and reflects base salary across our national hiring scale. Final offers are based on several factors, including the candidate's skills and experience, internal pay equity, work location, market conditions for the role, and the specific scope and responsibilities of the position. The top of the range is reserved for candidates who notably exceed the requirements; the lower end applies to those with less experience or fewer preferred qualifications. For positions based in higher-cost zones (e.g., California, New York, New Jersey), actual compensation may exceed the posted range; your recruiter will share specifics during the process
For more information on benefits and what we offer please visit us at https://www.exlservice.com/us-careers-and-benefits
Job Overview:
We are seeking an experienced and visionary Senior Full-Stack Data Engineer to lead the architecture, development, and optimization of a next-generation data platform. This is a critical role for an individual with over 10 years of deep data engineering expertise, capable of driving technical direction, mentoring team members, and delivering high-impact solutions in a fast-paced project environment..
Responsibilities- Platform Strategy & Leadership
- Technical Direction: Define and champion the architectural roadmap and best practices for our end-to-end data pipelines, ensuring scalability, reliability, and security across the platform.
- Team Mentorship & Project Velocity: Act as a primary technical mentor, guiding a team of engineers, conducting code reviews, and aggressively driving the project timeline to ensure rapid delivery of data products.
- Stakeholder Collaboration: Partner with Data Scientists, Analysts, and business stakeholders to translate complex requirements into robust, production-ready data solutions.
- Collaboration with Data Scientists and ML Engineers: Data Accessibility, Support for Model Development, Data Quality Assurance
- Data Pipeline Development & Management
- Ingestion & Transformation: Design, build, and optimize high-volume data ingestion and transformation jobs using tools like dbt Core, AWS Glue, ensuring data quality and integrity.
- Workflow Orchestration: Develop and maintain sophisticated data pipelines using orchestrators such as Dagster, focusing on modularity and reusability.
- Streaming & Real-time Integration: Implement and manage real-time data flows utilizing Confluent platforms or native AWS streaming services (e.g., Kinesis) for immediate data availability.
- Data Security and Privacy: Data Anonymization, Compliance with Regulations
- Be well versed with DataOps and DevOps fundamentals
- Assist and drive the Data Ecosystem Management & Monitoring
- Open Table Formats & Management: Implement and maintain the Iceberg open table format, utilizing tools for efficient schema evolution and data management.
- Compute Engine Optimization: Optimize query performance and cost efficiency across our primary compute engines: Snowflake, Amazon Redshift, and AWS Athena.
- Observability & Monitoring: Integrate comprehensive monitoring and observability into all pipelines using Splunk to ensure high availability, rapidly identify bottlenecks, and troubleshoot production issues
- 10+ Years of hands-on, progressive experience in Data Engineering, Data Architecture, or a closely related Full-Stack Data role
- Deep conceptual understanding of core data engineering principles, ETL/ELT patterns, and metadata management
- Proven track record of building and managing petabyte-scale data infrastructure in a cloud-native environment
- Insurance industry experience preferred but not mandatory
- Tools:
- Cloud Environment: AWS (S3, IAM, VPC, etc.)
- Experience with Talend, dbt Core, Iceberg, AWS Glue Catalog, Snowflake, Redshift, Athena, Splunk, AWS streaming services, Git
Strong SQL, Pyspark and Python
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Build and lead production AI/ML, Generative AI, and data engineering solutions for healthcare analytics and patient safety platforms. Develop full-stack applications, microservices, APIs, data pipelines, dashboards, self-service analytics, RAG systems, and event-driven architectures using cloud-native technologies. Apply engineering best practices including testing, CI/CD, observability, security, and performance optimization. Mentor engineers, lead technical initiatives, and collaborate with product, analytics, business, and platform teams.
Top Skills:
Amazon KinesisAWSAzureAzure Event HubsCi/CdDatabricksDelta LakeGCPGenerative AiJavaJavaScriptKafkaLarge Language Models (Llms)Microsoft FabricMlopsPysparkPythonReactRest ApisRetrieval-Augmented Generation (Rag)SnowflakeSparkSpring BootSQLTypescriptVector Search
Information Technology
Designs, develops, validates, and documents analytics data solutions for diverse client environments. Builds Snowflake, SQL, dbt, and AWS ingestion and transformation pipelines; monitors data quality and performance; troubleshoots complex technology issues; and supports reporting and business insights. Manages project scope, budgets, documentation, governance, client communication, implementation planning, and quality assurance while collaborating across teams and identifying opportunities for additional solutions.
Top Skills:
Analytics Data MartsAWSData GovernanceData IngestionData PipelinesData PrivacyData TransformationDbtMedallion ArchitectureQa TestingRelational DatabasesSnowflakeSQL
Artificial Intelligence • Insurance • Machine Learning • Software • Analytics
Build and maintain automated data quality testing for customer data pipelines. Responsibilities include developing dbt, Soda, and SQL checks; validating schemas, loads, sequencing, and downstream reconciliations; detecting anomalies; managing severity levels and customer overrides; reporting quality metrics; and partnering with engineering teams to embed testing into new pipelines. The role requires strong backend testing, SQL, Python, Airflow, and Databricks experience, along with excellent debugging and communication skills.
Top Skills:
Apache AirflowCi/CdCypressDatabricksDbtDelta LakeGitPlaywrightPythonSodaSpark SqlSQL
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
