Design, build, and optimize end-to-end ETL/ELT pipelines using Python and PySpark for large structured and unstructured datasets. Ensure data quality through validation and reconciliation, troubleshoot performance issues, implement CI/CD and code review practices, and collaborate with IT and business stakeholders to translate requirements into scalable data solutions.
Responsibilities: • Development and Maintain Data Pipelines: Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. • Utilize Python and Pyspark: Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement. • Ensure Data Quality and Integrity: Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle. • Collaborate with Stakeholders: Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions. • Troubleshoot and Optimize: Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability. • Adhere to Best Practices: Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment. Skills: • Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred) • Experience with big data frameworks (Hadoop, Spark). • Experience with cloud platforms ( AWS, Azure or GCP) • Experience with Code versioning tools ( Bitbucket, Github ) • Experience with CI/CD and setting up pipelines. • Solid understanding of database design principles, data modelling, schemas and data warehousing solutions. • Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.
Similar Jobs
Information Technology • Consulting
Lead design and implementation of data ingestion, data lake, and ETL solutions using Big Data technologies. Build performant cloud data pipelines, integrate multiple data sources, optimize query performance, enable analytics and data science teams, and lead offshore engineering teams.
Top Skills:
Apache FlumeSparkApache StormAPIsAWSAzureBig DataCassandraData LakeDatabricksETLGCPHbaseJavaKafkaMongoDBNoSQLPysparkPythonRabbitMQScalaSpark StreamingSQLWeb ServicesXML
Information Technology • Consulting
Lead development of enterprise data and analytics systems using Snowflake and Python. Build and optimize data pipelines on AWS (Lambda, Step Functions, ECS), write advanced SQL, and apply domain knowledge of asset management and financial markets.
Top Skills:
Amazon EcsAws LambdaAws Step FunctionsPythonSnowflakeSQL
Information Technology • Consulting
Lead data engineer responsible for designing, building, and operating production data pipelines using Python, Airflow, Snowflake, DBT, and AWS. Automate ETL, implement CI/CD, testing and DBT validations, troubleshoot pipelines, and work client-facing to deliver reliable data solutions.
Top Skills:
Ai IdeApache AirflowAws EcsAws IamAws S3Ci/CdCsvDbtGitJinjaJSONPythonSnowflakeSQLXML
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
