The Data Engineer will design and maintain scalable data pipelines, ensure data quality, implement governance standards, and collaborate with scientists and engineers for AI applications.
At Agile Dream Team, we harness the power of data to drive intelligent business decisions. We are looking for a highly skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure for AI-driven applications.
Learn more about us at: www.agiledreamteam.com
Role Overview
As a Data Engineer, you will be responsible for building and optimizing data pipelines, ensuring data quality, and enabling real-time and batch processing for analytics and AI models. You will work closely with Data Scientists, AI Engineers, and Software Developers to develop scalable data solutions that support business intelligence and machine learning applications
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
- Build data architectures that support batch and real-time data processing.
- Optimize data storage, retrieval, and performance for analytics and AI applications.
- Work with big data processing frameworks such as Apache Spark, Apache Flink, or Kafka.
- Implement data governance, security, and compliance standards.
- Ensure high data quality through data validation, monitoring, and anomaly detection.
- Automate data ingestion, transformation, and processing for AI/ML workflows.
- Deploy and manage data solutions in AWS, Azure, or GCP cloud environments.
- Collaborate with Data Scientists and AI Engineers to enable data-driven AI models.
- Utilize SQL and NoSQL databases for efficient storage and retrieval of large datasets.
Required Skills & Experience
- Proficiency in data engineering frameworks: Apache Spark, Hadoop, Airflow, DBT.
- Strong knowledge of SQL, NoSQL, and data modeling techniques.
- Experience in ETL/ELT pipeline development using Python, Scala, or Java.
- Hands-on experience with cloud data platforms (AWS Redshift, BigQuery, Snowflake, Databricks).
- Expertise in real-time streaming technologies (Apache Kafka, Kinesis, Pulsar).
- Experience with data warehouse and lakehouse architectures.
- Strong understanding of data partitioning, indexing, and performance tuning.
- Familiarity with containerized deployments (Docker, Kubernetes) for data pipelines.
- Experience in CI/CD automation for data workflows.
Preferred Qualifications
- Experience with machine learning data pipelines and feature engineering.
- Hands-on knowledge of data lake technologies (Delta Lake, Iceberg, Hudi).
- Familiarity with Terraform, Ansible, or other infrastructure-as-code (IaC) tools.
- Understanding of graph databases (Neo4j, ArangoDB) and time-series databases.
- Experience in data observability, lineage tracking, and governance.
Why Join Us?
- Work with cutting-edge data technologies in a forward-thinking AI/ML company.
- 100% remote role with a flexible schedule.
- Opportunities for growth and continuous learning in Data Engineering and AI.
- Engage in high-impact data projects that power real-world AI applications.
- Competitive salary
Ready to shape the future of data engineering? Apply now!
Top Skills
Airflow
Apache Kafka
Spark
Aws Redshift
BigQuery
Ci/Cd
Databricks
Dbt
Docker
Hadoop
Java
Kinesis
Kubernetes
NoSQL
Pulsar
Python
Scala
Snowflake
SQL
Similar Jobs
AdTech • Cloud • Digital Media • Information Technology • News + Entertainment • App development
As a Data Engineer II, you will design and maintain data pipelines, troubleshoot production issues, and collaborate in an Agile environment to support Fandango's business needs.
Top Skills:
Amazon MwaaApache AirflowAthenaAWSDynamoDBEmrGlueGlueInformaticaJavaLambdaPysparkPythonRedshiftS3ScalaSQLTalend
Cloud • Healthtech • Professional Services • Software • Pharmaceutical
Data Engineer will design, develop, and deploy SQL code; create ETL processes; and provide analytics reporting using various tools while ensuring compliance with quality standards.
Top Skills:
.NetAWSAzureBiC#CloudData AnalyticsData LakeData ModellingData PipelinesData WarehouseDb2ETLHTMLJavaJreviewOraclePl/SqlPythonQlikRSQLT-SqlTeradata
Big Data • Cannabis • eCommerce • Logistics • Database • Business Intelligence • Big Data Analytics
Design and maintain a data analytics platform, build ETL pipelines, standardize and model data, optimize database performance, and ensure data quality.
Top Skills:
AirbyteAWSDbtExcelGithub ActionsGlueHevoMetabaseModeMySQLPostgresPower BIPythonSnowflakeSparkSQLTerraformThoughtspot
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
