Build and maintain large-scale data processing pipelines using Apache Spark (PySpark/Scala) and Python. Design ETL workflows, implement distributed data processing, use version control (Git), and work with cloud platforms (GCP/AWS/Azure).
Job Title: Data Engineer (PySpark / Scala / Python)
Location: Remote
Job Description:We are hiring a Data Engineer with strong hands-on experience in PySpark, Scala, and Python. You must have solid expertise in Apache Spark, as it will be the core technology used for building and managing large-scale data processing pipelines.
Experience with cloud platforms like Google Cloud Platform (GCP), Microsoft Azure, or AWS is a plus.
Required Skills:Strong hands-on experience with Apache Spark
Proficient in PySpark
Experience in Scala and Python
Knowledge of ETL processes and data pipeline design
Understanding of distributed data processing
Familiarity with version control tools like Git
Basic knowledge of cloud platforms (GCP, AWS, or Azure)
Experience with cloud-native data tools (e.g., Dataproc, Glue, EMR, BigQuery)
Familiarity with workflow/orchestration tools like Airflow or Cloud Composer
Experience with CI/CD for data engineering
Exposure to both structured and unstructured data
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Design and operate scalable batch and streaming data platforms supporting machine learning and generative AI. Build pipelines for structured, unstructured, OCR, document, and image data; develop RAG, semantic search, and LLM-powered solutions; and establish data quality, observability, governance, orchestration, and deployment practices. Partner with stakeholders, lead platform scalability and cost optimization, mentor engineers, and translate ambiguous needs into production-ready technical roadmaps while securely handling sensitive data.
Top Skills:
AirflowAmazon KinesisSparkAWSAzureAzure Event HubsChart.JsDatabricksDeequDelta LakeDockerGithub ActionsGCPGreat ExpectationsJavaKafkaKubernetesLlmsMlopsPlotlyPysparkPythonRagScalaSeabornSnowflakeSQLTerraform
Artificial Intelligence • Information Technology • Professional Services • Software • Analytics • Generative AI • Big Data Analytics
Configure and maintain Adobe Experience Platform and Real-Time CDP data pipelines, XDM schemas, datasets, identity resolution, Profile enablement, segmentation readiness, and destination activation. Validate data quality, troubleshoot ingestion and identity issues, manage sandboxes, and implement privacy, consent, governance, and access controls. Partner with architects, data engineers, consultants, analysts, data scientists, and marketing teams to support reporting, personalization, and machine learning readiness.
Top Skills:
Adobe AnalyticsAdobe Experience PlatformAdobe I/O RuntimeAdobe Journey OptimizerAdobe Real-Time CdpAdobe Source ConnectorsAdobe TargetAPIsAWSAzureBatch IngestionBigQueryCcpaData PrepEltETLGCPGdprJavaScriptPythonQuery ServiceRedshiftSalesforce CdpSegmentSnowflakeSQLStreaming IngestionWeb SdkXdm
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Lead cloud data modernization by designing and building cloud-native ETL/ELT pipelines, data warehouses, and lakehouse architectures using Azure, Snowflake, and Databricks. Ensure data quality, security, governance, CI/CD, and healthcare interoperability while mentoring engineers and collaborating with stakeholders to deliver scalable, HIPAA-compliant analytics platforms.
Top Skills:
AdlsApache AirflowAzureAzure Data Factory (Adf)Azure Data Lake Storage Gen2 (Adls Gen2)Azure OpenaiCi/CdCptDatabricksDatabricks GenieDatabricks Mosaic AiDockerEltETLFhirGitGithub ActionsHcpcsHl7Icd-10KafkaKubernetesLlmsLoincPysparkPythonRag PipelinesSnowflakeSnowflake CortexSparkSQL ServerSsisTerraformVector StoresVisioX12 Edi (837/835/834)
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
