Build scalable ETL/ELT pipelines in Databricks using PySpark, Spark SQL, Delta Lake, Delta Live Tables, Workflows, and Unity Catalog. Ingest batch and streaming data across medallion architecture layers, develop transformations and data quality rules, and implement orchestration, monitoring, alerting, and automation. The role requires cloud data engineering, data modeling, performance tuning, CI/CD, and strong knowledge of lakehouse architecture, governance, and distributed systems.
DATAECONOMY is one of the fastest-growing Data & Analytics company with global presence. We are well-differentiated and are known for our Thought leadership, out-of-the-box products, cutting-edge solutions, accelerators, innovative use cases, and cost-effective service offerings.
We offer products and solutions in Cloud, Data Engineering, Data Governance, AI/ML, DevOps and Blockchain to large corporates across the globe. Strategic Partners with AWS, Collibra, cloudera, neo4j, DataRobot, Global IDs, tableau, MuleSoft and Talend.
Senior/Lead Data Engineer — Databricks
Rutherford, NJ/ Jersey City, NJ
Full-time
- Build scalable, production-grade ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Live Tables, Workflows).
- Ingest structured, semi-structured, and streaming data into Bronze, Silver, and Gold layers.
- Develop optimized transformations, data quality rules, and reusable framework components.
- Implement best practices for job orchestration, monitoring, alerting, and automation.
- Hands-on experience: Spark, Delta Lake, Workflows, Unity Catalog.
- Strong SQL programming and performance tuning skills.
- Experience with cloud environments (AWS/Azure/GCP).
- Experience with modern data lakehouse concepts and distributed systems.
- Strong understanding of Lakeflow Connect, LSDP/Lakehouse, Medallion Architecture, Data Validations, Genie, and Agent Bricks/RAG use cases.
- Should be able to explain these concepts using real project examples and architecture decisions.
- Knowledge of medallion architecture, DLT and unity catalog within Databricks.
Requirements
- Strong Python (PySpark) and SQL programming
- Databricks — Spark, Delta Lake, Workflows, Unity Catalog
- ETL/ELT pipeline development — Medallion Architecture (Bronze/Silver/Gold)
- Delta Live Tables, Auto-Loader, Structured Streaming
- Data modeling — dimensional (star/snowflake), normalization/denormalization
- CI/CD, Git, job orchestration
- Cloud experience — AWS, Azure, or GCP
- 7–10+ years in data engineering
- Knowledge of medallion architecture, DLT and unity catalog within Databricks.
Nice-to-Have Skills
- Lakeflow Connect, LSDP/Lakehouse, Genie, Agent Bricks/RAG use cases
- Data governance, metadata management, Unity Catalog advanced features
- Airflow, dbt, or similar orchestration tools
- Data security, compliance, and access models
- Cost optimization and performance tuning in cloud environments
- Corporate/enterprise data warehousing background
Similar Jobs
Aerospace • Artificial Intelligence • Machine Learning • Robotics • Software • Defense Technology
Lead operational reliability and platform enablement for Databricks: build monitoring, CI/CD, deployment standards, compute and job policies, observability, runbooks, and governance to support secure, cost-aware, production data workloads across regulated environments. Mentor engineers and align platform with cloud/infrastructure and compliance requirements.
Top Skills:
Ci/CdDatabricksDatabricks Asset BundlesDatabricks WorkflowsDelta LakeInfrastructure-As-CodeService PrincipalsUnity CatalogVersion Control (Git)
Information Technology • Analytics
The Sr. Data Engineer - Databricks designs and maintains scalable data solutions using Apache Spark and Databricks, ensuring data quality and performance for analytics and reporting.
Top Skills:
SparkAWSAzureDatabricksDelta LakeEltETLGCPUnity Catalog
Artificial Intelligence • Automotive • Greentech • Information Technology • Machine Learning • Software • Cybersecurity
Processes, codes, audits, and indexes service-provider invoices for fleet billing operations. Responsibilities include resolving invoice discrepancies, verifying approval limits, completing WACC requirements, processing rebills, performing VMRS coding and costing, managing purchase-order inquiries, and responding to overflow calls. The role requires accurate high-volume data entry, attention to detail, and communication with service providers and internal teams.
Top Skills:
TmcareVmrs
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
