Design, develop, and optimize scalable batch and real-time data pipelines on GCP. Implement Medallion Architecture, build Python and PySpark transformations, optimize BigQuery and SQL workloads, deploy Dataflow pipelines, and orchestrate workflows with Cloud Composer and Airflow. Manage GCS storage, CI/CD, version control, IAM, data governance, security, reliability, scalability, and cost efficiency across enterprise data platforms.
This is a remote position.
KANINI is seeking a highly skilled Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and modern data architecture. The ideal candidate will have hands-on experience designing scalable data pipelines, implementing Medallion Architecture, and building robust enterprise-grade data solutions.
This role requires strong technical proficiency in BigQuery, PySpark, Dataflow, and Airflow, along with a solid understanding of cloud data governance, performance optimization, and CI/CD practices.
Key Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines on GCP
- Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data processing
- Build high-performance data transformations using Python and PySpark
- Develop and optimize complex SQL queries for analytical workloads
- Work extensively with BigQuery for large-scale data processing and performance tuning
- Develop and deploy pipelines using Cloud Dataflow
- Orchestrate workflows using Cloud Composer (Apache Airflow)
- Manage data storage and lifecycle using Google Cloud Storage (GCS)
- Implement version control and CI/CD pipelines using Git-based tools
- Ensure data security, governance, and access control using GCP IAM
- Optimize data solutions for performance, scalability, reliability, and cost-efficiency
Required Skills & Experience
- Strong hands-on experience with Google Cloud Platform (GCP)
- Expertise in BigQuery (partitioning, clustering, query optimization)
- Proven experience implementing Medallion Data Architecture
- Strong programming skills in Python and PySpark
- Hands-on exposure on Java
- Advanced proficiency in SQL (complex joins, window functions, performance tuning)
- Hands-on experience with Cloud Dataflow
- Experience with Cloud Composer (Airflow) for orchestration
- Experience working with Google Cloud Storage (GCS)
- Knowledge of version control systems (Git) and CI/CD practices
- Strong understanding of GCP IAM and cloud security best practices
Preferred Qualifications
- Experience working with large-scale enterprise data platforms
- Knowledge of data warehousing and data lake concepts
- Familiarity with real-time streaming frameworks
- Experience in data governance and data quality frameworks
- Exposure to
Similar Jobs
Cloud • Information Technology • Security • Software • Cybersecurity
Leads technical customer Proof of Value engagements for Zero Trust Branch solutions. Builds advanced physical and virtual networking labs, troubleshoots complex L2/L3 networking escalations, authors deployment guides and reference architectures, and delivers technical enablement for systems engineers, professional services, and partners. The role requires deep enterprise networking expertise, packet analysis, lab automation, and customer-facing technical validation experience.
Top Skills:
AnsibleAws Transit GatewayAws VpcAzure Virtual WanBashDockerEsxiEve-NgFirewallsGcp Cloud RoutersGns3Infrastructure As CodeIpsecKubernetesKvmNatNetwork EmulationPythonRest ApisTcp/IpTcpdumpTerraformVxlanWireguardWireshark
Cloud • Information Technology • Security • Software • Cybersecurity
Provide first-line technical support for enterprise customers, independently troubleshooting complex security, connectivity, application, and network performance issues. Collaborate with Engineering on testing and resolutions, document incidents, develop team training, mentor support engineers, and provide product feedback. The role requires expertise in security and networking protocols, packet captures, enterprise escalations, and cloud platforms.
Top Skills:
Ai/MlAPIsAWSAzureDlpEsxiFirewallsGCPGre TunnelsHar TracesHTTPIp Sla MonitoringIpsIpsec VpnNgfwPac FilesPacket Capture AnalysisSAMLSandboxingScimSd-WanSslSsl VpnSwgTcp/Ip
Artificial Intelligence • Cloud • Consumer Web • Productivity • Software • App development • Data Privacy
Build and maintain application security tooling, automation, libraries, and custom Semgrep or CodeQL rules. Integrate secure development controls into SDLC workflows, conduct targeted code reviews, support vulnerability remediation, and advise engineering, product, GRC, and AI/ML teams across web, mobile, and desktop products. The role also contributes to threat modeling, security education, penetration-test and bug-bounty response, and potentially security on-call rotations.
Top Skills:
Access ControlCi/Cd PipelinesCloud-Native EnvironmentsCodeqlData ProtectionEncryptionGenerative AiMicroservicesSemgrep
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
