Own and lead the big data infrastructure platform, including deploying, upgrading, and operating HDFS, object storage, Iceberg, monitoring, and alerting. Drive infrastructure modernization, migrate Iceberg catalogs, implement multi-tenant quotas, improve observability, and support large-scale data architecture changes. Collaborate with a parallel infrastructure engineer, share solutions, provide coverage, and influence platform architecture.
As a Big Data Infrastructure Engineer in the Big Data Platform team, you’ll own the big data infrastructure of our international contour and be its technical lead. You’ll deploy, upgrade and operate the core services of the platform: HDFS, S3-compatible object storage (Apache Ozone), Iceberg and others. You’ll start by helping run an infrastructure pilot that changes how the platform is used: a small number of very large datasets will become thousands of small tables. You’ll introduce compute and storage quotas for business units, migrate us from the default Iceberg catalog to a more capable one (Lakekeeper or Gravitino), and rebuild monitoring and alerting so alerts mean something again. You’ll work closely with an engineer who runs the same stack in a parallel contour. You’ll share approaches and solutions with them and cover for each other. Our infrastructure runs on the latest versions of everything, and we actively replace outdated tools. The role suits an experienced engineer who wants real ownership and a big say in architecture.
Requirements
- 5+ years of experience designing, deploying and operating big data platforms.
- Hands-on experience with HDFS / Hadoop: deployment, version upgrades, troubleshooting.
- Hands-on experience deploying and operating S3-compatible object storage. Apache Ozone is ideal, but MinIO or similar solutions also count.
- Experience with Apache Iceberg.
- Experience setting up monitoring and alerting for data infrastructure.
- English B2.
- Ability to work in the GMT+3 time zone (up to 3 hours difference).
Nice to have
- Experience with Trino, Kafka or Flink.
- Experience with Iceberg REST catalogs (Lakekeeper, Gravitino, Polaris or similar).
- Experience setting up resource quotas and cost allocation in a multi-tenant data platform.
- Hands-on use of AI tools, such as Copilot or agents, to speed up your work.
Benefits
- Flexible schedules and opportunity to work remotely.
- Ambitious and supportive team who love what they do, appreciate each other, and grow together.
- A structured onboarding, a personal development plan, and regular internal lectures and demos from the team.
- Internal programs for adaptation and training, development of soft skills, and leadership abilities.
- Partial compensation for participating in external training and conferences.
- Corporate English school: Group and individual lessons, speaking clubs with colleagues from all over the world.
- Corporate prices on hotels and travel services.
- MyTime Day Off - an extra non-working day without loss of compensation.
Similar Jobs
Cloud • Security • Software • Cybersecurity • Automation
Develop and improve GitLab’s self-service growth funnel across registration, trials, onboarding, upgrades, and purchases. Build Ruby on Rails backend services and Vue.js frontend experiences, design and evaluate A/B tests, analyze product data, and maintain experimentation tooling. Collaborate asynchronously with Product, UX, Data, and Engineering teams to ship measurable improvements, document decisions, and optimize customer conversion across GitLab.com and self-managed GitLab.
Top Skills:
A/B TestingExperimentation FrameworksFeature FlagsGitJavaScriptPostgresRubyRuby On RailsVue
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Build and ship AI-native product capabilities, including models, tools, retrieval, and evaluation. Lead technical delivery for product areas, design and review code, partner with product owners, support CI/CD and production readiness, instrument features, improve shared engineering patterns, and mentor AI Product Engineers. The role requires strong software engineering fundamentals, production systems experience, and hands-on AI/ML or generative AI integration.
Top Skills:
Ai/MlSparkAPIsCi/CdDistributed Web SystemsGenerative Ai
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Analyzes enterprise data across PostgreSQL, Salesforce, Splunk, and integrated systems. Writes complex SQL queries, investigates logs and data discrepancies, performs reconciliation and validation, identifies root causes, and recommends improvements. Collaborates with technical and business stakeholders to translate requirements into actionable insights, document findings, resolve cross-system issues, and improve data quality and operational efficiency in a financial services environment.
Top Skills:
APIsConfluenceEtl/EltJIRAPostgresSalesforceSOQLSplunkSQL
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
