Build and maintain scalable web crawlers to extract, validate, and structure product data from eCommerce sites. Use AI-driven techniques, browser automation, proxies, and APIs to handle dynamic content and anti-bot measures. Monitor performance, clean large datasets, collaborate with Product/Engineering/Data teams, and continuously improve crawling frameworks and automation.
Data Engineer - ConsultantRole Overview
Part 1: The Fundamentals
Work Model
Investors
Visit Us
You will be responsible for building and maintaining Rubick's data acquisition engine by extracting, validating, and structuring product information from eCommerce websites and marketplaces. This role focuses on developing scalable web crawling solutions that power Product Discovery, Search, Catalog Intelligence, and Market Intelligence platforms.
The role involves three areas: Part 1: The Fundamentals | Part 2: AI-Driven Data Crawling Excellence | Part 3: Innovation & Improvements
Part 1: The Fundamentals
- Develop, maintain, and optimize web crawlers to extract product data from eCommerce websites and marketplaces.
- Collect structured and unstructured product information, including product details, pricing, images, specifications, and availability.
- Clean, validate, and organize extracted datasets to ensure accuracy and consistency.
- Monitor crawler performance, identify failures, and resolve data extraction issues.
- Collaborate with Product, Engineering, and Data teams to ensure reliable and timely data collection.
- Maintain documentation for crawling processes, extraction rules, and data quality standards.
- Leverage AI-powered extraction techniques to improve data accuracy and extraction efficiency.
- Build intelligent crawling workflows using Python and modern web automation frameworks.
- Develop scalable solutions for handling dynamic websites, JavaScript-rendered content, and anti-bot mechanisms.
- Utilize browser automation, APIs, proxies, and scheduling tools to maximize crawl success rates.
- Implement automated data validation and monitoring systems to maintain high-quality datasets.
- Collaborate with AI and Data Engineering teams to support Machine Learning models and Product Intelligence systems.
- Continuously optimize crawling performance for speed, scalability, and reliability.
- Identify opportunities to automate repetitive extraction and validation workflows.
- Improve data collection strategies by adopting new crawling technologies and AI-assisted solutions.
- Build reusable crawling frameworks and standardized extraction pipelines.
- Stay updated with the latest web scraping libraries, browser automation tools, and industry best practices.
- Contribute to knowledge repositories, documentation, and process improvements across the data acquisition function.
- 1+ year of experience in Web Crawling, Data Scraping, Product Matching, Data Extraction, or similar roles.
- Strong proficiency in Python for web scraping and automation.
- Hands-on experience with BeautifulSoup, Scrapy, Selenium, Playwright, or similar frameworks.
- Good understanding of HTML, CSS, XPath, JSON, and DOM structures.
- Familiarity with REST APIs, proxies, browser automation, and dynamic website scraping.
- Strong analytical and problem-solving skills.
- Ability to work with large datasets while maintaining high data quality.
- Experience in eCommerce, Retail, Marketplace Operations, or Catalog Management is preferred.
- Exposure to AI-assisted data extraction, Product Intelligence, or Machine Learning data pipelines is a strong advantage.
Rubick OS is a leading platform for brands and marketplaces, enabling seamless management of catalog automation, pricing and competitive intelligence, product listing intelligence, CAST systems, and brand and assortment intelligence.
Our solutions are used across five countries by major brands and marketplaces, including Amazon, Myntra, Reliance Group, Nykaa, Flipkart, Rare Rabbit, Decathlon, Celio, The Bay, Kiabi, Myer, Jumbo, and many more.
Work Model
Work from office – Monday to Saturday
Team Size250+ Employees
Invested by Innospark US, Betatron Hong Kong, MJV India, 1Crowd India, and other investors.
Rubick.ai is a place to work if you want to build something foundational rather than incremental—it’s aiming to become the AI operating layer for ecommerce, solving complex, real-world problems across cataloging, pricing, and marketplace operations at scale.
You get high ownership, direct impact, and exposure to various parts of the business, making you an entrepreneur in-house. Rubick operates in the AI segment, which means you will be at the forefront of the technological revolution.
Rubick is profitable and growing and is looking to work with 10,000 brands in the next 2–3 years.
Join us in this exciting journey.
Visit Us
www.rubick.ai
Similar Jobs
Artificial Intelligence • Enterprise Web • Information Technology • Productivity • Sales • Software • Database
Lead Apollo’s application security strategy across engineering, product, platform, data, and AI teams. Responsibilities include security architecture reviews, threat modeling, secure code review, penetration testing, exploit development, vulnerability management, secure-by-default building blocks, AI security standards, automation, and mentoring security engineers and champions. The role requires solving ambiguous, high-scale AppSec problems, influencing cross-functional roadmaps, and delivering measurable reductions in security risk.
Top Skills:
AIApplication SecurityDockerElasticsearchExploit DevelopmentGCPKubernetesLinuxMongoDBOauthPenetration TestingPythonReactRubyRuby On RailsScaTerraformThreat Modeling
Artificial Intelligence • Enterprise Web • Information Technology • Productivity • Sales • Software • Database
Leads Apollo’s detection and response strategy while remaining hands-on with detection engineering, security observability, threat hunting, response automation, and incident investigations. The role owns detection architecture, telemetry coverage, incident response readiness, playbooks, tabletop exercises, and performance metrics. It leads high-severity incidents across cloud, SaaS, and corporate systems, mentors engineers, partners with technical and business teams, and establishes AI-assisted security workflows using human oversight.
Top Skills:
AtlassianCi/CdClaudeClaude CodeCloudflareCrowdstrikeCursorDockerGCPGitGitGleanGoogle WorkspaceKandjiKubernetesMitre Att&CkOktaPantherPythonSIEMSlackSnowflakeTerraform
Artificial Intelligence • Enterprise Web • Information Technology • Productivity • Sales • Software • Database
The Senior Application Security Engineer will strengthen Apollo’s secure software development lifecycle through application security reviews, threat modeling, code analysis, vulnerability management, penetration testing, remediation, and AppSec tooling. The role includes developing automation, securing AI-powered features, improving security observability, enabling developers, and partnering across Engineering, Product, Platform, Data, Legal, and Security teams. This senior individual contributor will own long-term AppSec goals, resolve ambiguous risks, and drive cross-functional security improvements.
Top Skills:
Ai ApisCsrfGoogle Cloud Platform (Gcp)LinuxOauthPythonRubySastSoftware Composition Analysis (Sca)Ssrf
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
