Lead engineering team to architect, build, and operate large-scale, distributed Node.js web scraping systems. Design data extraction pipelines, solve anti-bot challenges (CAPTCHA, fingerprinting, IP rotation), manage proxies and storage, optimize cloud-native, containerized deployments, implement monitoring and CI/CD, ensure legal compliance, and drive technology roadmap and team growth.
We are seeking a Director of Software Engineering with deep expertise in Node.js development and large-scale web scraping. This role will lead the engineering team, designing and optimizing high-performance, distributed web scraping systems. The ideal candidate has extensive experience in handling anti-bot measures, data pipeline optimization, and scalable cloud-based architectures.
Key Responsibilities- Software Engineering & Web Scraping Leadership:
- Architect, develop, and maintain scalable and distributed web scraping systems using Node.js.
- Design and implement data extraction pipelines to process large volumes of structured and unstructured data.
- Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation.
- Optimize scraping processes for performance, reliability, and efficiency while managing proxy services(residential, datacenter, rotating).Oversee data storage and processing strategies, ensuring high availability and consistency.
- Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications.
- Implement best practices for microservices, API integrations, and real-time data streaming.
Key Responsibilities- Scalability, Security & DevOps:
- Lead the transition to cloud-native, containerized, and serverless architectures for web scraping.
- Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.).Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping.
- Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks.
- Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability.
Key Repsonsibilities- Engineering Team Management & Strategy:
- Lead, mentor, and grow a high-performance engineering team.
- Define and execute the technology roadmap, aligning with business objectives.
- Foster a culture of continuous learning, collaboration, and innovation.
- Implement agile development methodologies (Scrum, Kanban) to optimize project execution.
- Ensure code quality, security, and best practices across all engineering efforts.
Qualifications & Experience- Technical Expertise:
- 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
- Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
- Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
- Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
- Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).
Qualifications & Experience- Leadership & Soft Skills:
- Proven track record of scaling engineering teams and leading complex projects.
- Strong problem-solving and debugging skills, especially for scraping challenges and performance bottlenecks.
- Excellent communication and stakeholder management skills.
- Passion for mentorship, team development, and continuous learning.
Preferred Qualifications:
- Experience with machine learning for data extraction and NLP.
- Knowledge of browser fingerprinting and bot detection mechanisms.
- Familiarity with enterprise-scale web crawling frameworks (Scrapy, Colly, Apify, etc.).
- Prior leadership experience in data-driven businesses or web scraping startups.
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Software • Biotech
Lead biopharma business development across the US West Coast and Central States, acquiring pharmaceutical and biotechnology customers and expanding strategic accounts. Build and manage Salesforce pipeline, develop proposals, negotiate agreements, and close projects involving genomic assays, biomarker discovery, clinical trial assays, precision medicine, and companion diagnostics. Collaborate with scientific, commercial, and operational teams to shape customer solutions. Travel domestically and internationally approximately 30% of the time.
Top Skills:
Companion Diagnostics (Cdx)Next-Generation Sequencing (Ngs)SalesforceSophia Ddm
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Lead incident response consulting engagements, including client scoping, forensic and malware investigations, threat hunting, remediation planning, and executive communications. Coach junior analysts conducting host and network forensics across Windows, macOS, and Linux. Develop recommendations to improve clients’ security posture, produce reports for stakeholders and legal counsel, contribute thought leadership, and travel up to 30% as needed.
Top Skills:
Artificial IntelligenceAWSBro/ZeekGoogle Cloud PlatformLinuxmacOSMalware AnalysisAzureNetwork ProtocolsSuricataWindows
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Leads proactive media relations strategies for cybersecurity products, business initiatives, and executives. Builds relationships with cybersecurity, technology, business, and financial reporters; develops and pitches compelling stories; secures high-impact earned coverage; prepares executives for media engagements; translates complex cybersecurity trends into newsworthy narratives; monitors industry and competitor activity; and collaborates with leadership, marketing, technical teams, regional PR groups, and agencies.
Top Skills:
Ai Tools
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
