Job Brief:
The Data Acquisition team operates at the core of our platform, engineering efficient systems that empower all our clients through all our products. We design and run large scale, highly distributed pipelines capable of scheduling, orchestrating, and executing tens of millions of data collection tasks daily across a diverse and constantly evolving landscape of sources. This involves tackling complex challenges in high-throughput web crawling, adaptive extraction, intelligent batching, fault tolerant processing, and seamless distribution of mission critical datasets. We are seeking a Software Engineer who thrives on building resilient, scalable systems, solving real world data engineering puzzles, and pushing the boundaries of automation and efficiency in large-scale information retrieval.
Essential Function & Responsibilities:
- Design and implement advanced capabilities to detect and counter anti-bot protection methods.
- Engineer complex network requests that accurately mimic real user behavior and environments.
- Develop network services leveraging HTTP(S) protocol versions 1.1, 2, and 3.
- Implement and manage egress proxy networks with MITM traffic interception and routing optimization.
- Extend Chrome DevTools Protocol (CDP) with custom commands to support specialized use cases.
- Build and maintain scalable WebSocket infrastructures optimized for short-lived, high-volume connections.
- Design distributed caching systems to support high throughput containerized worker pools.
- Deploy and manage geographically distributed proxies with automated failover and load balancing.
- Optimize compute, storage, and traffic costs for high-volume cloud-based services.
- Proficiency with data parsing, transformation, and storage technologies.
- Unwavering focus on delivering measurable value to customers
Location:
This is a remote position open to candidates based in the United States.
Compensation:
The salary for this role is $130,000-160,000. Please note that our salary ranges are based on current market data and the offered compensation may vary based on experience, skills, location, and other factors permitted by law.
Wayvia is an equal opportunity employer that is committed to inclusion and diversity.
Top Skills
Wayvia (formerly PriceSpider) Irvine, California, USA Office
4000 Barranca Pkwy, Irvine, CA , United States, 92604
Similar Jobs
What you need to know about the Los Angeles Tech Scene
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
