Backend Engineer, Collection Platform

At Vannevar, we build mission software for the national security, defense, and intelligence communities. The Collection Platform team builds the systems that acquire mission-critical data from websites, APIs, feeds, and third-party platforms.

In this role, you’ll be a hands-on technical leader for collection and ingestion systems. You’ll build web scrapers, third-party integrations, and data pipelines that produce complete, correct, and timely data despite unreliable upstream systems. You’ll partner across engineering, product, and mission teams to meet mission needs.

• Shape the architecture and technical direction of collection and ingestion systems, looking 6–12 months ahead to identify needs and develop plans to address them.
• Build and operate production web scrapers, third-party integrations, and data pipelines, delivering major services and infrastructure with impact across the organization.
• Design for schema changes, rate limits, authentication requirements, and partial failures. Set collection schedules based on source change frequency, user needs, downstream capacity, and cost.
• Build reliable retry, backfill, deduplication, idempotency, and replay mechanisms.
• Define and monitor data completeness, correctness, freshness, duplication, and provenance.
• Develop APIs, databases, and shared services for collection workflows; simplify systems and processes so teams can deliver more with less work.
• Make sustained improvements to the stability, performance, and scalability of mission-critical collection systems.
• Partner with mission, product, and platform teams to evaluate sources, prioritize what to collect or repair, and measure whether collected data meets user needs.
• Guide engineers through design and code reviews, technical mentorship, and clear communication of decisions and standards.

• You have 8+ years of backend engineering experience, including deep, hands-on experience building and operating production collection or ingestion systems—for example, web crawlers, third-party data integrations, ETL pipelines, or change data capture systems.
• You have led the design and delivery of large services, complex libraries, or major infrastructure, with impact beyond a single team.
• You have a record of improving the stability, performance, and scalability of business-critical systems and maintaining data quality across large, changing datasets.
• You have designed systems that tolerate upstream failures and changing external interfaces.
• You make thoughtful architectural tradeoffs among reliability, scalability, performance, maintainability, and cost.
• You communicate technical decisions clearly and work well across engineering, product, and mission teams.
• U.S. Person status is required, as this role requires access to U.S.-only data systems and the ability to obtain a security clearance.

• Experience with distributed task processing, workflow orchestration, or large-scale data pipelines.

• Health, dental, and vision insurance
• Remote friendly with WeWork access
• Unlimited PTO, shared downtime during the federal holiday calendar, and company-wide off time at the end of each year
• 401(k) match
• Lifestyle & wellbeing stipends
• Salary top-up during military reserve duty
• Fully paid parental leave
• Child and pet care reimbursement during travel