Site Reliability Engineer

About Subzero Labs

Subzero Labs is building the next generation of decentralized infrastructure.

Site Reliability Engineer

We're building the infrastructure behind a next-generation decentralized programmable network with reliability, observability, and confidentiality baked in from the ground up. As a Site Reliability Engineer, you'll ensure the scalability, performance, and reliability of our large-scale blockchain applications and infrastructure.

Position Overview

Combining software engineering and systems administration expertise, you'll adopt a proactive, software-centric approach to tackle operational challenges. Your responsibilities include detecting issues, automating failure handling, devising disaster recovery plans, maintaining system uptime, and mitigating broken systems to prevent future disruptions. You'll leverage coding, automation, and engineering principles to build resilient, self-healing systems that scale to meet growing demands.

What You'll Do

Network Infrastructure Reliability: Design fault-tolerant systems to run validators, nodes, and indexers across cloud and bare-metal environments. Build self-healing mechanisms that recover automatically from faults, crashes, and partitions.

Infrastructure as Code: Define production systems using Terraform, Helm, Kubernetes, or Pulumi—supporting reproducible deployments, rapid scaling, and multi-region HA clusters.

TEE-Backed Secure Computation: Deploy and manage trusted execution environments (TEEs) such as Intel TDX, AMD SEV-SNP, or Azure Confidential VMs for secure blockchain operations.

Observability & Alerting: Build comprehensive Grafana dashboards and AlertManager alerts to monitor chain liveness, network performance, and quality metrics. Instrument services with tracing, metrics, and logs down to the hardware level.

Performance & Resource Tuning: Profile and tune workloads under sustained high throughput—optimizing CPU/memory/disk I/O pressure. Build tools to detect degraded validators or slow block propagation in real time.

Security Hardening & Key Management: Engineer hardened signing pipelines integrating TEEs, HSMs, or cloud-native KMS systems. Manage key lifecycle (rotation, expiration, revocation) with zero downtime while reducing attack surface area.

CI/CD & Safe Rollouts: Build GitHub Actions workflows testing and shipping changes across multiple environments. Own release engineering across devnet, testnet, and mainnet, ensuring protocol compatibility and seamless validator upgrades.

Incident Response & Chaos Engineering: Run fire drills, simulate node failures and partitions, and lead incident postmortems. Design for failure and validate assumptions under pressure.

Cross-Functional Communication: Work closely with engineers, product managers, node operators, and partners to support deployments, debug edge cases, and share best practices as a key interface between core protocol teams and the network operator ecosystem.

Requirements

• 10+ years in DevOps or SRE roles with focus on tooling, automation, and infrastructure

• Proficiency in systems languages: Rust, Go, Python, Shell scripting

• Experience writing and reviewing code, developing documentation and disaster recovery plans, debugging complex problems on live blockchain systems

• Advanced knowledge of cloud infrastructure: networking, orchestration tools, containerization, compute, and storage systems

• Proven ability to design, develop, and deploy systems enhancing throughput, latency, reliability, availability, and security

• Clear communication skills: ability to explain technical concepts simply

• Self-starter mindset: continuous learning and critical thinking under pressure

Preferred Qualifications

• Background in distributed systems and consensus protocols

• Experience with monitoring and observability platforms

• Knowledge of security best practices for distributed systems