Define the architectural foundation for processing unstructured data for LLM applications. Oversee the core platform's development and mentor engineering teams.
About the Role
Unstructured is looking for a Senior Principal / Distinguished Engineer to define the architectural foundation for how enterprises process and transform unstructured data for large-language-model (LLM) applications.
This is a high-impact, hands-on role for someone who thrives in deep systems work—someone who could write the book on Kubernetes, lives and breathes Python, and understands Postgres at the index and transaction-log level. You’ll own the technical direction for our core platform, guiding how data moves through every layer—from ingestion to transformation to retrieval.
You’ll partner closely with the Head of Engineering, Head of Infrastructure, and product leaders to ensure Unstructured’s systems are performant, resilient, and ready to support the next generation of AI-driven workloads.
SF/Bay Area preferred - not required.
Responsibilities
Define and evolve the end-to-end architecture for Unstructured’s data transformation and retrieval platform.
Build and scale distributed systems that process massive volumes of unstructured data across diverse formats and sources.
Serve as the company-wide authority on Kubernetes orchestration, cluster design, performance tuning, and reliability.
Lead Python architecture and best practices—ensuring performance, modularity, and maintainability across services.
Design and optimize Postgres schemas, queries, and indexing strategies to support large-scale metadata and retrieval pipelines.
Mentor senior engineers through design reviews and code guidance, raising the bar for technical excellence across the org.
Partner with the infrastructure and product teams to translate research prototypes into production-grade systems.
Evaluate emerging technologies and open-source tools in LLM infrastructure, retrieval, and orchestration—deciding where and how to integrate them.
You Might Be a Fit If You
Have 15+ years of software engineering experience with a focus on distributed systems, infrastructure, or data architecture.
Are a Python expert—capable of building frameworks and performance-critical services from scratch.
Have deep Kubernetes expertise; you can design, deploy, and debug at scale and could teach others how to productionize it securely.
Are fluent in Postgres—you understand query planning, partitioning, and tuning for high-throughput environments.
Are obsessed with clean, scalable architecture and can lead design reviews that shape how entire systems evolve.
Have experience in high-performance data or AI/ML systems—especially those involving retrieval pipelines, embeddings, or hybrid workloads.
Thrive in fast-moving, ambiguous environments where technical depth and judgment matter more than process.
Bonus Points/Nice to Have
Experience building or scaling LLM-powered or RAG systems in production.
Familiarity with open-source orchestration frameworks, vector databases, or hybrid cloud infrastructure.
Contributions to open-source projects in Python, Kubernetes, or distributed systems.
Why This Role Matters
At Unstructured, we’re building the core infrastructure layer that powers enterprise-grade retrieval-augmented generation and unstructured data transformation. The systems you architect will define how data moves through every layer of that stack. You’ll shape the foundation that enables organizations, and the models that serve them, to reason over the world’s information.
Compensation
Competitive salary + equity + full benefits package. Level and title (Senior Principal vs. Distinguished) will be based on experience and scope of impact.
Top Skills
Kubernetes
Postgres
Python
Similar Jobs
Fintech • Machine Learning • Payments • Software • Financial Services
The Distinguished Engineer will drive innovation in network connectivity and infrastructure automation, mentoring teams and providing thought leadership in engineering.
Top Skills:
AnsibleAWSAzureBgpCloud NetworksGCPGoIp NetworksIs-IsJavaJavaScriptPythonSaltstackSd-WanSwiftTypescriptVxlanWi-Fi
Fintech • Machine Learning • Payments • Software • Financial Services
Lead the engineering community at Capital One, focusing on developer enablement, mentoring, and defining technical strategy to enhance productivity and software quality across the organization.
Top Skills:
ArtifactoryAWSAws CloudformationAzureCncf BuildpacksGCPGitGoHarnessJavaJavaScriptJenkinsPythonSonarSwiftTypescript
Fintech • Machine Learning • Payments • Software • Financial Services
The Director, Distinguished AI Engineer will lead teams to design, develop, and deploy AI solutions, enhancing customer interactions and business operations at Capital One.
Top Skills:
AWSAzureGoGCPHuggingfaceJavaNemo GuardrailsPythonPyTorchScalaVectordbs
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
