Senior Machine Learning Engineer, Defensive Agent
Horizon3 AI4 hours ago
Remote, United StatesSenior
Base Salary
$211k - $249k/yr
Responsibilities
- Build and own training and post-training pipelines covering data preparation, fine-tuning, preference optimization, experiment tracking, artifact management, and reproducibility.
- Build and operate model inference and serving infrastructure with provider routing, fallback, regional pinning, batching, and caching.
- Own versioned model-layer releases, including prompts, model selections, tool definitions, shadow deployments, canaries, and rollback.
- Develop model-layer monitoring for behavioral drift, regressions, output quality, latency, token usage, and per-tenant cost budgets.
- Build data, context, retrieval, and embedding pipelines with end-to-end tenant isolation.
- Optimize inference cost and latency and communicate tradeoffs using measurable data.
- Develop ETL and GraphQL product features for model outputs, run history, and evaluation results.
- Partner with AI researchers to productionize prototypes and feed operational constraints and failure data back into research.
Requirements
- Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.
- At least 5 years of professional software engineering experience with strong production Python.
- Demonstrated experience taking ML or LLM-backed systems from prototype to production and operating them.
- Hands-on experience with ML pipelines and tooling, including training or fine-tuning workflows, experiment tracking, artifact and model registries, and reproducible data preparation.
- Experience building and operating production inference or model-serving infrastructure, including latency and cost optimization.
- Experience building applications on cloud platforms such as AWS, Azure, or GCP and using Docker and Kubernetes.
- Solid SQL proficiency and experience with production data pipelines.
- Preferred qualifications include experience with LLM or agentic systems, post-training methods, model gateways, multi-provider routing, self-hosted or customer-hosted inference, GPU infrastructure, quantization, inference optimization, PostgreSQL, Neo4j, GraphQL backends, Datadog, Prometheus, Grafana, distributed tracing, and regulated or air-gapped environments.
Benefits
- Remote and hybrid work models are available depending on role and location, with some Chicago roles requiring regular in-office presence.
- Health, vision, and dental insurance are available for employees and their families.
- Flexible vacation policy and generous parental leave are provided.
- Full-time roles are eligible for equity in the form of stock options.
- The company offers growth and career development opportunities in an inclusive, collaborative environment.
Tech Stack
AWSAzureDatadogDockerGoogle Cloud PlatformGrafanaGraphQLKubernetesNeo4jPostgreSQLPrometheusPythonSQL
Categories
About Horizon3 AI
Horizon3 AI builds NodeZero, a proactive security platform that autonomously performs penetration testing and validates attack paths across on‑prem, cloud, and hybrid environments. The company sells its software by subscription to security and IT teams at enterprises and government agencies, helping prioritize and verify fixes. Founded in 2019 and headquartered in San Francisco, it is privately held and serves customers across regulated industries and the public sector.