3 days ago
Toronto, Canada or Dallas, TX, USAEntry Level / Mid Level
H1B sponsor
Responsibilities
- Write code and automation to improve GenAI product and platform resilience through retries, timeouts, circuit breakers, rate limiting, graceful degradation, and model/provider fallback.
- Run load tests, failure-injection tests, chaos or game-day exercises, and recovery tests to identify and fix weaknesses before production incidents.
- Harden Kubernetes services through health checks, autoscaling, disruption budgets, resource limits, and multi-zone deployment.
- Plan capacity for model-provider quotas, GPU and inference capacity, databases, launches, and growth.
- Maintain and regularly test backup, restore, and disaster-recovery plans.
- Define and track service-level indicators and objectives covering availability, latency, error rate, token cost, and quality scores.
- Instrument services with logs, metrics, and traces; build actionable dashboards and alerts for user-facing failures.
- Support release readiness, CI/CD and GitOps deployments, canary rollouts, and fast rollback.
- Participate in on-call rotations, respond to incidents, write blameless post-incident reviews, and track corrective actions to completion.
- Write and maintain runbooks and convert recurring operational issues into automation or permanent fixes.
- Support self-hosted open-weight and small language models, including serving, GPU usage, scaling, rollout, rollback, patching, secrets, and data-boundary requirements.
- Collaborate with AI engineers, product managers, platform teams, security teams, and product owners on reliability risks and production operations.
Requirements
- Requires 1–4 years of experience in site reliability, DevOps, platform, or software engineering with hands-on production systems work.
- Requires a bachelor’s degree in computer science, engineering, or a related field, or equivalent experience.
- Requires proficiency in Python or Go and solid version control, testing, and code review practices.
- Requires experience with Linux, containers, Kubernetes, and a major cloud platform, with Google Cloud preferred.
- Requires experience with CI/CD and infrastructure as code, such as Terraform.
- Requires working knowledge of monitoring, logging, tracing, and alerting.
- Requires understanding of distributed-system failure modes and resilience patterns.
- Requires experience with on-call or incident response and willingness to join an on-call rotation.
- Requires familiarity with LLM applications, including model APIs, retrieval, agents, and tool calls.
- Requires clear written and verbal communication.
- Preferred experience includes running GenAI or ML systems in production, managed model platforms such as Vertex AI, or model serving such as vLLM.
- Preferred experience includes OpenTelemetry, Prometheus, Grafana, PagerDuty, GitOps, Argo CD, Helm, canary deployments, load testing, chaos engineering, Vault, security scanning, and automation or AI agents that reduce operational work.
Benefits
- Employee Assistance Program is available to all employees.
- US employees may enroll in health, life, disability, and voluntary insurance plans; retirement plans; accrued paid time off; and paid holidays.
- Canada employees may enroll in healthcare, life, disability, and optional benefits; DCPP, RRSP, and TFSA savings or retirement plans; vacation; personal time; and paid holidays.
- The role is offered in the Toronto, Canada, and Dallas, United States offices; benefits are subject to eligibility and plan terms.
Tech Stack
Categories
Site Reliability
About Equinix
Equinix is a public digital infrastructure company that operates a global platform of colocation and interconnection data centers for enterprises, cloud and network providers. It sells space, power, and connectivity, plus services such as Equinix Fabric, Network Edge, and Equinix Metal for hybrid multicloud and edge deployments. Founded in 1998 and headquartered in Redwood City, California, it is listed on NASDAQ (EQIX) and runs hundreds of IBX facilities across major metros worldwide.