Senior Software Engineer II - Applied AI and Evaluations (Remote Eligible)
Smartsheetabout 2 hours ago
Remote, United States
Senior / Staff+
H1B Sponsor
Base Salary
$175k - $245k/yr
Responsibilities
- Own agent quality end-to-end, including diagnosis, improvement, and validation.
- Identify and prioritize failure modes across various quality dimensions.
- Drive quality improvements through prompt and context engineering.
- Extend and mature the evaluation framework for production traffic.
- Ensure every change has a measurable quality signal.
- Collaborate with the Agent Architecture lead on quality problem solutions.
- Establish a repeatable methodology for quality improvement.
Requirements
- 8+ years of software engineering experience, with 2 years in LLMs.
- Deep experience with prompt and context engineering.
- Strong knowledge of RAG architectures and failure diagnosis.
- Experience building LLM evaluation frameworks.
- Fluency in agent system design and architectural tradeoffs.
- Strong Python skills and experience in data-heavy environments.
- Ability to communicate complex findings to technical and non-technical stakeholders.
- Strong cross-functional judgment and decision-making skills.
- A bias for clarity in ambiguous situations.
- Legally eligible to work in the U.S. on an ongoing basis.
- BS or MS in Computer Science or equivalent experience.
Benefits
- Employer subsidized medical, vision, and dental coverage.
- 401k Match to help save for the future.
- Monthly stipend for work and productivity support.
- Flexible Time Away Program and Sick Time Off.
- Life insurance and disability plans for US employees.
- 12 paid holidays per year.
- Up to 24 weeks of Parental Leave.
- Personal paid Volunteer Day.
- Opportunities for professional growth and access to online courses.
- Company Funded Perks including counseling membership and local discounts.
- Teleworking options from any registered location in the U.S.
Tech Stack
DatabricksMLflowPython
Categories
AI & MLData Science