AI Evaluation Engineer
Yes Energy3 months ago
Bucharest, RomaniaSenior
Responsibilities
- Own and continuously improve the end-to-end evaluation strategy for Yes AI.
- Design benchmarks and automated test suites for hallucination, factual drift, retrieval failures, accuracy, and reliability.
- Build and maintain scalable evaluation pipelines that run against prompt changes, model updates, and Snowflake data changes.
- Partner with power market domain experts and senior engineers to define and codify evaluation criteria.
- Lead prompt engineering and RAG experimentation and use evaluation results to improve answer quality.
- Make AI quality measurable for engineering and product decisions and help establish the AI evaluation discipline at Yes Energy.
- Work directly on projects that integrate systems and serve energy data to customers.
Requirements
- Bachelor's degree in Computer Science, Data Science, Mathematics, or a related field.
- 5+ years of software engineering experience with significant focus on testing, evaluation, quality systems, or AI/ML tooling.
- Strong Python skills and experience building production-grade evaluation or testing pipelines.
- Understanding of LLM evaluation concepts including benchmarking, hallucination detection, retrieval-augmented generation quality, and semantic similarity.
- Experience designing and owning quality frameworks in ambiguous, fast-moving environments.
- Ability to work with non-engineering subject matter experts to define and codify quality criteria.
- Strong judgment in distinguishing meaningful evaluation problems from noise.
- Strong written and verbal communication skills for technical and non-technical audiences.
- Preferred experience with RAGAS, LangSmith, PromptFlow, production data pipelines using RAG or vector databases, and data-intensive or regulated environments with strict data governance.
- Interest in learning the energy industry or applying experience from equities or commodities trading environments.
Benefits
- Net salary of 16.000 - 19.000 RON per month.
- Full-time position in Bucharest, Romania.
- Hybrid work requiring 2-3 office days per week.
- Private medical insurance, wellness/gym benefits, flexible vacation, and flexible work schedules.
- Company-funded formal and informal professional development.
- Real bonuses and a diverse, inclusive workplace.
Tech Stack
PythonSnowflake
Categories