Senior Software Engineer – Annotation & Evaluation (AI Platform)
Datadog
8 months ago
Paris, France
Senior
H1B Sponsor
Responsibilities
- Design and build systems for automated evaluation of AI models using production-like telemetry.
- Lead the development of benchmark suites and evaluation pipelines with trust and safety metrics.
- Integrate labeling systems and coordinate annotation workflows.
- Collaborate with Applied AI teams for fast iteration and reproducible experimentation.
- Develop data pipelines to track model behavior metrics and alerts.
- Implement best practices for safe model deployment, including bias checks and hallucination detection.
Requirements
- 6+ years of software engineering experience, with 2+ years in ML/AI products.
- Experience in building backend or data infrastructure systems for model evaluation.
- Comfortable working across the stack, from data processing pipelines to UI integrations.
- Solid understanding of ML concepts, including performance metrics and trust & safety.
- Fluent in Python or Go, with familiarity in cloud-native development patterns.
- Bonus: experience with open-source evaluation frameworks and human feedback loops.
Benefits
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP).
- Continuous professional development and career pathing opportunities.
- Intradepartmental mentor and buddy program for networking.
- Inclusive company culture with Community Guilds for employee resource groups.
- Access to internal panel discussions on inclusion.
- Free global mental health benefits for employees and dependents age 6+.
- Competitive global benefits.
Tech Stack
GoPython
Categories
AI & MLBackendData EngineeringData Science