Senior Software Engineer, Data Science
Procore Technologies14 hours ago
Auckland, New ZealandSenior
Responsibilities
- Build and maintain dataset, ground-truth, labeling, and evaluation infrastructure for multiple AI products.
- Diagnose and resolve measurement-path issues between offline evaluations and production accuracy.
- Own prompt optimization from candidate selection through production rollout and post-launch validation.
- Automate evaluation pipelines and build reusable tooling for error analysis, audits, and experimentation at scale.
- Analyze the downstream impact of ontology and schema changes across models and infrastructure.
- Prioritize work by likely impact and collaborate with engineering, product, and AI teams.
Requirements
- Strong professional experience using Python for data manipulation, analysis, and tooling, including tools such as pandas and NumPy.
- Solid SQL and database experience, including querying, data modeling, and working with large datasets.
- Data science background with knowledge of LLM accuracy, particularly visual reasoning.
- Experience building evaluation datasets, running and debugging evaluations with tools such as Braintrust or Langfuse, and conducting error analysis.
- Experience designing ground-truth and labeling systems and measuring label agreement and data quality.
- Familiarity with prompt engineering, prompt lifecycle management, ontology design, and schema design.
- Construction or industrial inspection domain knowledge is a plus.
Benefits
- Based in Auckland with at least two days per week in the office.
- Requires good overlap with the 9–5 schedule, occasional on-call support, and occasional domestic and international travel.
- May be eligible for equity compensation and/or bonus incentive compensation.
Categories
About Procore Technologies
Procore Technologies builds a cloud platform for construction project management used by general contractors, specialty contractors, and owners to manage bidding, scheduling, documents, financials, and field safety via web and mobile apps. It sells its software by subscription and integrates with common accounting and ERP systems. Founded in 2002 and headquartered in Carpinteria, California, Procore is a public company (NYSE: PCOR) whose software supports projects worldwide.