Responsibilities
- Design, build, and operate multi-agent workflows and tool-enabled agents with orchestration logic, state management, safety guardrails, and fallback strategies.
- Architect and maintain end-to-end RAG systems covering document ingestion, chunking, embeddings, vector retrieval, reranking, and answer synthesis.
- Evaluate and integrate managed and in-house LLM and generative AI services across cost, performance, and privacy considerations.
- Develop, version, and optimize prompting strategies with automated prompt testing and regression tracking.
- Define and own evaluation frameworks using automated metrics, LLM-as-judge methods, human evaluation, hallucination detection, and drift monitoring.
- Apply classical NLP techniques where appropriate and monitor data distribution shifts affecting production model behavior.
- Build and operate scalable, secure AI infrastructure on AWS using well-architected and infrastructure-as-code practices.
- Own model and agent deployment lifecycles, including CI/CD, testing, observability, and rollback procedures.
- Ensure data quality through validation and augmentation and source datasets for training, fine-tuning, and evaluation.
Requirements
- Production experience designing multi-agent systems with tool use, memory or state management, and fault-tolerant routing.
- Hands-on experience building end-to-end RAG pipelines with chunking, embeddings, vector databases, retrieval tuning, and answer synthesis at production scale.
- Strong experience defining and running generative AI evaluation pipelines, including automated scoring, human evaluation, hallucination mitigation, and drift monitoring.
- Experience with foundation models and generative AI providers including AWS Bedrock, OpenAI, Anthropic, and Meta, including fine-tuning, instruction tuning, and prompt engineering.
- Grounding in classical NLP techniques such as named entity recognition, text classification, intent detection, and topic modeling.
- Hands-on AWS Bedrock experience with foundation model APIs, Bedrock Agents, Knowledge Bases, Guardrails, and preferably Bedrock Model Evaluation.
- Proficiency with SageMaker, Lambda, ECS or EKS, S3, OpenSearch, IAM, CloudWatch, and VPC networking.
- Familiarity with model registries, CI/CD for ML, feature stores, canary deployments, monitoring, and rollback.
- Experience with Terraform or AWS CDK for infrastructure provisioning.
- Production-quality Python experience including packaging, pytest, type hints, asynchronous programming, and clean ML pipeline abstractions.
- Familiarity with traditional machine learning frameworks and fine-tuning workflows.
- Experience with Langfuse, Arize, LangSmith, or equivalent experiment-tracking tools.
- Ability to translate ambiguous business goals into technical solutions and communicate tradeoffs to non-technical stakeholders.
- Strong collaboration across engineering, product, and data teams, with an evidence-driven approach to measuring, testing, and monitoring systems.
- Master’s degree in Machine Learning or Computer Science, preferably specializing in NLP.
Categories
About Mitratech
Mitratech is a trusted, global technology partner empowering corporate legal, risk & compliance, and HR professionals to maximize productivity, control expense, and mitigate risk by deepening operational alignment, and increasing collaboration & visibility across their organization. With Mitratech’s proven portfolio of end-to-end solutions, organizations worldwide are able to proactively identify and implement best practices and standardize processes across all lines of business to manage risk and ensure business continuity. Mitratech partners with more than 10,000 companies worldwide, from small nimble teams needing tech-enablement, to the largest organizations in the world trying to centralize, streamline and collaborate across their legal and risk functions. Mitratech's user base spans more than 160 countries. Innovate now by visiting: www.mitratech.com