15 days ago
Pune, IndiaStaff+
Responsibilities
- Design, build, and maintain automated testing frameworks for AI/ML systems, LLMs, RAG pipelines, and traditional ML models.
- Create automated evaluation pipelines for hallucination rates, context retention, retrieval precision, semantic similarity, and answer relevance.
- Validate data quality, integrity, schema compliance, ETL processes, and data flows into AI models.
- Integrate AI quality gates into enterprise CI/CD pipelines using Tekton, Harness, UDeploy, and Jenkins.
- Lead adversarial testing for prompt injection, jailbreaking, data leakage, and other AI vulnerabilities.
- Validate real-time safety guardrails against ethical AI guidelines, corporate policies, and regulatory standards.
- Implement monitoring and observability for model drift, performance degradation, and latency bottlenecks in production.
- Collaborate with data scientists, ML systems engineers, front-end engineers, business leaders, and other stakeholders to define quality benchmarks and resolve quality risks.
Requirements
- 5–8 years of professional experience implementing data-intensive solutions or test automation frameworks, including 2–3 years directly validating AI/ML, generative AI, or data-intensive systems.
- Strong Python, shell scripting, and Jupyter Notebook scripting experience; Java is a secondary language.
- Experience with machine learning and AI evaluation technologies including Scikit-learn, Keras, TensorFlow, PyTorch, spaCy, NLTK, RAGAs, DsPy, Hugging Face Transformers, and RASA.
- Experience with MLOps and lifecycle platforms including MLFlow, Dagster, Google Cloud Vertex AI, and Amazon SageMaker.
- Experience with relational and NoSQL databases including Oracle, MySQL, PostgreSQL, and MongoDB.
- Experience with CI/CD, DevOps, version control, containers, and orchestration tools including UDeploy, Harness, Tekton, Jenkins, Maven, Git, Subversion, Docker, OpenShift, and Amazon ECS.
- Experience with data engineering and processing technologies including Kafka, RabbitMQ, Sqoop, PySpark, Hive, Hadoop, HDFS, Avro, Parquet, and Iceberg.
- Experience with scheduling, web frameworks, and web infrastructure including Autosys, Cron, FastAPI, Flask, and NGINX.
- Exceptional communication and diplomacy skills for engaging stakeholders, articulating quality risks, negotiating solutions, and aligning on quality benchmarks.
- Bachelor’s or university degree, or equivalent experience.
Tech Stack
Apache HadoopApache HiveApache KafkaDockerFastAPIFlaskGitHarnessHugging Face TransformersJavaJenkinsJupyter NotebookKerasMavenMLflowMongoDBMySQLNLTKOpenShiftPostgreSQLPythonPyTorchRabbitMQscikit-learnspaCySubversionTektonTensorFlow
Categories
About Citi
Citi's mission is to serve as a trusted partner to our clients by responsibly providing financial services that enable growth and economic progress. Our core activities are safeguarding assets, lending money, making payments and accessing the capital markets on behalf of our clients. We have over 200 years of experience helping our clients meet the world's toughest challenges and embrace its greatest opportunities. We are Citi, the global bank – an institution connecting millions of people across hundreds of countries and cities. For information on Citi’s commitment to privacy, visit on.citi/privacy.