2 months ago
Pune, IndiaStaff+
Responsibilities
- Design, build, and maintain automated testing frameworks for AI/ML systems, LLMs, RAG pipelines, and traditional ML models.
- Create automated evaluation pipelines for hallucination rates, context retention, retrieval precision, semantic similarity, and answer relevance.
- Validate data quality, integrity, schema compliance, ETL processes, and data flows into AI models.
- Integrate AI quality gates into enterprise CI/CD pipelines using Tekton, Harness, UDeploy, and Jenkins.
- Lead adversarial testing for prompt injection, jailbreaking, data leakage, and other AI vulnerabilities.
- Validate real-time safety guardrails against ethical AI guidelines, corporate policies, and regulatory standards.
- Implement monitoring and observability for model drift, performance degradation, and latency bottlenecks in production.
- Collaborate with data scientists, ML systems engineers, front-end engineers, business leaders, and other stakeholders to define quality benchmarks and resolve quality risks.
Requirements
- 5 –8 years of professional experience implementing data-intensive solutions or test automation frameworks, including 2–3 years directly validating AI/ML, generative AI, or data-intensive systems.
- Strong Python, shell scripting, and Jupyter Notebook scripting experience; Java is a secondary language.
- Experience with machine learning and AI evaluation technologies including Scikit-learn, Keras, TensorFlow, PyTorch, spaCy, NLTK, RAGAs, DsPy, Hugging Face Transformers, and RASA.
- Experience with MLOps and lifecycle platforms including MLFlow, Dagster, Google Cloud Vertex AI, and Amazon SageMaker.
- Experience with relational and NoSQL databases including Oracle, MySQL, PostgreSQL, and MongoDB.
- Experience with CI/CD, DevOps, version control, containers, and orchestration tools including UDeploy, Harness, Tekton, Jenkins, Maven, Git, Subversion, Docker, OpenShift, and Amazon ECS.
- Experience with data engineering and processing technologies including Kafka, RabbitMQ, Sqoop, PySpark, Hive, Hadoop, HDFS, Avro, Parquet, and Iceberg.
- Experience with scheduling, web frameworks, and web infrastructure including Autosys, Cron, FastAPI, Flask, and NGINX.
- Exceptional communication and diplomacy skills for engaging stakeholders, articulating quality risks, negotiating solutions, and aligning on quality benchmarks.
- Bachelor’s or university degree, or equivalent experience.
Tech Stack
Apache HadoopApache HiveApache KafkaDockerFastAPIFlaskGitHarnessHugging Face TransformersJavaJenkinsJupyter NotebookKerasMavenMLflowMongoDBMySQLNLTKOpenShiftPostgreSQLPythonPyTorchRabbitMQscikit-learnspaCySubversionTektonTensorFlow
Categories
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.