Kyndryl

Lead Data Engineer

Kyndryl
Apply
2 days ago
Bengaluru, India or Hyderābād, IndiaStaff+

Responsibilities

  • Architect and scale RAG pipelines that process unstructured IT logs and documentation into vector embeddings.
  • Maintain high-performance vector database infrastructure and optimize retrieval for agentic reasoning workflows.
  • Build knowledge graphs, semantic layers, metadata-driven ELT frameworks, reusable pipeline templates, and data-quality frameworks.
  • Create automated guardrails to identify noise, bias, PII, and other data-quality issues before data reaches models.
  • Develop, deploy, and maintain production data infrastructure and associated pipeline automation.
  • Apply performance tuning, partitioning, incremental processing, schema evolution, and large-volume data-processing techniques.
  • Support healthcare data requirements including patient identity, effective dating, PHI handling, lineage, and auditing.

Requirements

  • 7+ years of experience in data engineering or data warehousing.
  • 4+ years of hands-on experience with Databricks/Spark and Snowflake.
  • Advanced hands-on experience with Databricks, Python, SQL, Spark/PySpark, Delta Lake, and Snowflake.
  • Strong experience with Databricks Workflows/Jobs, notebooks, Unity Catalog, Git, and CI/CD.
  • Experience building metadata-driven ELT frameworks, reusable pipeline templates, data-quality frameworks, reconciliation utilities, and automated testing.
  • Knowledge of dimensional and normalized data modeling, troubleshooting, debugging, and root-cause analysis.
  • Experience with healthcare data, including patient identity, effective dating, PHI handling, lineage, and audit requirements.
  • Preferred: experience with GenAI/RAG applications, agentic AI, generative AI, LLMs, prompt engineering, embeddings, vector search, MLflow, AI/ML lifecycle practices, enterprise LLM platforms, agent frameworks, and productionizing AI solutions.
  • Preferred: healthcare domain experience and familiarity with PHI-sensitive datasets.

Benefits

  • Hybrid-friendly work culture.
  • Professional certifications and qualifications, including learning opportunities involving Microsoft, Google, and Amazon.
  • Be Well programs supporting financial, mental, physical, and social health.
  • Personalized development goals, coaching, continuous feedback, and career-path tools.
  • Opportunities to grow into other technical or non-technical roles.

Tech Stack

Apache SparkDatabricksGitMLflowPythonSnowflakeSQL

Categories

Data Engineering
Kyndryl

About Kyndryl

10,000+ employees

Kyndryl is a public IT services company that designs, runs, and modernizes mission-critical infrastructure for large enterprises and governments. Spun off from IBM in 2021, it is headquartered in New York City and trades on the NYSE (ticker: KD). The company provides managed infrastructure, cloud migration and operations across AWS, Azure, and Google Cloud, plus mainframe, network, security, data, and digital workplace services.

Contact me