Kyndryl

Lead Data Engineer

Kyndryl
Apply
2 days ago
Bengaluru, India or Hyderābād, IndiaStaff+

Responsibilities

  • Architect and scale RAG pipelines that transform unstructured IT logs and documentation into optimized vector embeddings.
  • Maintain the health and performance of vector databases such as Pinecone, Milvus, and Weaviate for sub-second retrieval.
  • Build knowledge graphs and semantic layers that provide contextual information for AI agents.
  • Create automated data guardrails to detect noise, bias, and personally identifiable information before data reaches models.
  • Identify and resolve data-quality issues at their source.
  • Build, deploy, and maintain data-infrastructure CI/CD pipelines.
  • Develop metadata-driven ELT frameworks, reusable pipeline templates, data-quality frameworks, reconciliation utilities, and automated testing.
  • Optimize partitioning, incremental processing, schema evolution, performance, and large-volume data processing.
  • Apply healthcare data practices involving patient identity, PHI, lineage, effective dating, and audit requirements.

Requirements

  • At least 7 years of experience in data engineering or data warehousing.
  • At least 4 years of hands-on experience with Databricks, Spark, and Snowflake.
  • Advanced hands-on experience with Databricks, Python, SQL, Spark, PySpark, Delta Lake, and Snowflake.
  • Strong experience with Databricks Workflows/Jobs, notebooks, Unity Catalog, Git, and CI/CD.
  • Knowledge of dimensional and normalized data modeling.
  • Strong troubleshooting, debugging, and root-cause-analysis skills.
  • Preferred: experience building GenAI or RAG applications on Databricks, including agentic AI, LLMs, prompt engineering, embeddings, and vector search.
  • Preferred: experience with MLflow, AI/ML lifecycle practices, Azure OpenAI or equivalent enterprise LLM platforms, and agent frameworks.
  • Preferred: experience taking AI or GenAI solutions from proof of concept to production, including monitoring, security, and governance.
  • Preferred: healthcare domain experience and familiarity with PHI-sensitive datasets.

Benefits

  • Hybrid-friendly work culture.
  • Professional certifications and qualifications, including learning opportunities involving Microsoft, Google, and Amazon.
  • Be Well programs supporting financial, mental, physical, and social health.
  • Personalized development goals, career-path tools, coaching, continuous feedback, and hands-on learning experiences.

Tech Stack

Apache SparkDatabricksGitMLflowPythonSnowflakeSQL

Categories

Data Engineering
Kyndryl

About Kyndryl

10,000+ employees

Kyndryl is a public IT services company that designs, runs, and modernizes mission-critical infrastructure for large enterprises and governments. Spun off from IBM in 2021, it is headquartered in New York City and trades on the NYSE (ticker: KD). The company provides managed infrastructure, cloud migration and operations across AWS, Azure, and Google Cloud, plus mainframe, network, security, data, and digital workplace services.

Contact me