AI Data & Knowledge Engineer – Senior Manager
State Street23 hours ago
Hyderābād, IndiaStaff+
Responsibilities
- Design, build, and maintain scalable batch and streaming data pipelines for AI and analytics use cases.
- Own end-to-end ETL/ELT workflows, including ingestion, cleansing, transformation, enrichment, and publishing.
- Build AI-ready datasets and retrieval pipelines supporting RAG, GraphRAG, semantic search, and knowledge-driven applications.
- Develop and manage knowledge graphs, ontologies, metadata layers, and data contracts.
- Implement vector indexing, chunking, hybrid retrieval, and backend data preparation workflows.
- Integrate source systems, data platforms, and AI application layers, including MCP server integrations where relevant.
- Monitor pipeline health, data quality, latency, throughput, and reliability; troubleshoot production issues and optimize performance.
- Mentor junior engineers and contribute to data engineering, data governance, and AI data enablement best practices.
Requirements
- Bachelor’s or master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.
- 5–10 years of experience in data engineering, platform engineering, or backend data systems.
- Strong hands-on experience with Python and SQL.
- Experience building production ETL/ELT pipelines and scalable data integration workflows.
- Experience with big data and distributed processing frameworks such as Spark, Databricks, Hadoop ecosystem components, or equivalent platforms.
- Familiarity with workflow orchestration tools such as Airflow or similar.
- Experience with cloud data platforms, warehouses, or lakehouses such as Snowflake, Databricks, BigQuery, Redshift, or equivalent.
- Understanding of data contracts, metadata management, data lineage, and data governance practices.
- Preferred exposure to RAG, GraphRAG, vector databases, or knowledge graphs in enterprise settings.
- Experience with streaming or event-driven patterns using Kafka, Kinesis, or similar tools is a plus.
- AWS and/or Azure cloud certifications are a plus.
Benefits
- On-premise work schedule.
- Inclusive development opportunities, flexible work-life support, paid volunteer days, and employee networks.
Tech Stack
Amazon RedshiftApache AirflowApache HadoopApache KafkaApache SparkAWSAzureDatabricksGoogle BigQueryPythonSnowflakeSQL
Categories
Data Engineering
About State Street
State Street is a global custodian bank and asset manager serving institutional investors with investment servicing, fund accounting, trading/FX, data, and index/active strategies. Its products include State Street Global Advisors’ funds and ETFs, the Charles River Investment Management Solution, and the front-to-back State Street Alpha platform. Founded in 1792 and headquartered in Boston, the company is publicly traded on the NYSE (ticker: STT) and operates across major markets worldwide.