17 hours ago
Coimbatore, IndiaMid Level / Senior

Responsibilities

  • Own the six-stage processing pipeline from ingestion through output.
  • Design the async Python orchestration framework with parallel/sequential execution, retries, and dead-letter queues.
  • Build connectors for enterprise APIs, databases, file systems, and SaaS platforms.
  • Implement PII/PHI detection, data integrity checks, format compliance, and input sanitization.
  • Build the reconstruction module for Markdown, vector representations, and context maps.
  • Design event-driven architecture with message queuing, backpressure handling, and graceful degradation.
  • Implement immutable audit-trail logging across all pipeline stages.
  • Ensure strict multi-tenant data isolation at every pipeline stage.
  • Develop and operate high-volume pipelines processing 100K+ documents or records per batch.

Requirements

  • 4–8 years of relevant experience, including 2+ years building data-intensive pipelines.
  • Strong distributed Python experience with asyncio, aiohttp, and Celery; scripting experience alone is insufficient.
  • Experience with Kafka, SQS, RabbitMQ, or Redis Streams for messaging and streaming.
  • Experience designing REST or GraphQL APIs with versioning, rate limiting, and authentication.
  • Experience with AWS services including Lambda, SQS/SNS, MSK, EKS, Step Functions, ECS, S3, DynamoDB, and RDS.
  • Experience with multi-tenant SaaS architecture and customer data isolation patterns.
  • Experience with comprehensive testing, CI/CD, monitoring, and incident response.
  • Experience building reliable systems for high-volume data processing.

Tech Stack

Amazon DynamoDBApache KafkaDockerElasticsearchFastAPIGitHub ActionsGrafanaGraphQLPostgreSQLPrometheusPythonRabbitMQRedisTerraform

Categories

BackendData Engineering
Aivar Innovations Private Limited

About Aivar Innovations Private Limited

51-200 employees

Aivar Innovations Private Limited is an AI-first technology partner that builds and operates data and ML platforms, generative-AI pipelines, and cloud-native infrastructure for enterprises. The company delivers consulting and engineering services—covering async Python platforms, multi-tenant SaaS architectures, compliance/PII safety layers, DevOps/SRE, and AWS modernization—often via reusable accelerators. Founded in 2024 and headquartered in Menlo Park, California, Aivar raised a $4.6M seed round in January 2026.

Contact me