Gen AI Data Engineer
Tiger Analytics Inc.over 1 year ago
Remote, United StatesSenior
Responsibilities
- Architect and build distributed data systems and large-scale data warehouses capable of processing petabytes of data.
- Design and manage real-time and batch ingestion pipelines and data platforms.
- Build document-ingestion, indexing, retrieval, and vector-search pipelines for scalable RAG solutions.
- Develop and maintain data persistence and retrieval systems across relational, vector, bucket, graph, and knowledge-graph stores.
- Integrate external databases, APIs, and knowledge graphs into RAG systems.
- Create ontologies, schema-level constructs, and Open Cypher-based knowledge-retrieval solutions.
- Curate and collect data from structured, unstructured, traditional, and non-traditional sources.
- Conduct experiments, evaluate RAG workflow effectiveness, and iterate for improved performance.
- Monitor and optimize system performance, reliability, scalability, and security.
- Apply infrastructure automation, containerization, CI/CD, disaster recovery, and cloud security practices.
Requirements
- 8+ years of experience in data engineering, platform engineering, or related fields.
- Deep expertise designing and building distributed data systems and large-scale data warehouses.
- Proven experience architecting data platforms for petabyte-scale processing and real-time and batch ingestion.
- Strong experience building data pipelines for document ingestion, indexing, and retrieval for RAG solutions.
- Proficiency with Python, SQL, PySpark, Apache Airflow, cloud platforms, and GitHub.
- Experience with Snowflake, NoSQL, Neo4j, Hadoop, Spark, Streamlit, APIs, and vector databases.
- Experience with information retrieval, vector search, FAISS, Pinecone, Elasticsearch, or Milvus.
- Experience with graphs, graph algorithms, LLMs, optimization algorithms, relational databases, and diverse data formats.
- Experience building knowledge graphs and ontologies, including higher-level classes, punning, property inheritance, and Open Cypher.
- Experience with AWS services including SageMaker, Lambda, OpenSearch, S3, RDS, AWS Batch, and CloudFormation, or with GCP services including Vertex AI, BigQuery, and GKE.
- Advanced skills in Terraform and CloudFormation, plus experience with Docker, Kubernetes, Jenkins, and GitHub Actions.
- Experience with Airflow DAGs, AutoSys, CronJobs, unstructured data, monitoring, scalability, disaster recovery, and cloud security.
- Familiarity with generative AI tools and techniques and a basic understanding of machine learning concepts.
- Strong analytical, problem-solving, communication, presentation, and collaboration skills.
Benefits
- Remote, full-time position in the United States.
- Opportunity for significant career development in a fast-growing, challenging entrepreneurial environment with a high degree of individual responsibility.
Tech Stack
Apache AirflowApache HadoopApache SparkAWSDockerElasticsearchGitHub ActionsGoogle BigQueryGoogle Cloud PlatformJenkinsKubernetesLinuxNeo4jPythonSnowflakeSQLTerraform
Categories
Data Engineering
About Tiger Analytics Inc.
Tiger Analytics Inc. provides data and AI consulting services that deliver machine learning, analytics, and cloud data engineering solutions for large enterprises across industries, including life sciences. The company engages on strategy and implementation projects, from modern BI and data platforms to MLOps, helping Fortune 100 clients turn data into business outcomes. Founded in 2011 and headquartered in Santa Clara, it is privately held with operations across North America, Europe, and Asia-Pacific.