Data Platform Engineer
BrightAI Corporation16 days ago
Palo Alto, CA, USASenior
Responsibilities
- Design, build, and maintain scalable, reliable, high-throughput ingestion pipelines for structured and semi-structured data.
- Implement secure, cost-effective data lake and SQL-based storage architectures.
- Develop internal data ingestion tools and frameworks using Python, Golang, and SQL.
- Define data contracts, schemas, and retention policies with Cloud, Edge, Product, and AI teams.
- Use AWS services to support end-to-end data workflows.
- Manage data platform infrastructure with Terraform and Infrastructure-as-Code practices.
- Monitor pipeline quality, latency, and failures using CloudWatch, SumoLogic, and DataDog.
- Optimize storage, partitioning, and query performance across large-scale datasets.
- Participate in architecture reviews and uphold security, compliance, and best-practice standards.
Requirements
- 5+ years of professional software engineering or data engineering experience.
- Strong programming skills in Python and Golang.
- Deep understanding of SQL and modern data lake architectures, including Parquet, Iceberg, or Delta Lake.
- Hands-on experience with AWS services such as S3, Lambda, Glue, Kinesis, Athena, and RDS.
- Proficiency with Terraform for infrastructure deployment and management.
- Experience with real-time or batch ingestion at scale and fault-tolerant ETL/ELT pipelines.
- Familiarity with event-driven architectures and messaging systems such as Kafka or Kinesis.
- Strong debugging and optimization skills across cloud, network, and application layers.
- Excellent collaboration, communication, and documentation skills.
- Experience with industrial-scale time-series or IoT sensor data is a bonus.
- Familiarity with Redshift, Snowflake, gRPC, protobuf, ML pipelines, feature stores, or Kubernetes is a bonus.
- Prior startup experience or comfort in fast-paced, iterative environments is a bonus.
Tech Stack
Categories
Data Engineering
About BrightAI Corporation
BrightAI builds an AIoT platform and autonomous inspection solutions that connect sensors, drones, and edge devices to monitor and manage physical infrastructure. Its Stateful platform supports predictive diagnostics, digital twins, and AI-enabled workflows for operators in water, power, gas compression, HVAC, manufacturing, and pest control. Founded in 2019 and headquartered in Palo Alto, the privately held company focuses on retrofitting legacy systems and deploying robotics at real infrastructure sites.