Staff Engineer / AI Builder - NYC
Robots and Pencils6 hours ago
Base Salary
$166k - $228k/yr
Responsibilities
- Design and build single- and multi-agent workflows with reasoning loops, tool/function calling, and orchestration.
- Build and maintain RAG pipelines covering chunking, embeddings, vector search, re-ranking, and knowledge-source refresh handling.
- Integrate AWS Bedrock and Agent Core, including MCP-based tool design, agent registration, memory/session management, and gateway provisioning.
- Write production system prompts and build LLM evaluation and observability using golden datasets, RAGAS-style metrics, LLM-as-judge, and tracing.
- Develop Python and Node.js backend services, serverless architectures, RESTful APIs, and full-stack integrations.
- Design DynamoDB single-table schemas for conversation state, agent memory, and session history.
- Implement event-driven orchestration with Step Functions, SQS, and EventBridge.
- Design reliability patterns for LLM-backed systems, including retries, backoff, jitter, circuit breakers, fallback models, and user-facing degradation handling.
- Support deployment, monitoring, debugging, hardening, and production troubleshooting in AWS and Docker environments.
- Provide technical leadership, contribute to architecture decisions, mentor engineers, and own features and releases end-to-end.
Requirements
- 6+ years of professional software engineering experience.
- Meaningful professional experience shipping GenAI/LLM-powered systems in production.
- Hands-on depth with agentic AI, reasoning loops, tool/function calling, and single- and multi-agent orchestration.
- Practical RAG expertise including chunking strategies, embeddings, vector databases, cosine similarity search, and re-ranking.
- Experience with LLM evaluation and observability, including golden datasets, LLM-as-judge, RAGAS or comparable metrics, and LangFuse/LangSmith or similar tracing tools.
- Strong production prompt-engineering skills, including structured prompts, output constraints, and few-shot design.
- Hands-on experience with the AWS GenAI stack, including Bedrock, Agent Core, Lambda, DynamoDB single-table design, S3, SQS, EventBridge, and Step Functions.
- Strong Python and Node.js skills with experience building full-stack applications and RESTful APIs.
- Understanding of production reliability patterns for LLM-backed systems, including retries/backoff, circuit breakers, and fallback models.
- Experience with Docker, containerization, and cloud-native deployment.
- Ability to own ambiguous, integration-heavy problems and provide technically deep architectural judgment.
- Helpful experience includes regulated or high-stakes domains, Step Functions or similar state-machine orchestration, and production LLM observability tooling.
Benefits
- Comprehensive benefits package including paid time off, medical, dental, and vision insurance.
- 401(k) benefits are available to eligible employees.
- The role is based in NYC.
- Employment may be conditional on successful completion of a background check.
Categories
About Robots and Pencils
Robots & Pencils is an applied AI engineering firm that designs, builds, and runs production AI systems and agentic co-workers for enterprise operations, with a focus on AWS. The privately held company, founded in 2009 and headquartered in Chagrin Falls, Ohio, provides consulting and implementation services spanning data readiness, cloud modernization, and AI enablement. It is an AWS Advanced Tier Services Partner with delivery teams across the U.S., Canada, Eastern Europe, and Latin America.