AI Software Engineer in Test
GreenSky Administrative Services LLC6 days ago
Alpharetta, GA, USAMid Level
Base Salary
$115k - $145k/yr
Responsibilities
- Use AI tools to generate, modify, refactor, debug, and maintain automated tests across web, mobile, API, backend, data platform, and AI-enabled applications.
- Design, develop, and maintain scalable automated test frameworks, reusable components, shared libraries, test utilities, and framework standards.
- Develop test strategies and automated evaluations for GenAI, LLM-powered, RAG, conversational AI, and agentic systems, including accuracy, relevance, hallucination, toxicity, bias, fairness, and policy validation.
- Create evaluation datasets, prompt test suites, golden answer sets, scoring rubrics, and regression approaches for prompts, model upgrades, and agent workflows.
- Integrate automated tests into CI/CD pipelines and establish quality gates for build validation, coverage, security checks, test execution, and release readiness.
- Collaborate with engineering, QA, product, architecture, security, data science, and platform teams on requirements, test strategy, code quality, and responsible AI practices.
- Advocate for testability, maintainability, observability, secure coding, quality-left practices, and improved release confidence.
Requirements
- Bachelor’s or master’s degree in Computer Science, Software Engineering, Information Systems, Data Science, Artificial Intelligence, or a related technical discipline.
- At least 2 years of hands-on experience in software development, test automation, or quality engineering.
- Experience building and maintaining automated test frameworks.
- Experience with AI development lifecycle delivery models, including Scrum and Kanban.
- Experience working in fast-paced delivery environments with short turnaround times.
- 2 to 4+ years of database experience, preferably SQL.
- Experience with AI-assisted coding tools such as Claude Code, Kiro, Copilot, or similar tools is preferred.
- Strong software engineering, quality engineering, data validation, and automated testing skills.
Benefits
- Hybrid work arrangement in Alpharetta, Georgia, with 3 days per week on-site.
- Medical, dental, vision, disability, and life insurance.
- 401(k) retirement benefits.
- Paid time off, paid holidays, and paid personal/sick time.
- Eligible for an annual bonus within a comprehensive total rewards package.