How You'll Make An Impact: As a Software Engineer in AI Quality, you will play a role in ensuring the quality, reliability, and performance of our cutting-edge Generative AI features. This is an exciting opportunity to work at the intersection of software quality engineering and AI, where you'll help define testing strategies and evaluation frameworks for Agentic applications. You'll grow your skills in both software engineering and emerging AI quality practices while contributing to products that push the boundaries of what's possible with AI.

What You Will Do:

• Develop and execute comprehensive test plans for Generative AI features, including prompt testing, output validation, and behavioral testing of Agentic systems.

• Design and implement evaluation frameworks to assess AI output quality, consistency, accuracy, and reliability across different use cases and scenarios.

• Write automated tests to validate AI feature behavior, performance, and edge cases.

• Collaborate closely with other team members and product stakeholders to understand AI feature requirements and identify potential quality risks early in the development cycle.

• Perform both manual and automated testing of AI applications, documenting findings and communicating issues clearly to the development team.

• Experiment with and leverage AI evaluation tools and frameworks such as AWS Bedrock evaluations, LangSmith, or custom-built testing solutions.

• Contribute to the development of quality standards and best practices specifically tailored for Generative AI applications.

• Monitor and analyze AI system behavior in development and production environments, identifying patterns and anomalies that may indicate quality issues.

• Create clear, comprehensive documentation including test plans, evaluation criteria, test results, and quality reports.

• Stay current with emerging trends, tools, and techniques in AI quality assurance and LLM evaluation.
  

The Experience You Bring

• AI & Gen AI Interest: Strong curiosity and foundational knowledge of Generative AI concepts. Eagerness to deepen your understanding of how AI systems work and how to effectively test them.

• Software Engineering Foundation: Solid grasp of core computer science concepts including data structures, algorithms, debugging techniques, and API interactions. Understanding of software development lifecycle and quality assurance principles.

• Programming Skills: Hands-on coding experience in Python.

• Testing & Quality Mindset: Understanding of testing methodologies including functional testing, integration testing, and test case design. Willingness to learn prompt-based evaluation techniques, LLM-specific testing approaches, and AI quality metrics.

• Evaluation & Analysis: Strong analytical skills with attention to detail. Ability to assess AI outputs critically, identify edge cases, and think creatively about potential failure modes in AI systems.

• Versatility & Adaptability: Comfortable working across different types of tasks—from manual exploratory testing to writing automated test scripts to creating documentation. Willingness to experiment with new tools and approaches in an evolving field.

• Problem-Solving: Good troubleshooting abilities and systematic approach to identifying root causes of quality issues.

• Communication & Collaboration: Strong written and verbal communication skills. Ability to document findings effectively and collaborate with cross-functional teams using tools like Jira and Confluence.

• Learning Orientation: Self-motivated learner who stays curious and proactive about acquiring new skills in AI, testing frameworks, and quality practices.
  

Education & Experience

• Bachelor's degree in Computer Science, Software Engineering, or a related field with upto 2 years of experience in software testing, software development in Gen AI applications.
  

Bonus points if you have:

• Hands-on experience testing Gen AI applications or working with Gen AI & Agentic evaluation tools.

• Familiarity with prompt engineering or conversational AI systems.

• Experience with test automation frameworks (pytest, unittest, Selenium, etc.).

• Understanding of CI/CD pipelines and automated testing integration.

• Participation in AI-related projects, hackathons, or coursework focused on machine learning or natural language processing.

#LI-ML-1