We’re looking for a AI QA Engineer
Job Description:
We are looking for an AI QA Engineer to ensure the reliability, safety, and quality of intelligent applications powered by LLMs, RAG pipelines, and AI agents. You will design and execute functional, non-functional, and AI-specific tests covering accuracy, hallucinations, groundedness, bias, security, prompt injection, and workflow correctness. AI QA engineers own evaluation strategy, test automation, golden datasets, and go/no-go recommendations for AI releases.
Requirements and Duties:
Design test strategies for AI applications including LLMs, RAG systems, conversational AI, and agentic workflows. Build automated evaluation pipelines for accuracy, groundedness, context relevance, reasoning, tool-use correctness, and latency. Create and maintain golden datasets, test scenarios, and evaluation metrics for regression testing. Implement LLM-as-a-Judge, schema validation, tracing, and calibrated scoring for multi-turn and long-horizon tasks. Test for hallucinations, bias, harmful content, policy violations, PII leaks, and brand drift using filters and guardrails. Conduct red-team testing for prompt injection, jailbreaking, data exfiltration, and adversarial inputs. Embed evaluation harnesses into CI/CD pipelines and monitor production telemetry for behavioral drift and failures. Analyze model and integration failures to identify root causes, controls, and mitigations. Translate business risks into measurable quality criteria, thresholds, and release recommendations. Collaborate with engineers, product managers, and security teams on testability, observability, and governance.
Skill and Ability:
3+ years of experience in QA engineering, test automation, ML QA, AI evaluation, or MLOps. Strong proficiency in Python and test frameworks such as Pytest, with experience building automated test suites. Hands-on experience testing LLM applications, RAG pipelines, or AI agents in production or pre-production environments. Understanding of AI failure modes: hallucinations, grounding errors, prompt sensitivity, context limits, and tool misuse. Familiarity with AI evaluation tools and platforms such as MLflow, Galileo, PyRIT, LangSmith, or equivalent. Knowledge of security testing concepts including prompt injection, jailbreaking, and data leakage risks. Experience working with CI/CD, containers, cloud environments, and observability tools. Strong analytical skills to interpret evaluation results and communicate risks clearly. Experience with multi-agent systems, autonomous workflows, or long-horizon task evaluation. Knowledge of responsible AI, fairness testing, and regulatory compliance for AI systems. Background in software development, data engineering, or ML engineering. Contributions to AI testing frameworks, open-source evaluation tools, or internal QA standards.
Konsatnt Infosolutions
Pvt Ltd
Information Technology & Service
Language
English
Location
Jaipur
Department
Quality Assurance
Job Type
Full time
Experience
3+ Years
Job Posted
26, Oct 2026
Current Openings
Current Job Vacancies
Can’t find your dream job role? Send us your resume to [email protected]
Hey! Come and join us!
Be a part of the most
happening team
around!
We are hunting expert web and app developers, ingenious graphic designers and solution-oriented project managers to work with us on the various challenging and innovative projects of our clients. If you are ready to become a part of our brainy, tech-savvy team, get in touch with us now.
We Just Need Some Basic Information, And We’ll Take It From There.
We'll schedule a call to discuss your idea. After discovery sessions, we'll send a proposal, and upon approval, we'll get started.
