Optimizing an AI-Powered Platform with Luxe Quality’s QA Expertise

AI-powered web platform that transforms natural language queries into SQL requests to simplify access to business data and streamline BI analytics.

• Before our QA joined the project, testing was conducted by the development team.
• There was testing documentation in the form of a checklist, and no visual references to guide UI validation.
• AI responses often missed the point of user queries, and inconsistencies in SQL logic led to unpredictable results.

Challenges

• AI responses lacked consistency; identical queries to AI often resulted in outputs with different SQL logic.
• The absence of UI mockups made it challenging to validate interface elements precisely.
• There was no detailed documentation to guide the testing process.
• Lack of reference outputs made verifying the correctness of AI-generated responses challenging.

Solutions

• We designed flexible test cases to validate SQL logic relevance rather than exact SQL templates.
• SQL queries were manually validated using DBeaver to check both logic and data accuracy in responses.
• Clear UI issues were reported through bug reports.
• We compiled a complete documentation set, including a test strategy and a detailed test plan with test cases.

• DBeaver is used to validate SQL queries and ensure the accuracy of database responses.
• Browsers: Google Chrome was the primary browser for functional and UI testing, and Firefox and Safari were also used for cross-browser compatibility checks.
• Xbox Game Bar / Zoom is used for screen recording and capturing defects.
• Slack served as the main channel for direct team communication.
• Jira and Confluence are used for task tracking, documentation management, and storing test cases.

Testing Approaches

• Developed and documented regression testing scenarios to cover critical user flows and system functionalities.
• Maintained a list of failed or inaccurate SQL queries as a supporting artifact for root cause analysis and AI model improvement.
• Conducted UI/UX testing without design mockups, relying on UX best practices, interface consistency, and common sense.

• There was a high tolerance for variation in correct answers. Due to AI's generative nature, multiple SQL queries could produce equally valid results.
• Test cases tailored for working with generative AI, focusing on SQL logic relevance rather than exact query structure.

• Over 200 detailed test cases were created to verify AI-generated SQL queries and UI functionality.
• Up to 150 bugs were reported.
• The accuracy of AI responses improved by up to 40%—the system started generating answers that matched the intent of the user's questions.
• The probability of receiving incorrect or logically irrelevant SQL queries decreased.
• AI passed the red teaming review after the final testing cycle.
• Response generation became more consistent across similar queries.
• A complete QA documentation package was delivered: a test strategy and detailed test plan with test cases.
• The client gained better visibility into product quality and a structured foundation for further development and quality control.