ISTQB CT-GenAI
Evaluating AI Output and Refining Prompts
Easy
Metrics for GenAI results
- What is the purpose of metrics when a test team uses GenAI for test tasks?
- Which metric compares the overall correctness of an AI output with a reference created by experts?
- Which metric asks whether the model found all relevant cases?
- What does the precision metric express for AI results?
- What does the diversity metric assess for generated test cases?
- What does the execution success rate measure?
- What does the time efficiency metric compare?
- When do generated test cases have good relevance and contextual fit?
- Which statement about metrics for evaluating GenAI results is correct?
- In which two ways can testers collect metrics for AI results?
- Why should metrics for GenAI results be based on several runs?
- Which of the following measures is a metric for evaluating GenAI results in testing?
Evaluating and refining prompts
- How do you proceed with iterative prompt modification?
- What happens in an A/B test of prompts?
- What is the goal of output analysis when refining prompts?
- Whose feedback is particularly helpful when refining test prompts?
- Which statement about the length of a prompt is correct?
- How can a test team best pass on proven prompts?
- What is the goal of regular sessions in which a team evaluates and optimizes prompts?
- What is the decision about which prompt variant is better in an A/B test based on?
Medium
Metrics for GenAI results
- An LLM generates 40 test scripts for a cinema booking system. 34 of them run in the test environment without changes. Which metric describes this?
- There are six equivalence partitions for age when buying a cinema ticket. The generated tests cover only three of them. Which metric is weak here?
- An LLM marks ten entries in the log of a parcel locker as anomalies. Only four of them are real problems. Which metric is low?
- The 25 generated tests for a driving school app all check appointment booking in the normal case, only with different student names. Which metric turns out poorly?
- For the order shop of a German bakery, an LLM generates tests with dollar prices and US tax rates. Which metric is affected?
- With an LLM and subsequent review, a team needs two hours for a test suite, previously eight by hand. Which metric is evaluated with this?
- A team compares AI-generated test cases with a suite written by an experienced test analyst. Which metric is it mainly collecting?
- A tester runs a prompt once, finds the result very good and declares the prompt finished. What is problematic about this?
- How can the execution success rate of generated test scripts be collected automatically?
- Which reference is suitable for an automatic comparison to measure the accuracy of generated test cases?
- An LLM is to generate test data for an IBAN field in a banking app. Which indicator is a task-specific metric?
- What must a team include to measure the time efficiency of GenAI fairly?
Evaluating and refining prompts
- A first prompt delivers only generic tests for a club management system. The tester adds a glossary of club terms and a system description and checks again. Which technique is she applying?
- Two prompt variants each run ten times with the same requirements. The execution success rate is evaluated each time. Which technique is this?
- Output analysis reveals that the LLM regularly invents fields in API tests that do not exist in the interface description. What is a sensible next step?
- The testers report: the generated test cases are too fine-grained, every click is a separate case. How do you use this feedback?
- A team gives the model the complete manual of a library software. The answers tend to get worse as a result. What should it try?
- Each tester in the team maintains their own prompts, and the same errors keep occurring. What is the best recommendation?
- What is part of a meaningful A/B test of two prompts?
- Many generated scripts fail because the LLM outputs explanatory text before and after the code. Which prompt change helps most?
Hard
Evaluating and refining prompts
- Why do you change only one aspect per round, if possible, in iterative prompt modification?
- Variant B performs slightly better than variant A in a single run. What follows from this?
- The error analysis shows that boundary values are missing in 70% of the faulty test cases for a travel expense app. Which refinement is most effective?
- A prompt contains five detailed example test cases. The model copies them almost word for word and hardly varies them. What is a sensible attempt?
- Which order makes sense for refining a test prompt?
- The team switches to a new version of the language model. How does it deal sensibly with its prompt library?
- In a club management system, users are called "members". In the tests, however, the LLM writes "customers" and "accounts", which confuses the business department. Which adjustment fits?
- Two testers give opposing feedback: one wants more detail in the test cases, the other less. What is a sensible way of dealing with this?
Metrics for GenAI results
- An LLM is to find anomalies in server logs of a daycare management system. It marks almost every line as suspicious. How are precision and recall typically to be assessed?
- A model reports only two anomalies, both are genuine. In fact, however, there are eight. Which assessment is correct?
- There are eight equivalence partitions, valid and invalid, for the weight field of a parcel tracking system. The generated tests hit six of them. How high is recall?
- Almost all generated scripts run without errors, but they check functions that are not required in the test basis at all. Which conclusion is correct?
- A team optimizes its prompt only for high diversity. The tests become more and more unusual. What risk exists?
- When compared with an expert suite, some AI test cases count as wrong although they check sensible cases that the experts missed. What does this show?
- What does it mean in practice that metrics for GenAI must be based on statistically relevant data?
- An LLM generates tests in five minutes. Reviewing and correcting takes three hours; by hand it would be two hours. What does time efficiency show?
- An LLM is to generate regression test scripts for a CI pipeline. Which metric first shows whether the scripts are usable there at all?
- What does a team need to be able to calculate the recall of generated test cases?
- What benefit do evaluation metrics have beyond optimizing prompts?
- Which statement about manual and automatic evaluation of AI output is correct?