ISTQB CT-GenAI

Evaluating AI Output and Refining Prompts

60 questions

Easy

Metrics for GenAI results

  1. What is the purpose of metrics when a test team uses GenAI for test tasks?
  2. Which metric compares the overall correctness of an AI output with a reference created by experts?
  3. Which metric asks whether the model found all relevant cases?
  4. What does the precision metric express for AI results?
  5. What does the diversity metric assess for generated test cases?
  6. What does the execution success rate measure?
  7. What does the time efficiency metric compare?
  8. When do generated test cases have good relevance and contextual fit?
  9. Which statement about metrics for evaluating GenAI results is correct?
  10. In which two ways can testers collect metrics for AI results?
  11. Why should metrics for GenAI results be based on several runs?
  12. Which of the following measures is a metric for evaluating GenAI results in testing?

Evaluating and refining prompts

  1. How do you proceed with iterative prompt modification?
  2. What happens in an A/B test of prompts?
  3. What is the goal of output analysis when refining prompts?
  4. Whose feedback is particularly helpful when refining test prompts?
  5. Which statement about the length of a prompt is correct?
  6. How can a test team best pass on proven prompts?
  7. What is the goal of regular sessions in which a team evaluates and optimizes prompts?
  8. What is the decision about which prompt variant is better in an A/B test based on?

Medium

Metrics for GenAI results

  1. An LLM generates 40 test scripts for a cinema booking system. 34 of them run in the test environment without changes. Which metric describes this?
  2. There are six equivalence partitions for age when buying a cinema ticket. The generated tests cover only three of them. Which metric is weak here?
  3. An LLM marks ten entries in the log of a parcel locker as anomalies. Only four of them are real problems. Which metric is low?
  4. The 25 generated tests for a driving school app all check appointment booking in the normal case, only with different student names. Which metric turns out poorly?
  5. For the order shop of a German bakery, an LLM generates tests with dollar prices and US tax rates. Which metric is affected?
  6. With an LLM and subsequent review, a team needs two hours for a test suite, previously eight by hand. Which metric is evaluated with this?
  7. A team compares AI-generated test cases with a suite written by an experienced test analyst. Which metric is it mainly collecting?
  8. A tester runs a prompt once, finds the result very good and declares the prompt finished. What is problematic about this?
  9. How can the execution success rate of generated test scripts be collected automatically?
  10. Which reference is suitable for an automatic comparison to measure the accuracy of generated test cases?
  11. An LLM is to generate test data for an IBAN field in a banking app. Which indicator is a task-specific metric?
  12. What must a team include to measure the time efficiency of GenAI fairly?

Evaluating and refining prompts

  1. A first prompt delivers only generic tests for a club management system. The tester adds a glossary of club terms and a system description and checks again. Which technique is she applying?
  2. Two prompt variants each run ten times with the same requirements. The execution success rate is evaluated each time. Which technique is this?
  3. Output analysis reveals that the LLM regularly invents fields in API tests that do not exist in the interface description. What is a sensible next step?
  4. The testers report: the generated test cases are too fine-grained, every click is a separate case. How do you use this feedback?
  5. A team gives the model the complete manual of a library software. The answers tend to get worse as a result. What should it try?
  6. Each tester in the team maintains their own prompts, and the same errors keep occurring. What is the best recommendation?
  7. What is part of a meaningful A/B test of two prompts?
  8. Many generated scripts fail because the LLM outputs explanatory text before and after the code. Which prompt change helps most?

Hard

Evaluating and refining prompts

  1. Why do you change only one aspect per round, if possible, in iterative prompt modification?
  2. Variant B performs slightly better than variant A in a single run. What follows from this?
  3. The error analysis shows that boundary values are missing in 70% of the faulty test cases for a travel expense app. Which refinement is most effective?
  4. A prompt contains five detailed example test cases. The model copies them almost word for word and hardly varies them. What is a sensible attempt?
  5. Which order makes sense for refining a test prompt?
  6. The team switches to a new version of the language model. How does it deal sensibly with its prompt library?
  7. In a club management system, users are called "members". In the tests, however, the LLM writes "customers" and "accounts", which confuses the business department. Which adjustment fits?
  8. Two testers give opposing feedback: one wants more detail in the test cases, the other less. What is a sensible way of dealing with this?

Metrics for GenAI results

  1. An LLM is to find anomalies in server logs of a daycare management system. It marks almost every line as suspicious. How are precision and recall typically to be assessed?
  2. A model reports only two anomalies, both are genuine. In fact, however, there are eight. Which assessment is correct?
  3. There are eight equivalence partitions, valid and invalid, for the weight field of a parcel tracking system. The generated tests hit six of them. How high is recall?
  4. Almost all generated scripts run without errors, but they check functions that are not required in the test basis at all. Which conclusion is correct?
  5. A team optimizes its prompt only for high diversity. The tests become more and more unusual. What risk exists?
  6. When compared with an expert suite, some AI test cases count as wrong although they check sensible cases that the experts missed. What does this show?
  7. What does it mean in practice that metrics for GenAI must be based on statistically relevant data?
  8. An LLM generates tests in five minutes. Reviewing and correcting takes three hours; by hand it would be two hours. What does time efficiency show?
  9. An LLM is to generate regression test scripts for a CI pipeline. Which metric first shows whether the scripts are usable there at all?
  10. What does a team need to be able to calculate the recall of generated test cases?
  11. What benefit do evaluation metrics have beyond optimizing prompts?
  12. Which statement about manual and automatic evaluation of AI output is correct?

Certifications & exams

Which exam comes next?

You can already practise for 51 exams, and 16 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications