ISTQB CT-GenAI

Hallucinations, Reasoning Errors and Biases

60 questions

Easy

Error types of generative AI

  1. What is called a hallucination in a language model?
  2. An LLM generates test cases for a hotel booking portal. What most likely indicates a hallucination?
  3. What is meant by a reasoning error of an LLM?
  4. Why do LLMs easily make mistakes in multi-step arithmetic and logic tasks?
  5. For which test task is the risk of reasoning errors by an LLM especially high?
  6. Where do biases in an LLM's outputs mainly come from?
  7. An LLM generates test data for a registration form and uses almost only names like "Max Müller" with a German address. What type of error is this?

Detecting errors in AI output

  1. What is part of cross-verification to find hallucinations?
  2. Why are domain experts involved in detecting hallucinations?
  3. What does a consistency check examine in AI output?
  4. How can reasoning errors in generated test scripts be uncovered most directly?
  5. What is assessed in the logical validation of an AI output?
  6. How do you recognize bias in AI-generated test cases?
  7. What determines how thoroughly a team examines AI output for these error types?

Countermeasures and non-determinism

  1. Which measure most directly reduces the risk of hallucinations?
  2. How does spreading a complex task over several prompts help against reasoning errors?
  3. Which input form makes a correct evaluation easier for an LLM?
  4. What does the non-deterministic behavior of an LLM mean?
  5. What is the effect of lowering an LLM's temperature?
  6. What is the purpose of setting a fixed seed for an LLM, if the environment allows it?

Medium

Error types of generative AI

  1. An LLM derives test cases from user stories of a library app. One checks renewal by voice assistant, which is required nowhere. What is this?
  2. An LLM is to add up the effort for four test tasks of 3, 5, 2 and 4 hours and reports 16 hours. What type of error is this?
  3. Rule of an online shop: discount only for members AND from an order value of 50 €. In one test case the LLM expects a discount for a non-member with 60 €. What happened?
  4. For a fitness studio app used internationally, an LLM suggests only test data with German postal codes and phone numbers. How should this be classified?
  5. A tester corrects a prompt and the hallucination disappears. The next day it reappears in a new conversation. Why?
  6. What are hallucinations, reasoning errors and biases fundamentally due to?

Detecting errors in AI output

  1. A tool compares AI test cases for a municipal utility portal with the requirements catalog and flags test cases without a matching requirement. Which method is this?
  2. An LLM generates test cases for the dosage calculation of pharmacy software. Which detection method is especially important here?
  3. Two LLM-generated test cases for the same function expect different results for the same input. Which check uncovers this?
  4. An LLM generates test scripts for the software of a ticket machine. What is the most reliable way to find out whether the scripts test correctly?
  5. A tester reads the justification of an AI-created test prioritization and finds that one step contradicts the previous one. Which method is the tester applying?
  6. An LLM delivers synthetic test data for an insurance calculator; all persons are 30 to 50 years old. The test strategy requires boundary values for age. Which check finds this?
  7. For which task is a rather lean check of the AI output most likely sufficient?

Countermeasures and non-determinism

  1. An LLM is to determine boundary values for a tariff calculator but does not know the tariff table and invents values. Which measure helps most?
  2. A tester has an LLM analyze requirements, derive test conditions and write test cases in a single prompt. Errors in the analysis go unnoticed. What improves this?
  3. A project's requirements exist as photographed notes with arrows and margin remarks. What is advisable before handing them to an LLM?
  4. A team is unsure whether the equivalence partitions calculated by an LLM are correct. Which measure helps detect errors in the output?
  5. A team wants to obtain test data from an LLM that stays as stable as possible for a regression suite. Which settings help?
  6. What is the drawback of a very low temperature when an LLM is to provide ideas for exploratory tests?
  7. Why is non-deterministic behavior especially noticeable in long outputs, such as a whole test suite?

Hard

Error types of generative AI

  1. For the criterion "password at least 12 characters", an LLM adds a test case for an account lockout after three failed attempts, which is not required. Why is this more treacherous than an obviously wrong test case?
  2. An LLM takes all facts correctly from the specification but concludes that a high-risk test should run last because it takes the longest. How should this be classified?
  3. Which statement about biases in LLM output is correct?
  4. A colleague says: "With a larger model, hallucinations disappear completely." How should this be assessed?
  5. Why is a reasoning error in an AI-created effort estimate often not noticed immediately?
  6. Which assignment is correct? (1) A field "customer category" that the system does not know appears in test data. (2) The total of an estimate is wrong. (3) All test persons have male first names.

Detecting errors in AI output

  1. A team automatically checks only whether AI-generated test scripts run without errors. What is the weakness of this approach?
  2. Why is an automatic cross-verification against the requirements catalog not always sufficient?
  3. An LLM generates 40 test cases for a web shop. 38 concern credit card payment; purchase on invoice and direct debit hardly appear. What fits?
  4. Which method best uncovers reasoning errors in an AI-created test prioritization?
  5. A team uses GenAI for test data of medical device software and for drafts of internal release notes. How should the checking effort be distributed?
  6. Why is a passed consistency check alone no proof of correct AI output?
  7. For a banking app, an LLM suggests only test cases for transfers, balances and standing orders. Which bias is likely?

Countermeasures and non-determinism

  1. A team gives an LLM all requirements of a timetable service but still receives wrong expected transfer times. Which measure fits next?
  2. When is it especially useful to give the same task to several LLMs and compare the results?
  3. A team uses a general chat model to generate test scripts in a little-used test language, and they are often faulty. Which countermeasure is obvious?
  4. A team sets the temperature to the lowest value. Which expectation is realistic?
  5. What trade-off does lowering the temperature involve?
  6. How does automated checking of AI output help in dealing with non-determinism?
  7. An LLM prioritizes test cases for a veterinary appointment planner sometimes correctly, sometimes wrongly. Which combination of measures makes most sense?

Certifications & exams

Which exam comes next?

You can already practise for 51 exams, and 16 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications