Qwissit Artificial Intelligence

ISTQB CT-AI

Testing AI Systems and GenAI

60 questions

Fundamentals of testing AI-based systems

  1. What characterizes a locked AI-based system? Easy
  2. Which is an example of an adaptive AI-based system? Easy
  3. Which type of AI-based system is far easier to test? Easy
  4. How is an updated locked AI-based system typically treated? Easy
  5. Why can a single test, such as a cat classified as a dog, not reliably reflect a model's functional correctness? Easy
  6. What does the test oracle problem describe? Easy
  7. What is a proxy oracle? Easy
  8. Which is an example of defining output boundaries as a solution to the test oracle problem? Easy

Testing generative AI

  1. Which test approach is common when testing generative AI? Easy
  2. What is called the "input explosion problem" when testing GenAI? Easy
  3. What are benchmark suites used for when testing generative AI? Easy
  4. What is red teaming (RT)? Easy
  5. Why is red teaming particularly important for generative AI? Easy
  6. When is red teaming most effective? Easy
  7. What is blue teaming? Easy

Test levels and risk-based testing

  1. Which two specialized test levels address ML-specific risks? Easy
  2. Which data does input data testing consider? Easy
  3. To which systems should risk-based testing be applied? Easy
  4. Into which three areas can ML-specific risks be divided along the ML workflow? Easy
  5. What is ML model testing concerned with? Easy

Fundamentals of testing AI-based systems

  1. Where do many GenAI systems such as LLM-based chatbots stand between locked and adaptive systems? Medium
  2. Why can even locked AI systems with large neural networks exhibit non-deterministic behavior in practice? Medium
  3. How can an adaptive AI system be tested before deployment? Medium
  4. How can one check that the adaptations of an adaptive AI system in operation are safe? Medium
  5. Which reasons speak for a statistical approach to testing AI systems? Medium
  6. With which statistical means can a model's accuracy, fairness and robustness be quantified? Medium
  7. Why must a statistically significant sample of scenarios from the operational data be tested? Medium
  8. What causes the test oracle problem in AI systems? Medium
  9. Which approaches exist for the test oracle problem? Medium
  10. Why must testers specify environmental boundaries such as lighting, temperature or network latency? Medium
  11. Which limitation does consulting domain experts as a test oracle have? Medium

Testing generative AI

  1. What is evaluated when testing generative AI? Medium
  2. Which parameters besides the prompts influence the output of a GenAI system? Medium
  3. Which non-functional measures are relevant for GenAI systems? Medium
  4. Which harmful results is red teaming meant to provoke, for example? Medium
  5. Which steps does the typical red teaming approach comprise? Medium
  6. With which approaches can comprehensive coverage be achieved in red teaming? Medium

Test levels and risk-based testing

  1. What is part of component integration testing of an ML system? Medium
  2. What does system testing of an ML system check, among other things? Medium
  3. Which product risks belong to the "model" area? Medium

Fundamentals of testing AI-based systems

  1. An automotive supplier uses a fixed trained DNN for traffic sign recognition and deploys a retrained model after one year. Which statements are correct? Hard
  2. A team tests a locked image classification model on GPUs and in rare cases receives slightly different outputs for identical inputs. How is this to be classified? Hard
  3. A tester is asked to write test cases in advance for all future behaviors of an adaptive recommender system. What is there to say about this? Hard
  4. Why is testing adaptive AI systems before deployment more complex than testing locked systems? Hard
  5. A team has checked an image model with 20 test images, all classified correctly, and now wants to report "100% correct". What speaks against this? Hard
  6. A manufacturer of a medical AI system must demonstrate to an authority that a fairness threshold is met. Which statements are correct? Hard
  7. A team testing a virtual assistant cannot agree on "correct" answers because users have very different expectations. Which cause of the test oracle problem is present, and what helps? Hard
  8. For an image classification model there are many unlabeled test images but no expected results. Which approach fits? Hard
  9. A self-learning pricing system was accepted with a set of expected results. After a few weeks, many of these tests fail although the system works well. What is the reason? Hard
  10. A model for assessing complex protein structures delivers results that humans can hardly verify. Which statements are correct? Hard

Testing generative AI

  1. A team has the images of an image generator checked automatically by an image recognition system. Which statements about this are correct? Hard
  2. A tester receives different answers from a chatbot to the same user prompt depending on the session. Which factors are possible causes? Hard
  3. How is "passed" or "failed" decided when testing GenAI if the outputs are reviewed manually? Hard
  4. A company wants to use red teaming to check an LLM with RAG integration for security vulnerabilities before launch. Which attacks should be considered? Hard
  5. A healthcare chatbot is to be checked for safety using red teaming. What does this aim at, unlike security red teaming? Hard
  6. How does red teaming relate to benchmarks? Hard
  7. A team plans red teaming only for security. Which other areas can red teaming cover, and what role does regulation play? Hard

Test levels and risk-based testing

  1. A company uses a cloud provider's language model as a service and integrates it into its CRM. Which test levels and activities are intended for this? Hard
  2. A DNN was compressed for a smartphone to make it smaller. Which test level is particularly important here, and what is checked? Hard
  3. Which mappings of risk and area along the ML workflow are correct? Hard

Certifications & exams

Which exam comes next?

You can already practise for 19 exams, and 21 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications