Qwissit Artificial Intelligence

ISTQB CT-AI

ML Model Testing: Risks, Performance, Attacks

60 questions

ML model risks

  1. Which mitigation fits the risk "adversarial examples"? Easy
  2. Which mitigation helps against unacceptable data or concept drift? Easy
  3. Which mitigation helps if deploying an updated model could cause immediate failure? Easy
  4. Which mitigation helps against inadequate performance efficiency of the ML model? Easy
  5. Which mitigation helps against poor model documentation? Easy
  6. Which mitigation helps against a defect in the model API? Easy

Model documentation

  1. Why is the documentation of ML models so important? Easy
  2. What are model cards? Easy
  3. What do "datasheets for datasets" describe? Easy
  4. What role does a documentation audit play for high-risk systems? Easy
  5. Which section of ML model documentation do the deployment plan, monitoring metrics and rollback plan belong to? Easy

Functional performance testing

  1. What does ML functional performance testing evaluate? Easy
  2. What does the abbreviation MoE stand for in connection with statistical acceptance criteria? Easy
  3. How should a test report on the functional performance of a probabilistic ML model be worded? Easy
  4. Which requirements must the test dataset for functional performance testing meet? Easy

Adversarial testing

  1. What are adversarial examples? Easy
  2. What is the goal of adversarial testing? Easy
  3. What distinguishes an adversarial attack from an accidental adversarial example? Easy
  4. Why does white-box knowledge make adversarial testing easier? Easy
  5. Which challenges does adversarial testing involve? Easy

ML model risks

  1. Which types of risk does ML model testing address? Medium
  2. Which mitigations help against the test oracle problem for ML models? Medium
  3. Which mitigations help with poor system requirements? Medium
  4. Which mitigations help against a lack of model robustness due to unexpected inputs? Medium
  5. Which mitigations fit model updates that introduce defects or decrease functional performance? Medium
  6. Against which risks does red teaming help? Medium
  7. Which mitigations fit the risks "model causes side-effects" and "model exhibits reward hacking"? Medium

Model documentation

  1. What are the goals of reviewing ML model documentation against a checklist? Medium
  2. Which contents belong in the "datasets" section of ML model documentation? Medium
  3. Which contents belong in the "usage" section of ML model documentation? Medium
  4. What role does transparency play for model documentation? Medium

Functional performance testing

  1. What is a precondition for functional performance testing of probabilistic ML systems? Medium
  2. What happens if the confidence level is fixed as the number of tests increases? Medium
  3. What happens if the margin of error is fixed as the number of tests increases? Medium
  4. How many test cases are needed to confirm "98% accuracy with a ±4% margin of error at a 95% confidence level" under all conditions? Medium
  5. What does sequential testing enable in functional performance testing? Medium
  6. Why does functional performance testing of probabilistic ML systems go beyond a simple "passed/failed"? Medium

Adversarial testing

  1. How can adversarial testing be performed as black-box testing? Medium
  2. Which assumption underlies transferring adversarial inputs from an equivalent model to the original model? Medium
  3. How can adversarial testing be performed? Medium

ML model risks

  1. After a model update, a team reports: the new model crashes immediately after deployment, and in another version previously correct cases no longer work. Which mitigations fit these two risks? Hard
  2. Which mappings of model risk and mitigation are correct? Hard
  3. A chatbot occasionally provides dangerous guidance and discloses personal data. Which mitigation fits these risks? Hard
  4. A team has neither a test oracle nor precise requirements for a new model. Which mitigations fit these two risks? Hard

Model documentation

  1. When reviewing model documentation, it becomes apparent that details of the test dataset, the independence of testing and the adversarial test activities are missing. Which section of the documentation is incomplete, and which review goal is affected? Hard
  2. Which mappings of content and section of ML model documentation are correct? Hard
  3. A team is to describe the model and training data of a high-risk system to a regulator in a standardized way. Which documentation frameworks are suitable, and for what? Hard
  4. Why is a thorough review of the model documentation also useful for systems that are not considered high-risk? Hard

Functional performance testing

  1. For the criterion "98% accuracy with a ±4% margin of error at a 95% confidence level", 601 test cases were planned and all executed, 580 of them passed. Is the criterion met? Hard
  2. For a safety-critical system, "99% reliability with 95% confidence" applies. Which statements are correct? Hard
  3. After 601 test cases, the observed accuracy is 98%. What applies to the measured margin of error at the 95% confidence level? Hard
  4. A team uses sequential testing for the criterion "98% accuracy with ±4% at a 95% CL", and the observed accuracy consistently remains at 98% or more. When can testing stop, approximately? Hard
  5. A team measures a model's accuracy on a sample from the training dataset and reports "96% ±3% at a 95% CL". What should be criticized? Hard
  6. Which test report is appropriate for probabilistic ML systems? Hard
  7. A stakeholder only requires "accuracy ≥ 95%" for acceptance. What is missing for a statistically defined acceptance criterion, and what is it used for? Hard
  8. A project manager asks why as many as 601 test cases are planned for "98% accuracy with ±4% at a 95% CL", although the model achieves about 98% anyway. What is the reason? Hard

Adversarial testing

  1. A team only has API access to a vendor's image model and wants to test it for adversarial examples. Which approaches are possible? Hard
  2. A traffic sign recognizer misclassifies a stop sign after a few pixels, barely visible to humans, were changed. Which statements are correct? Hard
  3. A development team has full access to the architecture, parameters and training process of its model. Which approach to adversarial testing is then typically easier? Hard
  4. A company performed adversarial testing once two years ago and considers the topic settled. What speaks against this? Hard

Certifications & exams

Which exam comes next?

You can already practise for 19 exams, and 21 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications