Qwissit Artificial Intelligence

ISTQB CT-AI

ML Model Testing: Metamorphic, Drift, A/B

60 questions

Metamorphic testing

  1. What is metamorphic testing (MT)? Easy
  2. What does a metamorphic relation (MR) describe? Easy
  3. When is metamorphic testing particularly useful? Easy
  4. In which AI application areas has metamorphic testing already been used? Easy
  5. Which test objective of metamorphic testing describes that outputs remain stable under perturbations of the input? Easy
  6. From what do testers derive metamorphic relations? Easy

Drift and model fit

  1. What is data drift? Easy
  2. What is concept drift? Easy
  3. Which is an example of data drift? Easy
  4. What does dynamic drift testing rely on? Easy
  5. What is overfitting? Easy
  6. What is underfitting? Easy
  7. When should testing for overfitting and underfitting take place? Easy
  8. What does static drift testing compare? Easy

Comparative testing

  1. What is A/B testing? Easy
  2. Which is a simple example of A/B testing? Easy
  3. Which statistical techniques are most popular for A/B testing? Easy
  4. What is back-to-back testing? Easy
  5. What needs to be generated for back-to-back testing? Easy
  6. Why should the pseudo-oracle and the system under test ideally not share common software components? Easy

Metamorphic testing

  1. When is metamorphic testing typically selected over traditional oracle-based testing? Medium
  2. How can metamorphic relations be validated? Medium
  3. Which limitations does metamorphic testing have? Medium
  4. Which is an example of a monotonic metamorphic relation? Medium
  5. For which types of testing can metamorphic testing be used? Medium
  6. What is checked in metamorphic testing when a function is too complex to generate expected results? Medium

Drift and model fit

  1. Which is an example of concept drift? Medium
  2. In a film recommendation system, what is direct and what is indirect user feedback? Medium
  3. How is it determined in dynamic drift testing whether drift is present? Medium
  4. How is overfitting tested for? Medium
  5. How can overfitting and underfitting be detected during testing? Medium
  6. How does underfitting show in the learning curves? Medium
  7. Which values indicate underfitting? Medium

Comparative testing

  1. How does A/B testing help with the test oracle problem? Medium
  2. Which precondition is needed to test updates of an AI system with A/B testing? Medium
  3. How can A/B testing be used for self-learning systems? Medium
  4. Which statements about the pseudo-oracle in functional back-to-back testing are correct? Medium
  5. In which situations does back-to-back testing provide particular value? Medium
  6. What is a significant difference between A/B testing and back-to-back testing? Medium
  7. Where can the test inputs for back-to-back testing come from? Medium

Metamorphic testing

  1. An image classifier has correctly identified a photo as "cat". Which follow-up test cases result from an invariance relation? Hard
  2. A creditworthiness model is to be tested. Which metamorphic relation is a monotonicity relation? Hard
  3. A search engine returns 800 hits for "running shoes". Which follow-up test case results from a sensible metamorphic relation derived from domain knowledge? Hard
  4. A team derives metamorphic follow-up test cases from a source test case that has itself failed. What is the problem? Hard
  5. For an insurance model, a team uses the relation "higher age always results in a higher premium" and finds no defects. Later it turns out that age and pre-existing conditions interact. What happened here? Hard
  6. A team plans to test a route optimization system exclusively with metamorphic testing. What is recommended? Hard
  7. A team derives several metamorphic relations for an ML system. Which statements about this are correct? Hard
  8. A speech recognition system has correctly transcribed a sentence. Which follow-up test cases can be derived from metamorphic relations whose expected result is the same transcript? Hard

Drift and model fit

  1. Which mappings of situation and type of drift are correct? Hard
  2. A fraud detection system receives no feedback at all on whether its decisions were correct. How can drift still be tested? Hard
  3. A streaming service wants to check whether its recommendation model is drifting and uses the users' viewing data for this. Which statements are correct? Hard
  4. A model achieves 95% accuracy on the validation dataset but only 71% on a separate test dataset with less common examples. What can be concluded from this? Hard
  5. A model achieves 61% accuracy on the training data and 59% on the validation data; even with more epochs, the values hardly improve. Which statements are correct? Hard

Comparative testing

  1. A city wants to check whether an update to its AI-based traffic routing system improves journeys. Which approach fits? Hard
  2. A team expects an A/B test to provide the test cases for a new model. What is there to say about this? Hard
  3. A self-learning pricing system makes a change overnight. What does the use of A/B testing look like here? Hard
  4. As a pseudo-oracle for back-to-back testing, a team builds a second model with the same open-source library and the same algorithm. Both models always deliver matching results. What must be considered? Hard
  5. A team is to build a pseudo-oracle for functional back-to-back testing on a tight budget. Which statements help? Hard
  6. An ML system is migrated from the development to the production environment. The team wants to ensure that it behaves the same there, especially for unusual inputs. Which technique is suitable? Hard
  7. A team wants, on the one hand, to find out which of two model variants has the better accuracy and, on the other, to detect defects in a new implementation. Which techniques fit each purpose? Hard

Certifications & exams

Which exam comes next?

You can already practise for 19 exams, and 21 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications