Qwissit Artificial Intelligence

ISTQB CT-AI

Testing ML Development and Deployment

60 questions

ML development risks

  1. What does ML development testing focus on? Easy
  2. Which mitigation helps against incorrect or unintended use of library or framework APIs such as TensorFlow or PyTorch? Easy
  3. Which mitigation helps against a defective framework installation or build? Easy
  4. Which mitigation helps against poor usability of the framework? Easy
  5. Which mitigation helps against security vulnerabilities in the framework? Easy
  6. Which mitigation helps against sub-optimal framework selection? Easy
  7. Which mitigation helps against poor user documentation of the framework? Easy
  8. Which mitigation helps against defective allocation of data to the training, validation and test datasets? Easy
  9. Which test shows before full deployment whether a new model is an improvement over the current one? Easy
  10. Which mitigation helps against poor performance efficiency, for example a framework that is slow to respond? Easy

Deployment testing

  1. What does installability testing check? Easy
  2. What does rollback testing check? Easy
  3. When must rollback testing be performed? Easy
  4. What is canary testing? Easy
  5. What is shadow testing? Easy
  6. What does model conversion testing check? Easy
  7. What does cross-device testing check? Easy
  8. What does API testing check when deploying an ML system? Easy
  9. In which target environments must an ML system function correctly and reliably when deployed? Easy
  10. Which real-time metrics are monitored in canary testing? Easy

ML development risks

  1. Which mitigations help against a defective implementation of the evaluation by the framework? Medium
  2. Which mitigations help against a defect in a library used by the framework or a defective algorithm implementation? Medium
  3. Which mitigations help against sub-optimal algorithm selection? Medium
  4. Which mitigations help against sub-optimal hyperparameter selection, for example network structure or learning rate? Medium
  5. Which mitigation helps against poor selection of the evaluation approach, for example k-fold cross-validation? Medium
  6. Which mitigations help against a deployment defect, for example when generating a modified version for a target platform? Medium
  7. Which mitigations help if a deployed model is incompatible with the operational environment? Medium
  8. Which mitigation helps against problems in the algorithm, model or framework that introduce systemic unfairness? Medium
  9. Why does ML functional performance testing help against incorrect interpretation of test results? Medium
  10. Against which risks does smoke testing help? Medium
  11. Which test types serve to verify robustness and efficiency in ML development testing? Medium

Deployment testing

  1. What is checked during installability testing of an ML system, among other things? Medium
  2. What can rollback testing cover? Medium
  3. Which defects can shadow testing uncover before full deployment? Medium
  4. What does model conversion testing check besides predictive accuracy? Medium
  5. With which external systems are integration workflows checked during API testing of an ML system? Medium
  6. What distinguishes canary testing from shadow testing? Medium
  7. What is the goal of canary testing? Medium
  8. A model was converted from the training format into a compact runtime format for a smartphone. Which deployment test is intended for this? Medium
  9. An ML system is to run on Android smartphones, on edge devices in factories and on cloud servers. Which deployment test is particularly relevant? Medium

ML development risks

  1. A team suspects that a framework's metric calculation delivers wrong F1 values. Which mitigations fit this risk? Hard
  2. After a PyTorch update, the same model with the same data delivers different results. Which risk is present, and which mitigations fit? Hard
  3. A team finds that some records appear in both the training and the test dataset. Which risk is present, and which mitigation fits? Hard
  4. Two training runs with identical configuration yield slightly different accuracy values, and a developer draws far-reaching conclusions about a hyperparameter change from this. Which risk and which mitigation fit? Hard
  5. Which mappings of risk and mitigation in ML development are correct? Hard
  6. A team chose k-fold cross-validation with an unsuitable split and considers the results reliable. Which mitigation helps against this development risk? Hard
  7. An error occurred when generating an adapted model version for a target platform. Which mitigations fit? Hard
  8. A team chose a framework that is unsuitable for the planned use on edge devices, and its responses are also too slow. Which mitigations fit? Hard
  9. A developer calls a function of an ML library with swapped parameters without an error being reported. Which risk is present, and which mitigation fits? Hard

Deployment testing

  1. A bank wants to compare a new fraud detection model with live transactions without customers being affected by possible wrong decisions of the new model. Which deployment test fits? Hard
  2. An online retailer wants to introduce a new recommendation model gradually and detect regressions early, before all users receive it. Which deployment test fits, and how does it work? Hard
  3. A team only wants to check the rollback once a deployment has actually failed. What is there to say about this? Hard
  4. After a failed model update, the data pipeline is also to be reset to its previous state. Which statements about rollback testing are correct? Hard
  5. A model was compressed for an edge device and converted into another format. On the device it is faster, but some predictions deviate. Which test covers this, and what does it check? Hard
  6. An ML system cannot be started on a new server because a suitable GPU driver is missing. Which deployment test should have detected this? Hard
  7. A model works correctly in the cloud but produces errors on older smartphones. Which deployment test addresses this risk? Hard
  8. A customer reports that the interface of an ML service returns no understandable error message for faulty inputs and is not standards-compliant. Which test is responsible, and what does it cover? Hard
  9. A team wants to compare a new model with the existing one using live data and also detect data drift and performance regressions without affecting users. Which statements are correct? Hard
  10. Which mappings of deployment risk and deployment test are correct? Hard
  11. A company plans to introduce a new model in three steps: first in parallel without affecting users, then for a small subset of users, then for everyone. Which tests fit the first two steps? Hard

Certifications & exams

Which exam comes next?

You can already practise for 19 exams, and 21 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications