ISTQB CT-GenAI

LLM-Powered Test Infrastructure

60 questions

Easy

Architecture and RAG

  1. What is the task of the front end in an LLM-powered test infrastructure?
  2. Which task does the back end of an LLM-powered test infrastructure typically take on?
  3. How can the language model be provided in an LLM-powered test infrastructure?
  4. What is a vector database used for in an LLM-powered test infrastructure?
  5. What does retrieval-augmented generation (RAG) achieve?
  6. Why are large bodies of text split into smaller sections (chunks) for RAG?
  7. What benefit does RAG have in test analysis?

LLM-powered agents

  1. What distinguishes an LLM-powered agent from a simple AI chatbot?
  2. What is meant by "tools" in LLM-powered agents?
  3. What characterizes an autonomous agent?
  4. What characterizes a semi-autonomous agent?
  5. What is called orchestration in multi-agent systems?
  6. Which weakness do LLM-powered agents share with language models?

Fine-tuning and LLMOps

  1. What is meant by fine-tuning a language model?
  2. Why can it make sense to adapt a small language model (SLM) instead of an LLM through fine-tuning?
  3. What does overfitting mean in fine-tuning?
  4. What is especially important in fine-tuning for test tasks to avoid biased results?
  5. What does LLMOps refer to?
  6. Which of the following is a common way of bringing generative AI into an organization's test processes?
  7. Are the routes AI chatbot, GenAI test tool and in-house development mutually exclusive?

Medium

Architecture and RAG

  1. How does an LLM-powered test tool differ from a general AI chatbot?
  2. A back end checks the model's raw answer and brings it into the format of the test management tool before it is displayed. What is this step called?
  3. A team wants to store test cases with fixed fields like ID, priority and status and additionally find defect reports related in content to a question. Which combination fits?
  4. In which order do the RAG steps run after a question has been asked?
  5. What happens in RAG during preprocessing, that is, before requests are made?
  6. A health insurer changes its reimbursement rules every month. The test team wants test cases always generated to match the current rules. Which approach fits best?
  7. What characterizes an answer supported by RAG?

LLM-powered agents

  1. An agent reads a user story, creates test cases in the test management tool and starts them in the CI pipeline. Which capability makes this possible?
  2. An agent is to create and run test cases for the dosage calculation of a medical app. How can the risk of faulty agent results best be reduced here?
  3. A team has an autonomous agent evaluate regression tests at night. Which measure reduces the risk that wrong evaluations go unnoticed?
  4. A tool uses one agent each for test analysis, test design and execution, which pass results to each other. What is the expected advantage?
  5. How does test automation change with agent-based AI assistants in the test tool?
  6. Which test activities can AI assistants based on agents support?

Fine-tuning and LLMOps

  1. An energy supplier wants a model to write defect reports using in-house terms such as "installment" and "meter change" in a fixed house format. Which approach targets exactly this?
  2. A fine-tuned model delivers strange test cases, and nobody can understand why. Which challenge of fine-tuning shows here?
  3. A small test team wants to adapt a large LLM itself through fine-tuning. Which hurdle must especially be considered?
  4. A team wants to use an AI chatbot for test tasks. What is the main LLMOps focus on this route?
  5. What must an organization additionally check when it introduces a vendor's test tool with GenAI functions?
  6. What does the route of developing a GenAI test tool in-house require above all?
  7. How does fine-tuning fundamentally differ from RAG?

Hard

Architecture and RAG

  1. Why is the architecture of an LLM-powered test infrastructure considered more than a classic client-server model?
  2. A team splits its requirements manual for RAG into very large sections of 20 pages each. Which consequence is most likely?
  3. A tester asks a RAG system about "payment abort". The requirements only say "payment transaction is cancelled". Why does the system still find the passage?
  4. A RAG system delivers test cases for a rule that was changed two months ago, still in the old version. What is the most likely cause?
  5. A team compares an LLM's outputs for a test task with and without RAG. Which observation is most likely?
  6. Where in an LLM-powered test infrastructure should it be controlled which tester may access which project data?
  7. Which statement about the data sources in the back end of an LLM-powered test infrastructure is correct?

LLM-powered agents

  1. An agent is to reset outdated test data in an isolated test environment every night. The task is clearly defined and errors are easy to fix. Which degree of autonomy is most justifiable here?
  2. In which case is an AI chatbot not enough and an LLM-powered agent needed?
  3. Which statement about LLM-powered agents in the test process is false?
  4. An agent evaluates test results and wrongly marks a real defect as a "known issue". Why is this particularly critical in an automated pipeline?
  5. What does an LLM-powered agent essentially use the language model for?
  6. A team uses an agent-based assistant that derives and runs tests directly from new user stories. Which benefit is to be expected, and what remains to be considered?

Fine-tuning and LLMOps

  1. A railway company wants a model to master the stable domain logic of passenger rights reliably while always taking the current timetable into account. Which combination fits best?
  2. A model fine-tuned on an online shop's test cases writes good test cases for the shopping cart but fails with the new returns feature. Which challenge is most likely evident here?
  3. For fine-tuning, a team uses only test cases from a project in which negative tests hardly occurred. What is the most likely consequence?
  4. In which order does fine-tuning for a test task sensibly proceed?
  5. A public authority may under no circumstances pass test data to third parties and still wants to use GenAI in testing. Which route offers the greatest control over privacy and security?
  6. The provider of the LLM in use rolls out a new model version, and the generated test cases change noticeably. To which area of responsibility does handling this mainly belong?
  7. A company uses an AI chatbot for ideas on test conditions and at the same time develops its own tool with RAG for test case creation. How should this be assessed?

Certifications & exams

Which exam comes next?

You can already practise for 51 exams, and 16 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications