Contact number:

+886 2 77182788

What is test coverage?

As the name suggests, test coverage is a metric that measures the scope of testing for a software program, used to evaluate the completeness of testing. The most commonly used test coverage evaluation methods are requirement-based test coverage and code-based test coverage.

Requirement-based test coverage refers to the fact that each design requirement has at least one test case to verify it. In actual evaluations, there are two methods: one is to consider the test case corresponding to the design requirement as having achieved test coverage for that requirement after execution, or to only consider the requirements corresponding to test cases that passed as covered. Typically, both approaches are acceptable. The level of requirement-based test coverage primarily depends on whether testers have created and executed test cases specifically targeting each design requirement.

Code-based test coverage is a metric that measures the scope of test execution at the code level, assessing the completeness of testing by counting how much/which code was executed during testing. Depending on the statistical criteria used, it is often divided into various types of test coverage, such as statement coverage, branch coverage, modified condition/branch coverage (MC/DC), function coverage, and function call coverage. Code-based test coverage provides a more precise quantitative metric for testing completeness.

Why measure test coverage?

In short, test coverage ensures testing completeness by quantifying “where in the software has been tested and where has not been tested.” Of course, there is no such thing as perfect software or 100% comprehensive test cases, but users can at least use test coverage metrics to evaluate whether the completeness of testing has met expectations. Clearly, if there are still a significant number of requirements or code segments that have not been covered by testing after the testing process is complete, this is an issue that requires careful consideration.

Therefore, for software systems with high reliability or safety requirements, evaluating test coverage to improve test quality is an effective and necessary approach.

On the other hand, whether it is functional safety SIL/ASIL or airworthiness certification, the IEC 61508/EN 5018/ISO 26262/DO-178B, DO-178C standards explicitly stipulate requirements for software test coverage. As shown in the following figure illustrating SIL certification requirements:

1629789287973228.png

-- HR: Highly Recommended

-- R: Recommended

Entry Points Coverage: Entry point coverage. The most basic type of test coverage, now more commonly implemented in many industries using the standards of “function coverage” and “function call coverage,” which require that the functions contained in the software are at least executed and called to avoid obvious omissions in testing or the appearance of honor codes.

Statement Coverage: Statement coverage. Certification levels such as SIL Level 2, ASIL Level B, DO-178B, and DO-178C Level C or above require software testing to achieve statement coverage, ensuring that each executable line of code is executed at least once during testing. For example, the following statement can be covered by a single test case:

if((a || b) && c)

Branch Coverage: Branch Coverage. SIL Level 3, ASIL Level C, DO-178B, DO-178C Level B and above require branch coverage testing, focusing on each branch decision statement—the result of which can be TRUE or FALSE—to ensure that each branch is traversed at least once. Using the following if statement as an example:

if((a || b) && c)

To meet the branch coverage requirement for this statement, at least two test cases are needed to cover its TRUE and FALSE branches. Therefore, when designing test cases, we need to consider the conditional statements in the if statement and create test input parameters that meet the requirements.

MC/DC Coverage: Modified Condition/Branch Coverage. SIL Level 4, ASIL Level D, DO-178B, DO-178C Level A, or nuclear safety level standards, in addition to requiring the above test coverage, also require MC/DC coverage. This requires higher test coverage and more stringent coverage conditions. MC/DC coverage requires that each sub-condition in the conditional statement independently affects the conditional result. In other words, each sub-condition in the conditional statement changes the conditional result while the other conditions remain unchanged. For example,

if((a || b) && c)

To meet the MC/DC coverage requirements for the above if conditional statement with three sub-conditions (a, b, and c), we need to design at least 4 (n+1) test cases, forming 3 pairs of test cases, so that a, b, and c each independently influence the conditional result. As shown in the figure below.

1629789322108819.png

Even the most experienced test engineers would find this task difficult to accomplish. Therefore, MC/DC coverage is generally only required in projects with the highest safety requirements, i.e., systems where software errors could cause numerous deaths and have a relatively high probability of occurring.

Demands and Challenges

  • Statistical code coverage testing requires the insertion of test code, but excessive insertion expansion, compiler incompatibility, or immature insertion algorithms can result in the inability to compile or run normally after insertion.

  • Only the test coverage of the host computer software can be calculated; embedded environments are not supported.

  • Only the test coverage of a single test phase can be calculated, such as the coverage of unit tests, and the system test coverage cannot be calculated.

  • Test coverage cannot be accumulated across multiple test executions; repeated集中执行 is required.

  • Test coverage reports are not intuitive and difficult to understand.

Solutions

  • Test coverage and traceability based on requirements. By establishing traceability management between design requirements and test cases, we ensure that “requirements-based testing” cases fully cover the requirements.

  • Automatically calculate test coverage for modules such as unit tests and integration tests.

  • Test coverage of automatic statistical system functional testing

  • Supports test coverage aggregation and incremental analysis between different levels of test phases.

  • Fully supports various embedded target machine environments and host machine platforms.

  • Common types of test coverage, such as statement coverage, branch coverage, MC/DC coverage, function coverage, and function call coverage.

  • Dynamic test coverage statistics at runtime.

  • Animation playback test coverage process.

  • Solutions that fully comply with high reliability industry certification standards such as airworthiness and functional safety.

Related Products

RELATED RESOURCES

Download Request

Do you need technical support?

Verification code

Warm tips：

We will send you the download address by e-mail. Please check whether your work email is correct.

Submit