Moderator: Hello and welcome to a new episode of "Applied FuSa," a podcast for FuSa pragmatists. Confirmation Measures, which means Confirmation Review, Audits and Assessments. In today's episode, we'll examine their significance and how to implement them in practice. We'll pay special attention to the topic of Level of Independence. Do audits and assessments always have to be conducted by external providers? Are internal auditors and assessors independent enough? These are questions we will answer.
Expert: Confirmation Measures consist of Confirmation Reviews on one side, and Audits and Assessments on the other. The division into two groups makes sense for the following reason. In contrast to Audits and Assessments, Confirmation Reviews fundamentally relate to individual Work Products. Put simply: Confirmation Reviews are used to check whether individual Work Products meet the requirements of ISO 26262. In an audit, it is examined whether the safety-relevant processes conform to the standard, and during an assessment, the technical contents of the Work Products are evaluated to determine whether the functional safety concepts have been completely and correctly implemented. This expresses a fundamental character of functional safety clearly. It is never about right or wrong. It is about risk minimization, and the question of whether this goal has been achieved can only be answered objectively in part. There always remains a subjective component in the argumentation. Is this acceptable? Should a product not be able to be considered safe precisely when all requirements of ISO 26262 are demonstrably fulfilled? Well, there are also requirements that require precisely this subjective evaluation. Take as an example the probably most important work product, the safety case. Let's look at some definitions from volume 1 of the safety standard. 3.136, safety case: argument that functional safety is achieved for items, or elements, and satisfied by evidence compiled from work products of activities during development; 3.67, functional safety: absence of unreasonable risk due to hazards caused by malfunctioning behavior of E/E-systems; 3.176, unreasonable risk: risk judged to be unacceptable in a certain context according to valid societal moral concepts. End of quote. So, the safety case must provide an argument that functional safety has been completely achieved for a product. Functional safety is defined as the absence of unreasonable risk. Risk is designated as unreasonable when it must be evaluated (!) as unacceptable. This means: The argument that must be provided in the safety case ultimately goes back to the evaluation of what is to be considered an unacceptable risk. Evaluation criteria are not formulated in ISO 26262, for the simple reason that evaluation criteria can change over time. How must risk then be evaluated? Often the best engineering practice is cited as an argument, and this is not only legally a generally accepted practice. However, we must be clear about this, and this must be emphasized again and again, that the safety of a product or the residual risk that always remains must be evaluated by humans. Humans who can make mistakes, but engineering judgment often remains the final authority. Why this excursus? Because, as we will see, it is important for understanding some important aspects of the topic of confirmation measures. To give an example: As mentioned, it is about risk minimization, and whether risk has been sufficiently minimized can sometimes only be evaluated through engineering judgment. The application of certain methods or the avoidance of other methods has the exclusive goal of reducing risk. Here too, an example. If I as a software engineer check my own source code, then the risk that a safety-relevant error is NOT found is usually significantly higher than if a completely independent and sufficiently experienced person does it. The choice of independence alone is a way to minimize risk. But how can one decide whether the person is sufficiently independent and experienced? Well, there are hardly any objective criteria. This too is a subjective evaluation. When it is checked in the course of a confirmation review whether a work product meets the requirements of ISO 26262, then engineering judgment is also required again and again in these cases, since there is sometimes room for interpretation. The same applies to the requirements for the process, which are checked during an audit. Finally, one sometimes cannot avoid the subjective evaluation of an implementation during an assessment when it comes to finding out whether a concrete, technical solution correctly implements the underlying safety concept. This may sound very unsatisfactory to the inexperienced. However, it should be noted that every measure can always only achieve a minimization of risk, because there will never be perfectly safe products. A residual risk for malfunctions always remains. The most important goal is that this residual risk is a generally accepted one, which is also a subjective evaluation. Note: Standards, by the way, have the overarching goal of ensuring precisely that generally acceptable level regarding quality, safety and security for a product.
Expert: There are confirmation reviews for the following nine work products: Impact analysis; HaRa; safety plan; functional safety concept; technical safety concept; integration and test strategy; safety validation; safety analysis and dependent failure analysis; and safety case. ISO 26262 defines in Volume 3, Chapter 6.4.9 a total of 5 generic requirements, which must be fulfilled by all nine confirmation reviews. Let's go through these one by one. Requirement 6.4.9.1: The functional safety of the item and its elements shall be confirmed, based on: A, confirmation reviews to judge whether the key work products, i.e. those included in Table 1, provide sufficient and convincing evidence of their contribution to the achievement of functional safety, considering the corresponding objectives and requirements of the ISO 26262 series of standards, in accordance with Table 1 and 6.4.10; B, a functional safety audit to judge the implementation of the processes required for functional safety, in accordance with Table 1 and 6.4.11; and C, a functional safety assessment to judge the achieved functional safety of the item, or the contribution to the achievement of functional safety by the developed elements, in accordance with Table 1 and 6.4.12. End of quote. These first three requirements define the basis on which the achievement of functional safety for a product should be evaluated. They demand what has already been said. The achievement of functional safety should be demonstrated with the help of successfully completed Confirmation Reviews, Audits and Assessments. The goal of the next two requirements is that the Confirmation Measures do not lose efficiency or effectiveness because needed persons, documents or tools are not available. Requirement 6.4.9.2: The persons who carry out a confirmation measure shall have access to, and shall be supported by, the persons and organizational entities that carry out safety activities during the item development. End of quote Requirement 6.4.9.3: The persons who carry out a confirmation measure shall have access to the relevant information and tools. End of quote Interim summary: The requirements for Confirmation Measures demand that the achievement of Functional Safety is demonstrated by means of Confirmation Reviews, Audits and Assessments, and that all needed persons, documents and tools are available for these activities. Sounds somewhat trivial, but is important because it sets the framework for conducting the Confirmation Reviews. Table 1 as well as further requirements specifically for Confirmation Reviews, Audits and Assessments provide significantly more details. So let's look next at what is defined in Table 1.
Expert: Table 1 lists for all Confirmation Measures the required independence of the persons who carry out the respective Confirmation Measure. The degree of independence is also dependent on the ASIL of the Work Products that are the subject of the Confirmation Measures. The degree of independence can take 4 values: I0, the confirmation measure should be performed; however, if the confirmation measure is performed, it shall be performed by a different person in relation to the person(s) responsible for the creation of the considered work product(s); I1, the confirmation measure shall be performed, by a different person in relation to the person(s) responsible for the creation of the considered work product(s); I2, the confirmation measure shall be performed, by a person who is independent from the team that is responsible for the creation of the considered work product(s), i.e. by a person not reporting to the same direct superior; and I3, the confirmation measure shall be performed by a person who is independent, regarding management, resources and release authority, from the department responsible for the creation of the considered work product(s). In the first two cases, I0 and I1, it is required that a Confirmation Measure is carried out by persons who were not involved in the creation of the respective work products. The only difference between I0 and I1 is that for I0 the Confirmation Measure is recommended (should), while it is required for I1 (shall). Otherwise: the main thing is that no persons are involved who developed the work product. Developers and reviewers may be on the same team. For I2, a Confirmation Measure is required, and additionally, that the persons who developed the work products and those who carry out the Confirmation Measure may not belong to the same team. They may not have the same direct superior. The requirement for I3 is a bit more complicated for good reason. A Confirmation Measure is required. The persons must be independent from the entire department within which the work products were developed. This means that they have no common superiors with any of the departments responsible for management, resources and release. Since I3 is the strongest requirement, it follows overall that nothing speaks against carrying out confirmation measures internally, as long as the required independence is ensured. For the Safety Case, it is absolutely necessary that compliance with the required independence is proven and documented.
Expert: We will now consider the requirements for confirmation reviews. ISO26262 has specified five requirements for confirmation reviews. So, let's look at these in detail and see what conclusions we must or can draw. Requirement 6.4.10.1: A person responsible to perform the confirmation review shall be appointed, in accordance with 5.4.4 and 5.4.2.7, for each confirmation review that is included in Table 1 and required by the safety plan. This person shall provide a report that contains a judgement of the achieved contribution to functional safety by the work product. End of quote. That persons need to be appointed at all is obvious, because otherwise the Confirmation Measures could not be carried out. The actual requirement is rather in the two references. Requirement 5.4.2.7 demands for the Confirmation Measures that the responsible persons be equipped with all necessary authorities to be able to do their job. Chapter 5.4.4 defines requirements for Competence Management. A topic that is often interpreted to mean that sufficient competence must be demonstrated for every person involved in safety-relevant activities. However, this is not exactly what ISO 26262 expects. Chapter 5.4.4 specifies requirements for how competence is ensured, not that it already exists. From a FuSa perspective, it is absolutely acceptable if an employee is not yet sufficiently trained in FuSa, as long as there are higher-level verification measures for work products that were created by inexperienced employees. Proven competence should minimize risk entirely in the spirit of FuSa. Those who are competent make fewer mistakes. Reviews can minimize these mistakes just as well, provided they are carried out by a sufficiently competent person. Back to the topic... Briefly summarized, this first requirement means that those responsible must be provided with sufficient authorities, and that there must be Competence Management in the sense of the ISO. Requirement 6.4.10.2: The confirmation reviews shall be finalized before the release for production. End of quote. Well, this is more of a requirement for Release for Production. Basically, the requirement states that Release for Production may not be granted until all Confirmation Reviews have been successfully completed. A similar requirement accordingly also exists in Chapter 6.4.13 Release for Production, so that this requirement here can be considered redundant. Reference is gladly made at this point to the episode about Release for Production. Listen in. It's only a few minutes, because there really isn't much to say about it. Requirement 6.4.10.3: A confirmation review may be based on performing a judgement of whether the corresponding objectives of the ISO 26262 series of standards are achieved. End of quote. This defines the actual goal of every Confirmation Review. It should be checked for the respective work product whether the associated objectives were achieved; that is, whether all applicable requirements of ISO 26262 for the work product are demonstrably implemented. One possibility to provide this proof is to use checklists in the Confirmation Review that contain all work product specific requirements. However, it is far more effective if existing verification measures and their results are checked, as a simple example should show. If there are detailed Review Reports for the Technical Safety Concept, and the reviews were carried out by sufficiently independent and experienced persons, then this is proof that will minimize risk just as much as if the responsible persons verify all requirements again during the Confirmation Review. In that case, sufficient competence would also have to be demonstrated. The effort would therefore be at least comparable, or even higher. Requirement 6.4.10.4: One or more assistants may be appointed to support the performance of a confirmation review in accordance with 6.4.9.2 and 5.4.4. Such persons may lack independence from the developers of the corresponding item, elements or work products, but their independence shall be at least I1, as defined in Table 1, and the reviewer shall appraise their input to ensure an unbiased opinion is given. End of quote. This requirement points out on one hand that it is allowed and sometimes necessary to bring additional persons into a confirmation review. These persons must be available when the responsible person needs them. The requirements for competence management must also be fulfilled for persons brought in. On the other hand, an independence of at least I1 is required, which is a bit irritating, because the difference between I0 and I1 is not relevant at all at this point since it is only about whether the confirmation review should take place or not. What it's really about is that persons brought in do not have to fulfill the degree of independence required for the confirmation review. Requirement 6.4.10.5: A confirmation review and a verification review may be combined, provided the review is performed with sufficient independence in accordance with Table 1. End of quote. We will talk about the difference between confirmation and verification review at the end of the episode. What this requirement wants to say is that the persons who carried out a Verification Review must fulfill the required degree of independence if results of the Verification Review are to be used for the Confirmation Review. Not in the sense that the results should be reviewed, but in the sense that they should be reused, that is, the Confirmation Review is tailored.
Expert: In Volume 1, Definition 3.181, the term Verification Review is defined as "activity to ensure that the result of a development activity fulfills the project requirements, or technical requirements, or both". End of quote. In other words, a verification review checks the fulfillment of requirements for a work product and is not limited to safety-relevant requirements. Any kind of activity is meant here, as emerges from Definition 3.180 (Verification). Confirmation Reviews relate, as we have seen, primarily to the fulfillment of the requirements from ISO 26262 relevant for a work product. In general, there will be overlaps in projects. If only because many work products for which requirements can be found in ISO 26262 are not FuSa-specific work products (for instance, source code). Work products that are also created when a product is not safety-relevant at all. In such cases, verification reviews will generally take place, whose results are reused for confirmation reviews when the mentioned requirement 6.4.10.5 is fulfilled. It is in any case sensible to consider this option when creating the Safety Plan in order to be able to save effort, which can be considerable. It must also not be forgotten that a confirmation review that has the same content as a completed verification review represents an unnecessary risk, because mistakes can be made that are actually avoidable. A quite general principle is hidden in this. Every safety-relevant activity that is carried out but did not have to be carried out represents an unnecessary risk.
Moderator: Applied FuSa – a podcast for Functional Safety pragmatists. Get your new piece of FuSa every other week.