BS ISO/IEC TR 13233:1995

Foreword
Introduction
0 Scope of this Technical Report
1 Scope
2 References
3 Definitions
4 Organization and management
    4.A Use of commercial reference implementations
    4.B Proficiency testing
5 Quality system, audit and review
    5.A Maintenance procedures
    5.B Documentation of MOT and test tool validation
6 Personnel
    6.A Maintaining Competence
7 Accommodation and environment
    7.A Separation of data partitions
    7.B Identification of environment
    7.C Remote testing over a network
    7.D Checking the testing or validation environment
8 Equipment and reference materials
    8.A Means of testing and test tool validation records
    8.B Procedures for handling errors in the MOT and
          test tools
    8.C MOT and test tool maintenance
    8.D Identification of equipment
9 Validation and traceability (Measurement traceability
    and calibration)
    9.A The validation concept
    9.B MOT and test tool validation
    9.C Validation of test software in the system under
          test
    9.D.1 Traceability of validations
    9.D.2 Traceability of test results
    9.D.3 Traceability of test cases
    9.D.4 Traceability of test tools
    9.E Confidence checks
    9.F Calibration of test tools used to measure
          software timers
10 Test methods (Calibration and test methods)
    10.A Introduction to test methods
    10.B Repeatability and reproducibility
    10.C Internationally agreed test methods
    10.D Measurement uncertainty
    10.E Configuration management
11 Handling of test items (Handling of calibration and
    test items)
    11.A Protection against viruses and other agents of
          corruption
12 Records
    12.A Confidentiality and integrity of records
13 Certificates and reports
    13.A Objectivity
    13.B Clear and unambiguous test reports
14 Sub-contracting of calibration or testing
15 Outside support services and supplies
    15.A Provision of MOT and test tool validation
          services
16 Complaints
Annexes
A Glossary of Terms
B Bibliography

Gives guidance to assessors and testing laboratories on specific interpretation of accreditation requirements applying to testing (including validation of means of testing and test tools) in Information Technology and Telecommunications (IT & T) in relation to software and protocol testing services. Not applicable to accreditation of inspection, certification and quality assurance assessment activities.

0.1 This Technical Report provides guidance for assessors and testing laboratories on the specific interpretation of the accreditation requirements applicable to testing (including validation of means of testing and test tools) in the field of Information Technology and Telecommunications (IT&T), specifically in relation to software and protocol testing services. This Technical Report does not apply to the accreditation of inspection, certification and quality assurance assessment activities. 0.2 However, ISO/IEC Guide25 and any other applicable ISO/IEC Guides take precedence over the interpretation given in this Technical Report. 0.3 This Technical Report covers the use by accredited testing laboratories of services for the validation of means of testing (MOT) and test tools, and also applies to the possibility of accreditation of MOT and test tool validation services, because such a validation service is just a specialised form of software testing service. NOTE — In many areas of IT&T, it may be impractical to require the use of accredited MOT and test tool validation services, both economically and given the state of the art in the particular area. It is important to recognise that the mere existence of an applicable accredited validation service does not mean that relevant accredited testing laboratories should be required to use it, as other suitable forms of MOT and test tool validation may exist. Other factors outside the scope of this Technical Report will determine if and when use of accredited MOT and test tool validation services might become a requirement. 0.4 The aim is that it should be generally applicable across the whole software and protocol testing area, whenever accreditation to ISO/IEC Guide25 applies. However, it does not cover all the requirements of ISO/IEC Guide25. Laboratories are reminded that, in order to obtain and maintain accreditation, they shall fully comply with ISO/IEC Guide25. This Technical Report interprets the ISO/IEC Guide25 requirements in this field; it does not in any way replace them. Furthermore, there may be other interpretations of ISO/IEC Guide25 which are sector independent, maybe focusing on just one aspect of accreditation, in which case such generally applicable interpretations continue to apply, and are not replaced by this interpretation. 0.5 This interpretation applies to conformance testing and other types of objective testing of software. Specific guidance is provided for OSI, telecommunications protocols, product data exchange (as defined by ISO TC184), graphics, POSIX and compilers. The testing of physical properties of hardware is outside the scope of this interpretation, but may be covered elsewhere. Evaluation of systems and products, as in IT&T Security and Software Quality evaluation (ISO/IEC9126), is also not included in the scope of this interpretation. Safety-critical software and general application software testing are also not included in this edition. 0.6 Specific text is given in this interpretation for conformance testing. However, the general interpretations given in this Technical Report are applicable to all types of objective testing, including measuring some objective aspects of performance (e.g. as in compiler testing for some programming languages) and types of testing that are particular to a single area within the IT&T field. Analysis by the test operator in order to produce the final result for a test case, in accordance with procedures that lead to objective results, is included in this interpretation. NOTES Normally, each individual test case in a test suite (set of test cases) will be designed to yield a test verdict, that is a statement of pass, fail or inconclusive. Conformance testing involves testing the implementation against the conformance requirements specified in one or more standards (or other normative specifications). The standards against which implementations are tested for conformance will often be International Standards, although they may be ITU-T Recommendations, regional or national standards, or even a manufacturer\'s specification when the manufacturer is seeking independent confirmation that the implementation conforms. The test cases to be used in conformance testing may also be standardized, but (in the fields of software and protocol testing) are usually distinct from the standards which specify the requirements to which implementations are supposed to conform. Each test verdict should be made with respect to the purpose of the test case and the requirements of the relevant standard(s). Optionally, a particular test suite may specify various classes of pass, fail or inconclusive test verdict (e.g. fail class 1 : severe non-conformance; fail class 2: invalid behaviour but satisfied the test purpose), but this does not alter the general points about test verdicts. Requirements in ISO/IEC Guide25,IT&T Interpretations and Definitions Guidance and Examples 1 Scope No IT&T specific interpretation is required for clause1 of ISO/IEC Guide25. See clause0 for the scope of this Technical Report. Note that this Technical Report applies to testing laboratories but not to calibration laboratories. The relevant laboratories, however, include validation laboratories that offer validation services for means of testing and/or test tools to be used by testing laboratories; in this case, the item to be validated is to be regarded as a system or implementation under test. 2 References No IT&T specific interpretation is required for the references of ISO/IEC Guide25. The following standards contain provisions which, through reference in this text, constitute provisions of this Technical Report. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to agreements based on this Technical Report are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below. Members of IEC and ISO maintain registers of currently valid International Standards. ISO/IEC Guide25: 1990, General requirements for the competence of calibration and testing laboratories. ISO9000-3: 1991, Quality management and quality assurance standards- Part3: Guidelines for the application of ISO9001 to the development, supply and maintenance of software. These are the only normative references required by this interpretation. Informative references used in this Technical Report are given in AnnexB. 3 Definitions No IT&T specific interpretation is required for the definitions of ISO/IEC Guide25. However, ISO/IEC Guide25 subclause3.7 is referenced in 10.A.1 and subclause3.15 is referenced in 4.B. Additional definitions are required in this Technical Report; for these, see AnnexA. As far as possible standard definitions are used. Even where this is not possible, the intent is not to standardize new definitions but rather to explain the meaning of terms as used in this Technical Report. The distinction between a means of testing (MOT) and a test tool is important in this interpretation. The complexity of MOT and test tools varies from one area of software testing to another. For example, in OSI and telecommunications protocol testing, each MOT is a very complex hardware and software system which plays a major part in the testing, whereas in compiler testing, in addition to the test suite (of programs) itself, only a few ancillary software test tools are used. For the purposes of this Technical Report, a means of testing is hardware and/or software, and the procedures for its use, including the executable test suite itself, used to carry out the testing required. In an accredited testing service, the MOT is run under the control of the testing laboratory. For the purposes of this Technical Report, a test tool is hardware and/or software, excluding the test suite itself, used to carry out or assist in carrying out the testing required. It may be concerned with running the test cases, analysing the results, or both. Those concerned with running the test cases may also involve parameterization, selection or even generation of the test cases. 4 Organization and management 4.1 No IT&T specific interpretation is required for sub clause4.1 of ISO/IEC Guide25. 4.2 ISO/IEC Guide25 subclause4.2 is interpreted in 4.A and 4.B. 4.A Use of commercial reference implementations In ISO/IEC Guide25, subclause4.2, requires the following: \'The laboratory shall... have arrangements to ensure its personnelare free from any commercial, financial andother pressures which might adversely affectthe quality of their work; be organized in such a way that confidencein its independence of judgement and integrityis maintained at all times;...\' In IT&T, these requirements shall be interpreted as follows. 4.A.1 If a commercial implementation (not designed to be a reference implementation) is used by a testing laboratory or validation laboratory as a reference implementation for the purposes of MOT validation within the laboratory, then the adequacy of the technical coverage with respect to the other implementations that are available shall be kept under review by the laboratory, in consultation with the accreditation body. If it is agreed that the technical coverage has become inadequate compared to other implementations, then the commercial implementation should be replaced, within a time period to be agreed with the accreditation body, by one of the following, as appropriate: another implementation with better coverage; a set of implementations from different suppliers; or an implementation which is designed to be a reference implementation. See9.B for a description of the use of reference implementations in MOT validation. It is recognised that for some IT&T standards there may be no alternative to using a normal commercial implementation as a reference implementation against which to validate the MOT. In such cases, the publication of the identity of the reference implementation (in order to be open about the nature of the MOT validation conducted) may inadvertently give commercial advantage to the supplier. The decision to use a normal commercial implementation as a reference implementation, and the choice of which commercial Implementation to use in this way, are decisions to be made by the laboratory. In some cases, it may be necessary to use multiple reference implementations for MOT validation, in order to ensure that adequate coverage of the MOT behaviour is checked. This arises because a given commercial implementation may only support a subset (or \'profile\') of the relevant specification(s). This may be acceptable as a temporary solution, particularly if the market Is primarily interested in that subset, but is inadequate as a longer term solution if the testing service is to cover the specification(s) in full. If a set of implementations is chosen, the set shall be chosen to give better technical coverage of the relevant specification(s) and not for commercial reasons. 4.B Proficiency testing In ISO/IEC Guide25, subclause4.2j) requires the following: \'The laboratory shall, where appropriate,participate in interlaboratory comparisons andproficiency testing programmes. \' The definition of proficiency testing given in subclause3.15 of ISO/IEC Guide25 is as follows: \'Determination of the laboratory ... testingperformance by means of interlaboratorycomparisons. \' In ISO/IEC Guide25, subclause5.6 requires the following: \'In addition to periodic audits the laboratory shallensure the quality of results ...by implementingchecks. These checks ... shall include, asappropriate, but not limited to: .. participation in proficiency testing or otherinterlaboratory comparisons;\' In IT&T, these requirements shall be interpreted as follows. 4.B.1 If a laboratory claims to offer a harmonised testing or validation service, it shall provide evidence of its participation in the relevant inter-laboratory comparisons to ensure that the declared harmonisation is achieved and maintained. There may be numerous inter-laboratory comparison schemes organised for IT&T. IT&T Agreement Groups have been and are being formed to operate mutual recognition agreements whereby the group of testing laboratories establish the means to recognise the mutual equivalence of their corresponding testing services. Such Agreement Groups provide one formalized way of participating in inter-laboratory comparisons. They may require that testing service harmonization and demonstrations of equivalence are carried out, and that all participating testing laboratories become accredited for the services they offer (within a reasonable period of time). Agreement Groups may also provide inter-laboratory comparison schemes for MOT validation services. If it is not practical or economic for the laboratory to participate in inter-laboratory comparisons, then the laboratory shall not claim that the service is harmonised.

A laboratory may decide not to join an Agreement Group and therefore not to claim to provide a harmonised testing or validation service. It may nevertheless be required by the accreditation body to participate in some informal inter-laboratory comparison exercises, in order to overcome any doubts there may be about the objectivity of its test or validation results. 5 Quality system, audit and review 5.1 No IT&T specific interpretation is required for this subclause. 5.2 ISO/IEC Guide25 subclause5.2 is interpreted in 5.A and 5.B. 5.3 No IT&T specific interpretation is required for this subclause. 5.4 No IT&T specific interpretation is required for this subclause. 5.5 No IT&T specific interpretation is required for this subclause. 5.6 ISO/IEC Guide25 subclause5.6 is interpreted in 4.B. 5.A Maintenance procedures In ISO/IEC Guide25, subclause5.2 requires the following: \'The quality manual and related qualitydocumentation shall also ... reference procedures for... verification andmaintenance of equipment; ... In ISO/IEC Guide25, subclause8.2 requires the following: \'All equipment shall be properly maintained.... \' In ISO/IEC Guide25, subclause9.1 requires the following: \'All... testing equipment having an effect on theaccuracy or validity of... tests shall be... verifiedbefore being put into service. The laboratory shallhave an established programme for the ...verification of its... test equipment.\' In IT&T, these requirements shall be interpreted as follows. 5.A.1 A testing laboratory shall have procedures defining the checking to be performed whenever major or minor changes are made to the MOT or other test tools, in order to ensure that harmonisation is maintained as appropriate with other testing laboratories and that correctness is maintained with respect to the relevant standard(s) or specification(s). 5.A.2 A validation laboratory, or a testing laboratory which conducts its own MOT or test tool validations, shall have procedures defining the checking to be performed whenever major or minor changes are made to the reference implementation or other means of validation, in order to ensure that harmonisation is maintained as appropriate with other validation laboratories and that correctness is maintained with respect to the relevant standard (s) or specification (s). 5.B Documentation of MOT and test tool validation In ISO/IEC Guide25, subclause5.2 requires the following: \'The quality manual and related qualitydocumentation shall also contain: . . . m) reference to procedures for... verificationand maintenance of equipment; . . . In ISO/IEC Guide25, subclause10.1 requires the following: \'The laboratory shall have documentedinstructions on the use and operation of allrelevant equipment...\' In ISO/IEC Guide25, subclause9.2 requires the following: \'The overall programme of ... validation ofequipment shall be designed and operated so asto ensure that, wherever applicable,measurements made by the laboratory aretraceable to national standards of measurementwhere available: In the context of MOT and test tool validation, these requirements shall be interpreted as follows. 5.B.1 The procedures for carrying out MOT and test tool validations shall be documented by the laboratory. If an accredited external validation service is used, then the procedures merely need to refer to the use of that service. If a non-accredited external validation service is used, then the laboratory should provide adequate procedures for the selection and monitoring of the results of the service (see15.A.2). If the laboratory carries out its own validations, then the procedures should include those for selecting which test cases to run. 5.B.2 If, for a given MOT or test tool, there is no suitable validation service available outside the testing laboratory to which accreditation is applicable, and there is no suitable reference implementation that could be used by the testing laboratory to validate the MOT or test tool, then the testing laboratory shall define and document the procedures and methods that it uses to check on the correct operation of the MOT or test tool, and provide evidence that these procedures and methods are applied whenever the MOT or test tool is modified. The suitability of an external validation service may depend not only on its relevance to the given MOT or test tool, but also on the cost-effectiveness of using the service compared to alternative means of validation that may be available and acceptable.

The locally defined procedures could involve arbitrarily complex arrangements of other hardware and software tools. They could also involve some checking of the MOT or test tool by one or more other testing laboratories. ISO/IEC Guide25, subclause9.3, cites inter-laboratory comparison as one of the means of providing satisfactory evidence of correlation of results. Such checking is required to fulfil the requirements in ISO/IEC Guide25 subclauses8.2 and 9. See9.A and 9.B on the validation of the MOT and test tools. 6 Personnel 6.1 ISO/IEC Guide25 subclause7.2 is interpreted in 6.A. 6.2 ISO/IEC Guide25 subclause7.2 is interpreted in 6.A. 6.3 No IT&T specific interpretation is required for this subclause. 6.A Maintaining Competence In ISO/IEC Guide25, subclause6.1 requires the following: \'The testing laboratory shall have sufficientpersonnel having the necessary education,training, technical knowledge and experience fortheir assigned functions. \' In ISO/IEC Guide25, subclause6.2 requires the following: \'The testing laboratory shall ensure that thetraining of its personnel is kept up-to-date. \' In IT&T, these requirements shall be interpreted as follows. 6.A.1 The testing laboratory shall have a procedure to define how to maintain competence of its test operators, particularly in the absence of clients for a given testing or validation service. It is recommended that in order to maintain their competence in operating a particular testing or validation service, the relevant test operators sh