Third-party software certification should attest that the software product satisfies the required confidence level according to certification standards such as ISO/TEC 9126, ISO/IEC 14598 or ISO/IEC 25051. In many application areas, especially in mission-critical applications, certification is essential or even mandatory. However, the certification of software products using common off-the-shelf (COTS) components is difficult to attain, as detailed information about COTS is seldom available. Nevertheless, software products are increasingly being based on COTS components, which mean that traditional certification processes should be enhanced to take COTS into account in an effective way. This paper proposes a mean to help in the certification of component-based systems through an experimental risk assessment methodology based on fault injection and statistical analysis. Using the proposed methodology the certification authority or the system integrator can compare among components available the one that best fit for the system that is assembling a component that provides a specific functionality. Based on the results it is also possible to decide whether a software product may be considered certified or not in what concerns the risk of using a COTS into the system. The proposed approach is demonstrated and evaluated using a space application running on top of two alternative COTS real-time operating systems: RTEMS and RTLinux. © Springer-Verlag Berlin Heidelberg 2007.4746 LNCS 179197Amland, S., Risk-based Testing: Risk analysis fundamentals and metrics for software testing including a financial application case study (2000) The Journal of Systems and Software, 53, pp. 287-295Arlat, J., Fault Injection and Dependability Evaluation of Fault Tolerant Systems (1993) IEEE Transaction on Computers, 42 (8), pp. 919-923Bach, J., Heuristic Risk-Based Testing (1999) Software Testing and Engineering MagazineBasili, V., Briand, L., Melo, W., Measuring the Impact of Reuse on Quality and Productivity in Object-Oriented Systems (1995), Technical Report, University of Maryland, Dep. Of Computer Science, CS-TR-3395Chillarege, R., Orthogonal Defect Classification, Ch. 9 of Handbook of Software Reliability Engineering (1995) IEEE Computer Society, , M. Lyu Ed, McGraw-HillChristmansson, J., Chillarege, R., Generation of an Error Set that Emulates Software Faults-Based on Fields Data (1996) Proc. of 26th Int. Symp. on Fault-Tolerant Computing, pp. 304-313. , Sendai, Japan, ppColombo, R., Guerra, A.: The Evaluation Method for Software Product. In: Proc. of Int. Conf. on Software \& Systems Engineering \& Applications - ICSSEA '2002, Paris, France (2002)Councill, B., Third-Party Certification and Its Required Elements (2001) Lecture Notes in Computer Science (LNCS, , Proc. of The 4th Workshop on Component-Based Software Engineering CBSE, Springer, Heidelberg, CanadaDurães, J., Madeira, H., Definition of Software Fault Emulation Operators: A Field Data Study (2003) Proc. of The International Conference on Dependable Systems and Networks, DSN2003, pp. 105-114. , San Francisco, USA , William Carter Award for the best student paperDurães, J., Madeira, H.: Emulation of Software Faults: A Field Data Study and a Practical Approach. IEEE Transactions on Software Engineering 32(11) (November 2006), ISSN: 0098-558El Emam, K., Benlarbi, S., Goel, N., Rai, S., Comparing Case-based Reasoning Classifiers for Predicting High Risk Software Components (2001) Journal of Systems and Software, 55 (3), pp. 301-320General Terms and Definitions Concerning Standardization and Related Activities (1993), EN 45020, CEN, BrusselsFenton, N., Ohlsson, N., Software Metrics and Risk (1999) Proc. of The 2nd European Software Measurement Conference (FESMA 99)Halstead, M., (1977) Elements of Software Science, , Elsevier Science Inc, New YorkHosmer, D., Lemeshow, S., (1989) Applied Logistic Regression, , John Wiley \& Sons, Chicester(1998), Health & Safety Commission The use of computers in Safety-critical Applications. Technical Report, UKHudepohl, EMERALD: A Case Study in Enhancing Software Reliability (1998) Proc. of IEEE Eight Int. Symposium on Software Reliability Engineering - ISSRE98, 98, pp. 85-91ISO/IEC 12119. International Organization For Standardization ISO/IEC 12119, Information Technology - Software packages - Quality requirements and testing, p. 16, Geneve (1994)International Organization For Standardization ISO/IEC 14598-1 Information Technology - Software product evaluation - Part 1: General OverviewGeneve ISO (1999), ISO/IEC 14598-1International Organization For Standardization ISO/IEC 9126-1, Software Engineering - Software product quality - Part 1: Quality ModelGeneve ISO (2001), ISO/IEC 9126-1Software Engineering - Requirements for quality of Commercial Off-The-Shelf (COTS) software product and instructions for testing, Final Draft (2006), ISO/IEC 25051 International StandardIyer, R., Experimental Evaluation (1995) Special Issue FTCS-25 Silver Jubilee, 25th IEEE Symposium on Fault Tolerant Computing, pp. 115-132Jacobson, I., Griss, M., Jonsson, P., (1997) Software Reuse: Architecture, Process and Organization for Business Success, , Addison-Wesley, LongmanKhoshgoftaar, Process Measures for Predicting Software Quality (1997) Proc of High Assurance System Engineering Workshop - HASE'97Leveson, N., (1995) Safeware, System Safety and Computers, , Addison-Wesley Publishing Company, ReadingThe linux kernel. Accessed on Feb/06 (2006), http://www.kernel.orgLyu, M., Chen, J., Avizienis, A., Experience in Metrics and Measurements for N-Version Programming. Int (1994) Journal of Reliability, Quality and Safety Engineering, 1 (1), pp. 41-62Lyu, M., Handbook of Software Reliability Engineering (1996) IEEE omputer Society Press, , McGraw-Hill, Los AlamitosMadeira, H., Vieira, M., Costa, D., On the Emulation of Software Faults by Software Fault Injection (2000) Proc. of The Int. Conf. on Dependable Systems and Networks, , NY, USAMenzies, T., Greenwald, J., Frank, A., Learning Defect Predictors. Journal (submitted, 2006), , http://menzies.us, accessed February/2006Moraes, R., Durães, J., Martins, E., Madeira, H., A field data study on the use of software metrics to define representative fault distribution (2006) Proc. of The International Conference on Dependable Systems & Networks - DSN2006, , IEEE Computer Society Pres, Los AlamitosMoraes, R., Durães, J., Barbosa, R., Martins, E., Madeira, H.: Experimental Risk Assessment and Comparison using Software Fault Injection. In: The International Conference on Dependable Systems and Networks - DSN 07, Edimburgo (2007)Morris, J., Lee, G., Parker, K., Bundell, G., Lam, C., Software Component Certification (2001) IEEE Computer, 34 (9), pp. 30-36Musa, J., (1996) Software Reliability Engineering, , McGraw-Hill, New YorkMunson, J., Khoshgoftaar, T., Software Metrics for Reliability Assessment (1995) Handbook of Software Reliability Engineering, IEEE Comp, , Michael, R, ed, Society Press, Los AlamitosKitchenham, B., Pfleeger, S., Fenton, N., Towards a framework for software measurement validation (1995) IEEE Transactions on Software Engineering, 21 (12), pp. 929-944Rodriguez-Dapena, P., Software Safety Certification: A Multidomain Problem (1999) IEEE Software, 16 (4), pp. 31-38Methodology for Software Reliability Prediction and Assessment (1992), Rome Laboratory RL, Technical Report RL-TR-92-52, 1 and 2Rosenberg, L., Stapko, R., Gallo, A., Risk-based Object Oriented Testing (2000) Proc of. 13th International Software / Internet Quality Week-QW, p. 2. , San Francisco, California, USAhttp:// msquaredtechnologies.com/m2rsm/rsm.htm, Resource Standard Metrics, Version 6.12005, Last access(2006) Real-Time Operating System for Multiprocessor Systems, , http://www.rtems.com, February, accessedRushby, John Modular Certification. Langley Research Center. Report Number: NAS 1.26212130 NASA CR-2002-212130, SRI-11003Sherer, S., A Cost-Effective Approach to Testing (1991) IEEE Software, 8 (2), pp. 34-40Singpurwalla, N., (1999) Statistical Methods in Software Engineering: Reliability and Risk, , 1st edn. Springer, HeidelbergStafford, J., Wallnau, K., Is Third-Party Certification Necessary? (2001) Proceedings of the 4th ICSE Workshop on Component-Based Software Engineering, Toronto, Canada, May, Toronto, Canada, pp. 13-17Systems Integration Requirements Task Group Certification Considerations for Highly-Integrated or Complex Aircraft Systems, Technical Report AS-1C, ASD, SAE 1996Tang, M., Kao, M., Chen, M., An Empirical Study on Object-Oriented Metrics (1999) Proceedings of the Sixth International Software Metrics Symposium, pp. 242-249(2000) Technical Assessment Guide T/AST/046, , Nuclear Safety Directorate Computer Based Safety Systems, UK(2006), http://www.testwell.fi, Testwell Oy Ltd. Accessed on March/06Voas, J., Certifying Off-the-Shelf Software Components (1998) IEEE Computer, 31 (6), pp. 53-59Voas, J., Certifying Software for High-Assurance Environments (1999) IEEE Software, 16 (4), pp. 48-54Weyuker, E., Testing Component-Based Software: A Cautionary Tale (1998) IEEE SoftwareYang, Y., Boehm, B., Clark, B., Assessing COTS Integration Risk Using Cost Estimation Inputs (2006) Proc. of 28th International Conference on Software Engineering, , Shangai, China