The idea of creating a general 'framework' for evaluating translation tools came into being as the result of evaluators having to design their own evaluations from scratch. Over the years, a great deal of literature has focused on the purposes, criteria and measurements in machine translation evaluation. Each time an evaluation was required, the evaluation literature had to be extensively searched to find suitable evaluation criteria, measurement and methods. Hence the non-existence of a standard evaluation method available for use and/or adaptation to all evaluators has been viewed as a serious problem (Hovy, King and Popescu-Belis 2002b: 44). As a result, a number of projects have been initiated to establish some form of general framework, method or set of guidelines to evaluate natural-language processing tools.

Most of the frameworks described here incorporate software standards from ISO as the foundation for a general evaluation framework. ISO is the world's largest developer of standards with the aim of ensuring that the development, manufacture and supply of products and services are efficient, safe and environment-friendly. In ISO, the quality and evaluation of natural-language processing tools is regulated by two series of standards, ISO 9126 (Software Product Quality) and ISO 14598 (Software Product Evaluation). Machine translation and computer-aided translation tools fall under the large umbrella of natural-language processing tools and, therefore, are required to achieve the standards set by these series. In order to fulfil the standard requirement of software quality, most evaluation frameworks existing today are based on these standards. Each series is briefly described below.

The ISO 9126 series (Software Product Quality) provides definitions of six key characteristics used in evaluating the quality of software products (see Trujillo 1999; Hovy, King and Popescu-Belis 2002a and 2002b):

• Functionality: meeting stated or implied needs of an end-user when functions of the system operate under specific conditions.

• Reliability: maintaining the level of performance by the system when operating under specific conditions.

• Usability: the ease of operating, understanding and learning each task of the system as a whole.

• Efficiency: the performance of the system in relation to the amount of resources available.

• Maintainability: the capability of the system to undergo modifications such as corrections, improvement and adaptations for different requirements and working environments.

• Portability: the ability to transfer the system from one environment to another such as to different operating systems.

The ISO 14598 series (Software Product Evaluation) provides guidance on the practical implementation of software evaluation that takes into consideration different points of view. It can also be used in conjunction with the six key characteristics described in ISO 9126. The evaluation process may include the evaluation of specific components within a system or of the entire system, a process broken up into five stages as shown in Figure 1.