The evaluation requirement stage of the process shown in Figure 1 is where criteria are identified for evaluation purposes. The criteria are for measuring the capability of a specific machine translation system in which its strengths and weaknesses are discovered. It means that an evaluation is performed either on selected components of a system or an entire system. A list of criteria includes adequacy to evaluate user-friendliness, efficiency to evaluate the consumption of resources, and linguistic quality to evaluate the coverage of the lexicon, syntax and semantics of the source and target languages of the system (Vertan and Hahn 2003). Since it is not easy to measure quality, the characteristics suggested in the ISO 9126 series are used as a quality measurement.

In the specifications stage of the evaluation, the type of measurement used is often dictated by pre-determined criteria and quality characteristics such as scale-based scores, to measure items that include word or sentence errors in the target-language text (Tomás, Ảngel Mas and Casacuberta 2003). 

Figure 1 Example of an evaluation process

Based on the description of the components undergoing evaluation, plans are scheduled and methods are selected in the evaluation design stage that include the length of time to complete an evaluation, the order of components being evaluated and appropriate methods such as test suite or black-box. All these will be determined by whether the evaluation is for end-users or researchers. Once the evaluation design is agreed upon, the actual task of evaluating can take place. Measurement refers to the scores that are obtained from the metrics defined in the second stage. Rating concerns the interpretation of the scores, and assessment is the written summary of the ratings (see Tomás, Àngel Mas and Casacuberta 2003). Upon completion of the evaluation, the reports of the findings are written up. In addition, a detailed description of the problems occurring in the system during the evaluation is also given. In some cases, the whole evaluation process is reviewed to ensure that the initial objectives of the evaluation are met and the problems encountered are confirmed (see Vertan and Hahn 2003).

As I have mentioned previously, ISO evaluation standards were used as the basis for producing this kind of standard for machine translation evaluation. A number of related projects were involved in this process over time; these are all described below. The sequence of development started with EAGLES, in which ISO standards were adapted to the translation environment; the resulting new standards were later carried forward by the International Standards for Language Engineering (ISLE) and the Test-bed Study of Evaluation Methodologies: Authoring Aids (TEMAA). Another initiative was the Framework for the Evaluation of Machine Translation (FEMTI), an extension of ISLE. 

责任编辑：admin