INTEGRATED METHOD OF EXTRACTION, FORMALIZATION AND AGGREGATION OF COMPETITIVE AGENTS EXPERT EVALUATIONS IN A GROUP

Context. The problem of extraction, formalization and aggregation of expert evaluations performed during selection of the best solution from possible alternatives set was considered. The problem actuality is defined by different application areas, additional analysis necessity and group evaluation under uncertainty. Objective. The research objective was to raise quality of the decisions made by group of experts due to increase of individual expert evaluation process efficiency and improvement of evaluation aggregation process. Method. The integrated method which consists of individual and group evaluation was proposed for the problem solution. The modified method of extraction and formalization of individual expert evaluations which is based on the analytic hierarchy process modification and includes absolute and relative evaluation phases was proposed. It evaluates expert competence based on confidence coefficient for expert judgments. The modified method of agent group evaluation based on summation of individual expert evaluations of every alternative was proposed. It makes total evaluation based on relative quantitative importance of the agents and confidence coefficients for judgments of every group participant. It gives preference to judgments of qualified experts to increase solution quality. Results. The experimental investigation of the proposed methods confirmed availability of the developed mathematical support. The proposed methods are able to detect and control nontrivial issues in tasks which are solved. Conclusions. Scientific novelty of the paper consists in the proposed integrated method of extraction, formalization and aggregation of expert evaluations in group which enables to define integrated assessment of competitive agents as well as to evaluate confidence coefficient for expert judgments directly during individual evaluation process and to use it in the following group decision-making phase. Practical significance of the paper results consists in the developed information technology which made it possible to put the method into practice for solving of the tender support tasks and experiment results.


NOMENCLATURE
A -pairwise comparison matrix which elements are criteria or agents; Ag -set of competitive agents; a rp -result of comparison of the r object with the p object according to general criterion; B -pairwise comparison matrix of competitive agents according to criteria; ij b -evaluation of the i-th agent according to the j-th criterion; C -criterion set; D -set of competitive agents evaluations; best d -agent which is considered most preferable; d i -part of the i-th agent in the total estimate; f 1 -function which extracts element with the largest key value; f 2 -functional dependence; g gw -global priority vector for the g-th expert; best g gw -agent which is considered most preferable by the g-th expert; Q -set of "control" questions; k q -comparative evaluation of random agent pair from the k matrix; S -set of experts; s g -the g-th expert from the set S; Sum i -total evaluation of the i-th competitive agent; g v -confidence coefficient for judgments of the expert s g ; ) (k v g -confidence coefficient for judgment of the g-th expert on the k-th question; W -local priority vector; r w -the r-th element of local priority vector W .

INTRODUCTION
The problems of selection of one solution from the set of possible variants are known in different activity areas. These problems are often characterized by the fact that the choice has significant impact on problem-solving environment state. At the same time most commonly it isn't possible to select the best solution from the set of alternatives in such problems without additional analysis of all possible variants as well as factors which impact on solution quality.
As factors which are evaluation criteria can be not only quantitative but also qualitative, and evaluation is executed under uncertainty, automatic decision-making isn't possible.
Hence expert knowledge are used for selection of the best variant from the set of alternatives.
To raise objectivity of evaluation procedure it should be done by more than one expert. However one part of experts in group may turn out to be more competent than others in the problem which is solved, so judgments of this part of expert group should be preferred during aggregation of group evaluation.
So the development of integrated method which enables to extract and aggregate expert evaluations taking into account competence of every expert is actual.
Object of the research is process of formalization and aggregation of expert evaluations. Subject of the research is methods of evaluation performed during selection of solutions from the set of alternatives (which are considered as competitive agents).
The research objective is to raise quality of the decisions made by group of experts due to increase of individual expert evaluation process efficiency and improvement of evaluation aggregation process.
The following tasks were defined to achieve the research objective: -improvement of the method of extraction and formalization of expert judgments; -development of the method of expert competence evaluation during individual evaluation process; -improvement of the method of group evaluation based on individual expert evaluations taking into account expert competence; -practical investigation of the proposed methods efficiency using the developed information technology.
1 PROBLEM STATEMENT Suppose we have the set Ag of competitive agents, the set S of experts who can estimate these agents and the set C of the evaluation criteria.
Then the problem of competitive agents selection is to define estimates } ,..., Method of determination of the dependence should be proposed in the paper for competitive agents selection problem solving.
It should be noted that the given problem includes also competitive agent selection problem which consists in selection of competitive agent with the biggest part in total evaluation

REVIEW OF THE LITERATURE
When decision is made by expert group, collaboration using open dynamic discussion has substantial disadvantages which can impact on decision, making it subjective [1][2][3][4]. Therefore in some circumstances it is more efficient to question experts individually, permitting them to use only personal experience making judgments [5][6]. At the same time expert should be proposed to place all possible solutions in descending order of its correspondence to the task as well as to estimate solutions quantitatively determining this correspondence level or relative priority of each alternative comparing it with another one. To obtain such estimations there is a lot of traditional methods: the Schulze method [7], the method of preference [8], the analytic hierarchy process [9], fuzzy methods and the COPRAS method [10]. But the described methods have substantial disadvantages which are critical for the problem which is solved. So the Schulze method and the method of preference make evaluations only by one criterion and limit scale of correspondence to the task by natural numbers. The analytic hierarchy process is sufficiently laborious and doesn't enable to take into account correspondence level of considered variants to the problem objective. Fuzzy methods may be unefficient for practical tasks, because these methods can make inaccuracy for similar alternative evaluations during transformation of triangular numbers to real ones. The COPRAS method solves tasks of multicriterion alternatives evaluation based on its correspondence to the task objective, but this method doesn't enable to detect its priorities comparing with each other. The methods [11][12][13][14][15][16][17][18][19] enable to make evaluations according to several criteria based on association rules synthesis [11], decision trees [12], radial basis neural networks [13], production rules [14], neuro-fuzzy networks [15], cascade systems [16] and sequential [11,17] or parallel [18,19] computing. However these methods are applicable only when training data samples, which contain collected information about objects and processes under consideration, are available, that makes difficult and in some circumstances eliminates possibility of its applicability for solving of tasks which need expert evaluation.
As analysis result it should be noted that regarding the problem which is considered decomposition approach of the analytic hierarchy process enables to develop the most appropriate problem model [20], since experts can use quantitative and qualitative criteria for evaluation of agents using qualitative criteria based on clear scale.
Thus disadvantages of existing expert evaluation methods cause necessity of improvement of methods which conduct individual evaluation of competitive agents by expert group and further evaluations aggregation taking into account competency of every expert.

MATERIALS AND METHODS
Currently only one approach is usually used for extraction and formalization of expert judgments at a time. Possible approaches include the following ones: expert can estimate agent as consistent entity based on its correspondence to the given problem or expert can compare all agents with each other creating relative rating. Both these approaches give only one-track evaluation. For integrated evaluation of alternatives as competitive agents within the scope of the current problem it was proposed to make evaluation using two approaches [21][22]: -absolute evaluation as determination of competitive agents correspondence to the customization demands for each criterion; -relative evaluation based on comparison of agents with each other according to each criterion.
At the beginning of the proposed method of extraction and formalization of individual expert evaluations level of correspondence of competitive agents to the problem statement according to each criterion is evaluated (absolute evaluation). This phase is made by absolute percentage evaluation (from 0 to 100%) of each agent according to each criterion comparing to the problem statement requirements. After evaluation obtained values has to be normalized using division by 100. The result of this stage is the matrix B of the size KNxKM which consists of evaluations ] ... . Then the modified analytic hierarchy process is used to evaluate agents comparatively according to each criterion (relative evaluation).
Relative evaluation process is generally represented on activity diagram ( fig. 1). The method of paired comparisons [9] is used to compare criteria according to the given task and to compare agents according to criteria. According to this method pairwise comparison matrices (PCM) are obtained from the results of comparison of every row element with every column element.
So the matrix A of size KAxKA is obtained from elements a rp which are presented by real numbers: Every expert that takes part in the evaluation process has to compare all elements with each other presenting evaluations as numeric values in accordance with fundamental ratio scale [23] and has to complete corresponding PCM.
One of the main disadvantages of the method of paired comparisons is its laboriousness. It is necessary to analyze fundamental properties of PCM [9] to reduce the number of comparison operations which have to be executed by expert.
The main property of PCM is the following: for each r, p=1,...,KA elements a rp >0. Besides that under r=p a rp =1, because these elements present comparisons of each element with oneself and have to correspond to equal significance.
The third property is inverse symmetry of the matrix about the main diagonal. For each r, p=1,...,KA a pr =1/a rp , so expert has to complete only evaluations which are situated above the main diagonal.
Thus the PCM A will take the following form: for each g-th expert, local priorities of each evaluated agent are multiplied by priority of the corresponding criterion and results are added to each other.
Then the corresponding values of the global priority gi gw of every agent are compared with each other. Agent which is considered most preferable by the g-th expert will have the biggest value of the global priority: The tasks which are solved in the context of the considered problem are seldom completely identical, because every task has its own features and differences: for example technical or time constraints. Application of the analytic hierarchy process demands familiarity with these features, constraints and its influence on the final result from experts. However it isn't always possible because of the wide specialization of experts who take part in questioning in practice.
In this case it is necessary to evaluate additionally expert competence and confidence in decisions made by him [24]. Expert competence level should be then used for aggregation of group evaluation [25].
Confidence coefficient g v corresponds to quantification of the level of confidence in competence of expert S s g ∈ ( KS g ,..., 1 = ) in the presented problem. It is proposed to add set of «control» questions Q to pairwise comparison sequence to evaluate confidence coefficients for expert judgments. Each element of the set Q is defined as is comparative evaluation of random pair of agents from the bottom part of the k-th PCM where agents are compared according to criteria. So the set Q is defined in the following form: «Control» question is a question which was restated and repeated to expert. Such questions enable to determine confidence of expert answering main questions in agents evaluation process.
Since all elements of the matrix A are inverse symmetrical that is for each r, p=1,..., KA a pr =1/a rp , it is possible to verify if expert judgments which were gotten comparing random pair of agents are equal to expert judgments which were gotten during primary comparisons. This verification is based on evaluation of PCM elements which are situated under the main diagonal as inverse proportion to the values of elements which are situated in the matrix part which was filled by expert.
For each k-th PCM random pair of agents p and r is selected from the elements of the matrix which are situated under the main diagonal and is proposed to expert for comparison.
If values rp a and pr a are bigger or lower than 1 at the same time, then confidence coefficient for the given question should be specified as 0 ) ( = k v g (the g-th expert showed his incompetence answering the k-th question). Otherwise confidence coefficient for the given question should be calculated using the following expression:  After all experts of the group made own evaluations, global priority vector gw g and confidence coefficient for expert judgments g v are defined for each expert from number KS, the method of group agent evaluation has to be executed. This method calculates total evaluation Sum i for each i-th agent based on summation of evaluations gi gw which were given to the agent by the g-th expert. At the same time evaluations has to be multiplied by confidence coefficients for expert judgments: Thus the bigger value confidence coefficient g v for expert judgments has, more impact g-th expert will have on overall evaluation.
After summation is done, overall rating of competitive agents D is defined. Each individual evaluation is multiplied by confidence coefficient for expert judgments and part d i of each i-th agent in the total evaluation is calculated using the following formula: Agents are ordered by decrease of d i values, in that way defining function 1 f which can be implemented as binary heap. The result of this operation is agent rating, where agent which is considered most preferable by expert group heads the list. Based on the obtained rating responsible person makes final decision on the best agent. In most cases it is agent which is the first in the list and it becomes solution of the competitive agents selection problem.

EXPERIMENTS
The proposed methods were used for the development of the competitive agents selection information technology.
The tasks of competitive agents selection were solved for experimental investigation of practical availability of the proposed methods using the developed information technology.
The investigation was made based on data collected from tender tasks, participant offers and tender results accumulated by design companies LLC "SK Inzhenernye systemy" and "VST" (Ukraine) and evaluations of experts who were selected and invited based on the tender tasks.
The following competitive agents (companies) were considered in the task of tender for design, supply and mounting of metalware and barriers: -АО "Metallist SMK", Kharkiv; -Atlas Word (Germany), branch in Kyiv; -Borga Hale, Gdynia, Poland.
The following competitive agents were considered in the task of supply of materials and equipment for mounting of shopping center buildings: -Master Profi, Dnipro; -Remko, Kyiv; -Atlas Word, Kyiv; -Askon, Kamyanske; -IBT, Kyiv. Design companies traditionally used pricing appraisal or oral discussion method for evaluation during tender process. However since tender results can be determined in long time horizon [26] it is necessary from one to six months after tender is ended for its efficiency evaluation. During experimental investigation the results of traditional approach were compared with the results obtained by the proposed integrated method.

RESULTS
Solving the task of tender for design, supply and mounting of metalware and barriers the derived proposals were formalized as competitive agents and were proposed to the group of three experts for evaluation. Expert evaluations were used for agents rating aggregation.
The rating of competitive agents obtained by the traditional approach (without taking into account confidence coefficient for expert judgments) can be summarized in the following list ( fig. 2a): -Borga Hale -35,60%; a b Figure 2 -The results of competitive agents evaluation in the task of tender for design, supply and mounting of metalware and barriers: a -without taking into account confidence coefficient for expert judgments; b -taking into account confidence coefficient for expert judgments -Atlas Word -35,35%; -АО "Metallist SMK" -29,05%. The rating of competitive agents obtained using the modified methods (taking into account confidence coefficient for expert judgments) can be summarized in the following list ( fig. 2b): -Atlas Word -35,60%; -Borga Hale -35,50%; -АО "Metallist SMK" -28,9%. Solving the task of supply of materials and equipment for mounting of shopping center buildings the derived proposals were formalized as competitive agents and were proposed to the group of four experts for evaluation.
Before the beginning of evaluation process experts were proposed to examine tender task using the developed information technology. Then traditional approach based on oral discussion was used for evaluation. Based on oral discussion and simple voting methods expert commission elected the tender offer of "Master Profi" because of its minimum cost and delivery time.
Then agents evaluation was repeated by the same expert commission using competitive agents information technology based on the proposed methods ( fig. 3). Besides that individual expert preferences and confidence coefficients for expert judgments were defined ( fig. 4).

DISCUSSION
Results of competitive agents evaluation in the task of tender for design, supply and mounting of metalware and barriers showed that taking into account confidence coefficient for expert judgments changed positions of participants in rating. It should be noted that the bigger the dispersion of confidence coefficients for expert judgments is, more impact experts with bigger values of confidence coefficient will have on tender results.
As the result of the proposed integrated method application to the tender for supply of materials and equipment for mounting of shopping center buildings the tender offer of the company «Master Profi» which was chosen during oral discussion occupied only the third place in the rating. It is caused by its failures which were missed during basic analysis: delayed starting date of execution phase and small production volume regarding competitors.
Thus investigation results confirm availability of the developed competitive agents selection information technology in whole and the proposed integrated method particularly for practical tasks solving since the proposed problem solution helps to detect nontrivial problems and failures of the considered competitive agents.
Analysis of the obtained results showed that the proposed integrated method of extraction, formalization and aggregation of expert evaluations enabled to raise accuracy of group evaluations due to formalization of individual evaluations derivation process as well as implementation of expert competence evaluation mechanism. In turn it ensures group decision making process efficiency and enables to raise quality of the final solution.
The proposed integrated method can be used for practical expert evaluation tasks solving in many areas of industry, economics and medicine: tender processes in all areas of state and business activity, complicated clinical decision support, expert evaluation of techware quality in industry.
In what follows it is planned to investigate availability of the developed method for solving of practical tasks in other activity areas and the dependence of the method efficiency on application domain.

CONCLUSIONS
In this paper actual problem of competitive agents selection was solved based on group expert evaluations.
Scientific novelty of the paper consists in the proposed integrated method of extraction, formalization and aggregation of expert evaluations in group which enables to define integrated assessment of competitive agents as well as to evaluate confidence coefficient for expert judgments directly during individual evaluation process and to use it in the following group decision-making phase. The proposed method gives preference to judgments of competent experts in the overall evaluation, at the same time increasing quality of the decision.
Practical significance of the paper results consists in the developed information technology and experiment results. Information technology implemented the proposed integrated method that made it possible to put the method into practice for solving of the tender support tasks. Experiments confirmed availability of the developed mathematical support and made it possible to recommend it for practical application.