MULTIMODAL DATA PROCESSING BASED ON ALGEBRAIC SYSTEM OF AGGREGATES RELATIONS

Context. In many engineering tasks where the monitoring of changes in the characteristics of an observation object, subject, or process is required, it is necessary to process multimodal data recorded with respect to time moments when these characteristics are registered. In this paper, the author presents a new approach to solving the problem of multimodal data structures timewise processing, which allows to simplify the processing of such data by using the mathematical apparatus of an algebraic system of aggregates and thereby reduce the requirements to computing resources. The algebraic system of aggregates operates with such specific data structures as aggregates and multi-images. These complex data structures can be obtained as a result of data measuring, generating, recording, etc. The processing of such multimodal data can also require discrete intervals processing. Objective. The goal of the work is to formalise the relations between basic mathematical objects defined in the algebraic system of aggregates, such as elements, tuples and aggregates, as well as the data structures based on these mathematical objects, namely, discreet intervals and multi-images. Method. The research presented in this paper is based on both the algebraic system of aggregates and the concept of multi-image which enable multimodal data timewise processing. A carrier of the algebraic system of aggregates is an arbitrary set of specific structures – aggregates. An aggregate is a tuple of arbitrary tuples, elements of which belong to predefined sets. Aggregates can be processed by using logical, ordering, and arithmetical operations defined in the algebraic system of aggregates. A multi-image is a non-empty aggregate, the first tuple of which is a tuple of time values. Such tuple of time values represents a certain discrete interval. To process discrete intervals and multi-images, a set of relations is defined in the algebraic system of aggregates. This set includes relations between tuple elements, relations between tuples, and relations between aggregates. The relations between tuples enable arithmetical comparison, frequency comparison, and interval comparison. This mathematical apparatus can be used for both complex representation of object (process) multimodal characteristics and further timewise processing of data represented as multi-images. Results. The approach to discrete intervals and multi-images processing based on relations, which are defined in the algebraic system of aggregates, has been developed and presented in the paper. The author provides examples of the developed approach practical implementation. Conclusions. The results obtained in the research presented in this paper has shown that the relations defined in algebraic system of aggregates enable processing of complex data structures named multi-images in data modelling, prediction and other tasks. To allow data processing with respect to time scale, discrete intervals can be employed. A discrete interval is a tuple of time values. In the paper, the author shows how relations for discrete intervals comparison can be used for solving practical tasks. Besides, the author presents the software tools which can be used for practical implementation of the given theoretical approach by employing the domain-specific language ASAMPL.


ABBREVIATIONS
ASA is the algebraic system of aggregates; ASAMPL is the programming language for mulsemedia data processing based on algebraic system of aggregates.

NOMENCLATURE
A is an aggregate; is a non-empty set (carrier); is a set of operations; is a set of relations; i, i 1 , i 2 , j, k, q, p are indices; a , 1 a , 2 a , j a , t a , p a are tuples of arbitrary values; Nowadays, there is a wide range of tasks in engineering, health care, education, and other fields [1][2][3], where data are complex structures, values of which are defined, measured, generated, recorded in terms of time. Since such data can be obtained in arbitrary time moments as well as they are a subject of digital processing in computer systems, time readings defining moments when certain data values are obtained are digital values which belong to discrete intervals.
Complex data in this context can be presented as a multi-image [4][5][6]. A multi-image is a complex representation of multiple data sets describing an object (subject, process) of observation which are obtained (measured, generated, recorded) in the course of time. In mathematical sense, the multi-image is an aggregate, the first data tuple of which is a non-empty tuple of time values. These values can be natural numbers or values of any other type which can be used for evident and monosemantic representation of time. The advantage of multi-image use is that since we have a complex representation of multimodal data sequences defined in terms of time, it gives new opportunities for data modelling, prediction and other similar tasks. To process multi-images, we need to operate with relations between them and their components. Since a multi-image is an aggregate, relations defined in the Algebraic System of Aggregates (ASA) are used. In this paper we present the relations of ASA and propose a general approach for their use.
The object of study is the process of multimodal data processing with respect to time stamps of data values.
The subject of study is relations between components of complex data structures, namely, between discrete intervals and between multi-images.
The purpose of the work is to formalise logical apparatus of the algebraic system of aggregates and elaborate an approach to its practical implementation.  2] be discrete intervals. Then the problem is to establish relations between these discrete intervals which enable their arithmetical, frequency, and interval comparison which can be used for multi-image logical processing.

REVIEW OF THE LITERATURE
The foundations of interval algebra and interval-based temporal logic were presented in [7] where Allen proposed 13 relations between intervals. Allen and Hayes [8] extended Allen's interval-based theory by formally defining the beginnings and endings of intervals which have properties normally associated with points.
Nebel and Bürckert, in [9], introduced a new subclass of Allen's interval algebra called ORD-Horn subclass. The authors proved that reasoning in the ORD-Horn subclass is a polynomial-time problem and showed that the path-consistency method is sufficient for deciding satisfiability. Allen and Ferguson, in [10], presented a representation of events and actions based on interval temporal logic. One of important features of the logic is that it can express complex temporal relations because of its underlying temporal logic.
In [11], Schockaert, De Cock, and Kerre formulated a notion of a fuzzy time interval and proposed fuzzy Allen relations which are the generalization of Allen's interval relations. The authors applied the relatedness measures to define fuzzy temporal relations between vague events.
In [12], Bozzelli et al. studied the expressiveness of Halpern and Shoham's interval temporal logic which is "interval-wise" interpreted and enables expressing properties of computation stretches, spanning a sequence of states, or properties involving temporal aggregations, which are inherently "interval-based". Grüninger and Li, in [13], identified the first-order ontology that is logically synonymous with Allen's interval algebra, so that there is a one-to-one correspondence between models of the ontology and solutions to temporal constraints that are specified using the temporal relations.
These and other similar researches consider time as intervals and moments as well as such time values considered as single data, without structuring with data of other types. Thus, in our research we work on another approach which stipulates complex representation of multimodal data as aggregates and multi-images.

MATERIALS AND METHODS
ASA is an algebraic system, a carrier of which is an arbitrary set of specific structures -aggregates [4,5]. Definition 1. An aggregate A is a tuple of arbitrary tuples, elements of which belong to predefined sets: where {A} is a tuple of sets M j , A is a tuple of elements tuples Since ASA is an algebraic system [14], it consists of sets , where is a non-empty set (carrier), elements of which are elements of the system; is a set of operations; is a set of relations. The carrier of ASA is an arbitrary set of specific structures called aggregates.
Operations on aggregates include logical operations, ordering operations, and arithmetical operations.
The logical operations on aggregates are: Union, Intersection, Difference, Symmetric Difference, and Exclusive Intersection [4].
The basic relations in ASA [4,5] includes Is Equal, Is Less, Is Greater, Is Equivalent, Includes, Is Included, Precedes, Succeeds. Let us present the whole set of the relations in detail.
Relations in ASA include: -Relations between tuple elements; -Relations between tuples; -Relations between aggregates.
Relations between tuple elements are Is Greater (>), Is Less (<), Is Equal (=), Proceeds ( ), Succeeds ( ). The first three relations (<, >, and =) are based on elements value and the last two relations ( and ) concern elements position in a tuple. Naturally, elements must belong to the same tuple.
Then we can establish the fact of the following relations between the tuple elements: Relations between tuples enable the following types of tuples comparison: The relation Is Strictly Greater between two tuples 1 a and 2 a is defined as follows: The relation Is Majority-Vote Greater between two tuples 1 a and 2 a can be defined as follows.
, if , n n n n n n n The relation Is Strictly Less between two tuples 1 a and 2 a is defined as follows: The relation Is Majority-Vote Less between two tuples 1 a and 2 a can be defined as follows.
The relation Is Strictly Equal between two tuples 1 a and 2 a is defined as follows: The relation Is Majority-Vote Equal between two tuples 1 a and 2 a can be defined as follows.
Let us consider the following tuples:  Then we can establish the fact of the following arithmetical relations between these tuples: Frequency comparison is based on the following relations: -Is Thicker ( ); -Is Rarer ( ); -Is Equally Frequent ( ).
The relation Is Thicker between two tuples 1 a and 2 a is defined as follows: The relation Is Rarer between two tuples 1 a and 2 a is defined as follows: The relation Is Equally Frequent between two tuples 1 a and 2 a is defined as follows: Thus, if there are three tuples: ; ;ã a a a a a .
To define how much thicker or how much rarer is a certain tuple in comparison with another tuple, we introduce a frequency measure which can be calculated as follows: For example, for tuples 1 a , 2 a and 3 a given above: Interval comparison can be applied to two tuples 1 a and 2 a , where  [6] is based on relations of Allen's Interval Algebra. However, it has a significant difference from Allen's Interval Algebra in that it operates with discrete intervals. Let us define this notion and introduce the compact notation for relations between discrete intervals. Definition 2. A discrete interval is a tuple, elements of which are unique values ordered either in ascending or in descending order.
The relation Is Before between two discrete intervals 1 a and 2 a is defined as follows: The relation Is After between two discrete intervals 1 a and 2 a is defined as follows: The relation Coincides With between two discrete intervals 1 a and 2 a is defined as follows: Note that this relation does not fully correspond to relation Equal of Allen's Interval Algebra because in ASA we deal with discrete intervals, thus, two discrete intervals can coincide in the first and last values, but other values can be unequal. For example, if we have two tuples 2,3,8,10 and 2,5, 6,10 , their discrete intervals coincide but are unequal.
The relation Meets between two discrete intervals 1 a and 2 a is defined as follows: The relation Is Met By between two discrete intervals 1 a and 2 a is defined as follows: The relation Overlaps between two discrete intervals 1 a and 2 a is defined as follows: The relation Contains between two discrete intervals 1 a and 2 a is defined as follows: The relation Starts between two discrete intervals 1 a and 2 a is defined as follows: The relation Is Started By between two discrete intervals 1 a and 2 a is defined as follows: The relation Finishes between two discrete intervals 1 a and 2 a is defined as follows: The relation Is Finished By between two discrete intervals 1 a and 2 a is defined as follows: (24) Relations between aggregates consist of relations between sets of aggregates and relations between tuples of aggregates.
Relations between sets of aggregates are Is Equivalent ( ), Includes ( ), Is Included ( ).
The result of these relations depends of the aggregates compatibility [4,5]. Let us consider aggregates A 1 , A 2 , A 3 , A 4 , A 5 , and A 6 defined according to (1): Thus, compatibility of the aggregates in (25) is as follows: Then relations between sets of these aggregates are: The relation Is Equivalent between two aggregates A 1 and A 2 can be defined as: The relation Includes between two aggregates A 1 and A 2 can be defined as: The relation Is Included between two aggregates A 1 and A 2 can be defined as: Relations between tuples of aggregates are identical to relations between single tuples defined above relations of three types: -Arithmetical relations; -Frequency relations; -Interval relations. However, possibility of their application depends of the aggregates compatibility: relations between tuples can be established only for compatible and quasi-compatible aggregates.
Hiddenly compatible aggregates must be first transformed to compatible [5] and then a relation between tuples can be considered.
Let us also note that if tuples to be a subject of interval comparison are not discrete intervals (see Definition 2), at first, they must be sorted by using operator Sorting [5] and next they can be compared.
Relations between tuples of aggregates can be established with evident indication of the tuples to be compared: If a certain relation is true for several tuples, it can be indicated in the following way: If a relation is established for all tuples, it can be declared as follows: 1 2 Let us give several examples. At first, we consider two aggregates A 1 , A 2 such as A 1 A 2 : The following relations can be established between these aggregates: Next, let us consider two aggregates A 3 , A 4 (A 3 A 4 ): These aggregates can be compared by using the following relations: In all examples given above we operate with integer elements, but any other data types can be handled in a similar way.
Besides both relations between aggregates and relations between their components, we consider relations between multi-images.
where T is a set of time values; , [1, , ] i n i N τ ≥ ∈ … . Since a multi-image, by definition, includes a tuple of time values as the first tuple, let us formulate the following lemma. Lemma 1. If I 1 and I 2 are multi-images, then I 1 I 2 .
Since compatibility is a special case of quasicompatibility, let us state Lemma 2 which follows from Lemma 1.
These lemmas allow us to conclude that all types of relations defined in ASA can be used for any set of multiimages.
Let us employ this theoretical background for solving practical tasks.

EXPERIMENTS
Let us consider the following discrete intervals:  Then, we can establish the fact of the following interval relations between these tuples ( Fig. 1 Now let us solve the task related to health care. There are two patients whose health status was being monitored during a month by using several digital sensors: thermometer, pulsometer, and sphygmomanometer. Four parameters, namely, temperature, pulse rate, systolic pressure and diastolic pressure values were being measured in the first patient and only two parameters (temperature and pulse rate) were being measured for the second patient.As a result of the monitoring, several data sequences have been obtained and composed as two multi-images: by one multi-image for each patient.
Our task is to compare these multi-images in order to let doctors conclude on comparative health status of two patients.
The data obtained from sensors belong to the following data sets: There is also T = [1, …, 31] which is a set of time values (days of a month).
Let the data collected from sensors during the monitoring process of the first patient's health status be as follows:  Then the obtained multi-image of the first patient's health status is: The relations (30), (31), and (32) give general comparation of multi-images and allow us to conclude that some of measurements in both aggregates belong to the same data sets and, therefore, they can be compared ({I 1 } ⊃ {I 2 }); data have been measured at the same pe- , where means that each tuple has been sorted in ascending order [5] before interval comparison). These relations are supposed to be applied to logical rules used in data analysis software which can be developed by employing a domain-specific programming language such as ASAMPL [6].

DISCUSSION
The proposed theoretical approach has been realized for multimodal data processing by using programming language ASAMPL. The experiments showed that the proposed approach of timewise aggregated data processing enables considerable decreasing of the code size (Fig. 2). To allow the work with program code in ASAMPL, the software tools for ASAMPL program code compilation and execution have been developed. They include the complier [15] and the integrated development environment (IDE) [16]. The developed ASAMPL compiler enables lexical analysis, parsing, and interpretation of the program code. The compiler is interconnected with the IDE. The developed ASAMPL IDE allows a programmer to develop code in programming language ASAMPL and debug it by analyzing the syntax errors. The developed IDE simplifies the work on a program code development by allowing the user to edit it in the full-fledged text editor with the functions of automatic code completion, color highlighting of key words, compiling and running developed programs. Fig. 3 shows the program code analysis and compilation process in ASAMPL IDE.

CONCLUSIONS
The Algebraic System of Aggregates provides the theoretical background for timewise multimodal data processing. In particular, it defines relations between tuple elements, tuples, and aggregates. This set of relations enables wide range of algorithms of processing the complex data structures such as multi-images. A multi-image is a complex representation of multiple data sets describing an object (subject, process) of observation which are obtained (measured, generated, recorded) in the course of time. Thus, the relations defined in ASA enable processing of complex data structures presented as multi-images in data modelling, prediction and other tasks.
To allow data processing with respect to time scale, discrete intervals can be employed. A discrete interval is a tuple of time values. In the paper, we show how relations for discrete intervals comparison can be used for solving practical tasks. Besides, we present the software tools which can be used for practical implementation of the given theoretical approach by employing the domainspecific language ASAMPL.
The scientific novelty of the obtained results consists in the development of a new mathematical approach to timewise multimodal data processing which differs from the theory of sets by both including the feature of ordering and introducing new relations between elements, tuples, and complex mathematical structures called aggregates.
The practical significance of the proposed approach consists in simplification of timewise multimodal data processing and minimisation of requirements to computing resources.