DOI: https://doi.org/10.15588/1607-3274-2020-1-17

### MULTIMODAL DATA PROCESSING BASED ON ALGEBRAIC SYSTEM OF AGGREGATES RELATIONS

Yevgeniya Sulema

#### Abstract

Context. In many engineering tasks where the monitoring of changes in the characteristics of an observation object, subject, or process is required, it is necessary to process multimodal data recorded with respect to time moments when these characteristics are registered. In this paper, the author presents a new approach to solving the problem of multimodal data structures timewise processing, which allows to simplify the processing of such data by using the mathematical apparatus of an algebraic system of aggregates and thereby reduce the requirements to computing resources. The algebraic system of aggregates operates with such specific data structures as aggregates and multi-images. These complex data structures can be obtained as a result of data measuring, generating, recording, etc. The processing of such multimodal data can also require discrete intervals processing.

Objective. The goal of the work is to formalise the relations between basic mathematical objects defined in the algebraic system of aggregates, such as elements, tuples and aggregates, as well as the data structures based on these mathematical objects, namely, discreet intervals and multi-images.

Method. The research presented in this paper is based on both the algebraic system of aggregates and the concept of multi-image which enable multimodal data timewise processing. A carrier of the algebraic system of aggregates is an arbitrary set of specific structures – aggregates. An aggregate is a tuple of arbitrary tuples, elements of which belong to predefined sets. Aggregates can be processed by using logical, ordering, and arithmetical operations defined in the algebraic system of aggregates. A multi-image is a non-empty aggregate, the first tuple of which is a tuple of time values. Such tuple of time values represents a certain discrete interval. To process discrete intervals and multi-images, a set of relations is defined in the algebraic system of aggregates. This set includes relations between tuple elements, relations between tuples, and relations between aggregates. The relations between tuples enable arithmetical comparison, frequency comparison, and interval comparison. This mathematical apparatus can be used for both complex representation of object (process) multimodal characteristics and further timewise processing of data represented as multi-images.

Results. The approach to discrete intervals and multi-images processing based on relations, which are defined in the algebraic system of aggregates, has been developed and presented in the paper. The author provides examples of the developed approach practical implementation.

Conclusions. The results obtained in the research presented in this paper has shown that the relations defined in algebraic system of aggregates enable processing of complex data structures named multi-images in data modelling, prediction and other tasks. To allow data processing with respect to time scale, discrete intervals can be employed. A discrete interval is a tuple of time values. In the paper, the author shows how relations for discrete intervals comparison can be used for solving practical tasks. Besides, the author presents the software tools which can be used for practical implementation of the given theoretical approach by employing the domain-specific language ASAMPL.

#### Keywords

Multimodal Data Processing, Aggregate, Multi-Image, Discrete Interval.

PDF

#### References

Alakbarov R. G., Pashaev F. H., Hashimov M. A. Development of the Model of Dynamic Storage Distribution in Data Processing Centers, International Journal of Information Technology and Computer Science, 2015, Vol. 7, No. 5, pp. 18–24.

Kubiak I. The Unwanted Emission Signals in the Context of the Reconstruct Possibility of Data Graphics, International

Journal of Image, Graphics and Signal Processing, 2014, Vol. 6, No. 11, pp. 1–9.

Dutta P. K., Mishra O. P., Naskar M. K. Improving Situational Awareness for Precursory Data Classification using Attribute Rough Set Reduction Approach, International Journal of Information Technology and Computer Science, 2013, Vol. 5, No. 12, pp. 47–55.

Dychka I., Sulema Ye. Logical Operations in Algebraic System of Aggregates for Multimodal Data Representation and Processing, KPI Science News, 2018, Vol. 6, pp. 44–52.

Dychka I., Sulema Ye. Ordering Operations in Algebraic System of Aggregates for Multi-Image Data Processing, KPI Science News, 2019, Vol. 1, pp. 15–23.

Sulema Ye. ASAMPL: Programming Language for Mulsemedia Data Processing Based on Algebraic System of Aggregates, Advances in Intelligent Systems and Computing, Springer, 2018, Vol. 725, pp. 431–442.

Allen J. F. Maintaining knowledge about temporal intervals, Communications of ACM, 1983, pp. 832–843.

Allen J. F., Hayes P. J. Moments and points in an intervalbased temporal logic, Computational Intelligence, 1989, Vol. 5, Issue 3, pp. 225–238.

Nebel B., Bürckert H.-J., Reasoning About Temporal Relations: A Maximal Tractable Subclass of Allen’s Interval Algebra, Journal of the Association for Computing Machinery, 1995, Vol. 42, No. 1, pp. 43–66.

Allen J. F., Ferguson G. Actions and Events in Interval Temporal Logic, Spatial and Temporal Reasoning, Springer, 1997, Part 3, pp. 205–245.

Schockaert S., De Cock M., Kerre E., Reasoning About Fuzzy Temporal and Spatial Information from the Web, Intelligent Information Systems, 2010, Vol. 3, 608 p.

Bozzelli L., Molinari A., Montanari A. et al. Interval vs. Point Temporal Logic Model Checking: an Expressiveness Comparison, ACM Transactions on Computational Logic, 2018, Vol. 20, № 1, Article № 4, 31 p.

Grüninger M., Li Zh., The Time Ontology of Allen’s Interval Algebra, Proceedings of 24th International Symposium on Temporal Representation and Reasoning (TIME 2017), Mons, 16–18 October 2017, Article No. 16, pp. 16:1–16:16.

Fraenkel A. A., Bar-Hillel Y., Levy A. Foundations of Set Theory, Studies in Logic and the Foundations of Mathematics. Elsevier, 1973, Vol. 67, 415 p.

Peschanskii V. Yu. ASAMPL compiler, Thesis of Bachelor in Software Engineering. Kyiv, Igor Sikorsky KPI, 2019, 110 p.

Krysiuk A. M. ASAMPL IDE, Thesis of Bachelor in Software Engineering. Kyiv, Igor Sikorsky KPI, 2019, 108 p.

#### GOST Style Citations

1. Alakbarov R. G. Development of the Model of Dynamic Storage Distribution in Data Processing Centers / R. G. Alakbarov, F. H. Pashaev, M. A. Hashimov // International Journal of Information Technology and Computer Science. – 2015. – Vol. 7, № 5. – P. 18–24.

2. Kubiak I. The Unwanted Emission Signals in the Context of the Reconstruct Possibility of Data Graphics / I. Kubiak // International Journal of Image, Graphics and Signal Processing. – 2014. – Vol. 6, № 11. – P. 1–9.

3. Dutta P. K. Improving Situational Awareness for Precursory Data Classification using Attribute Rough Set Reduction Approach / P. K. Dutta, O. P. Mishra, M. K. Naskar // International Journal of Information Technology and Computer Science. – 2013. – Vol. 5, № 12. – P. 47–55.

4. Dychka I. A. Logical Operations in Algebraic System of Aggregates for Multimodal Data Representation and Processing / I. A. Dychka, Ye. S. Sulema // KPI Science News. – 2018. – Vol. 6. – P. 44–52.

5. Dychka I. A. Ordering Operations in Algebraic System of Aggregates for Multi-Image Data Processing / I. A. Dychka, Ye. S. Sulema // KPI Science News. – 2019. – Vol. 1. – P. 15–23.

6. Sulema Ye. S. ASAMPL: Programming Language for Mulsemedia Data Processing Based on Algebraic System of Aggregates / Ye. S. Sulema // Advances in Intelligent Systems and Computing. – Springer, 2018. – Vol. 725. – P. 431–442.

7. Allen J. F. Maintaining knowledge about temporal intervals / J. F. Allen // Communications of ACM. – 1983. – Vol. 26, № 11. – P. 832–843.

8. Allen J. F. Moments and points in an interval-based temporal logic / J. F. Allen, P. J. Hayes // Computational Intelligence. – 1989. – Vol. 5, № 3. – P. 225–238.

9. Nebel B. Reasoning About Temporal Relations: A Maximal Tractable Subclass of Allen’s Interval Algebra / B. Nebel, H.-J. Bürckert // Journal of the Association for Computing Machinery. – 1995. – Vol. 42, № 1. –P. 43–66.

10. Allen J. F. Actions and Events in Interval Temporal Logic / J. F. Allen, G. Ferguson // Spatial and Temporal Reasoning. – Springer, 1997. – Part 3. – P. 205–245.

11. Schockaert S. Reasoning About Fuzzy Temporal and Spatial Information from the Web / S. Schockaert, M. De Cock, E. Kerre // Intelligent Information Systems. – 2010. – Vol. 3. – 608 p.

12. Interval vs. Point Temporal Logic Model Checking: an Expressiveness Comparison / [L. Bozzelli, A. Molinari, A. Montanari, et al.] // ACM Transactions on Computational Logic. – 2018. – Vol. 20, № 1. – Article № 4. – 31 p.

13. Grüninger M. The Time Ontology of Allen’s Interval Algebra / M. Grüninger, Zh. Li // Temporal Representation and Reasoning : 24th International Symposium, Mons, 16– 18 October 2017 : proceedings. – Schloss Dagstuhl : Leibniz Center for Informatics, 2017. – Article № 16. – P. 16:1– 16:16.

14. Fraenkel A. A. Foundations of Set Theory / A. A. Fraenkel, Y. Bar-Hillel, A. Levy // Studies in Logic and the Foundations of Mathematics. – Elsevier, 1973. – Vol. 67. – 415 p.

15. Peschanskii V. Yu. ASAMPL compiler : thesis … bachelor : software engineering / Peschanskii Vladyslav Yuriyovych. – Kyiv : Igor Sikorsky KPI, 2019. – 110 p.

16. Krysiuk A. M. ASAMPL IDE : thesis … bachelor : software engineering / Krysiuk Andrii Mykhaylovych. – Kyiv : Igor Sikorsky KPI, 2019. – 108 p.