Um sistema por processamento de fluxos aplicado à análise e monitoramento da rede

Abstract

Data processing is commonly performed by big data systems, in traditional architectures, performing data manipulation offline. However, with the need to get results with low latency, there is the use of other architectures, such as LAMBDA and Kappa for the implementation of big data systems, directed to the processing of data streams. Several studies in the literature begin to apply this new model of architecture for different purposes, as well as the use of different types of tools to make it possible to implement it. In This scenario some systems are developed with these molds to monitor and process the flow of data generated by network traffic, employing different types of analysis on the collected data, to get from information about network bandwidth consumption to identify anomalies that occur. In this context, this work aims to develop a system based on the Lambda architecture, applied to the monitoring and processing of the data flow of network traffic, performing the integration of different open source tools. Each tool is responsible for certain functionality implemented, from monitoring and collecting network traffic, information transport, normalization and data storage, to subsequently perform analyses thereof and detect anomalies originated by DDoS attacks, brute force, and port scanning on certain protocols. Regarding the connections classified as anomalous, information pertinent to the IP responsible for originating this connection will be obtained. The experimental analysis of the system occurs with the use of a controlled set of data that has several anomalies, as well as those that must be detected by the system. Shortly after this step, the system is applied to process data that was collected from a local network for eleven days, totaling more than 14 million connections. The experimental results obtained on the actual network traffic present the three types of anomalies that were considered in this study, as well as information about the IPs responsible for them, identifying the country and its respective organization.