A methodology for building a log management infrastructure

Αναστόπουλος, Βασίλειος

Master Thesis

Author

Αναστόπουλος, Βασίλειος

Date

2014-09-24

Abstract

The collection of log data is a challenging operation for organizations that wish to monitor their infrastructure for security or other reasons, while compliance to standards is often mandated by the type of its activities. In this thesis the problem of performing real-time security monitoring on a large scale infrastructure is approached through the proposal of a methodology for the implementation of a log management infrastructure. Already available and related work is used to compose parts of the proposed methodology, avoiding to “reinvent the wheel” where possible, while the discipline of social network analysis is employed to make and justify decisions that where formerly made either intuitively or based on experience and best practices. The methodology concludes at the creation of the repository of the necessary data, while their actual exploitation from their owner is not addressed. Security issues are an essential part of the methodology and embedded were necessary. The proposed methodology addresses all the critical aspects of a log management infrastructure starting from the documentation of the log requirements and the details of the infrastructure that will be monitored. It continues with the analysis of log generation issues, which devices will be used and what log data need to be generated, how these data will be collected and managed in storage. The additional and critical issues of time synchronization, data preprocessing and infrastructure scalability are also analyzed, concluding with the proposal of a performance measurements process to measure the efficiency of the log management infrastructure and adjust it where it is necessary. The security issues are also examined as separate steps of the methodology. The result and contribution of this master thesis is an innovative methodology that can be used as step-by-step guide for the implementation of a log management infrastructure in an organization. This work can be expanded with the addition of log analysis and visualization tasks, as well as managerial issues such as the definition of Standard Operating Procedures (SOP) and the assignment of roles to the personnel. The opportunities offered by the cloud and virtualization technologies are also included as future work.