Running Kafka clusters on Kubernetes
Λειτουργία Kafka cluster σε Kubernetes περιβάλλον
Master Thesis
Author
Ntosas, Dimitrios
Date
2019-02-25View/ Open
Keywords
Kafka ; KubernetesAbstract
Software design patterns have changed radically during the last decade. The emerge of Cloud Computing capabilities accompanied by more demanding business requirements, made the industry moves towards new architectures that promised to bring efficiency to the next level. As a consequence, functionalities like Message Queuing and Containerized Workload Orchestration became common requirements in modern software engineering. Kafka as Message Broker system and Kubernetes as Container Orchestrator bring great features to serve the above needs and they are already tested on mission-critical environments. Problems begin when it comes to maintain and use these systems. Kafka and Kubernetes along with their powerful offerings, introduce great complexity for both Administrators and End-Users that usually act as a burden for their adoption in existing or new setups. We want to challenge these problems by creating a Platform that combines Kafka with Kubernetes and provides their functionalities as a Service. An abstraction interface that brings simplicity and confidence in the way applications and users interact with these technologies. Our goal for Applications utilizing this platform is to rely on benefits starting from Reliability and end-to-end Performance to enhanced Fault Tolerance. Our aim for users acting as Operators is to reduce the burden of manual operations and bring a smoother experience to the administrative process. For users acting as Software Engineers, we want to provide the capability of an accountable reduction of complexity to their Deployment Patterns and a minimalistic way of declaring Message Queuing dependencies. Once configured, Kubernetes, Kafka, and other applications deployed through our Platform will scale and self-heal without any manual intervention.