Data pipeline optimization : a case study on the maritime domain
| dc.contributor.advisor | Theodoridis, Yannis | |
| dc.contributor.advisor | Θεοδωρίδης, Ιωάννης | |
| dc.contributor.author | Segkos, Nikolaos | |
| dc.contributor.author | Σέγκος, Νικόλαος | |
| dc.date.accessioned | 2025-11-25T12:45:49Z | |
| dc.date.available | 2025-11-25T12:45:49Z | |
| dc.date.issued | 2025-09 | |
| dc.identifier.uri | https://dione.lib.unipi.gr/xmlui/handle/unipi/18577 | |
| dc.format.extent | 45 | el |
| dc.language.iso | en | el |
| dc.publisher | Πανεπιστήμιο Πειραιώς | el |
| dc.rights | Αναφορά Δημιουργού - Μη Εμπορική Χρήση - Παρόμοια Διανομή 3.0 Ελλάδα | * |
| dc.rights | Αναφορά Δημιουργού - Μη Εμπορική Χρήση - Παρόμοια Διανομή 3.0 Ελλάδα | * |
| dc.rights.uri | http://creativecommons.org/licenses/by-nc-sa/3.0/gr/ | * |
| dc.title | Data pipeline optimization : a case study on the maritime domain | el |
| dc.title.alternative | Βελτιστοποίηση αγωγών δεδομένων : μια έρευνα στο ναυτιλιακό τομέα | el |
| dc.type | Bachelor Dissertation | el |
| dc.contributor.department | Σχολή Τεχνολογιών Πληροφορικής και Επικοινωνιών. Τμήμα Πληροφορικής | el |
| dc.description.abstractEN | Data pipelines play an increasingly important role in the modern computing ecosystem. Across all sizes, complexities, and use‑cases, they form the backbone of any system that processes data. However, the rate at which data is ingested now outpaces the capabilities of vertically scaled architectures, leading to increased computing costs and potential disruptions to business continuity. Frameworks such as Apache Kafka, a distributed data streaming platform that uses the publish-subscribe model, have gained popularity for addressing these challenges by providing fault tolerance, high throughput, and horizontal scaling. In this thesis, we analyze the data ingestion workflow of the AIS Antenna of University of Piraeus, with the goal of identifying performance bottlenecks and propose architectural optimizations. Database polling was found to be the primary cause of high end‑to‑end latency, with message loss exceeding 70%. To solve these problems, we propose an efficient architecture that employs Apache Kafka as the message transmission backbone and adds a Redis cache layer for rapid data initialization. Our experimental study over large real-world dataset streams shows that our proposed workflow effectively eliminates message loss and reduces end‑to‑end message latency by 98.9%, enabling horizontal scalability as well as real‑time visualization of maritime traffic. | el |
| dc.subject.keyword | Data ingestion | el |
| dc.subject.keyword | Scalable pipelines | el |
| dc.subject.keyword | Maritime domain | el |
| dc.subject.keyword | Apache Kafka | el |
| dc.subject.keyword | AIS Data | el |
| dc.date.defense | 2025-11-18 |
Αρχεία σε αυτό το τεκμήριο
Αυτό το τεκμήριο εμφανίζεται στις ακόλουθες συλλογές
Εκτός από όπου διευκρινίζεται διαφορετικά, το τεκμήριο διανέμεται με την ακόλουθη άδεια:
Αναφορά Δημιουργού - Μη Εμπορική Χρήση - Παρόμοια Διανομή 3.0 Ελλάδα
Αναφορά Δημιουργού - Μη Εμπορική Χρήση - Παρόμοια Διανομή 3.0 Ελλάδα