Development of a system for medical knowledge retrieval and generation from clinical studies based on deep learning and RAG architecture
Ανάπτυξη συστήματος ανάκτησης και παραγωγής ιατρικών γνώσεων με χρήση τεχνικών deep learning και RAG αρχιτεκτονικής
View/
Keywords
Retrieval-augmented generation ; Clinical trials ; Natural language processing ; Information retrieval ; Large language models ; Vector search ; ChromaDB ; LLaMA ; ClinicalTrials.gov ; BM25Abstract
Clinical trials are the primary mechanism through which new medical interventions are
evaluated, and ClinicalTrials.gov — the world's largest clinical trial registry with over
500,000 registered studies — provides public access to detailed trial protocols, eligibility
criteria, intervention designs, and statistical results. Despite this availability, effective
access to this information remains challenging: trial records are written in dense medical
language, span thousands of words of structured text, and cannot be queried with natural
language through existing interfaces.
This thesis presents the design, implementation, and evaluation of an open-source
Retrieval-Augmented Generation (RAG) system for natural language question answering over
clinical trial records. The system enables users to select any trial from a live
ClinicalTrials.gov dropdown, wait while a per-trial index is dynamically constructed,
and then pose arbitrary natural language questions that are answered with citations to
specific trial passages. All inference runs locally on a consumer-grade GPU using
open-source models, requiring no proprietary cloud API.
The primary technical contribution is a section-aware hierarchical chunking pipeline that
exploits the structured module hierarchy of ClinicalTrials.gov v2 JSON records, producing
80–400 semantically coherent chunks per trial with rich metadata enabling structured
retrieval filtering. The retrieval pipeline combines BM25 sparse retrieval and dense
BGE-M3 bi-encoder retrieval via Reciprocal Rank Fusion, followed by cross-encoder
re-ranking and parent document expansion. Advanced query routing techniques — including
Hypothetical Document Embeddings, query decomposition, and step-back prompting — handle
compound, vague, and reasoning-intensive questions. Generation is performed by Llama 3.1
8B Instruct (Q4_K_M quantization) served via Ollama, with citation enforcement and
hallucination detection in post-processing.
A controlled ablation across three embedding models, four retrieval strategies, and two
structurally contrasting trials yields the thesis's principal finding: no embedding model is
universally best for clinical trial retrieval. The biomedical MedCPT encoder is strongest on
a completed, results-dense trial (nDCG@10 = 0.394), while the general-purpose BGE-M3 is
strongest on a recruiting, protocol-only trial (nDCG@10 = 0.508) — a near-complete reversal
driven by the match between each embedder's training distribution and the trial's content. End-to-end evaluation across 30
questions balanced over five query types achieves aggregate faithfulness of 0.862 and
answer relevancy of 0.883, with each query-routing strategy delivering its intended benefit,
confirming that the system produces well-grounded, on-topic answers with low hallucination
rates using a compact open-source model.
Department
Σχολή Τεχνολογιών Πληροφορικής και Επικοινωνιών. Τμήμα ΠληροφορικήςNumber of pages
108Language
EnglishCollections
Except where otherwise noted, this item's license is described as
Αναφορά Δημιουργού - Μη Εμπορική Χρήση - Παρόμοια Διανομή 3.0 Ελλάδα
Αναφορά Δημιουργού - Μη Εμπορική Χρήση - Παρόμοια Διανομή 3.0 Ελλάδα