Introduction: Pediatric palliative care plays a crucial role in oncology by providing comprehensive support to children and their families. On-demand information support in this regard is vital for optimum healthcare delivery. Retrieval Augmented Generation (RAG) is an Artificial Intelligence (AI) technique that enhances large language models (LLMs) by retrieving relevant information from an external knowledge base during the generation process. This approach enhances LLMs’ capability in accessing and utilizing up-to-date information, improving their accuracy, factuality, and ability to handle dynamic knowledge, which can be used to provide access to the patients and their caregivers a 24/7 on-demand information support about their medications, answer their questions about their condition, or help them to make decisions about their care.

Methodology: A knowledge base (KB) consisting of relevant information and frequently asked questions was prepared using internet and generative pre-trained transformers (GPT). This KB was then used to train an LLM using a prototype server to create a front-end of an AI Agent available via a URL. A comprehensive suite of test questions was developed and AI Agent’s responses were assessed using a matrix of six benchmark questions by human evaluators via iterative testing and refinement.

Results: Using GPT-4o mini LLM, which is capable of conversing in more than 50 languages including Arabic, average final score after three iterations for Accuracy (Factual Accuracy and Relevance) was 100%, Comprehensiveness (Completeness, Depth) 98%, Coherence (Logical Flow, Readability) 98%, User Experience (Conversationality, Personalization and User Friendliness) 98%, Performance (Response Time, Scalability, Robustness) 98% and RAG Specific (Retrieval Effectiveness, Generation Quality) 100%.

Conclusion: Given the excellent performance exhibited by the AI Agents based on human evaluation, introduction of AI Agents for 24/7 on-demand information support is a viable option in efforts to improve Patient-Reported Outcome Measures (PROMs) for pediatric palliative care service.