Pengembangan Sistem Tanya Jawab Penyakit Pernapasan Menggunakan Model Phi-4 dengan Pendekatan Retrieval-Augmented Generation

Salsabila Widya; Hendri Ahmadian; Nurrizqa

Authors

Salsabila Widya Teknologi Informasi, Universitas Islam Negeri Ar-Raniry https://orcid.org/0009-0000-1066-7309
Hendri Ahmadian Teknologi Informasi, Universitas Islam Negeri Ar-Raniry https://orcid.org/0000-0002-4387-9436
Nurrizqa Teknologi Informasi, Universitas Islam Negeri Ar-Raniry

Keywords:

Retrieval-augmented generation, fine-tuning, question answering system, respiratory diseases, phi-4

Abstract

Large Language Model (LLM)-based question answering systems are capable of generating relevant responses; however, they still have the potential to produce information that is inconsistent with the underlying knowledge source. The Retrieval-Augmented Generation (RAG) approach addresses this limitation by retrieving relevant context before generating responses. This study aims to implement the Phi-4 model with the RAG approach for a respiratory disease question answering system and to evaluate the effect of the fine-tuning process on system performance. Fine-tuning was performed using the Low-Rank Adaptation (LoRA) method, while the RAG implementation employed the BAAI/bge-m3 embedding model and the ChromaDB vector database. The evaluation was conducted using the RAGAS framework with the Context Precision, Context Recall, Faithfulness, and Answer Relevancy metrics on 50 question-answer pairs, and the performance of the fine-tuned Phi-4 model was compared with that of the base Phi-4 model. The results show that the fine-tuned Phi-4 model achieved Context Precision, Context Recall, Faithfulness, and Answer Relevancy scores of 1.0000, 1.0000, 0.9855, and 0.6874, respectively. Compared with the base model, the fine-tuning process improved the Faithfulness and Answer Relevancy scores by 0.0776 and 0.0791, respectively, while the Context Precision and Context Recall scores remained at 1.0000. These results indicate that the fine-tuning process improves answer quality without affecting retrieval performance.

References

[1] N. Rachmat and D. P. Kesuma, “Implementasi Large Language Models Gemini Pada Pengembangan Aplikasi Chatbot Berbasis Android,” Jurnal Ilmu Komputer (JUIK), vol. 4, no. 1, pp. 40–52, 2024.

[2] L. Athota, V. K. Shukla, N. Pandey, and A. Rana, “Chatbot for Healthcare System Using Artificial Intelligence,” in 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), 2020, pp. 619–622. doi: 10.1109/ICRITO48877.2020.9197833.

[3] A. Calderaro, M. Buttrini, B. Farina, S. Montecchini, F. De Conto, and C. Chezzi, “Respiratory tract infections and laboratory diagnostic methods: a review with a focus on syndromic panel-based assays,” Microorganisms, vol. 10, no. 9, p. 1856, 2022, doi: 10.3390/microorganisms10091856.

[4] Z. Ji, T. Yu, Y. Xu, N. Lee, E. Ishii, and P. Fung, “Towards Mitigating Hallucination in Large Language Models via Self-Reflection,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 1827–1843.

[5] M. Omar et al., “Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support,” Communications Medicine, vol. 5, no. 1, p. 330, 2025, doi: 10.1038/s43856-025-01021-3.

[6] Q. Zhou et al., “GastroBot: a Chinese gastrointestinal disease chatbot based on the retrieval-augmented generation,” Front. Med. (Lausanne)., vol. 11, p. 1392555, 2024, doi: 10.3389/fmed.2024.1392555.

[7] N. Son, I. Kang, I. Kim, K. Lee, S. Nam, and D. Lee, “Development and Evaluation of a Retrieval-Augmented Generation-Based Electronic Medical Record Chatbot System,” Healthc. Inform. Res., vol. 31, no. 3, pp. 218–225, 2025, doi: 10.4258/hir.2025.31.3.218.

[8] M. A. Abdurrazzaq, E. L. Tjiong, A. Fasya, M. Hiu, and J. Tanuwidjaya, “An Indonesian Chatbot for Disease Diagnosis Using Retrieval-Augmented Generation,” INOVTEK Polbeng-Seri Informatika, vol. 10, no. 3, pp. 1877–1887, 2025.

[9] I. L. Kharisma, M. S. Hidayat, Somantri, and Kamdan, “Implementasi Retrieval Augmented Generation dalam Sistem Chatbot Dermatologi Berbasis Website,” Jurnal Teknik Informatika dan Sistem Informasi, vol. 11, no. 3, pp. 448–462, 2025, doi: 10.28932/jutisi.v11i3.12258.

[10] G. Samudra, A. T. Zy, and Ermanto, “Implementasi Retrieval Augmented Generation (RAG) Dalam Perancangan Chatbot Kesehatan Pencernaan,” JSAI: Journal Scientific and Applied Informatics, vol. 8, no. 1, pp. 181–188, 2025, doi: 10.36085.

[11] L. O. M. Y. Prayitno, A. Nurfadilah, S. B. Saudi, W. D. Tsunami, and A. M. Sajiah, “Conversational Agent for Medical Question-Answering Using RAG and LLM,” Journal of Artificial Intelligence and Engineering Applications (JAIEA), vol. 4, no. 3, pp. 1894–1899, 2025.

[12] Y. Chen et al., “Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs,” arXiv preprint arXiv:2504.10982, pp. 1–10, 2025.

[13] O. K. Gargari and G. Habibi, “Enhancing medical AI with retrieval-augmented generation: A mini narrative review,” Digit. Health, vol. 11, pp. 1–7, 2025, doi: 10.1177/20552076251337177.

[14] M. Abdin et al., “Phi-4 Technical Report,” arXiv preprint arXiv:2412.08905, pp. 1–36, 2024.

[15] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Adv. Neural Inf. Process. Syst., vol. 33, pp. 9459–9474, 2020.

[16] Z. Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 1–38, 2023, doi: 10.1145/3571730.

[17] S. Es, J. James, L. E. Anke, and S. Schockaert, “RAGAS: Automated Evaluation of Retrieval Augmented Generation,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 2024, pp. 150–158.