We kunnen er niet omheen dat Artificiële Intelligentie (AI) al een grote rol speelt in onze maatschappij. Toch is een groot deel van de bevolking nog niet op de hoogte wat AI allemaal kan, zowel de positieve als de negatieve invloeden in ons dagelijks leven. Het best gekende Large language model (LLM) is ChatGPT, welke in staat is om menselijke taal te gebruiken en zo een antwoord te geven op vragen. Ook voor vragen over geneesmiddelen en zelfzorg kun je bij de LLM’s terecht. Deze worden beantwoord met hoge accuraatheid waardoor een tripje naar de dokter of apotheker kan vermeden worden. Maar is dit wel ideaal? Het is niet allemaal rozengeur en maneschijn en de antwoorden van AI durven erg te variëren.
Daarom heb ik in mijn thesis onderzocht of large language models in staat zijn om vragen over zelfzorg met betrekking tot geneesmiddelen en supplementen correct te beantwoorden. Daarnaast besteedde ik extra aandacht aan de variabiliteit van de AI.
Hoe werkt een large language model (LLM) nu eigenlijk?
Large language models zijn vergelijkbaar met de hersenen, waarin vele neuronen met elkaar verbonden zijn. (Figuur 1) Deze verbindingen worden ook wel parameters genoemd. De hoeveelheid parameters bepaalt hoe groot en sterk een model is. De meest moderne taalmodellen bevatten miljarden en sommigen zelfs een biljoenen verbindingen, waardoor ze erg complexe opdrachten kunnen uitvoeren.
Een LLM werkt door een input (bijvoorbeeld een vraag) om te zetten naar een output (het antwoord). De LLM gaat dit doen door te voorspellen wat de kans is dat een bepaald woord zal voorkomen in een zin. Dit betekent dat een model als ChatGPT je vraag niet ‘begrijpt’ en gewoon een grote computer is die voorspelt welk antwoord het beste zal zijn op je vraag.
Figuur 1: een voorstelling van een neuraal netwerk. Een input (bijvoorbeeld een vraag) wordt hier omgezet naar een output (bijvoorbeeld een antwoord).
Om het proces van vraag naar antwoord tot een goed einde te brengen moet zo’n AI natuurlijk getraind worden. Dit gebeurt door de AI enorme hoeveelheden tekst te laten verwerken zoals alle mogelijke geschreven bronnen (boeken, geschriften, verslagen, onderzoeken, etc.), het volledige wereldwijde internet en informatiepagina’s zoals Wikipedia. Bedenk wanneer je ergens een tekst leest, je ervan uit mag gaan dat ChatGPT die ook ooit gezien zal hebben. Hierdoor beschikt het LLM over een brede waaier van kennis waardoor zelfs moeilijke vragen beantwoord kunnen worden.
Slaagt ChatGPT voor het zelfzorgexamen?
Om de kennis over zelfzorg van large language models te testen werd er een lijst van vragen opgesteld. Deze werden daarna aan zes LLMs gesteld:
GPT 3.5
GPT 4.0
Gemini
Gemini Advanced
Copilot
Perplexity
De antwoorden werden verzameld en daarna beoordeeld op een schaal van één tot vijf met behulp van betrouwbare wetenschappelijke bronnen (online bronnen en fysieke bronnen).
De resultaten geven aan dat taalmodellen zeer goed in staat zijn om zelfzorgvragen nauwkeurig te beantwoorden en beschikken over de nodige kennis om relevante gezondheidsinformatie te verstrekken. GPT 4.0 kwam naar voren als het meest betrouwbare model en leverde de meest nauwkeurige en uitgebreide antwoorden.
AI kan ook een slechte dag hebben.
Een belangrijke bevinding is de aanzienlijke variatie in reacties, beïnvloed door taal en de formulering van vragen. De meeste modellen presteerden beter bij vragen in het Engels dan in het Nederlands, wat wijst op een sterkere trainingsbasis in het Engels.
Hoe de vraag geformuleerd werd had ook een sterke invloed op de uitkomst, waarbij modellen vaak hun antwoorden aanpasten aan de waargenomen voorkeuren van de gebruiker. De LLMs lijken prioriteit te geven aan de tevredenheid van de gebruiker boven de juistheid van het antwoord. Dit kan tot gevaarlijke situaties leiden.
Eénzelfde vraag kan ook leiden tot verschillende antwoorden. Wanneer dezelfde vraag 60 dagen lang dagelijks opnieuw werd gesteld aan de taalmodellen werd een variabele kwaliteit waargenomen. (Figuur 2) Modellen zoals GPT 4.0 en Copilot vertoonden weinig variatie met enkele uitschieters. Modellen zoals Perplexity daarentegen vertoonden extreme variabiliteit waarbij 1 dag het verschil kan maken tussen een perfect en een gevaarlijk antwoord. Deze variabiliteit benadrukt het belang van zorgvuldige integratie van deze taalmodellen in de gezondheidszorg om desinformatie en mogelijk schadelijke beslissingen te voorkomen.
Figuur 2: Een grafiek die de evaluatie toont van de reacties van de geteste modellen op de vraag "Moet ik ibuprofen op een lege maag innemen?". De gegevens zijn verzameld gedurende 60 dagen.
Ondanks deze variabiliteit toont deze studie aan dat taalmodellen een cruciaal onderdeel van patiëntenzorg kunnen en zullen worden. Wanneer ze correct worden gebruikt, gaan deze modellen de gezondheidsuitkomsten voor patiënten aanzienlijk verbeteren door toegankelijke en nauwkeurige gezondheidsinformatie te verstrekken. Het vermogen van LLMs om de last voor zorgverleners te verlichten, het begrip van patiënten over medische aandoeningen te verbeteren en 24/7 assistentie te bieden, maakt hen een waardevol hulpmiddel in de moderne gezondheidszorg.
Nog werk aan de winkel
LLMs brengen nog enkele risico’s met zich mee. Door de snelle ontwikkeling van taalmodellen blijven fouten een veelvoorkomend probleem. Het wordt steeds moeilijker om deze fouten te herkennen. De modellen geven menselijke reacties op een erg autoritaire toon, waardoor ze erg betrouwbaar lijken. Deze zogenaamde “hallucinaties” zien er grammaticaal juist uit en lezen vlot, maar bevatten onjuiste informatie. Dit is vooral risicovol in zelfzorgsituaties, waar gebruikers vaak moeite hebben om goede van foute informatie te onderscheiden. Hierdoor kunnen ze per ongeluk verkeerde of zelfs schadelijke adviezen opvolgen. Zowel zorgverleners als patiënten moeten daarom altijd voorzichtig zijn en de informatie die ze van deze modellen krijgen, dubbel controleren.
Naarmate deze LLMs meer getraind gaan worden met specifieke medische en farmaceutisch informatie, hebben ze het potentieel om individuele zelfzorg te verbeteren door toegankelijke en nauwkeurige adviezen over medicijnen en supplementen te bieden. Dit kan patiënten in staat stellen om weloverwogen beslissingen te nemen over hun gezondheid, wat uiteindelijk de gezondheidsresultaten verbetert en de druk op zorgsystemen vermindert. Hun inzet moet echter voorzichtig worden benaderd, met zorgvuldige validatie en het aanpakken van mogelijke risico's om hun voordelen te maximaliseren en schade te minimaliseren.
1. Jairoun AA, Al-Hemyari SS, Shahwan M, Humaid Alnuaimi GR, Zyoud SH, Jairoun M. ChatGPT: Threat or boon to the future of pharmacy practice? Res Soc Adm Pharm [Internet]. 2023 Jul 1 [cited 2024 Apr 9];19(7):975–6 doi:10.1016/j.sapharm.2023.03.012. Available from: https://www.sciencedirect.com/science/article/pii/S1551741123001158
2. Sorich MJ, Menz BD, Hopkins AM. Quality and safety of artificial intelligence generated health information. BMJ [Internet]. 2024 Mar 20 [cited 2024 Apr 10];384:q596 doi:10.1136/bmj.q596. Available from: https://www.bmj.com/content/384/bmj.q596
3. Ye H, Liu T, Zhang A, Hua W, Jia W. Cognitive Mirage: A Review of Hallucinations in Large Language Models [Internet]. arXiv; 2023 [cited 2024 Apr 16]. doi:10.48550/arXiv.2309.06794. Available from: http://arxiv.org/abs/2309.06794
4. Navigli R, Conia S, Ross B. Biases in Large Language Models: Origins, Inventory, and Discussion. J Data Inf Qual [Internet]. 2023 Jun 22 [cited 2024 Apr 9];15(2):10:1-10:21 doi:10.1145/3597307. Available from: https://dl.acm.org/doi/10.1145/3597307
5. What is Artificial Intelligence (AI)? | IBM [Internet]. 2024 [cited 2024 May 6]. Available from: https://www.ibm.com/topics/artificial-intelligence
6. ChatGPT [Internet]. [cited 2024 Apr 26]. Available from: https://chat.openai.com
7. Definition of Large Language Models (LLMs) - Gartner Information Technology Glossary [Internet]. Gartner. [cited 2024 May 6]. Available from: https://www.gartner.com/en/information-technology/glossary/large-language-models-llm
8. What Are Large Language Models (LLMs)? | IBM [Internet]. 2024 [cited 2024 May 6]. Available from: https://www.ibm.com/topics/large-language-models
9. Introduction to Large Language Models | Machine Learning [Internet]. Google for Developers. [cited 2024 Apr 9]. Available from: https://developers.google.com/machine-learning/resources/intro-llms
10. AlphaFold [Internet]. Google DeepMind. 2022 [cited 2024 May 25]. Available from: https://deepmind.google/technologies/alphafold/
11. Ruffolo JA, Nayfach S, Gallagher J, Bhatnagar A, Beazer J, Hussain R, et al. Design of highly functional genome editors by modeling the universe of CRISPR-Cas sequences [Internet]. bioRxiv; 2024 [cited 2024 May 25]. p. 2024.04.22.590591 doi:10.1101/2024.04.22.590591. Available from: https://www.biorxiv.org/content/10.1101/2024.04.22.590591v1
12. Kung TH, Cheatham M, Medenilla A, Sillos C, Leon LD, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health [Internet]. 2023 Feb 9 [cited 2024 Apr 9];2(2):e0000198 doi:10.1371/journal.pdig.0000198. Available from: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000198
13. Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on Medical Challenge Problems [Internet]. arXiv; 2023 [cited 2024 Apr 9]. doi:10.48550/arXiv.2303.13375. Available from: http://arxiv.org/abs/2303.13375
14. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med [Internet]. 2023 Aug [cited 2024 Apr 9];29(8):1930–40 doi:10.1038/s41591-023-02448-8. Available from: https://www.nature.com/articles/s41591-023-02448-8
15. Sallam M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare [Internet]. 2023 Mar 19 [cited 2024 Mar 14];11(6):887 doi:10.3390/healthcare11060887. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10048148/
16. Clusmann J, Kolbinger FR, Muti HS, Carrero ZI, Eckardt J-N, Laleh NG, et al. The future landscape of large language models in medicine. Commun Med [Internet]. 2023 Oct 10 [cited 2024 Apr 9];3(1):1–8 doi:10.1038/s43856-023-00370-1. Available from: https://www.nature.com/articles/s43856-023-00370-1
17. Sinha S. From GPT-1 to GPT-4: A Look at the Evolution of Generative AI [Internet]. HGS. 2023 [cited 2024 May 6]. Available from: https://hgs.cx/blog/from-gpt-1-to-gpt-4-a-look-at-the-evolution-of-generative-ai/
18. Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Internet]. arXiv; 2019 [cited 2024 May 6]. doi:10.48550/arXiv.1810.04805. Available from: http://arxiv.org/abs/1810.04805
19. LaMDA: Towards Safe, Grounded, and High-Quality Dialog Models for Everything [Internet]. [cited 2024 May 6]. Available from: http://research.google/blog/lamda-towards-safe-grounded-and-high-quality-dialog-models-for-everything/
20. Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrou [Internet]. [cited 2024 May 6]. Available from: http://research.google/blog/pathways-language-model-palm-scaling-to-540-billion-parameters-for-breakthrough-performance/
21. Introducing LLaMA: A foundational, 65-billion-parameter language model [Internet]. [cited 2024 May 6]. Available from: https://ai.meta.com/blog/large-language-model-llama-meta-ai/
22. Large Language Model (LLM) | Learn how to interact with OpenAI models [Internet]. [cited 2024 Apr 9]. Available from: https://microsoft.github.io/Workshop-Interact-with-OpenAI-models/llms/
23. Elias J. Google’s newest A.I. model uses nearly five times more text data for training than its predecessor [Internet]. CNBC. 2023 [cited 2024 May 6]. Available from: https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html
24. Adminko. Llama 2 model sizes (7B, 13B, 70B) [Internet]. Llama 2. 2023 [cited 2024 May 6]. Available from: https://llama-2.ai/llama-2-model-sizes/
25. Yu K-H, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018 Oct;2(10):719–31 doi:10.1038/s41551-018-0305-z.
26. Menon P. Introduction to Large Language Models and the Transformer Architecture [Internet]. Medium. 2023 [cited 2024 Apr 20]. Available from: https://rpradeepmenon.medium.com/introduction-to-large-language-models-and-the-transformer-architecture-534408ed7e61
27. Sheu Y, Magdamo C, Miller M, Das S, Blacker D, Smoller JW. AI-assisted prediction of differential response to antidepressant classes using electronic health records. Npj Digit Med [Internet]. 2023 Apr 26 [cited 2024 Apr 29];6(1):1–12 doi:10.1038/s41746-023-00817-8. Available from: https://www.nature.com/articles/s41746-023-00817-8
28. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol [Internet]. 2017 Dec 1 [cited 2024 Apr 29];2(4) doi:10.1136/svn-2017-000101. Available from: https://svn.bmj.com/content/2/4/230
29. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med [Internet]. 2019 Jan [cited 2024 Apr 26];25(1):44–56 doi:10.1038/s41591-018-0300-7. Available from: https://www.nature.com/articles/s41591-018-0300-7
30. Gottlieb S, Silvis L. How to Safely Integrate Large Language Models Into Health Care. JAMA Health Forum [Internet]. 2023 Sep 21 [cited 2024 Apr 9];4(9):e233909 doi:10.1001/jamahealthforum.2023.3909. Available from: https://doi.org/10.1001/jamahealthforum.2023.3909
31. Mukherjee S. A.I. Versus M.D. The New Yorker [Internet]. 2017 Mar 27 [cited 2024 Apr 29]; Available from: https://www.newyorker.com/magazine/2017/04/03/ai-versus-md
32. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) [Internet]. 2017 [cited 2024 Apr 26]. p. 3462–71 doi:10.1109/CVPR.2017.369. Available from: http://arxiv.org/abs/1705.02315
33. Lin C-S, Liu W-T, Tsai D-J, Lou Y-S, Chang C-H, Lee C-C, et al. AI-enabled electrocardiography alert intervention and all-cause mortality: a pragmatic randomized clinical trial. Nat Med [Internet]. 2024 Apr 29 [cited 2024 Apr 30];1–10 doi:10.1038/s41591-024-02961-4. Available from: https://www.nature.com/articles/s41591-024-02961-4
34. Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology. Ophthalmol Sci [Internet]. 2023 May 5 [cited 2024 Apr 29];3(4):100324 doi:10.1016/j.xops.2023.100324. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10272508/
35. Lahat A, Shachar E, Avidan B, Glicksberg B, Klang E. Evaluating the Utility of a Large Language Model in Answering Common Patients’ Gastrointestinal Health-Related Questions: Are We There Yet? Diagnostics [Internet]. 2023 Jun 2 [cited 2024 Apr 29];13(11):1950 doi:10.3390/diagnostics13111950. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10252924/
36. Lim ZW, Pushpanathan K, Yew SME, Lai Y, Sun C-H, Lam JSH, et al. Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. eBioMedicine [Internet]. 2023 Aug 23 [cited 2024 Mar 22];95:104770 doi:10.1016/j.ebiom.2023.104770. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10470220/
37. Ayers JW, Zhu Z, Poliak A, Leas EC, Dredze M, Hogarth M, et al. Evaluating Artificial Intelligence Responses to Public Health Questions. JAMA Netw Open [Internet]. 2023 Jun 7 [cited 2024 Feb 18];6(6):e2317517 doi:10.1001/jamanetworkopen.2023.17517. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10248742/
38. Ravipati A, Pradeep T, Elman SA. The role of artificial intelligence in dermatology: the promising but limited accuracy of ChatGPT in diagnosing clinical scenarios. Int J Dermatol [Internet]. 2023 [cited 2024 Apr 30];62(10):e547–8 doi:10.1111/ijd.16746. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/ijd.16746
39. Rydzewski NR, Dinakaran D, Zhao SG, Ruppin E, Turkbey B, Citrin DE, et al. Comparative Evaluation of LLMs in Clinical Oncology. NEJM AI [Internet]. 2024 Apr 25 [cited 2024 Apr 30];1(5):AIoa2300151 doi:10.1056/AIoa2300151. Available from: https://ai.nejm.org/doi/10.1056/AIoa2300151
40. Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and Limitations. Radiology [Internet]. 2023 Jun [cited 2024 May 6];307(5):e230582 doi:10.1148/radiol.230582. Available from: https://pubs.rsna.org/doi/10.1148/radiol.230582
41. WHO guideline on self-care interventions for health and well-being, 2022 revision [Internet]. [cited 2024 Apr 29]. Available from: https://www.who.int/publications-detail-redirect/9789240052192
42. Divo MJ, Martinez CH, Mannino DM. Ageing and the epidemiology of multimorbidity. Eur Respir J [Internet]. 2014 Oct 1 [cited 2024 Apr 29];44(4):1055–68 doi:10.1183/09031936.00059814. Available from: https://erj.ersjournals.com/content/44/4/1055
43. Allegrante JP, Wells MT, Peterson JC. Interventions to Support Behavioral Self-Management of Chronic Diseases. Annu Rev Public Health [Internet]. 2019 Apr 1 [cited 2024 Apr 29];40(Volume 40, 2019):127–46 doi:10.1146/annurev-publhealth-040218-044008. Available from: https://www.annualreviews.org/content/journals/10.1146/annurev-publhealth-040218-044008
44. Riegel B, Dunbar SB, Fitzsimons D, Freedland KE, Lee CS, Middleton S, et al. Self-care research: Where are we now? Where are we going? Int J Nurs Stud [Internet]. 2021 Apr 1 [cited 2024 Apr 29];116:103402 doi:10.1016/j.ijnurstu.2019.103402. Available from: https://www.sciencedirect.com/science/article/pii/S0020748919302093
45. Riegel B, Moser DK, Buck HG, Dickson VV, Dunbar SB, Lee CS, et al. Self‐Care for the Prevention and Management of Cardiovascular Disease and Stroke. J Am Heart Assoc [Internet]. [cited 2024 Apr 29];6(9):e006997 doi:10.1161/JAHA.117.006997. Available from: https://www.ahajournals.org/doi/full/10.1161/JAHA.117.006997
46. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention Is All You Need [Internet]. arXiv; 2023 [cited 2024 Apr 9]. Available from: http://arxiv.org/abs/1706.03762
47. Transformer: A Novel Neural Network Architecture for Language Understanding [Internet]. [cited 2024 Apr 20]. Available from: http://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/
48. Alammar J. The Illustrated Transformer [Internet]. [cited 2024 Apr 9]. Available from: http://jalammar.github.io/illustrated-transformer/
49. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language Models are Few-Shot Learners [Internet]. arXiv; 2020 [cited 2024 Apr 9]. Available from: http://arxiv.org/abs/2005.14165
50. Drost D. Different ways of training LLMs [Internet]. Medium. 2023 [cited 2024 Apr 23]. Available from: https://towardsdatascience.com/different-ways-of-training-llms-c57885f388ed
51. An Introductory Guide to Fine-Tuning LLMs [Internet]. [cited 2024 May 6]. Available from: https://www.datacamp.com/tutorial/fine-tuning-large-language-models
52. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al. Training language models to follow instructions with human feedback [Internet]. arXiv; 2022 [cited 2024 Apr 26]. doi:10.48550/arXiv.2203.02155. Available from: http://arxiv.org/abs/2203.02155
53. Microsoft Copilot: Your everyday AI companion [Internet]. Microsoft Copilot: Your everyday AI companion. [cited 2024 Apr 29]. Available from: https://ceto.westus2.binguxlivesite.net/
54. Gemini: de chat-tool die je ideeën prikkelt [Internet]. Gemini. [cited 2024 Apr 29]. Available from: https://gemini.google.com
55. Perplexity [Internet]. [cited 2024 Apr 29]. Available from: https://www.perplexity.ai/
56. GPT-4 [Internet]. [cited 2024 Apr 30]. Available from: https://openai.com/research/gpt-4
57. Copilot Pro Plan & Pricing - Premium AI features plus GPT-4 & GPT-4 Turbo | Microsoft Store [Internet]. [cited 2024 Apr 30]. Available from: https://www.microsoft.com/en-us/store/b/copilotpro
58. Gemini - Google DeepMind [Internet]. [cited 2024 Apr 30]. Available from: https://deepmind.google/technologies/gemini/
59. Getting Started with Perplexity [Internet]. [cited 2024 Apr 30]. Available from: https://www.perplexity.ai/hub/getting-started
60. Payton H, Warren S. How to identify red-flag symptoms and refer patients appropriately [Internet]. The Pharmaceutical Journal. 2024 [cited 2024 May 31]. Available from: https://pharmaceutical-journal.com/article/ld/how-to-identify-red-flag-symptoms-and-refer-patients-appropriately
61. Sivan V. Top 20 ChatGPT prompts that every prompt engineers should know [Internet]. Coinmonks. 2023 [cited 2024 Apr 29]. Available from: https://medium.com/coinmonks/top-20-chatgpt-prompts-that-every-prompt-engineers-should-know-937b0ea5472
62. Prompt Engineering Guide – Nextra [Internet]. [cited 2024 Apr 29]. Available from: https://www.promptingguide.ai/
63. Briggs GG, Freeman RK, Towers CV, Forinash AB. Briggs Drugs in Pregnancy and Lactation: A Reference Guide to Fetal and Neonatal Risk. Lippincott Williams & Wilkins; 2021.
64. Preston CL. Stockley’s Drug Interactions: A Source Book of Interactions, Their Mechanisms, Clinical Importance and Management [Internet]. 12th ed. London: Pharmaceutical Press; 2019. (Stockley’s Drug Interactions Series). Available from: https://books.google.be/books?id=oT1QvQEACAAJ
65. Alison Blenkinsopp, Martin Duerden, John Blenkinsopp. Symptoms in the Pharmacy A Guide to the Management of Common Illnesses. 8th ed. Wiley Blackwell; 2018.
66. Krinsky DL. Handbook of Nonprescription Drugs: An Interactive Approach to Self-Care [Internet]. Washington: American Pharmacists Association; 2020. Available from: https://books.google.be/books?id=FgO4zQEACAAJ
67. Small P, Keith PK, Kim H. Allergic rhinitis. Allergy Asthma Clin Immunol [Internet]. 2018 Sep 12 [cited 2024 Mar 30];14(2):51 doi:10.1186/s13223-018-0280-7. Available from: https://doi.org/10.1186/s13223-018-0280-7
68. Schwinghammer TL, DiPiro JT, Ellingrod VL, DiPiro CV. DiPiro’s Pharmacotherapy Handbook, 12e [Internet]. Vol. 12. New York, NY: McGraw Hill; 2023 [cited 2024 May 1]. Available from: accesspharmacy.mhmedical.com/content.aspx?aid=1201205915
69. Blackboard Learn [Internet]. [cited 2024 May 2]. Available from: https://blackboard.uantwerpen.be/
70. Wikipedia.org articles by language 2023 [Internet]. Statista. [cited 2024 May 16]. Available from: https://www.statista.com/statistics/1427961/wikipedia-org-articles-language/
71. Definition of homeopathy - NCI Dictionary of Cancer Terms - NCI [Internet]. 2011 [cited 2024 May 9]. Available from: https://www.cancer.gov/publications/dictionaries/cancer-terms/def/homeopathy
72. Homeopathy [Internet]. nhs.uk. 2017 [cited 2024 May 9]. Available from: https://www.nhs.uk/conditions/homeopathy/
73. Cukaci C, Freissmuth M, Mann C, Marti J, Sperl V. Against all odds—the persistent popularity of homeopathy. Wien Klin Wochenschr [Internet]. 2020 May 1 [cited 2024 May 9];132(9):232–42 doi:10.1007/s00508-020-01624-x. Available from: https://doi.org/10.1007/s00508-020-01624-x
74. Brata C, Gudka S, Schneider CR, Clifford RM. A review of the provision of appropriate advice by pharmacy staff for self-medication in developing countries. Res Soc Adm Pharm RSAP. 2015;11(2):136–53 doi:10.1016/j.sapharm.2014.07.003.
75. Batista AD, A. Rodrigues D, Figueiras A, Zapata-Cachafeiro M, Roque F, Herdeiro MT. Antibiotic Dispensation without a Prescription Worldwide: A Systematic Review. Antibiotics [Internet]. 2020 Nov [cited 2024 May 24];9(11):786 doi:10.3390/antibiotics9110786. Available from: https://www.mdpi.com/2079-6382/9/11/786
76. Björnsdottir I, Granas AG, Bradley A, Norris P. A systematic review of the use of simulated patient methodology in pharmacy practice research from 2006 to 2016. Int J Pharm Pract [Internet]. 2020 Feb 1 [cited 2024 May 24];28(1):13–25 doi:10.1111/ijpp.12570. Available from: https://doi.org/10.1111/ijpp.12570
77. Watson MC Dr, Norris P, Granas AG. A systematic review of the use of simulated patients and pharmacy practice research. Int J Pharm Pract [Internet]. 2006 Jun 1 [cited 2024 May 24];14(2):83–93 doi:10.1211/ijpp.14.2.0002. Available from: https://doi.org/10.1211/ijpp.14.2.0002
78. Pizetta B, Raggi LG, Rocha KSS, Cerqueira-Santos S, de Lyra-Jr DP, dos Santos Júnior GA. Does drug dispensing improve the health outcomes of patients attending community pharmacies? A systematic review. BMC Health Serv Res [Internet]. 2021 Aug 2 [cited 2024 May 24];21(1):764 doi:10.1186/s12913-021-06770-0. Available from: https://doi.org/10.1186/s12913-021-06770-0
79. Xu R, Feng Y, Chen H. ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience [Internet]. arXiv; 2023 [cited 2024 May 23]. doi:10.48550/arXiv.2307.01135. Available from: http://arxiv.org/abs/2307.01135
80. ChatGPT and Search Engines: Understanding the Differences [Internet]. [cited 2024 May 23]. Available from: https://www.linkedin.com/pulse/chatgpt-search-engines-understanding-differences-richter-csp-cpae
81. Lewis P. ChatGPT vs Search Engines [Internet]. Medium. 2024 [cited 2024 May 23]. Available from: https://medium.com/@prakashlewis/chatgpt-vs-search-engines-6f73546924ed
82. Hello GPT-4o [Internet]. [cited 2024 May 30]. Available from: https://openai.com/index/hello-gpt-4o/
83. Snoswell CL, Falconer N, Snoswell AJ. Pharmacist vs machine: Pharmacy services in the age of large language models. Res Soc Adm Pharm [Internet]. 2023 Jun 1 [cited 2024 Apr 9];19(6):843–4 doi:10.1016/j.sapharm.2023.03.006. Available from: https://www.sciencedirect.com/science/article/pii/S1551741123000967
84. Adler-Milstein J, Redelmeier DA, Wachter RM. The Limits of Clinician Vigilance as an AI Safety Bulwark. JAMA [Internet]. 2024 Apr 9 [cited 2024 May 8];331(14):1173–4 doi:10.1001/jama.2024.3620. Available from: https://doi.org/10.1001/jama.2024.3620
85. De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health [Internet]. 2023 Apr 25 [cited 2024 Apr 9];11 doi:10.3389/fpubh.2023.1166120. Available from: https://www.frontiersin.org/journals/public-health/articles/10.3389/fpubh.2023.1166120/full
86. dataenthusiast.io. Language Models at Risk: Safeguarding AI with LLM Guard [Internet]. Medium. 2024 [cited 2024 May 18]. Available from: https://medium.com/@dataenthusiast.io/language-models-at-risk-safeguarding-ai-with-llm-guard-11a3e7923af5
87. Dhinakaran A. Safeguarding LLMs with Guardrails [Internet]. Medium. 2023 [cited 2024 May 18]. Available from: https://towardsdatascience.com/safeguarding-llms-with-guardrails-4f5d9f57cff2
88. Menz BD, Kuderer NM, Bacchi S, Modi ND, Chin-Yee B, Hu T, et al. Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis. BMJ [Internet]. 2024 Mar 20 [cited 2024 May 13];384:e078538 doi:10.1136/bmj-2023-078538. Available from: https://www.bmj.com/content/384/bmj-2023-078538
89. Keya AJ, Wadud MAH, Mridha MF, Alatiyyah M, Hamid MA. AugFake-BERT: Handling Imbalance through Augmentation of Fake News Using BERT to Enhance the Performance of Fake News Classification. Appl Sci [Internet]. 2022 Jan [cited 2024 May 13];12(17):8398 doi:10.3390/app12178398. Available from: https://www.mdpi.com/2076-3417/12/17/8398
90. Pegoraro A, Kumari K, Fereidooni H, Sadeghi A-R. To ChatGPT, or not to ChatGPT: That is the question! [Internet]. arXiv; 2023 [cited 2024 May 13]. doi:10.48550/arXiv.2304.01487. Available from: http://arxiv.org/abs/2304.01487
91. Mattu JL Julia Angwin,Lauren Kirchner,Surya. How We Analyzed the COMPAS Recidivism Algorithm [Internet]. ProPublica. [cited 2024 May 13]. Available from: https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
92. Zack T, Lehman E, Suzgun M, Rodriguez JA, Celi LA, Gichoya J, et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health [Internet]. 2024 Jan 1 [cited 2024 May 20];6(1):e12–22 doi:10.1016/S2589-7500(23)00225-X. Available from: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(23)00225-X/fulltext
93. Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. Npj Digit Med [Internet]. 2023 Jul 6 [cited 2024 May 20];6(1):1–6 doi:10.1038/s41746-023-00873-0. Available from: https://www.nature.com/articles/s41746-023-00873-0
94. Albalak A, Elazar Y, Xie SM, Longpre S, Lambert N, Wang X, et al. A Survey on Data Selection for Language Models [Internet]. arXiv; 2024 [cited 2024 May 18]. doi:10.48550/arXiv.2402.16827. Available from: http://arxiv.org/abs/2402.16827
95. Welbl J, Glaese A, Uesato J, Dathathri S, Mellor J, Hendricks LA, et al. Challenges in Detoxifying Language Models. In: Moens M-F, Huang X, Specia L, Yih SW, editors. Findings of the Association for Computational Linguistics: EMNLP 2021 [Internet]. Punta Cana, Dominican Republic: Association for Computational Linguistics; 2021 [cited 2024 May 18]. p. 2447–69 doi:10.18653/v1/2021.findings-emnlp.210. Available from: https://aclanthology.org/2021.findings-emnlp.210
96. Zhang S, Roller S, Goyal N, Artetxe M, Chen M, Chen S, et al. OPT: Open Pre-trained Transformer Language Models [Internet]. arXiv; 2022 [cited 2024 May 18]. doi:10.48550/arXiv.2205.01068. Available from: http://arxiv.org/abs/2205.01068
97. Dodge J, Sap M, Marasović A, Agnew W, Ilharco G, Groeneveld D, et al. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. In: Moens M-F, Huang X, Specia L, Yih SW, editors. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing [Internet]. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics; 2021 [cited 2024 May 18]. p. 1286–305 doi:10.18653/v1/2021.emnlp-main.98. Available from: https://aclanthology.org/2021.emnlp-main.98
98. Longpre S, Yauney G, Reif E, Lee K, Roberts A, Zoph B, et al. A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity [Internet]. arXiv; 2023 [cited 2024 May 18]. doi:10.48550/arXiv.2305.13169. Available from: http://arxiv.org/abs/2305.13169
99. Workowski KA, Bachmann LH, Chan PA, Johnston CM, Muzny CA, Park I, et al. Sexually Transmitted Infections Treatment Guidelines, 2021. MMWR Recomm Rep [Internet]. 2021 Jul 23 [cited 2024 May 1];70(4):1–187 doi:10.15585/mmwr.rr7004a1. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8344968/
100. Moudatsou M, Stavropoulou A, Philalithis A, Koukouli S. The Role of Empathy in Health and Social Care Professionals. Healthcare [Internet]. 2020 Mar [cited 2024 May 21];8(1):26 doi:10.3390/healthcare8010026. Available from: https://www.mdpi.com/2227-9032/8/1/26
101. Jubraj B, Barnett NL, Grimes L, Varia S, Chater A, Auyeung V. Why we should understand the patient experience: clinical empathy and medicines optimisation. Int J Pharm Pract [Internet]. 2016 Oct 1 [cited 2024 May 21];24(5):367–70 doi:10.1111/ijpp.12268. Available from: https://doi.org/10.1111/ijpp.12268
102. Luccioni AS, Jernite Y, Strubell E. Power Hungry Processing: Watts Driving the Cost of AI Deployment? [Internet]. arXiv; 2023 [cited 2024 May 14]. Available from: http://arxiv.org/abs/2311.16863
103. Training a single AI model can emit as much carbon as five cars in their lifetimes [Internet]. MIT Technology Review. [cited 2024 May 14]. Available from: https://www.technologyreview.com/2019/06/06/239031/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/
104. The Environmental impact of Large language models - Measuring the Carbon footprint of AI [Internet]. [cited 2024 May 14]. Available from: https://www.linkedin.com/pulse/environmental-impact-large-language-models-carbon-ai-shanmugam-vopfc
105. Strubell E, Ganesh A, McCallum A. Energy and Policy Considerations for Deep Learning in NLP [Internet]. arXiv; 2019 [cited 2024 May 14]. doi:10.48550/arXiv.1906.02243. Available from: http://arxiv.org/abs/1906.02243
106. Patterson D, Gonzalez J, Hölzle U, Le Q, Liang C, Munguia L-M, et al. The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink [Internet]. arXiv.org. 2022 [cited 2024 May 14]. Available from: https://arxiv.org/abs/2204.05149v1
107. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference [Internet]. ar5iv. [cited 2024 May 14]. Available from: https://ar5iv.labs.arxiv.org/html/2310.03003
108. Powering a Google search [Internet]. Official Google Blog. [cited 2024 May 14]. Available from: https://googleblog.blogspot.com/2009/01/powering-google-search.html
109. Russell RG, Lovett Novak L, Patel M, Garvey KV, Craig KJT, Jackson GP, et al. Competencies for the Use of Artificial Intelligence–Based Tools by Health Care Professionals. Acad Med [Internet]. 2023 Mar [cited 2024 May 13];98(3):348 doi:10.1097/ACM.0000000000004963. Available from: https://journals.lww.com/academicmedicine/fulltext/2023/03000/competencies_for_the_use_of_artificial.19.aspx
110. Machine Learning Glossary [Internet]. Google for Developers. [cited 2024 Apr 21]. Available from: https://developers.google.com/machine-learning/glossary
111. Combophen 500 mg - 150 mg film-coat. tabl. | Medicinal product database [Internet]. [cited 2024 Jun 2]. Available from: https://medicinesdatabase.be/human-use/overview/change-of-delivery-mode/62bc230c8ab5583c188cf8bf
112. Sheth A, Shalin VL, Kursuncu U. Defining and detecting toxicity on social media: context and knowledge are key. Neurocomputing [Internet]. 2022 Jun 14 [cited 2024 Jun 2];490:312–8 doi:10.1016/j.neucom.2021.11.095. Available from: https://www.sciencedirect.com/science/article/pii/S0925231221018087
113. Cleare A, Pariante C, Young A, Anderson I, Christmas D, Cowen P, et al. Evidence-based guidelines for treating depressive disorders with antidepressants: A revision of the 2008 British Association for Psychopharmacology guidelines. J Psychopharmacol (Oxf) [Internet]. 2015 May 1 [cited 2024 May 1];29(5):459–525 doi:10.1177/0269881115581093. Available from: https://doi.org/10.1177/0269881115581093
114. BCFI | Sildenafil [Internet]. BCFI. [cited 2024 May 1]. Available from: https://www.bcfi.be/nl/chapters/8?frag=19430&view=pvt&vmp_group=28688
115. Papich MG author. Saunders handbook of veterinary drugs : small and large animal [Internet]. Fourth edition. Edinburgh: Saunders; 2016. Available from: http://lib.ugent.be/catalog/ebk01:3820000000017712
116. Batur P, Kransdorf LN, Casey PM. Emergency Contraception. Mayo Clin Proc [Internet]. 2016 Jun 1 [cited 2024 Mar 17];91(6):802–7 doi:10.1016/j.mayocp.2016.02.018. Available from: https://www.sciencedirect.com/science/article/pii/S002561961600149X
117. Apotheek.nl [Internet]. [cited 2024 May 1]. Available from: https://www.apotheek.nl/
118. Werking [Internet]. Flamigel®. [cited 2024 May 1]. Available from: https://www.flamigel.be/werking/
119. Aronson J. Meyler’s Side Effects of Drugs: The International Encyclopedia of Adverse Drug Reactions and Interactions. 16th edition. 16th ed. Elsevier; 2015.
120. The National Asthma Council Australia [Internet]. [cited 2024 May 1]. Available from: https://www.nationalasthma.org.au/living-with-asthma/how-to-videos/how-to-use-autohaler
121. De Backer G. Voedingsaanbevelingen voor België.
122. Milman N, Paszkowski T, Cetin I, Castelo-Branco C. Supplementation during pregnancy: beliefs and science. Gynecol Endocrinol [Internet]. 2016 Jul 2 [cited 2024 Mar 26];32(7):509–16 doi:10.3109/09513590.2016.1149161. Available from: https://doi.org/10.3109/09513590.2016.1149161
123. Kominiarek MA, Rajan P. Nutrition Recommendations in Pregnancy and Lactation. Med Clin [Internet]. 2016 Nov 1 [cited 2024 Mar 26];100(6):1199–215 doi:10.1016/j.mcna.2016.06.004. Available from: https://www.medical.theclinics.com/article/S0025-7125(16)37295-9/abstract
124. Fewtrell M, Bronsky J, Campoy C, Domellöf M, Embleton N, Fidler Mis N, et al. Complementary Feeding. J Pediatr Gastroenterol Nutr [Internet]. 2017 [cited 2024 Mar 17];64(1):119–32 doi:10.1097/MPG.0000000000001454. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1097/MPG.0000000000001454
125. Donovan SM, Rao G. Health benefits of yogurt among infants and toddlers aged 4 to 24 months: a systematic review. Nutr Rev [Internet]. 2019 Jul 1 [cited 2024 May 1];77(7):478–86 doi:10.1093/nutrit/nuz009. Available from: https://doi.org/10.1093/nutrit/nuz009
126. Paul Rutter. Community pharmacy: symptoms, diagnosis and treatment. Vol. 4th. Elsevier; 2017. 400 p.
127. Mumcuoglu KY, Pollack RJ, Reed DL, Barker SC, Gordon S, Toloza AC, et al. International recommendations for an effective control of head louse infestations. Int J Dermatol [Internet]. 2021 [cited 2024 May 1];60(3):272–80 doi:10.1111/ijd.15096. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/ijd.15096
128. Head lice [Internet]. [cited 2024 Mar 18]. Available from: https://caringforkids.cps.ca/handouts/health-conditions-and-treatments/head_lice
129. Leung AKC, Lam JM, Leong KF, Barankin B, Hon KL. Paediatrics: how to manage pediculosis capitis. Drugs Context. 2022;11:2021-11–3 doi:10.7573/dic.2021-11-3.
130. Reisapotheek [Internet]. 2023 [cited 2024 May 1]. Available from: https://www.wanda.be/nl/a-z-index/geneesmiddelen-mee-op-reis
131. Shenoy ES, Macy E, Rowe T, Blumenthal KG. Evaluation and Management of Penicillin Allergy: A Review. JAMA. 2019 Jan 15;321(2):188–99 doi:10.1001/jama.2018.19283.
132. Australia T in. Tobacco in Australia [Internet]. [cited 2024 Mar 18]. Available from: https://www.tobaccoinaustralia.org.au/homepage_module/homepagec_content/homepagec_main_content/home_content.html
133. BCFI | Paracetamol + AINS [Internet]. BCFI. [cited 2024 May 1]. Available from: https://www.bcfi.be/nl/chapters/9?frag=8903114&trade_family=42515
134. BCFI | Specialiteiten [Internet]. BCFI. [cited 2024 May 1]. Available from: https://www.bcfi.be/nl/chapters/11?frag=8417&view=pvt&vmp_group=22855
135. Geneesmiddel | Cetirizine | Kinderformularium [Internet]. [cited 2024 Mar 30]. Available from: https://www.kinderformularium.nl/geneesmiddel/8/cetirizine
136. Bayless TM, Brown E, Paige DM. Lactase Non-persistence and Lactose Intolerance. Curr Gastroenterol Rep [Internet]. 2017 Apr 18 [cited 2024 Mar 31];19(5):23 doi:10.1007/s11894-017-0558-9. Available from: https://doi.org/10.1007/s11894-017-0558-9
137. Szilagyi A, Ishayek N. Lactose Intolerance, Dairy Avoidance, and Treatment Options. Nutrients [Internet]. 2018 Dec [cited 2024 Mar 31];10(12):1994 doi:10.3390/nu10121994. Available from: https://www.mdpi.com/2072-6643/10/12/1994
138. Doron S, Snydman DR. Risk and Safety of Probiotics. Clin Infect Dis [Internet]. 2015 May 15 [cited 2024 Mar 31];60(suppl_2):S129–34 doi:10.1093/cid/civ085. Available from: https://doi.org/10.1093/cid/civ085
139. Carazo A, Macáková K, Matoušová K, Krčmová LK, Protti M, Mladěnka P. Vitamin A Update: Forms, Sources, Kinetics, Detection, Function, Deficiency, Therapeutic Use and Toxicity. Nutrients. 2021 May 18;13(5):1703 doi:10.3390/nu13051703.
140. Campbell B, Kreider RB, Ziegenfuss T, La Bounty P, Roberts M, Burke D, et al. International Society of Sports Nutrition position stand: protein and exercise. J Int Soc Sports Nutr [Internet]. 2007 Sep 26 [cited 2024 May 1];4(1):8 doi:10.1186/1550-2783-4-8. Available from: https://doi.org/10.1186/1550-2783-4-8
141. Reynolds NJ, Franklin V, Gray JC, Diffey BL, Farr PM. Narrow-band ultraviolet B and broad-band ultraviolet A phototherapy in adult atopic eczema: a randomised controlled trial. The Lancet [Internet]. 2001 Jun 23 [cited 2024 Apr 1];357(9273):2012–6 doi:10.1016/S0140-6736(00)05114-X. Available from: https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(00)05114-X/fulltext
142. Baldwin H, Tan J. Effects of Diet on Acne and Its Response to Treatment. Am J Clin Dermatol [Internet]. 2021 Jan 1 [cited 2024 Apr 1];22(1):55–65 doi:10.1007/s40257-020-00542-y. Available from: https://doi.org/10.1007/s40257-020-00542-y
143. Olsen EA, Messenger AG, Shapiro J, Bergfeld WF, Hordinsky MK, Roberts JL, et al. Evaluation and treatment of male and female pattern hair loss. J Am Acad Dermatol [Internet]. 2005 Feb 1 [cited 2024 May 1];52(2):301–11 doi:10.1016/j.jaad.2004.04.008. Available from: https://www.jaad.org/article/S0190-9622(04)01058-8/abstract#secd71163793e732