Quality Metrics in domain-specific Large Language Models
ICT & Open Learning/Innovation
Client company:OpenMaze
Guilherme Oliveira
Project description
How we can validate the quality of text generation by AI after we finetune it? The project focuses on researching new ways to validate a fine-tuned model on a given domain; how well it was finetuned.
Context
The stakeholder requested this research so, in the upcoming years, they could use it as to advise future projects.
Results
We validated a few hypotheses but the most exciting one that we found to be true was the use of word-embedding plots to visualise the difference between the output of a base model compared to his fine-tuned counterpart.
About the project group
I am a semester 4 student in Open Learning. I have been working on this project alone since the second half of this semester due to some complications with my group mates.