This website presents the leaderboard of the SweClinEval benchmark, a tool for evaluating the progress of Swedish clinical NLP. The first version of SweClinEval was created by Thomas Vakili, Martin Hansson, and Aron Henriksson from the NLP Group at the Department of Computer and Systems Sciences at Stockholm University. The datasets are from the Health Bank infrastructure.
Abstract
The lack of benchmarks in certain domains and for certain languages makes it difficult to track progress regarding the state-of-the-art of NLP in those areas, potentially impeding progress in important, specialized domains. Here, we introduce the first Swedish benchmark for clinical NLP: SweClinEval. The first iteration of the benchmark consists of six clinical NLP tasks, encompassing both document-level classification and named entity recognition tasks, with real clinical data. We evaluate nine different encoder models, both Swedish and multilingual. The results show that domain-adapted models outperform generic models on sequence-level classification tasks, while certain larger generic models outperform the clinical models on named entity recognition tasks. We describe how the benchmark can be managed despite limited possibilities to share sensitive clinical data, and discuss plans for extending the benchmark in future iterations.