Labs

Benchmark datasets of biomedical questions, in English, along with gold standard (reference) answers constructed by a team of biomedical experts. The participants have to respond with relevant articles, and snippets from designated resources, as well as exact and "ideal" answers.

Biomedical experts pose unanswered questions for developing problems, such as COVID-19, receive the responses provided by the participating systems, and provide feedback, together with updated questions in an iterative procedure that aims to facilitate the incremental understanding of developing problems in biomedicine and public health.

A shared task on the automatic detection and normalization of mentions of four clinical entity types, namely diseases, symptoms, procedures and medications, in cardiology clinical case documents in Spanish, English, Italian and Dutch.

A shared task on nested named entity recognition (NER) in PubMed abstracts in Russian and English. The train/dev datasets include annotated mentions of disorders, anatomical structures, chemicals, diagnostic procedures, and biological functions. Participants are encouraged to apply cross-language (Russian to English) and cross-domain techniques.

Assess whether a statement, sourced from either a tweet or a political debate, warrants fact-checking.

Given a sentence from a news article, determine whether it is subjective or objective.

Given a news article and a list of 23 persuasion techniques organized into a 2-tier taxonomy, including logical fallacies and emotional manipulation techniques that might be used to support flawed argumentation, identify the spans of texts in which each technique occurs.

Determine the roles of entities within memes, categorizing them as “hero”, “villain”, “victim”, or “other” through a multi-class classification approach that considers the systematic modeling of multimodal semiotic.

Given a rumor expressed in a tweet and a set of authorities for that rumor, retrieve up to 5 evidence tweets from the authorities' timelines, and determine if the rumor is supported, refuted, or unverifiable according to the evidence.

The task is realised in five domains: style-based news bias assessment (HN), propaganda detection (PR), fact checking (FC), rumour detection (RD) and COVID-19 misinformation detection (C19). For each domain, the participants are provided with three victim models, trained for the corresponding binary classification task, as well as a collection of 400 text fragments. Their aim is to prepare adversarial examples, which preserve the meaning of the original examples, but are labelled differently be the classifiers.

This task will test and verify a model's understanding of an application domain and specific topic of interest.

This task will test how the truthfulness or veracity of automatically generated text can be assessed.

This task will test the capability of a model to handle input variation -- e.g. dialectal, sociolectal, and cross-cultural -- as represented by a set of equivalent but non-identical varieties of input prompts.

This task will explore whether automatically-generated text can be distinguished from human-authored text. This task will be organised in collaboration with the PAN lab at CLEF.

It consists of ranking sentences from a collection of user writings according to their relevance to a depression symptom. The participants will have to provide rankings for the 21 symptoms of depression from the BDI Questionnaire.

The challenge consists in performing a task on early risk detection of anorexia. The challenge consists of sequentially processing pieces of evidence and detect early traces of anorexia as soon as possible.

The task consists of estimating the level of features associated with a diagnosis of eating disorders from a thread of user submissions. For each user, the participants will be given a history of postings and the participants will have to fill a standard eating disorder questionnaire.

The first task is a binary classification. The systems have to decide whether or not a given tweet contains or describes sexist expressions or behaviors (i.e., it is sexist itself, describes a sexist situation or criticizes a sexist behavior).

This task aims to categorize the sexist messages according to the intention of the author in one of the following categories: (i) direct sexist message, (ii) reported sexist message and (iii) judgemental message.

The third task is a multiclass task that aims to categorize the sexist messages according to the type or types of sexism they contain (according to the categorization proposed by experts and that takes into account the different facets of women that are undermined): (i) ideological and inequality, (ii) stereotyping and dominance, (iii) objectification, (iv) sexual violence and (v) misogyny and non-sexual violence.

similar to task 1, task 4 is a binary classification task where participants must determine when a meme contains or describes sexist expressions or behaviors (i.e., it is sexist itself, describes a sexist situation or criticizes a sexist behavior).

finally, task 5 addresses the problem of categorizing a sexist meme according to the type of sexism that it enclose: (i) ideological and inequality, (ii) stereotyping and dominance, (iii) objectification, (iv) sexual violence and (v) misogyny and non-sexual violence.

it will focus on predicting the ALSFRS-R score (ALS Functional Rating Scale - Revised), assigned by medical doctors roughly every three months, from the sensor data collected via the app. The ALSFRS-R score is a somehow “subjective” evaluation performed by a medical doctor and this task will help in answering a currently open question in the research community, i.e. whether it could be derived from objective factors.

it will focus on predicting the self-assessment score assigned by patients from the sensor data collected via the app. If the self-assessment performed by patients, more frequently than the assessment performed by medical doctors every three months or so, can be reliably predicted by sensor and app data, we can imagine a proactive application which, monitoring the sensor data, alerts the patient if an assessment is needed.

it will focus on predicting a relapse using environmental data and EDSS (Expanded Disability Status Scale) sub-scores. This task will allow us to assess if exposure to different pollutants is a useful variable in predicting a relapse.

This task is currently at its 20th edition and will continue all the medical sub-tasks in 2023, namely: (i) caption task with medical concept detection and caption prediction, (ii) GAN task on synthetic medical images generated with GANs, (iii) MEDVQA-GI task for medical images generation based on text input, and (iv) Mediqa task with a new use-case on multimodal dermatology response generation.

This task is its third edition, having run at the Touché workshop since 2022 and now, for the first time, as a joint task of Touché and ImageCLEF. This year, the task is, given a set of arguments, to return for each argument several images that help to convey the argument's premise, that is, suitable images could depict what is described in the argument.

This is a task which focuses on content-recommendation for cultural heritage content. Despite current advances in content-based recommendation systems, there is limited understanding how well these perform and how relevant they are for the final end-users. This task aims to fill this gap by benchmarking different recommendation systems and methods.

The objective of ImageCLEFtoPicto is to provide a translation in pictograms from a natural language, either from (i) text or (ii) speech understandable by the users, in this case, people with language impairments as pictogram generation is an emerging and significant domain in natural language processing, with multiple potential applications, enabling communication with individuals who have disabilities, aiding in medical settings for individuals who do not speak the language of a country, and also enhancing user understanding in the service industry.

The aim of the LongEval Task 1 is to support the development of Information Retrieval systems able to face temporal evolution.

The primary aim of the LongEval Task 2 is to assess the persistence of stance detection models in the dynamic landscape of social media posts.

Given an English document, determine at which paragraphs the author changes. Examples vary in difficulty from easy to hard depending on topical homogeneity of the paragraphs.

Given a toxic piece of text, re-write it in a non-toxic way while saving the main content as much as possible. Texts are provided in 7 languages.

Given an English or Spanish online message, determine if it is a conspiracy theory or critical thinking. In former case, find the core elements of the conspiracy narrative.

Given a document, determine if the author is a human or a language model. In collaboration with the ELOQUENT lab.

This task focuses on finding the most relevant subset of features to train a Learning-To-Rank model. It relies on the ISTELLA and LETOR4.0 datasets and targets LambdaMART or a content-based Recommender Systems.

This task focuses on clustering of embeddings, e.g. BERT embeddings or user-item embeddings.

Retrieving passages to include in a simplified summary

Identifying and explaining difficult concepts

Simplify Scientific Text

Tracking the State-of-the-Art in Scholarly Publications (Task, Dataset, Metric, Score)

Given a text, for each sentence, detect which human values the sentence refers to and whether this reference (partially) attains or (partially) constrains the value.

Given a parliamentary speech in one of several languages, identify the ideology of the speaker's party and identify whether the speaker's party is currently governing or in opposition.

Given an argument, retrieve or generate images that help to convey the argument’s premise.