In this module, we will introduce the concept of an ARC (Applied Research Competitions) and how our group participates in CLEF.
Module 0 - What's an ARC?
See also: CLEF Venue
Applied Research Competitions
An Applied Research Competition is where teams of students participate in ML research competitions similar in spirit to Kaggle competitions. This course specifically serves primarily to help prepare students to focus on the CLEF competition in the Spring semesters.
For new researchers, ARCs are an excellent introduction to research.
- Identifying relevant research questions and projects is difficult for non-experts.
- Data acquisition may be difficult or impossible for researchers.
ARCs typically address these issue for the participants by providing a well-defined research problem, a dataset, and (usually) an automatic venue for evaluation. This way the participants can focus on what they do best: ML research and experimentation.
A skeptic may ask, "Is this real research?". An ARC may seem less authentic at first glance; some of the hardest parts of research have been addressed for you. Firstly, for some domains, such as medical imaging, ARC style conferences and forums are the only way for researchers to access high-quality datasets, without a direct relationship with a hospital or medical research center.
Secondly, the ML community in particular has always embraced competitions as a legitimate avenue for research. AlexNet, often considered the breakout moment for deep learning, was developed for the ImageNet competition. Competitions both inspire significant advancements in the field, and prove the effectiveness of new methods. The ARC competitions are no different. The research questions are real, the datasets are real, and the paper students publish are real.
CLEF: Conference and Labs of the Evaluation Forum
The Conference and Labs of the Evaluation Forum, or CLEF, is both a research conference and an umbrella organization for Applied Research Competitions.
The structure of CLEF labs
CLEF is not one competition. It contains multiple labs, each organized around a broad research area. A lab may then contain several tasks and subtasks, each with its own dataset, evaluation metrics, baselines, and leaderboards. Some labs are well established and have a very mature process, others are new and still evolving.
An individual CLEF lab is therefore much closer to a research workshop than to a single leaderboard. The lab organizers define the scientific problems, construct or curate datasets, establish evaluation procedures, and publish overview papers comparing the submitted systems. Participants submit experimental runs and may publish peer-reviewed working-note papers describing their methods and results.
Unlike a typical Kaggle competition, the primary product is not only a score. It is a reproducible research method, an experimental analysis, and a paper explaining what was learned.
Below is a summary of the CLEF 2026 labs, their primary research areas, representative task types, and the number of papers published in the lab overview proceedings. The "Share" column indicates the proportion of papers in the lab relative to the total number of papers in the proceedings. Note that some labs are new and do not yet have overview papers. This is just an estimate but it gives us a rough estimate of the relative size of each lab in the CLEF ecosystem.
| Lab | Primary research area | Representative task types | Total papers | Share |
|---|---|---|---|---|
| BioASQ | Biomedical information access | Biomedical retrieval, QA, summarization, coding, linking, extraction | 50 | 12.8% |
| CheckThat! | Fact-checking and disinformation | Source retrieval, claim verification, trace evaluation, fact-check writing | 53 | 13.5% |
| ELOQUENT | Generative language-model evaluation | Robustness, detector evasion, item generation, answer scoring | 13 | 3.3% |
| eRisk | Early risk prediction | Screening, user classification, temporal modeling, evidence retrieval | 18 | 4.6% |
| EXIST | Sexism identification in social networks | Meme/video classification, disagreement modeling, sensor fusion | 34 | 8.7% |
| FinMMEval | Financial AI evaluation | Financial QA, grounded generation, time-series reasoning, live agents | -- | -- |
| HIPE | Historical information extraction | Relation extraction, temporal reasoning, knowledge-base enrichment | -- | -- |
| ImageCLEF | Multimodal and visual AI | Medical imaging, VQA, captioning, deepfakes, remote sensing | 49 | 12.5% |
| JOKER | Computational humor and wordplay | Humor retrieval, pun translation, name wordplay, humor generation | 13 | 3.3% |
| LifeCLEF | Biodiversity monitoring | Re-identification, soundscapes, object detection, plant recognition | 34 | 8.7% |
| LongEval | Longitudinal information retrieval | Evolving retrieval, topic formulation, query simulation, temporal RAG | 13 | 3.3% |
| PAN | Stylometry and digital text forensics | AI-text detection, watermarking, style analysis, plagiarism retrieval | 61 | 15.6% |
| qCLEF | Quantum information retrieval | Quantum optimization, feature selection, recommendation, clustering | 6 | 1.5% |
| SimpleText | Scientific-text accessibility | Simplification, hallucination detection, research-area classification | 18 | 4.6% |
| TalentCLEF | Human-capital information retrieval | Job matching, resume ranking, skill retrieval, fairness, ontology ranking | 16 | 4.1% |
| Touché | Computational argumentation | Fallacies, causality, argument mining, ad detection in RAG | 14 | 3.6% |
Information Retrieval in CLEF
Machine learning is a huge and diverse field. In general, many conferences focus on a specific area and CLEF is no different. While the diversity of the labs have grown over the years, the conference is still primarily focused on information retrieval and related areas.
Indeed when going to the official CLEF website, and viewing books they have published we see the following books.
Two of the books have "information retrieval" in the title. The ImageCLEF book has "information retrieval" in the subtitle as well. The first book on the eRisk project seems different. However a cursory look at https://erisk.irlab.org (want to guess what the "ir" in the lab URL stands for?) shows that it is focused on early risk detection in online communities, yet another problem in information retrieval.
That being said, the term "Information Retrieval" is doing a lot of work here, as many of the IR tasks are applied across a very broad range of computer vision, natural language processing, and multi-modal tasks. IR does not just mean "search engines" or "document retrieval" or "ranking". Furthermore, there are tasks which have no IR component at all.
What kinds of ML should we expect in CLEF?
The table below summarizes an automated analysis of the CLEF 2026 organizer overview papers. This was generated via codex, so there may be some errors, but the point here is just to have a rough idea of the kinds of techniques that are being used in CLEF labs.
| Technique family | Report units | Frequency | Labs containing the technique | Typical role in a top system |
|---|---|---|---|---|
| Ensembling or score fusion | 39/49 | 79.6% | 15/16 | Model averaging, reciprocal-rank fusion, multimodal fusion, candidate-judge pipelines, and global-local feature fusion |
| Fine-tuning or parameter-efficient fine-tuning | 33/49 | 67.3% | 13/16 | Full fine-tuning, LoRA, QLoRA, task-adaptive pretraining, supervised output heads, and knowledge distillation |
| Generative LLMs or VLMs | 32/49 | 65.3% | 14/16 | Answer generation, reasoning, judging, query expansion, document structuring, and synthetic-data creation |
| Dense embeddings | 32/49 | 65.3% | 16/16 | Bi-encoder retrieval, semantic matching, contextual representations, CLIP-like embeddings, and metric representations |
| Classical models, rules, or engineered features | 20/49 | 40.8% | 14/16 | Regex, n-grams, temporal priors, logistic regression, random forests, LightGBM, and task-specific heuristics |
| Reranking or verification | 18/49 | 36.7% | 10/16 | Cross-encoders, LLM judges, NLI verification, reward models, and pointwise, pairwise, or listwise reranking |
| RAG or explicit grounding | 17/49 | 34.7% | 10/16 | Retrieved evidence, citation support, knowledge-grounded generation, and retrieval-conditioned prompting |
| Foundation vision or audio encoders | 13/49 | 26.5% | 3/16 | CLIP, DINO, SigLIP, Perch, WavLM, wav2vec, and Earth-observation foundation models |
| Sparse or lexical retrieval | 12/49 | 24.5% | 9/16 | BM25, TF-IDF, RM3 query expansion, lexical filtering, and first-stage candidate generation |
| Structured knowledge | 11/49 | 22.4% | 6/16 | Ontologies, knowledge graphs, taxonomies, dictionaries, UMLS, ICD, ESCO, and Wikidata |
| Multimodal fusion | 10/49 | 20.4% | 3/16 | Text-image, image-metadata, audio-time, report-scan, and video-transcript fusion |
| Agentic systems or tool use | 8/49 | 16.3% | 6/16 | Planner-writer-critic architectures, search tools, memory retrieval, multi-agent voting, and deployed decision agents |
| Contrastive or metric learning | 6/49 | 12.2% | 6/16 | InfoNCE, GIST, ArcFace, SphereFace, supervised contrastive losses, and pair-representation learning |
| Temporal modeling or memory | 5/49 | 10.2% | 5/16 | Longitudinal histories, recurrent models, temporal retrieval filters, session state, and persistent memory stores |
| Clustering, k-nearest neighbors, or transductive inference | 5/49 | 10.2% | 3/16 | FAISS or kNN retrieval, graph clustering, community detection, pseudo-label propagation, and transductive inference |
| Reinforcement learning or preference optimization | 3/49 | 6.1% | 3/16 | GRPO, DPO, reward-guided generation, and detector-reward optimization |
| Generative image, medical-image, or audio models | 2/49 | 4.1% | 1/16 | Synthetic CT generation, image generation, identity-conditioned generation, and speech cloning |
| Quantum or QUBO optimization | 1/49 | 2.0% | 1/16 | Quantum annealing, simulated annealing, and hybrid optimization over QUBO objectives |
A few things stand out:
- IR techniques are indeed very common.
- Many solutions are combining multiple data modalities.
- Foundation models are often used as feature extractors or embedding generators.
- Fine-tuning of foundation models is common practice.
While we will not focus exclusively on these topics, hopefully this gives you a sense of the kinds of techniques that are being used in CLEF labs and what we will focus on.
Optional Exercise
Identify a CLEF lab that you find interesting and read the overview paper from this year. You can find the CLEF papers at clef-staging.pages.dev.