Other Venues

Beyond Kaggle and CLEF, several other platforms and major academic conferences play vital roles in the applied AI/ML research landscape.

TREC (Text REtrieval Conference)

  • Focus: TREC is a long-standing series of workshops, initiated in 1992 and run by the U.S. National Institute of Standards and Technology (NIST). Its core mission is to support and encourage research within the information retrieval (IR) community by providing the necessary infrastructure for large-scale evaluation of text retrieval methodologies and to accelerate the transfer of technology from research labs to commercial products.
  • Characteristics: TREC is organized into "tracks," each focusing on a particular subproblem or variant of the retrieval task. Over the years, tracks have covered diverse areas such as ad-hoc retrieval, question answering, cross-language IR, genomics IR, legal IR, and web search. For TREC 2024, continuing tracks included AToMiC (Authoring Tools for Multimedia Content) and NeuCLIR (Neural Cross-Language Information Retrieval), while new tracks included RAG (Retrieval-Augmented Generation). NIST typically provides large text collections and a set of questions (topics). Participating groups run their retrieval systems on this data and submit their results (e.g., ranked lists of documents). NIST then performs uniform scoring, often using evaluation techniques like pooling, where relevance judgments are made on a subset of documents retrieved by multiple systems.
  • Outputs: Participants submit ranked lists of documents or other outputs specific to the track's task. The results, methodologies, and experiences are shared and discussed at the annual TREC workshop, and overview papers for each track are published in the TREC proceedings.
  • TREC has been instrumental in advancing IR research by creating valuable, large-scale test collections and fostering a collaborative evaluation environment. Its track-based structure allows for focused research on a wide array of IR challenges. The introduction of the RAG track in 2024 is a clear indication of TREC's responsiveness to current trends in AI, particularly the integration of LLMs with retrieval systems.

KDD Cup

  • Focus: The KDD Cup is the premier annual competition in data mining and knowledge discovery, organized by the Association for Computing Machinery's Special Interest Group on Knowledge Discovery and Data Mining (ACM SIGKDD). Its primary aim is to stimulate research and development in these fields by presenting challenging problems derived from diverse domains.
  • Characteristics: KDD Cup challenges are renowned for often involving large, complex datasets and tasks that push the boundaries of current data mining techniques. Historical examples include the 2010 KDD Cup, which focused on predicting student answer correctness using one of the largest educational datasets at the time. More recently, the KDD Cup 2024 featured the "Open Academic Graph Challenge (OAG-Challenge)" for academic graph mining and the "Multi-Task Online Shopping Challenge for LLMs" hosted by Amazon.
  • Outputs: Participants develop solutions to the posed problems, and the results and winning approaches are typically presented at a dedicated workshop during the annual KDD conference.
  • The KDD Cup holds significant prestige and its challenges often set new directions in data mining research. The 2024 LLM challenge for online shopping, for instance, highlights the platform's alignment with contemporary advancements in AI.

NeurIPS Competitions

  • Focus: Hosted as part of the Neural Information Processing Systems (NeurIPS) conference, one of the top-tier venues for machine learning research, these competitions aim to advance modern AI and ML algorithms. There is a strong encouragement for proposals that address clear scientific questions and have a positive societal impact, particularly those leveraging AI to support disadvantaged communities or to advance other scientific, technological, or business domains relevant to the NeurIPS community.
  • Characteristics: NeurIPS features a dedicated Competition Track, with each accepted competition typically having an associated workshop where results are presented and discussed by organizers and participants. The tasks are often novel, cutting-edge, and interdisciplinary. Examples include the 2024 challenge on predicting hi-resolution rain radar movies from multi-band satellite sensors, requiring data fusion and video frame prediction , and past competitions on causal structure learning, multi-agent reinforcement learning (e.g., the "Melting Pot Contest" ), and foundation model prompting for medical image classification.
  • Outputs: Competition results are presented at the NeurIPS workshops. Organizers and participants also have the option to submit post-competition analysis papers to the NeurIPS Datasets and Benchmarks (D\&B) track in the subsequent year.
  • NeurIPS competitions are situated at the forefront of ML research, frequently exploring emerging areas and placing a strong emphasis on scientific rigor, methodological innovation, and potential societal benefits.

CVPR/ICCV Challenges

  • Focus: The Conference on Computer Vision and Pattern Recognition (CVPR) and the International Conference on Computer Vision (ICCV) are the premier international conferences in the field of computer vision. Both conferences host a multitude of workshops, many of which include associated challenges.
  • Characteristics: These challenges cover an extensive range of computer vision tasks. Examples from CVPR 2024 workshops include challenges on 3D scene understanding (e.g., the ScanNet++ Novel View Synthesis and 3D Semantic Understanding Challenge ), efficient large vision models, human modeling and motion generation, multimodal learning, and various application-driven challenges in domains like agriculture (Agriculture-Vision ), sports (CVsports ), retail (RetailVision ), autonomous driving (WAD workshops often feature multiple challenges, e.g., End-To-End Driving at Scale, Occupancy and Flow ), and medical imaging. These challenges typically involve large-scale, highly specialized image or video datasets.
  • Outputs: Participants submit their solutions, which are evaluated based on task-specific metrics. Winners and notable solutions are often announced and presented at the corresponding workshops , and results may be summarized in workshop proceedings or overview papers.
  • Challenges at CVPR and ICCV are central to driving progress in computer vision, pushing the state-of-the-art in specific sub-fields, and providing crucial benchmarks for new algorithms and techniques. The sheer breadth of topics covered in the CVPR 2024 workshop list attests to the dynamism and scope of research in this area.