❓ Why the above instructions? If you just tell the agent to install Anserini, it'll likely figure everything out by itself, but there are some choice points it might get confused by, like whether to do a full clone for a dev environment, or just fetch the fatjar. The skills make this clear and provide instructions that make the installation process more token-efficient. If you don't explicitly point your agent at the skills, it may be overeager and wander around...

Also, tell the agent not to install the skill locally, since you only need it once. Otherwise, it may do unnecessary work.

Key:

• 🔑 = keyword queries
• "full" = full 32-bit floating precision
• "int8" = quantized 8-bit precision
• 🫙 = cached queries, 🅾️ = query encoding with ONNX

The MS MARCO V2.1 corpora (documents and segmented documents) were derived from the V2 documents corpus for the TREC 2024 RAG Track. Instructions for downloading the corpus can be found here. The experiments below capture topics and passage-level qrels for the V2.1 segmented documents corpus.

Note that all Arctic-embed-l shards use flat vector indexes.

Key:

• ☂️ = UMBRELA for RAG24, UMBRELA 2.0 for RAG25
• 🔑 = keyword queries
• 🫙 = cached queries, 🅾️ = query encoding with ONNX

The MS MARCO V2.1 corpora (documents and segmented documents) were derived from the V2 documents corpus for the TREC 2024 RAG Track. Instructions for downloading the corpus can be found here. The experiments below capture topics and document-level qrels originally targeted at the V2 documents corpus, but have been "projected" over to the V2.1 documents corpus. These should be treated like dev topics for the TREC 2024 RAG Track; actual qrels for that track were generated at the passage level. There are no plans to generate addition document-level qrels beyond these.

Key:

• 🔑 = keyword queries
• 🫙 = cached queries, 🅾️ = query encoding with ONNX

Sparse representations

Key:

• F1 = "flat" baseline (Lucene analyzer), keyword queries (🔑)
• F2 = "flat" baseline (pre-tokenized with bert-base-uncased tokenizer), keyword queries (🔑)
• MF = "multifield" baseline (Lucene analyzer), keyword queries (🔑)
• U1 = uniCOIL (noexp), cached queries (🫙)
• Spp = SPLADE++ CoCondenser-EnsembleDistil: cached queries (🫙), ONNX (🅾️)
• Sv3 = SPLADE-v3: cached queries (🫙), ONNX (🅾️)

See instructions below the table for how to reproduce results programmatically.

Dense representations

Key:

• BGE (flat) = BGE-base-en-v1.5 (flat vector indexes)
  • original (float32) indexes: cached queries (🫙), ONNX (🅾️)
  • quantized (int8) indexes: cached queries (🫙), ONNX (🅾️)
• BGE (HNSW) = BGE-base-en-v1.5 (HNSW indexes)
  • original (float32) indexes: cached queries (🫙), ONNX (🅾️)
  • quantized (int8) indexes: cached queries (🫙), ONNX (🅾️)

See instructions below the table for how to reproduce results programmatically.

To reproduce the above results programmatically, use the following commands to download and unpack the data:

wget https://rgw.cs.uwaterloo.ca/pyserini/data/$COLLECTION -P collections/
tar xvf collections/$COLLECTION -C collections/

Substitute the appropriate $COLLECTION from the table below.

Once you've unpacked the data, follow the linked reproduction pages above to run and verify the desired BEIR corpus/model combinations.

Substitute the appropriate $MODEL from the table below.

BRIGHT is a retrieval benchmark described here.

• BM25
• SPLADE-v3 = SPLADE-v3: cached queries (🫙), ONNX (🅾️)
• BGE (flat) = BGE-large-en-v1.5 (flat vector indexes): cached queries (🫙), ONNX (🅾️)

The BRIGHT corpora above were processed from Hugging Face with these scripts.

• Reproductions for Mr. TyDi (v1.1) baselines: ar, bn, en, fi, id, ja, ko, ru, sw, te, th
• Reproductions for MIRACL (v1.0) baselines: ar, bn, en, es, fa, fi, fr, hi, id, ja, ko, ru, sw, te, th, zh
• Reproductions for TREC 2022 NeuCLIR Track BM25 (query translation): Persian, Russian, Chinese
• Reproductions for TREC 2022 NeuCLIR Track BM25 (document translation): Persian, Russian, Chinese
• Reproductions for TREC 2022 NeuCLIR Track SPLADE (query translation): Persian, Russian, Chinese
• Reproductions for TREC 2022 NeuCLIR Track SPLADE (document translation): Persian, Russian, Chinese
• Reproductions for HC4 (v1.0) baselines on HC4 corpora: Persian, Russian, Chinese
• Reproductions for HC4 (v1.0) baselines on original NeuCLIR22 corpora: Persian, Russian, Chinese
• Reproductions for HC4 (v1.0) baselines on translated NeuCLIR22 corpora: Persian, Russian, Chinese
• Reproductions for NTCIR-8 ACLIA (IR4QA subtask, Monolingual Chinese)
• Reproductions for CLEF 2006 Monolingual French
• Reproductions for TREC 2002 Monolingual Arabic
• Reproductions for FIRE 2012 monolingual baselines: Bengali, Hindi, English
• Reproductions for CIRAL (v1.0) BM25 (query translation): Hausa, Somali, Swahili, Yoruba
• Reproductions for CIRAL (v1.0) BM25 (document translation): Hausa, Somali, Swahili, Yoruba

• Reproductions for Disks 1 & 2 (TREC 1-3), Disks 4 & 5 (TREC 7-8, Robust04), AQUAINT (Robust05)
• Reproductions for the New York Times Corpus (Core17), the Washington Post Corpus (Core18)
• Reproductions for Wt10g, Gov2
• Reproductions for ClueWeb09 (Category B), ClueWeb12-B13, ClueWeb12
• Reproductions for Tweets2011 (MB11 & MB12), Tweets2013 (MB13 & MB14)
• Reproductions for Complex Answer Retrieval (CAR17): v1.5, v2.0, v2.0 with doc2query
• Reproductions for TREC News Tracks (Background Linking Task): 2018, 2019, 2020
• Reproductions for FEVER Fact Verification
• Reproductions for DPR Wikipedia QA baselines: 100-word splits, 6/3 sliding window sentences

• Reproducing BM25 baselines for MS MARCO Passage Ranking
• Reproducing BM25 baselines for MS MARCO Document Ranking
• Reproducing baselines for the MS MARCO Document Ranking Leaderboard
• Reproducing doc2query results (MS MARCO Passage Ranking and TREC-CAR)
• Reproducing docTTTTTquery results (MS MARCO Passage and Document Ranking)
• Notes about reproduction issues with MS MARCO Document Ranking w/ docTTTTTquery

• Reproducing BM25 baselines on the MS MARCO V2 Collections

• Indexing AI2's COVID-19 Open Research Dataset
• Baselines for the TREC-COVID Challenge
• Baselines for the TREC-COVID Challenge using doc2query

• Working with the 20 Newsgroups Dataset
• Guide to BM25 baselines for the FEVER Fact Verification Task
• Guide to reproducing "Neural Hype" Experiments
• Guide to running experiments on the AI2 Open Research Corpus
• Experiments from Yang et al. (JDIQ 2018)
• Runbooks for TREC 2018: [Anserini group] [h2oloo group]
• Runbook for ECIR 2019 paper on axiomatic semantic term matching
• Runbook for ECIR 2019 paper on cross-collection relevance feedback
• Support for approximate nearest-neighbor search on dense vectors with inverted indexes

• v0.39.0: January 12, 2025 [Release Notes]
• v0.38.0: September 6, 2024 [Release Notes]
• v0.37.0: August 22, 2024 [Release Notes]
• v0.36.1: May 23, 2024 [Release Notes]
• v0.36.0: April 28, 2024 [Release Notes]
• v0.35.1: April 24, 2024 [Release Notes]
• v0.35.0: April 3, 2024 [Release Notes]
• v0.25.0: March 27, 2024 [Release Notes]
• v0.24.2: February 27, 2024 [Release Notes]
• v0.24.1: January 27, 2024 [Release Notes]
• v0.24.0: December 28, 2023 [Release Notes]
• v0.23.0: November 16, 2023 [Release Notes]
• v0.22.1: October 18, 2023 [Release Notes]
• v0.22.0: August 28, 2023 [Release Notes]
• v0.21.0: March 31, 2023 [Release Notes]
• v0.20.0: January 20, 2023 [Release Notes]
• v0.16.2: December 12, 2022 [Release Notes]
• v0.16.1: November 2, 2022 [Release Notes]
• v0.16.0: October 23, 2022 [Release Notes]
• v0.15.0: September 22, 2022 [Release Notes]
• v0.14.4: July 31, 2022 [Release Notes]
• v0.14.3: May 9, 2022 [Release Notes]
• v0.14.2: March 24, 2022 [Release Notes]
• v0.14.1: February 27, 2022 [Release Notes]
• v0.14.0: January 10, 2022 [Release Notes]
• v0.13.5: November 2, 2021 [Release Notes]
• v0.13.4: October 22, 2021 [Release Notes]
• v0.13.3: August 22, 2021 [Release Notes]
• v0.13.2: July 20, 2021 [Release Notes]
• v0.13.1: June 29, 2021 [Release Notes]
• v0.13.0: June 22, 2021 [Release Notes]
• v0.12.0: April 29, 2021 [Release Notes]
• v0.11.0: February 13, 2021 [Release Notes]
• v0.10.1: January 8, 2021 [Release Notes]
• v0.10.0: November 25, 2020 [Release Notes]
• v0.9.4: June 25, 2020 [Release Notes]
• v0.9.3: May 26, 2020 [Release Notes]
• v0.9.2: May 14, 2020 [Release Notes]
• v0.9.1: May 6, 2020 [Release Notes]
• v0.9.0: April 18, 2020 [Release Notes]
• v0.8.1: March 22, 2020 [Release Notes]
• v0.8.0: March 11, 2020 [Release Notes]
• v0.7.2: January 25, 2020 [Release Notes]
• v0.7.1: January 9, 2020 [Release Notes]
• v0.7.0: December 13, 2019 [Release Notes]
• v0.6.0: September 6, 2019 [Release Notes] [Known Issues]
• v0.5.1: June 11, 2019 [Release Notes]
• v0.5.0: June 5, 2019 [Release Notes]
• v0.4.0: March 4, 2019 [Release Notes]
• v0.3.0: December 16, 2018 [Release Notes]
• v0.2.0: September 10, 2018 [Release Notes]
• v0.1.0: July 4, 2018 [Release Notes]

• Anserini was upgraded to Lucene 10.4.0 at c6eed6 (2026/04/12) as part of v2.0.0. Lucene 9 indexes remain readable, but older code is unable to read indexes generated by Lucene 10.
• Anserini was upgraded from JDK 11 to JDK 21 at commit 272565 (2024/04/03), which corresponds to the release of v0.35.0.
• Anserini was upgraded to Lucene 9.3 at commit 272565 (8/2/2022): this upgrade created backward compatibility issues, see #1952. Anserini will automatically detect Lucene 8 indexes and disable consistent tie-breaking to avoid runtime errors. However, Lucene 9 code running on Lucene 8 indexes may give slightly different results than Lucene 8 code running on Lucene 8 indexes. Lucene 8 code will not run on Lucene 9 indexes. Pyserini has also been upgraded and similar issues apply: Lucene 9 code running on Lucene 8 indexes may give slightly different results than Lucene 8 code running on Lucene 8 indexes.
• Anserini was upgraded to Java 11 at commit 17b702d (7/11/2019) from Java 8. Maven 3.3+ is also required.
• Anserini was upgraded to Lucene 8.0 as of commit 75e36f9 (6/12/2019); prior to that, the toolkit uses Lucene 7.6. Based on preliminary experiments, query evaluation latency has been much improved in Lucene 8. As a result of this upgrade, results of all reproductions have changed slightly. To reproducible old results from Lucene 7.6, use v0.5.1.