The contamination argument against MMLU cuts both ways: HotpotQA and Bamboogle are also static, public benchmarks sitting in training corpora by now. If leaderboard freshness is the reason to trust Arena over MMLU, the same logic should make you suspicious of the multi-hop QA numbers used to back the RAG claims.
Note that if you pass a prompt to Anthropic’s API, the header should return your prompt tokens used, which can be used to do comparisons across different language datasets
The contamination argument against MMLU cuts both ways: HotpotQA and Bamboogle are also static, public benchmarks sitting in training corpora by now. If leaderboard freshness is the reason to trust Arena over MMLU, the same logic should make you suspicious of the multi-hop QA numbers used to back the RAG claims.
Note that if you pass a prompt to Anthropic’s API, the header should return your prompt tokens used, which can be used to do comparisons across different language datasets
Thanks for pointing that out. We'll look into that.