Yi Zhu

Yi Zhu

Cofounder at Boson AI

Posts by Tags

AI

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.

Chatbot Arena and the Elo rating system - Part 1

13 minute read

Published:

Chatbot Arena, developed by members from LMSYS and UC Berkeley SkyLab, is a benchmark platform designed to evaluate large language models (LLMs) through anonymous, randomized battles in a crowdsourced environment. Launched in May 2023, it has been continuously updated to reflect the latest advancements in the field. The platform’s leaderboard is widely regarded as one of the most credible sources for ranking LLMs. The screenshot below highlights the competitive landscape featuring major players in the LLM space.

Bootstrap

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.

Bradley-Terry model

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.

Chatbot arena

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.

Chatbot Arena and the Elo rating system - Part 1

13 minute read

Published:

Chatbot Arena, developed by members from LMSYS and UC Berkeley SkyLab, is a benchmark platform designed to evaluate large language models (LLMs) through anonymous, randomized battles in a crowdsourced environment. Launched in May 2023, it has been continuously updated to reflect the latest advancements in the field. The platform’s leaderboard is widely regarded as one of the most credible sources for ranking LLMs. The screenshot below highlights the competitive landscape featuring major players in the LLM space.

Elo

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.

Chatbot Arena and the Elo rating system - Part 1

13 minute read

Published:

Chatbot Arena, developed by members from LMSYS and UC Berkeley SkyLab, is a benchmark platform designed to evaluate large language models (LLMs) through anonymous, randomized battles in a crowdsourced environment. Launched in May 2023, it has been continuously updated to reflect the latest advancements in the field. The platform’s leaderboard is widely regarded as one of the most credible sources for ranking LLMs. The screenshot below highlights the competitive landscape featuring major players in the LLM space.

LLM evaluation

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.

Chatbot Arena and the Elo rating system - Part 1

13 minute read

Published:

Chatbot Arena, developed by members from LMSYS and UC Berkeley SkyLab, is a benchmark platform designed to evaluate large language models (LLMs) through anonymous, randomized battles in a crowdsourced environment. Launched in May 2023, it has been continuously updated to reflect the latest advancements in the field. The platform’s leaderboard is widely regarded as one of the most credible sources for ranking LLMs. The screenshot below highlights the competitive landscape featuring major players in the LLM space.

Maximum Likelihood Estimation

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.

Win rate

Chatbot Arena and the Elo rating system - Part 2

14 minute read

Published:

In our previous blog post on Elo rating system, we introduced the basics of the Elo rating system and its online linear update algorithm, hereafter referred to as “online Elo”. However, we identified a significant concern with online Elo: its instability and tendency to bias toward recent results. For example, a demonstration from Chatbot Arena showed substantial shifts in model ratings when Elo was recalculated using the reverse order of matches.