Cross-university platform portalShow a desktop notification when the AI TA finishes replying
Notify me when classmates post messages in the forum
Play an alert sound whenever there is a new notification
Explain how Uedu uses the ELO rating system and blind A/B comparisons to let users assess the response quality of different AI Prompts, similar to LM Arena / Chatbot Arena.
The Prompt Leaderboard (ELO Leaderboard) on Uedu is a Prompt evaluation tool centred on the ELO rating system, inspired by LM Arena (formerly Chatbot Arena). Through blind A/B comparisons, users judge which of two different Prompts produces the better AI response, and the system dynamically adjusts each Prompt's ELO score accordingly.
This mechanism enables teachers and students to use crowdsourced evaluation to identify the Prompt design best suited to a specific task, while also providing quantitative data for research into Prompt Engineering.
The ELO rating system was originally used to rank chess players. Its core idea is: after each game, adjust both players' scores according to the difference between the actual result and the expected result. The winner gains points, the loser loses points, and the margin is larger for an upset.
| Parameter | Value | Description |
|---|---|---|
| K value | 32 | Maximum score change in each match |
| Initial score | 1500 | Initial ELO score for the new Prompt |
| Minimum number of matches | 30 | A prompt must accumulate at least 30 blind tests before entering the leaderboard |
The blind test removed participants’ bias towards a particular Prompt author or name, ensuring that judgements were based purely on response quality. This is consistent with the spirit of double-blind experiments in academic research.
In a match between A vs. B, A's Expected Score is:
EA = 1 / (1 + 10(RB - RA) / 400)
Here, RA and RB are the current ELO ratings for A and B respectively. Likewise, B's expected score EB = 1 - EA.
Update A's ELO score according to the actual score in the match:
ΔRA = K × (SA - EA)
Here, SA is A's actual score:
| Result | SA | SB |
|---|---|---|
| A wins | 1.0 | 0.0 |
| B wins | 0.0 | 1.0 |
| Tie | 0.5 | 0.5 |
Assume Prompt A (ELO 1500) vs Prompt B (ELO 1600), and the user chooses A to win (upset):
Because A has a lower expected win rate, an underdog victory earns more points.
The questions used in the blind test are divided into the following types to ensure the Prompt is evaluated across different task dimensions:
| Type code | Description |
|---|---|
knowledge | Knowledge-based questions (fact recall, concept explanation) |
coding | Programming questions (writing code, debugging, explaining code) |
writing | Writing-related questions (writing, editing, translation) |
reasoning | Reasoning questions (logical deduction, mathematics, causal analysis) |
creative | Creative questions (brainstorming, story creation, proposal design) |
Each Prompt can be tagged with multiple labels, making it easier for users to filter and for researchers to analyse:
| Tags | Description |
|---|---|
beginner_friendly | Suitable for beginners |
coding_expert | Good at programming-related tasks |
detailed | Tends to provide detailed, comprehensive responses |
concise | Tends to provide concise, succinct responses |
academic | Suitable for academic research contexts |
The system collects basic demographic information from blind-test participants (user profiles) as background variables for research analysis, such as usage experience and disciplinary field.
To ensure ranking reliability, a Prompt must meet the following conditions before it appears on the public leaderboard:
Prompts below the threshold will continue to take part in blind-test pairings, but their ELO score will be marked as “provisional” and will not be included in the official ranking.
According to the central limit theorem, more than 30 independent observations can initially approximate a normal distribution, giving ELO scores a degree of statistical stability.
AI Prompt quality assessment uses the Uedu platform's ELO Leaderboard module, referring to the blind comparison mechanism of LM Arena (Chatbot Arena). The system randomly pairs two Prompts, generates AI responses for each, and presents them to users in a blind test, where users judge the winner or a tie. The ELO scoring parameters are K=32 and an initial score of 1500. Expected scores are calculated using the formula EA = 1/(1+10(RB-RA)/400), and scores are then updated using ΔR = K×(S-E). A Prompt must accumulate at least 30 blind-test rounds before entering the official ranking. Blind-test questions cover five types: knowledge, programming, writing, reasoning, and creativity. See https://uedu.tw/doc/leaderboard for the full methodology.
It is recommended to provide the following: