Cross-university platform portalShow a desktop notification when the AI TA finishes replying
Notify me when classmates post messages in the forum
Play an alert sound whenever there is a new notification
Explain the design principles of the pre- and post-test Quiz for Socratic Dialogue, including the AI automated question-generation mechanism, pre/post test pairing logic, Learning Gain calculation method, and research application recommendations.
The Socratic Dialogue pre- and post-test system asks Students to complete a short Quiz before and after each Socratic Dialogue (mode=3) session (default 5 multiple-choice questions), to quantify the teaching effectiveness of Socratic Dialogue.
The system supports two question-setting methods:
Each new conversation has its own independent pre-/post-test pairing, linked via chat_id. Students may have multiple conversations on the same topic, and each will generate a new set of pre-/post-test data.
The Student's workflow is as follows:
After the pre-test is completed, only the number of correct items is shown; the correct answers and explanations are not disclosed. This is to prevent Students from memorising the answers and affecting the measurement validity of the post-test. Only after the post-test is completed will the full item-by-item comparison and explanations be shown.
In the Socratic Dialogue topic card on the Course Management page, Instructors can manage pre- and post-tests:
This quiz does not count towards the student's grade and is for teaching research use only. Students will see this explanation before answering.
The system uses the GPT-4o-mini model (configurable) to automatically generate multiple-choice questions based on the four fields of the dialogue topic:
| Field | Purpose |
|---|---|
title | Main topic title, as the core concept for question setting |
prompt | Background information, providing conceptual context |
rule | Conversation rules to help AI understand the teaching objectives |
guide | Guiding questions to prompt the core knowledge points |
| Field | Type | Description |
|---|---|---|
question_text | String | Question text |
options | JSON | {"A": "...", "B": "...", "C": "...", "D": "..."} |
correct_answer | String | Correct answer (A/B/C/D) |
explanation | String | Answer Explanation |
source | Enum | ai or manual |
The pre-test and post-test use the same questions, and the scoring method is the number of correct items. After the post-test is completed, the system calculates the following indicators:
| Indicator | Formula | Description |
|---|---|---|
| Learning Gain | Post-test score - pre-test score | Difference in raw scores |
| Normalised Gain (Normalized Gain) | (Post-test - Pre-test) / (Full mark - Pre-test) | Consider standardised metrics that account for ceiling effects, range 0-1 |
Proposed by Hake (1998), widely used in physics education research. <g> < 0.3 indicates low gain, 0.3 ≤ <g> < 0.7 indicates medium gain, and <g> ≥ 0.7 indicates high gain. When the pre-test is already full marks, gain is defined as 1.0 (if the post-test is also full marks) or 0.0.
| Field | Type | Description |
|---|---|---|
topic_id | INT | Associated Socratic Dialogue topic ID |
question_order | TINYINT | Question order |
question_text | TEXT | Question text |
options | JSON | Four options |
correct_answer | VARCHAR(1) | Correct Answer |
explanation | TEXT | Answer Explanation |
source | ENUM | ai / manual |
| Field | Type | Description |
|---|---|---|
topic_id | INT | Conversation topic ID |
chat_id | INT | Associated dialogue session ID |
user_id | INT | Student ID |
phase | ENUM | pre / post |
answers | JSON | Answers for each item, such as {"1":"A","2":"C"} |
score | TINYINT | Number of correct answers |
time_spent | INT | Response time (seconds) |
Pre- and post-test data can be exported via Uedu Lab for the following research analyses:
Pre- and post-test data are Learning history data. Before use, please confirm that IRB review has been passed and that informed consent has been obtained from Students. On the pre- and post-test pages, the system has marked: "This Quiz does not count towards the grade and is for teaching research use only."