Forum Scoring
Forum Assessment Methodology

1. Overview

Uedu's discussion forum scoring system automatically calculates Students' participation grades in the Course Forum through a weighted points system. The system takes into account post type (topic post, comment, reply), Emoji feedback from instructors and AI, and BERT semantic quality evaluation, and finally produces grade scores through linear mapping.

Instructors can customise the scoring parameters for each Course (point weighting, thresholds, maximum score), and can also use the emoji mechanism to express approval or penalty for individual posts in real time. The entire marking process is automatically recalculated every hour by a scheduled task (Cron Job) to ensure marks are updated in real time.

2. Scoring model

2.1 Basic points

A Student's base points (total_points) are accumulated from three posting behaviours, each with a configurable point value:

Behaviour	Parameter name	Default value	Description
Start a topic	pts_thread	3	Create a new discussion topic in the Forum
Comments	pts_comment	2	Post a comment under someone else's topic
Reply	pts_reply	1	Reply to other people's comments

Basic integration formula:

total_points = (threads × pts_thread) + (comments × pts_comment) + (replies × pts_reply) + emoji_bonus

2.2 Scoring target

Users with the following identities are not counted in the assessment:

• Instructor (teacher)
• TA
• Administrator (admin)
• AI user

The Emoji behaviour of these privileged users affects Student points (see Section 3), but their own posts do not count towards scoring.

2.3 Configurable forums

Instructors can choose which discussion boards (boards) to include in the marking scope; posts on boards not selected will not count towards points. This setting is course-based.

3. Emoji rewards and penalties

3.1 Bonus Emoji

Privileged users (instructors, TAs, AI) can award bonus points by reacting with Emoji on student posts:

Emoji	Name	User enabled	Effect
👍	thumbsup	Teacher, TA, AI	Bonus (each time +1 bonus)
🎊	confetti	Teacher, TA	Bonus (each time +1 bonus)
🚀	rocket	Teacher, TA	Bonus (each time +1 bonus)

Each Student's total Emoji bonus is capped by bonus_max (default 10); any bonus above the cap will no longer accumulate.

3.2 Penalty Emoji

Emoji	Name	User enabled	Effect
😴	zzz	Teacher, TA, AI	Set this post's points to zero (0 points)

When a post is marked with the zzz emoji, the base points and all bonus Emojis for that post are excluded, and the entire post is scored as 0.

3.3 Maximum bonus points

To avoid Emoji bonuses from becoming excessively inflated, the system provides a bonus_max parameter (configurable by the Instructor). The total Emoji bonus a Student receives across the entire Course will not exceed this cap.

4. Instructor review mechanism

When AI automatically reacts to student posts with Emojis (such as 👍 or 😴), the Instructor may disagree with AI's judgement. The system provides the Eyes (👀) Emoji as an Instructor review mechanism:

• When the Instructor or TA clicks the 👀 (eyes) emoji on a post, all AI emojis on that post will be ignored
• The instructor's own Emoji are not affected
• This mechanism allows teachers to quickly override AI's automatic judgements without having to remove AI Emoji one by one

4.1 Student review request

Students can proactively apply to the Instructor for review, with two use cases:

Scenario 1: AI judges the quality to be insufficient (💤 zzz)

When a Student’s post is judged by AI to be of insufficient quality and reduced to zero, the Student can click the “Request Instructor Review” button. After review, the Instructor may choose:

• Review approved — the system automatically adds the instructor 👀 reaction to the post, AI's 😴 judgement is overridden, and normal scoring is restored
• Maintain AI determination — the post remains 0 points; on the Student side, “Teacher has maintained the AI determination” is displayed

Scenario 2: Request teacher attention

Even if a post is not marked by AI 😴, Students can still click “Request Instructor attention for this post”. Please pay special attention to the quality of this post’s content; after review, the Instructor may award extra Emoji points (e.g. 👍, 🎉, 🚀).

4.2 How instructors receive notifications

After a student submits a review request, the instructor and TA receive notifications in three ways:

1. Forum prompt — when an instructor enters the Forum, an orange banner at the top shows "N posts awaiting review", which can be clicked to view and handle them directly
2. Email notifications — automatically send emails to the Instructor and TA (can be turned off in Forum notification preferences)
3. Teacher Console badge — A red numeric badge shown next to "Course Forum" in the sidebar to indicate the number pending review

5. AI quality evaluation

The system integrates BERT semantic quality assessment, using an external API (https://aptel-api.uedu.tw/webhook) to carry out automated quality analysis of Students' posts.

5.1 Evaluation process

1. After a student posts in the Forum, the system sends the post content to the BERT quality assessment API
2. Quality scores and classification results returned by the API
3. AI automatically assigns the corresponding emoji to posts based on the assessment result (👍 indicates high quality, 😴 indicates low quality)
4. Instructors can override AI’s judgement through the 👀 review mechanism

5.2 Quality dimensions

BERT evaluates the semantic quality of posts, mainly considering depth, relevance and originality. The evaluation results serve as the basis for AI automatic Emoji assignment, but the final points are still determined by the Emoji mechanism.

6. AI originality detection

To encourage students to participate in discussions in their own words, the system adds typing speed analysis and LLM originality detection in addition to BERT quality assessment, to determine whether a post has been generated by AI and then pasted. Detection results are visible only to instructors and TAs; students cannot see them at all.

6.1 Stage 0: Typing speed analysis

The front end records the time from when a Student opens the editor to when they submit the post, and calculates the number of characters per minute (chars/min):

Condition	Determination
Word count > 100 and words per minute > 150	paste_suspected
Other	Normal input

6.2 Stage 2: LLM originality detection

For posts with a word count of ≥ 80, the system calls the LLM (GPT-5.4 mini) to analyse five language-feature dimensions:

Dimension	Description
Vocabulary diversity	AI text is usually rich in vocabulary but lacks colloquial expressions or typos
Structural regularity	AI tends to use full paragraph structures and transitional words such as first, secondly, and finally
Tone consistency	AI text remains consistently toned, unlike humans whose tone varies with the content and mood
Fillers density	Humans often use spoken fillers such as “um”, “then”, and “you know”; AI uses them less often
Specificity	AI tends to give general descriptions, while humans are more likely to mention personal experiences or classroom details

The LLM combines five dimensions to produce ai_likelihood (0.0–1.0), representing the likelihood that the content was generated by AI.

6.3 Integrated decision rules

Based on typing speed and LLM detection results, the system adopts a dual-threshold system to determine the final Emoji:

Word count	Typing speed	LLM results	Emoji	Scoring effect
< 80	Normal	Do not run	By BERT	Normal scoring
80–100	Normal	ai ≥ 0.7	😴 zzz	Not scored
80–100	Normal	ai < 0.7	By BERT	Normal scoring
> 100	Suspected paste	ai ≥ 0.5	😴 zzz	Not scored
> 100	Suspected paste	ai < 0.5	Do not grant	Raw score points (neither rewarded nor penalised)
> 100	Normal speed	ai ≥ 0.7	😴 zzz	Not scored
> 100	Normal speed	ai < 0.7	By BERT	Normal scoring

6.4 Error handling

• BERT fails: quality defaults to neutral (quality=1), and LLM detection continues to run
• LLM failure + suspected copy-paste: give no Emoji (neither reward nor penalty), to avoid mistakenly sending 👍
• LLM failure + normal speed: as per BERT results
• BERT + LLM both fail: no Emoji is given, the student receives the original points, and will not be misjudged because of a system fault

6.5 Instructor-side display

In the thread details page, Instructors and TAs can see the detection label (shown next to the author name):

• Suspected paste — Typing speed is abnormal; the tooltip shows words per minute
• AI 85% — ai_likelihood ≥ 0.7
• AI 55% — ai_likelihood 0.4–0.7
• AI 20% — ai_likelihood < 0.4

Students on the Student side cannot see any detection markers. The Instructor can override the detection result through the 👀 review mechanism.

7. Grade calculation

6.1 Linear mapping formula

The Student's final grade is converted from total points (total_points) into a grade score using a linear mapping. The Instructor can set two key parameters:

Parameter	Description	Default value
threshold	Minimum points required to achieve a perfect score	20
full_score_value	Maximum score	100

Formula:

score = (total_points ≥ threshold) ? full_score_value : (total_points / threshold) × full_score_value

• When total_points ≥ threshold, score = full_score_value (full marks)
• When total_points < threshold, score = total_points / threshold × full_score_value (linear increase)

6.2 Calculation example

Assume threshold = 20, full_score_value = 100:

Student	Topic	Comment	Replies	Emoji bonus	total_points	Grade
A	3	5	4	5	28	100 (full marks)
B	1	3	2	2	13	65
C	0	1	1	0	3	15

8. Batch update

Forum scores are automatically recalculated by a scheduled task (cron_forum_scoring.py) every hour:

1. Scan all courses with forum ratings enabled
2. Count the number of posts made by each Student in the specified forum (topics, comments, replies)
3. Calculate emoji bonuses and penalties (taking 👀 review into account)
4. Apply a linear mapping formula to update grades

This batch mechanism ensures that even if an instructor modifies the Emoji or adjusts the parameters in a non-real-time setting, the grade will be automatically reflected in the next hour.

9. Suggested research citation

It is recommended to provide the following:

• Scoring parameter settings for this Course (pts_thread, pts_comment, pts_reply, threshold, bonus_max)
• Discussion boards included in the scoring
• Data collection period and number of Students
• Activation status and review ratio for AI quality assessment