UCG cognomics quiz toolkit
Methodology

1. Theoretical basis

1.1 Educational Omics and Cognomics

The UCG (Uedu Cognomics) cognomics quiz toolkit is the core implementation of the Cognomics dimension within the Educational Omics framework of the Uedu platform. Cognomics focuses on learners' cognitive processes, directly measuring performance in core cognitive abilities such as attention, working memory and executive function through standardised cognitive quizzes, as a base variable for understanding learning behaviour and learning outcomes.

1.2 Three-factor model of executive function

The selection of UCG quizzes is underpinned by the three-factor model of executive function proposed by Miyake et al. (2000). Through latent variable analysis, the model divides executive function into three separable but related components:

• Inhibition: suppressing a dominant but inappropriate response, measured by the Stroop Task and the Flanker Task
• Updating: monitor and update information in working memory, measured by the N-back Task
• Shifting: flexibly switching between different task sets or mindsets, measured by Task Switching

Building on this, UCG expands to constructs such as processing speed, spatial cognition, language ability and metacognition, forming a complete toolkit of 6 major constructs and 14 quizzes.

1.3 Suitability of web-based cognitive tests

Traditional cognitive tests are often conducted in laboratory settings using dedicated software (such as E-Prime and PsychoPy). UCG uses a web-based implementation, mainly based on the findings of Reimers & Stewart (2015):

• The timing precision of performance.now() in modern browsers has reached the sub-millisecond level, sufficient to support reaction-time measurement
• Although absolute RT on web-based systems may be slightly higher than in the laboratory (around 20-40ms), the rank-order consistency of individual differences is sufficient for research
• The web-based approach can greatly increase sample size and ecological validity, making it suitable for large-scale administration in educational settings

2. Quiz toolbox architecture

The UCG toolkit includes 6 major constructs and 14 instrument tools, covering the full spectrum of cognitive ability from basic cognitive processing to advanced metacognition.

Construct	Quiz name	Core Paradigm	Measurement objective
Executive Function Executive Function	Stroop Task	Colour-word inconsistency interference	Inhibition
	Flanker Task	Wing interference	Inhibition
	Task Switching	Task set switching	Cognitive Flexibility (Shifting)
Processing Speed / Webcam Classroom Recording Processing Speed & Attention	Simple RT	Easy reaction time	Basic processing speed
	Choice RT	Select response time	Decision speed
	PVT	psychomotor vigilance task	Sustained attention
Working Memory Working Memory	N-back	N-back task	Working memory update (Updating)
	Corsi Block	Spatial sequence memory	Capacity of visual-spatial working memory
Spatial Cognition Spatial Cognition	Mental Rotation	Mental rotation	Spatial reasoning ability
	Symmetry Span	Symmetric span task	Spatial working memory capacity
	UFOV	Effective perspectives	Range of visual attention
Verbal Ability Verbal Ability	Grammatical Reasoning	Grammar reasoning	Linguistic reasoning
	Vocabulary	Vocabulary test	Crystallised intelligence (vocabulary knowledge)
Metacognition Metacognition	Confidence Calibration	Confidence calibration	Self-monitoring accuracy

3. Data collection procedure

3.1 Informed consent

All cognitive assessment data are collected in accordance with the research ethics plan approved under IRB 202507EM058. Students must complete an electronic informed consent form when they first enter the assessment toolbox. The consent form clearly states the purpose of data use, retention period and right to withdraw.

3.2 Trial-level raw data retention

UCG retains the raw data for every trial in full, including:

• Stimulus parameters: condition (congruent/incongruent), target, flankers, n-back level, etc.
• Response data: reaction time (ms), response key/position, accuracy
• Temporal data: stimulus onset timestamp, response timestamp, inter-trial interval

Trial-level raw data are available for researchers to conduct custom analyses (e.g. ex-Gaussian RT distribution fitting, serial effects analysis, etc.), without being limited by the platform's default scoring method.

3.3 Server-side pre-generated test order

The trial sequence for each Quiz is pre-generated and stored on the server, ensuring:

• Reproducibility: re-running the same session will produce the same trial sequence
• Balanced design: congruent/incongruent ratios, target positions, etc. are balanced in accordance with experimental design rules
• Prevent manipulation: the client side cannot predict or modify the upcoming trial

3.4 Client-side precise timing

To ensure the precision of reaction time measurement, UCG uses the following technical strategies:

• High-precision timing: uses performance.now() instead of Date.now(), achieving sub-millisecond precision
• Double requestAnimationFrame synchronisation: stimulus presentation uses nested requestAnimationFrame calls (double-rAF) to ensure DOM updates synchronise with screen rendering, aligning the timing start point with actual pixel drawing
• Render confirmation: RT timing starts from the callback of the second rAF, not at the point of DOM modification

3.5 Response capture

To support cross-device administration, UCG listens for multiple input events at the same time:

• Touch/mouse: use the pointerdown event (unifying touch and mouse input) to avoid the 300ms delay on touch devices
• Keyboard: use the keydown event, supporting desktop users to respond quickly with the keyboard
• Duplicate-trigger prevention: only the first valid response for each trial is accepted; subsequent inputs are ignored

4. Scoring method

Each quiz uses different core metrics according to its experimental paradigm. The following explains the scoring logic for each paradigm.

4.1 Stroop Task & Flanker Task (interference effect)

Indicator	Calculation method	Meaning
Interference Effect (RT)	mean_RT(incongruent) - mean_RT(congruent)	The smaller the effect size of interference, the better the inhibitory ability
Accuracy	Accuracy by condition	Reflects accuracy and can calculate the speed-accuracy tradeoff
Congruency Effect (Accuracy)	acc(congruent) - acc(incongruent)	Interference effect on accuracy

4.2 N-back Task (Signal Detection)

Indicator	Calculation method	Meaning
d-prime (d')	Z(hit rate) - Z(false alarm rate)	Signal detection sensitivity; higher values indicate stronger discrimination
Hit Rate	Proportion of correctly detected targets	Sensitivity to target
False Alarm Rate	The rate of misclassifying non-target as target	Response bias

4.3 Simple RT & Choice RT (processing speed)

Indicator	Calculation method	Meaning
Simple RT	Median RT for pure reaction	Basic neural conduction and motor reaction speed
Choice RT	Select the median RT of the responses	Decision processing speed
Choice - Simple Cost	median_RT(choice) - median_RT(simple)	Decision overhead, reflecting central processing speed

4.4 Task Switching (switch cost)

Indicator	Calculation method	Meaning
Switch Cost (RT)	mean_RT(switch trials) - mean_RT(repeat trials)	Time cost of task conversion
Switch Cost (Accuracy)	acc(repeat) - acc(switch)	Accuracy drop caused by conversion

4.5 Corsi Block & Symmetry Span (memory span)

Indicator	Calculation method	Meaning
Max Span	Longest sequence length of successful memories	Working memory capacity limit
Span Product	max_span × proportion_correct_at_max_span	Comprehensively assess capacity and stability

4.6 Mental Rotation (spatial reasoning)

Indicator	Calculation method	Meaning
RT by Angular Disparity	Average correct RT at each rotation angle	Slope of the linear function of mental rotation
Rotation Rate	Regression slope of RT against angle (ms/degree)	Mental rotation speed; the smaller it is, the faster the spatial reasoning
Accuracy	Accuracy at each angle	Assess correctness

4.7 PVT (psychomotor vigilance)

Indicator	Calculation method	Meaning
Median RT	Median RT of all valid trials	Overall alertness reaction time
Lapses	Number of trials with RT > 500ms	Number of attention lapses, reflecting the stability of sustained attention
1/RT (Reciprocal RT)	The average of 1000/RT for each trial	Speed metric, more sensitive to slow responses

4.8 Grammatical Reasoning

Indicator	Calculation method	Meaning
Accuracy	Proportion of correct judgements	Linguistic reasoning ability
RT	Median RT for correct trials	Reasoning processing speed

4.9 Confidence Calibration

Indicator	Calculation method	Meaning
Calibration Error	mean(|confidence - accuracy|) by confidence band	The smaller the gap between confidence judgements and actual performance, the better
Overconfidence Index	mean(confidence - accuracy)	A positive value indicates overconfidence; a negative value indicates insufficient confidence

4.10 Vocabulary

Indicator	Calculation method	Meaning
Total Correct	Number of correct answers	Vocabulary knowledge (crystallised intelligence indicator)

4.11 UFOV (Useful Field of View)

Indicator	Calculation method	Meaning
Accuracy by Display Duration	Accuracy under each presentation-time condition	Threshold for the processing speed of visual attention

4.12 MOT (Multiple Object Tracking)

Indicator	Calculation method	Meaning
Max Targets Tracked	Maximum number of targets successfully tracked	The concurrent tracking capacity of visual attention

5. Data quality control

To ensure the validity of cognitive test data, UCG applies quality control rules separately at the trial level and the session level.

5.1 Trial-level screening

Rules	Threshold	Processing method	Rationale
Expected response	RT < 150ms	Marked as anticipation, excluded from analysis	Below the shortest human reaction time, classified as a predictive key press
Outliers	RT > 2000ms	Marked as outlier; keep or exclude depending on analysis needs	May indicate inattention or a temporary absence

5.2 Session-level verification

Rules	Threshold	Processing method
Valid trial ratio	≥ 80% of trials must be valid (non-anticipation, non-timeout)	If below the threshold, the entire session is marked invalid
Condition accuracy	Any condition accuracy < 70%	The data for this condition are marked as flagged and require manual review

5.3 Re-test interval warning

5.4 Reliability Paradox notes

Hedge, Powell & Sumner (2018) point out that interference effects (such as the Stroop effect and Flanker effect) have large and stable effect sizes, but their reliability of individual differences (split-half reliability) may be low; this is the so-called 'Reliability Paradox'. The reason is that the robustness of the experimental effect comes precisely from the fact that almost everyone produces a similar interference effect, making inter-individual variation relatively small.

Therefore, when using UCG interference effect indicators for individual difference analysis, researchers should:

• Report split-half reliability or Cronbach's alpha
• Consider using accuracy-based metrics as an alternative or complement
• Increase the number of trials to improve measurement precision

6. How to cite

Individual quiz citation format

If the paper uses only part of the assessment, you may list the specific assessments used in the methodology description and cite the corresponding original literature. For example:

7. References

• Baddeley, A. D. (1968). A 3 min reasoning test based on grammatical transformation. Psychonomic Science, 10(10), 341-342.
• Ball, K., & Owsley, C. (1993). The Useful Field of View test: A new technique for evaluating age-related declines in visual function. Journal of the American Optometric Association, 64(1), 71-79.
• Corsi, P. M. (1972). Human memory and the medial temporal region of the brain (Doctoral dissertation). McGill University.
• Dinges, D. F., & Powell, J. W. (1985). Microcomputer analyses of performance on a portable, simple visual RT task during sustained operations. Behavior Research Methods, Instruments, & Computers, 17(6), 652-655.
• Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16(1), 143-149.
• Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166-1186.
• Kane, M. J., Conway, A. R. A., Miura, T. K., & Colflesh, G. J. H. (2007). Working memory, attention control, and the N-back task: A question of construct validity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33(3), 615-622.
• MacLeod, C. M. (1991). Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin, 109(2), 163-203.
• Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex "frontal lobe" tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49-100.
• Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: Evidence for a parallel tracking mechanism. Spatial Vision, 3(3), 179-197.
• Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47(2), 309-327.
• Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(3972), 701-703.
• Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18(6), 643-662.