Chat Embedding
semantic vector methodology

1. Overview

Chat Embedding is Uedu's dialogue semantic vectorisation module, forming the infrastructure for the Cognomics (cognitive process) and Linguomics (language expression) dimensions within the Educational Omics framework.

The system will convert each message (request) sent by Students to the AI TA into a high-dimensional semantic vector and store it in the database. These vectors can be used for:

• Semantic search: perform semantic retrieval in the conversation history using natural language, replacing traditional keyword search
• Similar question detection: identify semantically similar questions raised by different Students to help Instructors understand learning pain points
• Research clustering analysis: analyse Students' question patterns and cognitive trajectories through distributions in vector space

2. Vectorisation workflow

2.1 Embedded object

The system only embeds the messages sent by Students (request), and does not embed the AI's replies (response). This is because the research focuses on Students' questioning behaviour and cognitive expression, rather than the AI's output.

2.2 Text pre-processing

The message must undergo the following pre-processing steps before embedding:

1. Cleaning: Remove extraneous whitespace and formatting noise
2. Minimum length check: messages must contain at least 2 characters; messages that are too short (e.g. a single punctuation mark) will be skipped
3. Token truncation: uses the cl100k_base tokenizer to count tokens; messages over 8,000 tokens will be truncated (and marked was_truncated = 1)
4. Content hash: calculate a MD5 content hash for deduplication to avoid embedding identical content repeatedly

2.3 Embedding model

2.4 Dual-path processing

The system adopts a dual-path architecture of real-time + batch backfill, ensuring that new messages are vectorised immediately while historical data are also completed:

2.5 Skip message handling

Messages that do not meet the embedding criteria (too short, no substantive content) will be written to a NULL embedding record, to prevent the batch backfill process from repeatedly attempting to process them. This ensures each message is evaluated only once.

3. Semantic search mechanism

3.1 Search principle

Semantic search uses cosine similarity to calculate the similarity between the query vector and all embedding vectors in the database. The system uses numpy matrix operations for efficient computation:

similarity = (A · B) / (||A|| × ||B||)

3.2 Scope filtering

Search supports three range filters, and you can narrow the search scope according to research needs:

3.3 Search parameters

3.4 Memory cache

To improve search performance, the system maintains a set of in-memory cache entries for each scope key, with the following cache policy:

• TTL: automatically expires after 5 minutes
• Cache invalidation: when new embeddings are written to that range, the cache is invalidated immediately and reloaded
• Cached content: includes the vector matrix and corresponding metadata to avoid repeated database queries

4. System parameters

5. Batch processing

5.1 Batch backfill architecture

The batch backfill process handles messages missing from real-time embeddings (for example, messages generated during a service restart), as well as historical messages from before the system went live.

• Number of workers: 3 parallel workers
• Partition strategy: assign messages to different workers using log_id % 3 to avoid duplicate processing
• Polling interval: query the database every 15 seconds to obtain messages not yet embedded
• Batch size: process up to 50 messages at a time

5.2 Idempotency guarantee

The database uses log_id as a UNIQUE KEY, and INSERT IGNORE is used when writing. Even if the real-time path and the batch path process the same message at the same time, duplicate records will not be created.

6. Data storage

6.1 Data table structure

Embedding results are stored in the chat_message_embeddings table:

6.2 NULL Embedding record

Messages that do not meet the embedding criteria (insufficient length, no substantive content) will be written to a record with embedding = NULL. This allows the batch backfill process to quickly identify unprocessed messages via LEFT JOIN, avoiding repeated attempts.

7. Research citation guidance

It is recommended to provide the following:

• Embedded message types (Student request only or including AI response)
• Data collection period and number of valid messages
• Search parameter settings (top_k, threshold)
• Informed consent form version and IRB approval number
• If used for clustering analysis, explain the dimensionality reduction and clustering algorithms used