Pebb's Lab / tieSe · a research story
tieSe uses non-generative AI models to select dictionary meanings based on sentence context.
A deliberately small study
Three short sentence variations. Five target words in each. Three dictionary inventories, kept separate. Twelve models asked to work with the meanings each inventory offered.
Only 45 distinct sentence–word–dictionary contexts, each examined by 12 models. That is 540 repeated model outcomes—not 540 independent sentences or independent contexts.
“He is a huge fan.”
“He has a huge fan.”
“It is a huge fan.”
Decision probabilities, embedding similarities, and relevance values are different score families. They are not interchangeable confidence measures.
Scope follows the reconciled 12-model report. Dictionaries are identified only as D1, D2, and D3; exact model and dictionary names are withheld from this scientific summary. This page does not publish the full outcome matrix or restricted definitions.
First, a little fiction
Once upon a time, a cruise ship sank. Dramatically. With bubbles. Two twin brothers floated away with the same name—and eventually met in a wrestling ring with a dictionary for a referee.
The story and artwork are fiction. Their named AI characters and labels belong to the joke, not to a description of the experimental methods below.
Enter the arenaThe linguistic puzzle
Before a machine can choose the right meaning, two things must be true: the dictionary must offer that meaning, and the sentence must provide enough evidence to justify choosing it.
These are teaching examples with tiny, invented meaning inventories—not dictionary quotations, experiment records, or live AI results.
“The desk fan blows cool air.”
Mini-inventory: an air-moving device (noun); a supporter (noun); move air (verb).
The device meaning is present, and the sentence needs a noun. “Desk” and “blows cool air” also give evidence for that meaning. Availability, grammar, and context line up.
A meaning is offered. A reason to choose it.
“They fan the embers.”
Same mini-inventory: an air-moving device (noun); a supporter (noun); move air (verb).
A supporter is a meaning of the word in this inventory, but choosing that noun here would not fit the sentence's verb slot. A verb meaning is available; the problem is the incompatible choice, not a missing entry.
A meaning is offered, but its grammatical role does not fit.
“They have a fan.”
Same mini-inventory: an air-moving device (noun); a supporter (noun); move air (verb).
Both noun meanings are grammatically possible. Is this a cooling device or an admirer? The sentence alone does not settle it. Picking one would add information the sentence has not supplied.
Several readings fit. The sentence does not settle which one.
“They fan the embers.”
Smaller mini-inventory: an air-moving device (noun); a supporter (noun). No verb meaning is offered.
The sentence points to an action, but neither offered meaning can fill that verb slot. Rejecting this inventory may be appropriate—even though the context itself is informative. A model cannot select a meaning that is missing.
No suitable meaning is offered. Rejection may be appropriate.
A proposal, not a score
We propose 3S—the Semantic Sufficiency Scale: a human-annotation framework for describing how much meaning the available context supports, at a specified target word and level of detail.
These are illustrative applications of the proposed rubric, not validated annotations of the experiment.
3S is not a calibrated AI confidence score. It has not been validated and is not operational in this study. The 540 outcomes have not been assigned validated 3S scores.
Missing dictionary coverage is a separate issue from contextual sufficiency. An informative sentence can still lack a suitable meaning in the offered inventory; that is not automatically LOW.
Independent human readers would assess each context without seeing model predictions. They would separately record whether an appropriate dictionary meaning exists, assign a proposed 3S state, and list acceptable interpretations with reasons.
Readers would compare disagreements before settling a reference annotation. Only then would we evaluate whether model choices, qualifications, alternatives, or refusals match the evidence available. Broader, independently annotated evaluation is still needed.
What this leaves us asking
Exploratory observations—not a model leaderboard.
Having an entry for the word does not guarantee that the required meaning is among the options.
Passing a grammatical check is not the same as finding enough evidence for a particular interpretation.
A larger or more elaborate system cannot be assumed to recover distinctions absent from the context. This small comparison does not establish a general ranking.
When the inventory has no suitable meaning, rejection may be warranted. When context is insufficient, alternatives or abstention may be warranted instead. A recorded refusal is not automatically a correct one.
The dataset is small, and human semantic adjudication remains incomplete. Grammatical compatibility is not semantic accuracy, and repeated outcomes are not independent evidence.
Broader evaluation with independent human annotation is planned. We do not yet know whether 3S will be a reliable, useful framework across a wider range of sentences and dictionary granularities.
The interesting part might be knowing when not to decide.
Not quite ready to leave the lab
The pipeline is planned for publication on GitHub.
The repository is currently private; public access is not yet available. The link may be inaccessible until release.
Back to Pebb's Lab