Missionloops · research seat (open) · metacognition & calibration
Knowing what you actually know.
A research seat on the gap between how good people think they are and how good they actually are, and whether a second person watching can close that gap. The platform is built to follow an operator's own self-monitoring across months of real, high-stakes decisions, a long-run complement to the one-shot lab measures the field relies on.
The faculty this seat studies
This seat studies the monitoring layer of thinking: the read on what you know, the confidence attached to an answer, the sense of being ready to act. Calibration is whether that read is accurate, whether what you feel you know matches what you actually know. The two come apart more than people expect, and the divergence is systematic: the gap is widest in the people least able to notice it.
Can an outside read close that gap? Or is it a stable trait that feedback does not move? That is the question this seat is open to study.
What the literature already shows
Several findings in the metacognition literature are settled enough to build on.
- Calibration is generally poor, and poorest among the least skilled. The least skilled overestimate the most, and in the original studies what improved their self-assessment was gaining the skill itself. The size of that effect is contested. Its direction is the rival hypothesis any outside-read claim has to beat.
- Confidence follows cues more than accuracy. The confidence attached to an answer tracks how fluently the answer arrived, so a measure built on self-report can capture ease rather than knowledge. How the signal is operationalised and scored decides which one you are measuring.
- The feeling that an answer is right is a real, measurable cue. It drives whether a person stops to check or moves on, and it can be dissociated from whether the answer is actually right.
- Monitoring and control are separable. Noticing a gap is one thing and acting on it is another; the field's framework treats them as distinct processes, so a training claim has to say which of the two it moves.
Two boundaries worth drawing. Whether self-report and behavioural measures of the same capacity agree is a real and hard problem, but it belongs to the volition seat. And the quality of the operator's first-order reasoning while they work belongs to the judgment seat. This seat is the second-order layer: the faculty that monitors and controls thinking, and whether a second observer can calibrate it.
The proposal's bet is that the guide can supply the outside read the least-calibrated cannot generate alone. In the field's terms, the guide watches the operator's monitoring layer, the confidence their written reasoning carries, and intervenes at the control layer, prompting a rethink when a self-read runs too high or too low. What to do with the prompt stays the operator's decision. Whether the bet is right is not assumed anywhere. It is the study.
The open questions
The question this seat owns is simple to state. Can a second observer train calibration, or is the confidence-accuracy gap a stable trait that feedback does not move? The pilot is built to test it by comparison, with convergence between guided and solo operators as the pre-registered primary criterion. If operators who work with a guide see their self-ratings converge on the blind behavioural estimate over the cycles while solo operators' do not, that is evidence calibration can be trained. If both groups' legs stay equally divergent, the gap holds, and the self-assessment half of the platform's mastery model is measuring only confidence. The guided-versus-solo trajectory tells the two apart.
- Measurement honesty. When does self-assessment track real proficiency, and when does it drift from it? And how do you score a soft signal so the number means proficiency? This is what has to hold before the platform can honestly offer an operator a read of what they are ready for.
- Calibration under social pressure. Can a person hold an accurate read of their own view when the group leans toward a claim the evidence does not support? The target construct is private-model accuracy: whether you know what you think and why, even while you stay quiet. How to score it is the study's first design problem.
- Where defence names it. The IDEaS Cognition and Trust challenge sets real-time dynamic calibration as its target. That is where this seat meets the platform's defence case.
Why the platform is the instrument
Calibration is hard to study over time. The one-shot tasks that measure it cleanly run into the reliability paradox: a task robust enough to show the same effect in everyone leaves little stable between-person variance to study. A real-stakes platform offers a different substrate for the same construct. Run through its pilot, it would yield months of an operator's own monitoring and revision, a guide supplying the outside read, and a blind behavioural estimate to score the self-assessment against.
The platform's dual-path mastery model is designed to fuse a blind behavioural estimate from structural metadata with the operator's own self-assessment, and to score how well the two agree. The self-assessment leg is anchored to Bloom's taxonomy, so the operator rates themselves against named levels of mastery rather than reporting raw confidence. In the field's own terms, that is the design of a meta-reasoning calibration study at real-stakes scale.
The studies this seat can run
Take these as a starting menu: recast any toward your own published line, or bring a question we did not anticipate.
The seat, and the terms
The seat is open, and the ask is a conversation. The terms are a free option: tell us the question you would bring, and commit only if the work is funded.
We are looking for the researcher whose line is metacognition and meta-reasoning, covering calibration, the confidence-accuracy relationship, and the monitoring and control of one's own thinking. If that is your line, the instrument on offer is a platform built to produce a long-run record of monitoring under real stakes, a complement to the one-shot measures.
scott (at) missionloops (dot) ca
See also: All research seats · Defence · Member · Investor