Two Decide, a Thousand Agree

Dr. Marina Milyavskaya¹ asked two questions. “How exactly does this help Canadians resist cognitive warfare?” And, “How can an acute, singular training experience solve a population problem?”

She was deciding whether to join the research team we are assembling. As I was answering her second question, about spread, she answered the first one herself. “I think it’s a side effect of the training,” she said. Call them the resistance question and the spread question. I owe her both answers, and I take her own, the side effect, seriously, which means saying what would make it wrong.

The spread question is the plainer one to state. We say we are building a defence against cognitive warfare, an attack on a whole population’s ability to think for itself and to say what it thinks. What we actually do is train individuals, one at a time, on their own problems. Between those two there is a gap of about forty million people, and nothing on this site says how it gets crossed. The attack does not land on a whole population at once either. It reaches people one at a time, through whatever each of them already uses, and then it has to get from the few it has bent to the many. So the attacker has the same gap to cross, and how the attacker crosses it is where both answers start.

Cognitive warfare has no fixed form. It moves to whatever medium works, and we do not try to catalogue its forms, only to harden what it attacks, which is a person’s own judgment. But every form of it has that one step in common, from one person to many, and the step runs on the same machinery whatever the medium. Three economists described the machinery in 1992.

Two people can decide for a thousand

Sushil Bikhchandani, David Hirshleifer and Ivo Welch worked through a situation so simple it hardly seems worth a paper. People decide one at a time. Each has a private hunch about the right answer, and the hunch is right more often than not. Each can see what the people before them chose, but not what those people privately thought.

The first person goes on their own hunch. The second sees the first and mostly follows. The third is where it gets interesting. They see two people agreeing, and if their own hunch points the other way, it is two against one, so the sensible thing is to go along. From then on you cannot tell anyone’s hunch from what they do. The fourth person sees three in agreement, one hunch cannot outweigh that, and so it goes for everyone after. The authors called it an informational cascade, and it starts fast: after two people have acted there is at least a three in four chance one has begun, and after ten the chance one has not is under one in a thousand.

What surprised me is how little a cascade carries. A thousand people deciding independently would bring a thousand people’s worth of information to a question. A thousand people deciding in sequence carry the judgment of the two who started the cascade, and nothing from the rest.² It looks like a thousand independent judgments. It contains two.

Most of us treat a lot of people agreeing as evidence. In a cascade it is not, and this is why. The crowd may well be right: if each person’s hunch is better than a coin flip, most cascades land on the right answer. But when it is right, it is right because of the first two people, and the thousandth person who agrees is no more reason to believe it than the third was. The paper shows one more thing. A cascade is fragile. One piece of public information can break it, and so can one person late in the line who the others can tell knows more.

Everyone in that chain did the sensible thing. What produced the cascade was the situation, and it is a particular one: people who act in turn and can see what the others did, not what they thought. When that is the setup a cascade is likely, and which way it runs depends on the first two. The authors compare it with a version in which each person’s hunch is public as well, and there, if the first two happened to be wrong, the group corrects itself, because everyone’s hunch is on the record even when they went along with the crowd.³ That is the practical reason free speech matters. If people can say what they think, and why, the group gets to use everyone’s judgment.

It comes down to a ratio

The paper’s people stood in a line, each one seeing only the choices of those ahead of them. That is the simplest shape a group can have. Real groups are webs: each person sees the people around them, and those people see others, with no first and no last. The same copying happens in a web,⁴ and for any one person in it, whether they move comes down to a ratio. On one side is the share of the people around them who have already gone one way. On the other is how firmly they hold their own view of the thing they are being nudged on. When the share around them is high and their own view is loosely held, they move. When they actually know something, because they have checked, the share has to be much higher before it moves them, and it may never get there.

Epidemiologists have a word for how easily a person can be moved: susceptibility. It is the second side of the ratio seen from the other end, since the firmer someone holds their view, the less susceptible they are.⁵ We take it to be specific to the topic: a person can be immovable on the one thing they have worked on and easily moved on the next. It is like doing arithmetic at the shop shelf. Being good at it does not mean you work out the cost per gram of every pair of packages. You do it where the price matters to you. A person’s hold works the same way: firm where they have checked, loose where they have not. If that is right, a person who has checked their own affairs is still easy to move on everything else, and cognitive warfare is not limited to anyone’s own affairs. That is what makes the resistance question hard: training people on their own problems looks like no defence on the questions an attack chooses.

Be early, fake the crowd, attach a censor

An attacker who wants to bend a population rarely builds anything of its own. It takes over what people already reach for: a compromised AI, a music act with a quiet sponsor, an influencer, a topic that is trending because an adversary paid for it to trend. Taking over what already exists costs almost nothing, and it is hard to trace back to whoever paid. Most of what comes through those channels does not argue with you. It changes what feels normal to care about, a little, one person at a time. Whether what is spreading is a claim or a care, the step from one person to many is the same.

What makes it a population effect is the cascade. An attacker who works through cascades has more moves than anyone can list, but a few are obvious. Be early: the first few decide for everyone, so bend a few people in a peer group, and if they speak first the rest follow, the way the third person in the cascade did. Fake the crowd: where being early is not enough, show the group a crowd that already agrees, whether or not it exists, which supplies the share side of the ratio without anyone having to earn it. Solomon Asch measured how much of that conformity there is to work with. He showed people a line on a card, in a room where everyone else was in on it and named the wrong match, and about a third of the time they went along with the room. Our Four Walls briefing walks through it. Keep that room in mind. From here on it stands for the people around you whose choices you can see, whether that is a meeting, a group chat or a feed. And attach a censor, which is what makes it last: the message carries a label with it, a name for the kind of person who doubts it, so every copy of the message passes the insult along and doubting costs standing. From then on the group does the attacker’s work. Certain evidence becomes awkward to cite, certain questions become risky to ask, and anyone who actually knows something learns to keep it to themselves, which keeps them apart from each other without the attacker having to find them.⁶

What is left, once the censor has done that, is what we call a shaped silence: a cascade that has locked, held in place by the group itself. In the paper, one clear piece of public information breaks a cascade. A shaped silence is harder. By then the group is counting voices, not information. An argument is just one more voice, and the group already has the insult ready, the label that came with the message. One counter still works there. Say who is being shut out, and why. That tells the group two things it could not see: that another view was there, and that it disappeared because of the label, not because it was wrong. The censor can label the person who says so as well, which is why two people are harder to isolate than one.

Two more things make all of that cheap to run. The attacker does not have to work any of it out by hand. It can model who is connected to whom and how loosely each person holds each view, or buy that targeting from an advertising platform that already has the model. And an attack does not have to work the first time. An attack can be run in pulses, measured, adapted, and run again.

Work a problem of your own

Start with the person, because that is where the training happens. What we are building asks a person to take an intractable problem in their own life and work it in writing, around John Boyd’s observe, orient, decide, act loop:⁷ write down what is happening, what picture of the situation fits, and what to do next, then do it, write down what came back, and go around again. The first plan usually comes apart, and how it comes apart is the point. A plan is a guess about how the situation works, so when it fails, the way it fails shows the person a piece of their situation they could not see from where they started. Anyone who has chased a fault in a machine knows this: you are not fixing anything at the start, you are changing one thing at a time to find out what is wrong, and the fix comes only once the picture is right. We call a person who has done that with a problem grounded. They cared enough to go and look, and what they hold is what they found. That is where the firmness on the second side of the ratio comes from: a grounded person is not susceptible on the thing they checked. The resistance question comes down to how far that reaches. It reaches in three ways, two of them direct.

The first is where the attack lands on a person’s own affairs, and there the training is the defence itself. Say the picture of your situation came from outside, or the way you see it was handed to you, or the advice drifted a little each time you asked for it. A plan built on any of those breaks like a plan built on any other wrong guess, and because the problem is yours, the failure costs you something you cannot walk away from. That is what makes you take the result seriously. The point of disabling a population’s thinking and speech is to bend its choices about its own affairs away from what it would have chosen with both intact. The attack is usually looked for in public opinion, and it is there. Our bet, stated as a bet, is that the damage is done closer to home, in livelihoods, families and communities, where a bent choice costs someone something. If that is right, a Canadian acting on their own checked interest is working against the attack without ever having to name it, and the result tells them whether they were right, not the room.

The second is one step upstream of belief. The cognitive attack’s product, as we described it, is not a belief. It is a shift in what feels worth caring about, and the training works on that too. Working on something is how people come to care about it and want it.⁸ That lowers their susceptibility, because a want of their own is a firm hold on that part of their life, and it makes the goal easier to reach, because a goal they want does not have to be pushed along by willpower.⁹ And a person who wants something has a reason to say so when the room goes the other way, which is where the free-speech half of the defence starts.

The third is the side effect, the one Dr. Milyavskaya named, and the one we cannot yet claim. Most of what an attack targets is in nobody’s own affairs: an election, a grievance between communities, a question that marks whose side you are on. On those questions the training gives nobody a stake, and the research says a stake is what it would take. Dan Kahan found that on questions tied to group identity, sharper reasoning makes people more polarised, not less: being wrong costs nothing and breaking with the group costs a great deal.¹⁰ So the training does not try to make anyone reason better about public questions. A NATO study says the attack is aimed not at what people think but at “the way they think”,¹¹ and a way of thinking is what the training can leave: not sharper reasoning, a habit of checking before a crowd counts as evidence. Under stress, deliberation goes quiet, habit runs, and what is learned there is learned as habit.¹²

Whether that habit fires beyond a person’s own affairs is her hypothesis, with two conditions. The first is that a habit answers to the cues that built it, and this one was built on a person’s own problem, so a public question may not carry the cues that set it off. The transfer literature says the same in general terms, and we cite it against ourselves: what works in the lab does not carry into the field.¹³ The one thing it says helps is training in the real thing, which is the whole design. The second is that in a room that agrees, most people’s habit is going along, which checking must displace first. Her side effect is the first thing to measure. The cheap test would be Asch’s rigged room. The real one would be whether people who have practised under real stakes hold their judgment on identity-charged questions where people like them who have not practised do not. That is a candidate study we are recruiting researchers for.

That is one person. What one such person does to the people around them is where the spread question starts, and it carries the free-speech half of the defence, which is one more answer to the resistance question.

One person who checked and said so

A cascade runs on people going along with the count. The one person it has trouble getting past is someone who has checked for themselves, because they do not go along.

Take the ten people from the first figure and make the sixth one a grounded person, someone with a stake in the claim, who worked on it and found it false. Five people ahead of them saying true does not move them, and they care enough to say what they believe, so they say false. The seventh person’s room is the six ahead of them, and it does not agree. Five said true, one said false, and this false was said out loud, which the third person’s never was. The paper puts one condition on what happens next: the seventh has to be able to tell that the sixth checked. If they cannot, the paper treats the sixth’s false as a mistake: the seventh reads it as if the sixth had said true, and the cascade rolls on. If they can, the sixth’s false counts for more than a hunch. It roughly cancels the two hunches that started the cascade, since the three in between were only going along, and the seventh is back to weighing their own hunch. The count starts again from there, with the sixth person’s judgment in it. The sixth person did not have to win anyone over. They said what they found, and the people after them went back to their own hunches.

The paper gives this two endings, and which one you get depends on how much weight the sixth person carries. If their checked view roughly offsets the two hunches that started the cascade, the people after them go back to their own hunches, which is the run in the figure. If those people can tell the sixth is far better informed than that, they do something else: they follow the sixth, and a new cascade forms behind the person who checked.¹⁴ Both endings put more than two people’s judgment into the record, and both come from one person who did not move. So that is already a population effect, and nobody else had to be trained. A person who is not moved does not pass the thing on. In a cascade that is a break in the chain, and with enough such people in a group a cascade cannot run across it.¹⁵ The censor tries to label the sixth person to drop their credibility, so that the seventh can discount them.

Two are harder to isolate than one

Two people hold out against a crowd better than one, for a reason Asch measured. A crowd’s power is the appearance that everyone present independently agrees, and a single real dissenter breaks that appearance for the person being tested. With one partner in the room giving the right answer, going along fell from about a third of trials to about one in twenty. Asch then asked whether the partner has to be right, and found that a partner who gave a different wrong answer still cut going along, to about one trial in eleven.¹⁶ What does the work is breaking the unanimity, more than being right. A person alone is an outlier the crowd can label. Two people who agree with each other against it, a pair, are a disagreement it has to explain. Asch could not measure whether that disagreement also frees the people watching, the ones not being tested, because everyone else in his room was either in on it or being tested themselves. We want to measure it.¹⁷

The Missionloops practice is done in pairs. A member pairs with anyone they know in the network. On their own missions they are the operator, and on other members’ they are the guide, reading along. So pairs wink in and out of existence around missions, and one person is in several at once. The pair also answers the censor, in two ways. The first is that the attacker cannot find the Missionloops network. The map an attacker works from is the public one, the social graphs on LinkedIn and Facebook. In the network we are designing, who is paired with whom is never published. Each member would hold their own part of it and decide who participates in it, so there is no whole to mine and the censor has nothing to start from.

The second is that the censor puts a price on saying what a person found out loud, in front of the room. In the pair they say it to one person they chose, and there is no price. It does not have to be said in public to be acted on. A person who knows what is true about their own situation can keep it to themselves in the room and still act on it. Over time, in the parts of life where it matters, acting on what is true beats acting on what the room says.

Against the resistance question, dissent is one more way, and it reaches differently from the three above. Grounding is narrow: it holds a person on the affairs they checked. Dissent is wider: a person who visibly holds their own view frees the people watching on that question, because what broke was the unanimity, and Asch’s wrong partner showed it does not matter whether the dissenter was right. Her side effect is one person’s firmness carrying into the rest of their life. Ours is a room’s independence carrying to the next question, a bet, not something Asch measured.

What we are building, and how it spreads

A note on status. There are no paying members yet. What there is: a series of prototypes. We have used them on real problems, and so have a lot of people we have put them in front of. What follows comes from that use: it describes what the software is built to do, as we have used it, not a measured result from a running community.

The practice is built to produce grounded people and to spread on its own. The operator works one mission at a time in the software, in the loop described above, for as long as they have problems worth working, which does not end, and neither does the attack. It is a practice, not a course. Operator and guide are roles, not people, and a pair can swap them at any time. Either can bring someone they trust into the network, so that the person they lean on is inside the software and its training with them.

The guide drops in every few loops for a few minutes to read the mission. In the operator’s room, the guide is the one real witness whose presence breaks its unanimity, Asch’s partner. On the mission, the guide has read it from the start, so they have what nobody in the model has: the operator’s own hunch, written down before the pressure arrived. If the operator’s belief moves later and nothing they checked has changed, the move came from the room, not from the facts. From the inside that feels like a change of mind. The guide can see the difference, and says so. When a problem is done, the two go back over the mission together, and that review ends by asking for the story.

The story is what makes the practice spread, and it has to spread on its own, because there is no marketing budget in the design and no office running a marketing campaign. A solved hard problem is a story people tell, and it travels the way a good mechanic’s name does, from someone who used one to someone who needs one. The practice is cheap to start: no course for the operator or the guide, and a price low enough for any Canadian and high enough that nobody joins idly. The people it reaches are the ones already carrying a problem, who would go to a therapist, a coach or a consultant if they could afford it, and we are none of those. Members have their own reason to bring in good people, because the network of those who guide them and whom they guide is theirs to keep. So the practice spreads through the same thing the attack produces, people under pressure with a problem they cannot solve, and nobody has to believe the attack is real to join. Each person it reaches is one more grounded person in one more group.

A cascade of our own

Everything so far has been about breaking the attacker’s cascades. The same people can start one of their own. In the model, whoever moves first decides for everyone after, and the attacker’s craft is to be those first movers or to fake them. Our people are first movers of another kind: not first in time, but fixed. Everyone else in a cascade goes by the count. The people who do not move are the only part of it that stays put, so the undecided end up on their side.

There is an experiment on exactly that. In 2018 Damon Centola, Joshua Becker, Devon Brackbill and Andrea Baronchelli put people into online groups of twenty to thirty and paid them to agree with a randomly assigned partner on a name for a face, round after round, until each group had settled on one name. Then they added a few people who would not budge from a different name. Below about a quarter of the group, nothing happened. At a quarter or more, the whole group came over, in every group they ran, and in one case the difference was a single person: four committed people in a group of twenty failed, five succeeded.¹⁸ The theory they were testing says a minority’s power comes from its commitment, and the result fit it. They also say where they expect it to matter next: governments planting small committed groups in online communities to shift what gets talked about. That is the attacker’s first move, being early with a few bent people, seen from the other side.

Our minority is committed to something narrower and harder to argue with: what its members checked in their own lives. And it has a weight Centola’s planted minority did not. Not every voice around a person counts the same. Someone who has been right in front of you before, who said what they assumed and what broke and was then proved right, gets more weight next time. Joseph Henrich and Francisco Gil-White called that prestige in 2001, deference given freely to someone whose competence you have seen. So when a grounded person says what they found, the people who have watched them check before give it more weight than one more voice. If that holds, it should take fewer of them than Centola’s quarter to tip a group. We have not tested that.

The story is also what lets the seventh person tell that the sixth one checked, the condition the model put on breaking a cascade, which the figure could show to you but not to the seventh. A hunch and a checked view sound the same when they are said as conclusions. A listener can tell them apart when they are told as what happened: what I assumed, what broke, what turned out to be true. That is the form the review at the end of a mission asks for. Told that way, a story of a solved problem starts a cascade from tested judgment, with the operator and the guide as the two voices at the head of it. It is a way of handling a problem that worked, and the person who hears it is as free to check it as the teller was. The attacker’s cascade does not care whether it is true. This one was tested before anyone told it. From every group the network has reached, stories like this go out, each about its teller’s own life, and they need not agree with each other, because each only has to be true where it is told.

Rumours and habits

What an attacker plants travels like a rumour. One exposure is enough, it crosses any connection however thin, and a feed can carry it across a country in an afternoon. The 1992 paper gives the price of that: what spreads on one exposure breaks on one, and a small amount of public information can shatter a cascade that millions have joined. The practice does not spread like that. Damon Centola and Michael Macy showed in 2007 that a rumour crosses a network in a single exposure, but a practice that costs something is taken up only when two or more people close to a person already have it. A signal that crosses one long tie arrives alone, with no one to confirm it, and goes no further.¹⁹ So a feed can tell people the practice exists, but the practice itself spreads between people who know each other, into whatever groups have a reason to take it up, and it spreads slowly.

The practice is slower, but it holds. When Centola ran the mechanism as an experiment, seeding a health practice into online communities built to different shapes, the people who took it up after two of their contacts had were twice as likely to come back to it as those who took it up after one.²⁰ He calls that a correlation, and so do we. But it is the shape we would expect. What needs several people to catch is held in place by several people, and there is no single voice the attacker can bend to undo it. The attack spreads like a rumour and breaks like one. The practice spreads like a habit and holds like one.

Which groups take it up is not ours to choose. What we control is whether the practice is cheap to start. And because it crosses from one group to the next only when a few people belong to both, the design includes meetups, where members meet people they would never otherwise work with. A meetup is where those shared people come from. A pair that belongs to both groups is the narrowest bridge there is, since the people who know them in the second group hear it from two at once.

What would prove us wrong

Each question has its own way of proving us wrong. The spread question’s is the company. If the practice does not spread the way a habit does, we run out of members, and the company ends before any study reports. The studies on that side, whether the people watching a practising pair are harder to move than people with no pair near them, and whether our practice reaches someone with two practising contacts before someone with one, would say why. The market says whether.

Her side effect needs a real test, and it is not ours to design alone. Our researchers would first work with DRDC, whose concern is cognitive warfare used against civilian populations, to learn how it sees the threat and how it would want a defence tested, since the term on its own covers too much to test against. Then they would look for two things. Whether the practice defends our own members against that. And whether our members defend the people around them, through network effects: by speaking out, by being grounded where others are not, by starting cascades of their own, or by taking positions of leadership.²¹ I am an engineer, not a researcher. These are their questions to shape, and part of their job is to find the way to prove me wrong.

So, Dr. Milyavskaya’s two questions. How does this help Canadians resist cognitive warfare? Directly, where the attack lands on a person’s own affairs and on what they care about, because a checked picture and a want of their own leave it nothing to bend. And, if she is right, everywhere else, as a habit the training leaves behind. How does one person’s training become a population’s? The attack has the same shape: it reaches one person at a time and is held there by the people around them. A person who has checked is a break in that chain. A pair is a disagreement the crowd has to explain, and we bet it frees the people watching, on the next question too. A story of a solved problem starts a cascade from tested judgment, and what it spreads holds. None of that asks the crowd to change its mind, only that enough people in it know something, act on it, and say so.

The research behind this

• Bikhchandani, S., Hirshleifer, D. and Welch, I. (1992). A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of Political Economy 100(5). https://doi.org/10.1086/261849
• Watts, D. J. (2002). A simple model of global cascades on random networks. PNAS 99(9). https://doi.org/10.1073/pnas.082090499
• Noelle-Neumann, E. (1984). The Spiral of Silence: Public Opinion, Our Social Skin. University of Chicago Press.
• Asch, S. E. (1951). Effects of group pressure upon the modification and distortion of judgments. In H. Guetzkow (ed.), Groups, Leadership and Men. Carnegie Press.
• Asch, S. E. (1955). Opinions and social pressure. Scientific American 193(5). https://doi.org/10.1038/scientificamerican1155-31
• Asch, S. E. (1956). Studies of independence and conformity: I. A minority of one against a unanimous majority. Psychological Monographs 70(9). https://doi.org/10.1037/h0093718
• Allen, V. L. and Levine, J. M. (1971). Social support and conformity: the role of independent assessment of reality. Journal of Experimental Social Psychology 7(1). https://doi.org/10.1016/0022-1031(71)90054-0
• Henrich, J. and Gil-White, F. J. (2001). The evolution of prestige: freely conferred deference as a mechanism for enhancing the benefits of cultural transmission. Evolution and Human Behavior 22(3). https://doi.org/10.1016/S1090-5138(00)00071-4
• Centola, D. and Macy, M. (2007). Complex contagions and the weakness of long ties. American Journal of Sociology 113(3). https://doi.org/10.1086/521848
• Centola, D. (2010). The spread of behavior in an online social network experiment. Science 329(5996). https://doi.org/10.1126/science.1185231
• Milyavskaya, M., Nadolny, D. and Koestner, R. (2014). Where do self-concordant goals come from? The role of domain-specific psychological need satisfaction. Personality and Social Psychology Bulletin 40(6). https://doi.org/10.1177/0146167214524445
• Milyavskaya, M., Inzlicht, M., Hope, N. and Koestner, R. (2015). Saying “no” to temptation: want-to motivation improves self-regulation by reducing temptation rather than by increasing self-control. Journal of Personality and Social Psychology 109(4). https://doi.org/10.1037/pspp0000045
• Kahan, D. M., Peters, E., Dawson, E. C. and Slovic, P. (2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy 1(1). https://doi.org/10.1017/bpp.2016.2
• Schwabe, L. and Wolf, O. T. (2009). Stress prompts habit behavior in humans. Journal of Neuroscience 29(22). https://doi.org/10.1523/JNEUROSCI.0979-09.2009
• Arnsten, A. F. T. (2009). Stress signalling pathways that impair prefrontal cortex structure and function. Nature Reviews Neuroscience 10(6). https://doi.org/10.1038/nrn2648
• Lintern, G. and Boot, W. R. (2019). Cognitive training: transfer beyond the laboratory? Human Factors. https://doi.org/10.1177/0018720819879814
• Centola, D., Becker, J., Brackbill, D. and Baronchelli, A. (2018). Experimental evidence for tipping points in social convention. Science 360(6393). https://doi.org/10.1126/science.aas8827
• du Cluzel, F. (2021). Cognitive Warfare. NATO Innovation Hub, Allied Command Transformation, January 2021.
• Centola, D. (2018). How Behavior Spreads: The Science of Complex Contagions. Princeton University Press. ISBN 978-0-691-17531-7.