This is the second in a series of posted chapters for a project around Trust in Society.
These first three chapters explain the workings of our Human Operating System, HumanOS. In the previous post, I explored how people gain trust in their internal and external senses by explaining Friston’s Free Energy theory of active inference. This post expands on that to look at how the Markov Blanket, basically a mathematical Platonic Cave, allows us to develop misperceptions of the world and to maintain those misperceptions even in the face of glaring evidence to the contrary.
Here's what to expect going forward –
Operation Trust: Debriefing the Collapse of Trust in Society
Lying Eyes: Believing is Seeing ← You are here
Rhetorical Color: Signals and Emotional Shades
From Me to Us: Character
Friend Vs Foe: The Prospect of Trust
From Us to We: Communities of Myth
Status: Ability, Morality, and Culture
From We to Them: Culture Clash ← Writing Now
Fashion: Police Authority
Organizations: Brain of the Firm
Economic Geography: A Pattern Language
Let's Play a Game: The Map of Trust
Self-knowledge is like an eye that can see itself - Socrates
Believing is Seeing
Understanding the Markov Blanket
Previously, we examined how Friston’s Law, the process of active inference, governs our sensory perception and how we use that process to gain trust in our sense of the world and trust in our sense of self. To deepen our understanding, we need to fill out the Markov Blanket, the mechanism of active inference, to explore how our minds deal with surprises, and how it allows us to confidently make—and maintain—wrong predictions.
For new readers, and to remind old readers.
Your Markov Blanket is like a smart filter. Imagine you're at a crowded party. Your brain filters out most conversations (noise) to focus on the person you're talking to (signal). This filtering happens for both sensory information (what you see and hear) and social information (what you understand and feel). The filter gets better with experience - you learn what's important to pay attention to and what to ignore. In this visualization, the left side represents the external world. The boundary you see is our Markov Blanket, which you can think of as our personal Platonic Cave. The green areas represent our senses, which interact with active states (orange) – think of these as our subconscious. Finally, we have internal states (red), which are the memories we have of the world. Active Inference, which we discussed earlier, are the lines that explain how the green sensory inputs and the orange active thoughts work together to update our red internal predictions.
The Role of Hidden Variables
As we add more layers to this Markov Blanket Network, we introduce what are called Hidden Variables. These are sub-steps, steps 2 and 3 in the model above, between our layers of perceptions which allow our brains to perform remarkable feats, like transforming the 2D images our eyes receive into the rich 3D world we perceive. However, these hidden sub-steps are also where we are most vulnerable to being fooled – both by ourselves and by others.
While the Markov Blanket explains how individual minds process information, perhaps the best illustration of how different minds can process the same information differently comes from a classic movie, and one of my favorites, Rashomon.
Perception and Reality
The Rashomon Effect: Multiple Truths
The Rashomon Effect refers to how different individuals can witness the same event but recall and interpret it in conflicting ways. This term originates from Akira Kurosawa’s 1954 film Rashomon, which portrays four witnesses giving starkly different accounts of a murder during a court trial. Each testimony reflects the narrator’s self-interest and biases, illustrating how personal perspective shapes perceived reality.
For example, in the film:
• A bandit claims the killing was in self-defense after the victim’s wife seduced him.
• The wife says she sought her husband’s protection after the bandit attacked her.
• Through a medium, the dead husband alleges his wife conspired with the bandit to murder him.
• A farmer, the final witness, recounts a duel between the husband and bandit, prompted by honor, but admits to stealing from the victim’s body afterward.
Each account reveals not just differing memories but also the influence of trust—or lack thereof—on interpretation. The bandit’s story is dismissed due to his unsavory reputation, while the farmer, initially appearing impartial, is undermined by his theft. Ultimately, the truth remains elusive, as the judges’ decisions hinge on the credibility they assign to each witness.
This effect underscores the central argument: our sense of truth is inseparably tied to our sense of trust. Society has long recognized this, which is why courts rely on rules of evidence to mediate conflicting accounts. However, only recently have neuroscientists begun to unpack the cognitive processes underlying the Rashomon phenomena.
Pascal Wallisch and Michael Karlovich, neurologists from NYU, describe this cognitive mechanism as SURFPAD:
• Substantial
• Uncertainty
• Ramified (branching)
• Forked
• Prior
• Assumptions
• Disagreements
In short, when uncertainty intersects with branching assumptions rooted in prior experiences, it results in disagreements. As David McRaney, author of How Minds Change, eloquently puts it: “When the truth is uncertain, our brains resolve that uncertainty without our knowledge by creating the most likely reality they can imagine based on our prior experiences.”
The Rashomon Effect illustrates this principle vividly. The witnesses in the story interpret the same sensory input differently because their brains fill in gaps with hidden variables—subconscious assumptions that reflect their past experiences and emotional states. This same process explains disagreements over phenomena like “The Dress” and sheds light on how trust mediates perception.
The Dress: Hidden Variables in Action
In 2015, an image of a dress taken in a UK department store circulated around the world, providing a perfect real-world example of how our hidden variable SURFPAD can lead to drastically different perceptions of the same reality.
What made this image so captivating was that people could not agree on what color the dress really was. Some people saw it as White and Gold, while others saw it as Blue and Black. The intriguing question is: Why did you see it one way, and others saw it differently?
The answer is surprising: The people who saw the dress as white and gold had been less exposed to artificial light over their lifetimes than those who saw it as black and blue. In reality, it was a black and blue dress shot under artificial light with a low-quality cell phone camera.
Yes, really.
No one was more intrigued by The Dress than the aforementioned neurologists Pascal Wallisch and Michael Karlovich, as the neurology of color perception is precisely their area of study. Their research has led them to understand that our perceptions are heavily influenced by our past experiences — a key aspect of our personal Markov Blankets.
To illustrate this concept, consider language and color perception:
Many languages do not have a word for the color blue.
Others, such as Russian, have more than one word for blue.
If you present people from these different linguistic backgrounds with a color wheel of blue, where one shade is slightly under or over-saturated, an interesting pattern emerges. People who speak languages with no word for blue often cannot tell the difference, whereas those with more than one word for blue can spot the imposter more easily.
In the case of The Dress, Wallisch and Karlovich suspected a similar mechanism was at play, but this time with light exposure rather than language. They identified several key factors:
The photo was taken with a poor-quality, low-resolution flip phone camera.
Due to the low resolution, our brains have to do more work to interpret the image.
This extra cognitive effort activates our preconceptions of what things are 'supposed to look like' more strongly.
This process of interpreting ambiguous sensory input is called disambiguation. It's how our brain takes low-resolution input, like our 2D sight, and constructs the rich 3D world of our perception. To make this process more efficient, our brain fills in blanks based on what it has perceived before. These preconceptions are essentially the hidden variables in our Markov Blankets.
To test their theory about The Dress, Wallisch and Karlovich designed an experiment:
They brought in subjects and asked them if they saw the dress as black and blue, or white and gold.
They then compared these answers to a survey of where the subjects grew up.
Their findings were revealing:
People who grew up in the south, rural areas, or places where natural light predominates mostly saw white and gold.
This supported the first part of their theory: exposure to natural vs. artificial light explained the difference in how people perceived The Dress.
However, to conclusively prove their mechanism — that it's the pre-programming of our expectations of how colors should look that drives these differences — they needed a second example.
The Crocs Experiment: Testing Our Assumptions
To conclusively prove their theory about how our past experiences shape our perception, Wallisch and Karlovich needed a second experiment. This wasn't an easy test to devise. They needed a picture of something that everyone recognized but was of an indeterminate color, and a color that could be changed with a lighting source. The key was to find an object that was both instantly recognizable, but which had a non-obvious color, so that they could trigger that disambiguation process we discussed earlier.
The answer came from an unexpected source: the fashion crime that is socks with Crocs. To make it even more egregious, they chose pink Crocs with white tube socks under green light.
The researchers explained their findings:
"Under normal lighting conditions, all people see the Crocs as pink. When put under green light, most people see the same Crocs as gray. However, some people — those that believe the socks are white, even though they appear green — are able to look past appearances and see the Crocs as pink, just like they did under regular light."
To illustrate the point, in this example, the hidden SURFPAD variables are set by how much artificial light versus natural light a person has been exposed to over their lifetime.
Again, the key to understanding who saw what, pink or grey, was the type of lighting that the person was familiar with - more or less artificial light. This phenomenon is known as an Interjacent Bi-Stable Visual Illusion. It's "bi-stable" because everyone agrees on one of two possible interpretations, and "interjacent" because no one person can see both interpretations simultaneously.
This is a special case of the more famous Intrapersonal Bi-Stable Visual Illusions, where everyone agrees on one of two possible interpretations, but you can only see one of them at a time, like the duck-rabbit or the old-young woman.
Building on these findings, David McRaney explains why we can see a green sweater in a dark closet or a blue jacket on a dark night. Our brain does a little extra work that helps us see what we expect Ought To Be There. In other words, our brains add in a lot of guesswork to keep our visions consistent with our expectations. So when people incorrectly saw the dress as Gold and White, it was their brain telling them a lie based on their prior assumptions about light.
Critically, these Hidden SURFPAD Variables are something we are both unaware of and not something that can be observed outside of a freak accident and a lab test. Yet, these SURFPAD neurons act as the anchors of our understanding of the world, they are the guideposts around which we twist our perceptions of the world. They are both the literal and figurative sources of our blind spots.
The SURFPAD phenomenon illustrates a fundamental principle of perception: our brains prioritize consistency over accuracy. To make sense of the world, we unconsciously adjust what we see to align with prior experiences. While this usually works well, it can lead to stark disagreements when hidden assumptions differ.
In the context of trust, this has profound implications. Just as we trust our sensory systems to interpret ambiguous visuals, we trust others to provide reliable information about the world. When interpretations clash, trust becomes the deciding factor in resolving the conflict. The Dress serves as a reminder that our perceptions—and by extension, our beliefs—are deeply shaped by the unseen biases of our past experiences.
From Perception to Emotion
Four Basic Emotions
These perceptual experiments reveal more than just how we see - they show us how we feel about what we see. When someone insists The Dress is a different color than what you perceive, your emotional response - frustration, curiosity, or dismissal - reveals how deeply our perceptions are tied to our emotional processing. In essence, Friston's Law describes the physical logic of how minds make sense of the world and change. The mind seeks to minimize free energy— or the element of surprise —but what happens when free energy is released?
The SURFPAD variables that shape our perceptions operate through the same mechanism as our emotional responses - both are ways our brains handle prediction errors. When our SURFPAD assumptions are challenged, when we encounter information that conflicts with our priors, when we release of free energy, according to the Neurologist Mark Solms, that experience is emotion.
Specifically, Solms' research has found that Free Energy release can be divided into 4 basic human emotions -
• Joy (positive surprise): motivates us to seek more of what feels good.
• Fear (negative surprise): helps us avoid threats and uncertainty.
• Sadness (past negative surprise): discourages actions linked to past harm.
• Anger (reversed expectations): spurs action to correct perceived unfairness.
Emotional Decisions
The most commonly cited examples of how emotions affect human behavior come from patients who, for various reasons, have had the emotional part of their brain damaged. For example, Phineas Gage was a railroad construction foreman who survived a severe brain injury in 1848 when an iron rod pierced his skull, damaging his prefrontal cortex. Remarkably, Gage lived, but his personality and behavior changed dramatically. Before the injury, he was responsible and hardworking, but afterward, he became impulsive, unreliable, and most importantly struggled with decision-making.
A more recent example is an unnamed patient with similar prefrontal issues who was asked by his doctor what day would be best for his next appointment. After several minutes of pros and cons about each day, but without coming to a decision, the doctor finally said, ‘How about Tuesday?’ And that is when the patient agreed.
How we receive, act, and react to new information is driven by the impulses of emotion. To be clear, much of that free energy is channeled into bodily reactions. That is why we tend to describe emotions as experiences. But emotions are, as Dr. Laith Al-Shawaf says, a coordination mechanism for our attention. We channel our free energy into emotions that let us better fight, run, comfort, or atone for a mistake. Just like information or knowledge, emotions help us to better navigate our social and physical world.
These emotions form a feedback loop between perception and action. For example:
• Joy and anger fuel engagement and decision-making.
• Sadness and fear lead to withdrawal, reducing energy for further action.
While under lab conditions we can only define 4 emotions active inference implies a range of emotions and intensities. These go from the lowest levels where your emotions are beyond your conscious perception, small surprises, all the way to the ecstasy and heartbreak of missing or making the winning shot in the finals.
The critical point is that living organisms do not learn through reason, logic, or analysis alone; they learn through emotional feedback as well. That's not to say that humans don't use, and can't recognize good reasons or logic, but it does say by definition, a person can't change their mind without feeling an emotion.
This emotional feedback system parallels the dynamics of trust. Positive emotional exchanges, like shared joy or mutual anger at injustice, reinforce trust between individuals, while negative exchanges erode it. This is the foundation of social learning: emotional alignment fosters shared understanding.
Conclusion
There are a number of lessons we can draw from these stories, but the most important are these: disagreements often arise at a level of consciousness that neither you nor the person you disagree with can even recognize. Often people disagree because of not just where they grew up, but what kind of light they grew up with - are you a gamer or a grass toucher? If you are a gamer you will trust other gamers more, not because they are gamers too, but because they see the world like you do. A fellow gamer would never tell me they see a white and gold dress that is plainly a black and blue dress. Maybe those grass-touchers are lying, maybe they are mental, or maybe they just saw more sunlight than the backlight.
And that is the second lesson - what you think determines how you feel and that is how you (re)act. That is the 1-2-3 of active inference. If you are frustrated or angry that someone disagrees with you will act one way, but if you are simply surprised you might look at the phenomena closer, and find out that you see what you believe, and that you can not see what you do not already believe.
In the next section, we are going to learn how people … learn.
Specifically, we are going to put a technical foundation down around this idea of active inference with emotional feedback, what I am calling HumanOS, and see how two people, or agents, can communicate with one another to learn something new.
PS - The Next Chapter is Here!
BIBLIOGRAPHY
Beal, M. J. (2003). Variational Algorithms for Approximate Bayesian Inference. University of Cambridge.
Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181–204.
Damasio, A. R. (1994). Descartes’ Error: Emotion, Reason, and the Human Brain. Avon Books.
Friston, K. (2010). The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11(2), 127–138.
Friston, K. (2005). A theory of cortical responses. Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456), 815–836.
Harlow, J. M. (1848). Passage of an iron rod through the head. Boston Medical and Surgical Journal.
Heider, F., & Simmel, M. (1944). An experimental study of apparent behavior. The American Journal of Psychology, 57(2), 243–259.
Kurosawa, A. (Director). (1950). Rashomon [Film]. Daiei Film.
Girard, R. (1987). Things Hidden Since the Foundation of the World. Stanford University Press.
McRaney, D. (2022). How Minds Change: The Surprising Science of Belief, Opinion, and Persuasion. Portfolio.
Panksepp, J. (1998). Affective Neuroscience: The Foundations of Human and Animal Emotions. Oxford University Press.
Solms, M. (2021). The Hidden Spring: A Journey to the Source of Consciousness. W. W. Norton & Company.
Tomasello, M. (2009). Why We Cooperate. MIT Press.
Wallisch, P., & Karlovich, M. (2017). The dress phenomenon: Neural mechanisms behind perceptual disagreement. Journal of Vision Research.