How Do People Learn Judgment in Systems They Didn't Build?

Many of the systems we rely on every day are opaque by design.

Sometimes that opacity is technical. We interact with a platform, an automated tool, or an AI system without knowing exactly how it reached a decision.

Sometimes it is institutional. We are told that a process exists, but not how it works in practice, whose judgment carries the most weight, or which considerations will ultimately determine the outcome.

And sometimes it is organizational. The official instructions say one thing, experienced colleagues know something more nuanced, and the person encountering the system for the first time is expected to somehow understand the difference.

Most of us did not build these systems. We do not control them. We were not trained to reason about them. Yet we are routinely expected to exercise good judgment inside them.

I keep returning to the tension this creates: What happens when a system demands judgment from people without providing the conditions they need to develop it?

This question sits at the center of much of my work in responsible AI, product strategy, trust, and safety. But it extends well beyond technology. It appears anywhere people are asked to make consequential decisions within systems where authority is diffuse, feedback is uneven, and accountability is not shared equally.

In those environments, “use your best judgment” can sound less like permission and more like a test for which the instructions have been withheld.

The distance between the rules and the work

When organizations want people to make better decisions, the instinct is often to create more documentation.

We write policies. We develop training. We build onboarding programs. We add approval steps. We produce decision trees that attempt to account for every possible situation.

These tools matter. Clear rules can establish boundaries, create consistency, and make expectations easier to communicate. But rules are not the same thing as judgment.

A rule tells me what generally should happen. Judgment helps me determine what matters in this particular situation.

That distinction becomes especially important when the facts are incomplete, the available options are imperfect, or several legitimate priorities are in tension. Those are precisely the moments when people are most likely to be told to exercise judgment. They are also the moments least likely to be resolved by consulting one more page of documentation.

Imagine a team reviewing whether a new AI feature is ready to launch. The policy may identify the categories of risk that should be considered. The product requirements may define acceptable performance. An evaluation may show that the system meets the established threshold.

But someone still has to decide whether the remaining failures are tolerable.

Does the system perform differently across languages or communities? Are the errors merely inconvenient, or could they compound an existing disadvantage? Does the evaluation reflect how people will actually use the product? Is the threshold appropriate for this use case, or was it inherited from a different context?

The documentation can inform that decision. It cannot make the decision self-executing.

The same dynamic appears in less technical settings. A school leader may have a procurement policy and still need to decide whether a particular tool is appropriate for their students. A manager may understand the formal performance process and still have to interpret what support looks like for a struggling employee. A platform moderator may know the rule and still encounter a case that technically complies while clearly violating the spirit of the system.

The difficult work happens in the space between what the rule can specify and what the situation requires.

“You should have known better”

We tend to notice this gap only after something goes wrong.

A decision causes harm. An employee escalates too late. A user behaves in a way the product team did not anticipate. A reviewer follows the policy but reaches an outcome that feels indefensible.

Then comes the familiar response: They should have known better.

Sometimes that criticism is warranted. People do make careless decisions. They ignore clear information. They act against advice or prioritize convenience over responsibility.

But “they should have known better” can also conceal a design failure.

What opportunities did the person have to learn what better looked like? Were they shown examples of strong decisions, including the reasoning behind them? Did they receive feedback before the consequences became serious? Could they see how more experienced people handled ambiguity? Were they encouraged to ask questions, or did asking reveal that they did not already know the answer?

Too often, we treat judgment as an individual possession. A person either has it or they do not. If they make the right call, they are considered thoughtful or experienced. If they make the wrong one, the failure is attributed to their competence, character, or common sense.

I am increasingly convinced that this is incomplete.

People bring different experiences and capabilities into a system, certainly. But the environment also teaches them what to notice, what to ignore, what to question, and what will be rewarded.

If people repeatedly see speed valued over care, they learn that speed is the real priority, regardless of what the policy says. If raising a concern creates more work but quietly accepting risk earns praise, they learn which behavior the organization actually wants. If decisions arrive without explanations, people may learn the outcome, but they cannot learn the reasoning that produced it.

In that sense, systems are always teaching judgment. The question is whether they are teaching it intentionally.

Judgment develops in company

Think about how people usually learn to make good decisions in a new environment.

They watch what experienced people do. They ask why one case was handled differently from another. They borrow language that helps them describe what they are seeing. They test their interpretation with someone they trust. They make smaller decisions, receive feedback, and gradually take responsibility for larger ones.

This is not simply knowledge transfer. It is the development of a shared way of noticing.

Formal instruction can accelerate that process, but much of it happens socially. We learn through examples, analogy, conversation, correction, and observation. We begin to understand not only the rule, but the values and tradeoffs beneath it.

This is one reason experienced teams can appear to make complex decisions quickly. Their speed may look like intuition, but that intuition is often compressed experience. They recognize patterns because they have seen related situations before. They know which detail is unusual because they understand what usually happens. They can anticipate second-order effects because they remember what followed a similar decision last time.

The danger is that this knowledge often remains informal.

It lives in the minds of a few experienced people. It surfaces in private conversations but not in the official process. It is available to employees with the right relationships, confidence, or proximity to power, while everyone else is left to infer it.

That does not merely make learning inefficient. It can reproduce inequity.

When the real rules are unwritten, familiarity becomes an advantage. People who already understand the institution’s language and norms are more likely to be perceived as exercising good judgment. People encountering those norms for the first time may be evaluated against expectations they were never given a fair opportunity to learn.

Legibility is part of the design

This brings me to legibility.

A legible system does not need to expose every internal detail or eliminate every form of complexity. It does, however, help people understand how meaningful decisions are made.

It makes the relevant factors visible. It communicates where discretion exists. It shows how competing priorities are weighed. It provides examples that reveal how a principle changes across contexts. When possible, it closes the loop by explaining not only what was decided, but why.

This matters for both the people making decisions and the people affected by them.

If I am responsible for a decision, legibility helps me form better judgment. I can compare my reasoning with prior examples, identify what I may be missing, and understand when the situation warrants escalation.

If I am affected by a decision, legibility gives me a basis for evaluating it. I may still disagree with the outcome, but I am not forced to reverse-engineer the system from its effects.

This becomes especially important as organizations introduce more automated and AI-assisted decision-making. We often focus on whether a model can explain its output. That is an important technical and governance question. But model explainability alone does not make the larger system legible.

A system can explain which signals influenced an output while leaving unanswered who selected those signals, why the threshold was chosen, when a human can intervene, and who is accountable when the result is wrong.

The model is only one participant in the decision. Understanding the system requires us to examine the people, policies, incentives, interfaces, and institutional choices surrounding it.

Designing for judgment, not just compliance

If judgment is shaped by its environment, then improving judgment requires more than telling individuals to make better choices.

We can begin by asking different design questions.

Can people see examples of how difficult decisions were made? Do we preserve the reasoning behind decisions, or only record the final outcome? Are people able to practice on lower-stakes cases before being given consequential authority? Does feedback arrive soon enough to change future behavior? Can someone question a decision without being treated as an obstacle to progress?

We can also pay closer attention to what the system rewards.

No training program can compensate for incentives that punish the behavior it claims to encourage. If an organization tells people to raise risks but rewards only speed, the incentive is the more credible instruction. If a platform asks users to behave responsibly while optimizing relentlessly for engagement, its design is teaching a different lesson from its community guidelines.

Good judgment requires information, but it also requires permission: permission to pause, to ask, to surface uncertainty, and to revise an earlier view when new evidence emerges.

That does not mean every decision should become a group deliberation or every system should move slowly. The goal is not endless process. It is to make the moments that require judgment easier to recognize and the reasoning around them easier to access.

A collective capability

I am not interested in removing individual responsibility from decision-making. I am interested in placing that responsibility in context.

People remain accountable for the choices they make. But systems also shape which choices feel possible, which risks become visible, and which forms of reasoning are treated as legitimate.

When we treat judgment solely as a personal trait, we limit our options. We can hire for it, praise it, or punish its absence. When we treat judgment as a capability that can be developed, we can design environments that help more people practice it well.

That shift matters because the systems we are building are not becoming simpler. More decisions are being distributed across people, platforms, institutions, and automated tools. Authority is becoming harder to locate even as the consequences of failure remain very real.

Rules will continue to matter. Training will continue to matter. Compliance will continue to matter.

But none of them can substitute for the social and institutional conditions through which judgment grows.

So I find myself returning to a more useful question than whether people should know better:

What would we need to make visible, discuss, practice, and reinforce so that they actually could?