What Does it Mean to Design for Trust When Incentives are Misaligned?

“Trust us.”

Most systems do not say this quite so directly. The request is usually embedded in other language.

Your safety is our priority.

We value transparency.

We are committed to responsible innovation.

These statements make a promise about how an institution, platform, or product intends to behave. Sometimes that promise is sincere. The people making it may care deeply about building something worthy of trust.

But people do not experience intentions.

They experience decisions. They experience consequences. They notice what happens when something goes wrong, whether anyone explains the outcome, and whether they have any meaningful way to challenge it.

A system can say that safety matters.

Its incentives reveal how much.

Trust isn’t a byproduct of good intentions. It’s shaped by incentives, power, and lived experience. Designing for it means treating misalignment as a starting point, not a failure.

In practice, trust is shaped by more than whether a system works as intended. It develops through repeated encounters with what a system rewards, what it overlooks, and whose interests it protects when priorities come into conflict.

So when I hear that an organization wants to “build trust,” I find myself asking a different question:

What is the system giving people reason to believe?

The promise and the operating model

Many systems make trust part of their stated values while embedding incentives that undermine it.

A company may publicly prioritize user safety while rewarding teams primarily for growth. A platform may encourage careful review while evaluating performance through speed and volume. An institution may emphasize transparency while making its decisions difficult to understand or challenge. An organization may ask employees to raise concerns while quietly penalizing the people who slow down a launch.

None of these systems needs to be intentionally deceptive for trust to erode.

The contradiction can emerge from ordinary organizational choices. Metrics are assigned. Deadlines are set. Ownership is divided. Certain outcomes receive attention from leadership. Others appear only in risk reports, support tickets, or the experiences of people with limited power.

Each choice signals what matters.

Over time, those signals become more credible than the organization’s language.

This does not mean growth, speed, or efficiency are inherently opposed to trust. Systems need to operate. Products need to ship. Organizations need to make decisions without resolving every uncertainty.

The problem appears when one priority is formally described as essential but another is consistently rewarded.

Under those conditions, trust becomes aspirational while the competing incentive becomes operational.

People learn the difference.

People evaluate what systems do

Trust is sometimes discussed as if it were a feeling that can be produced through better communication.

Explain the system more clearly. Publish the principles. Add a notice. Give people more information about the organization’s intentions.

Communication matters. But communication cannot repair a contradiction the underlying system continues to produce.

If a platform explains that it values user control while repeatedly making its controls difficult to find, people will draw conclusions from the interface.

If an AI system is described as accountable but affected users cannot understand or contest consequential outputs, people will draw conclusions from the absence of recourse.

If employees are told to speak openly but see that disagreement carries professional costs, they will draw conclusions from what happens to those who speak.

Those conclusions may be called skepticism, resistance, or lack of trust.

They may also be accurate readings of the system.

This is an important shift for me. When people do not trust a system, the first question should not be how to persuade them. It should be whether the system has behaved in a way that makes their response reasonable.

People are constantly gathering evidence.

Can I predict what this system will do?

Will the same standard apply when the outcome is inconvenient for the institution?

If a mistake is made, will anyone acknowledge it?

Can I challenge a decision without making myself more vulnerable?

Does the system respond differently depending on who is affected?

The answers form an experience of trustworthiness long before anyone asks people whether they trust the system.

The burden is often placed on the person carrying the risk

There is also a power question embedded in every request for trust.

Who is being asked to trust whom?

And who bears the consequences if that trust is misplaced?

An organization introducing a new automated system may experience uncertainty as a manageable business risk. The person evaluated by that system may experience the same uncertainty as a threat to employment, access, safety, or opportunity.

A platform may understand an error as one imperfect outcome among millions. For the person affected, that single outcome may be the entire experience.

The stakes are not evenly distributed. Neither is the ability to recover.

Yet institutions often place the burden of trust on the people carrying the greater risk. Users are asked to consent. Employees are asked to embrace change. Communities are asked to believe that safeguards will work. People affected by automated decisions are asked to accept that the system is broadly accurate, even when little has been done to make their individual experience understandable.

Trust, in this framing, becomes a moral expectation.

A good employee trusts leadership. A reasonable user accepts the explanation. A cooperative community gives the new system a chance.

But trust cannot be made responsible simply by making distrust seem unreasonable.

When people have limited power, skepticism may be one of the few protective tools available to them. Disengagement may be a response to repeated evidence that participation will not change the outcome. Performative compliance may be what remains when a system demands agreement but offers no meaningful influence.

These responses are not always productive. But they are not always irrational either.

Sometimes they are feedback.

Misaligned incentives become visible through behavior

Organizations often acknowledge that their incentives are imperfect. The more difficult step is examining what those incentives cause people to do.

If speed is rewarded more consistently than care, teams will learn to treat care as negotiable.

If growth is celebrated while harms are discussed privately, people will learn which outcomes advance their careers.

If compliance is easier to demonstrate than understanding, people will optimize for evidence that the process was followed.

This is not simply a matter of individual ethics. Incentives shape what information reaches decision-makers, which risks receive attention, and how much friction a person must absorb to act against the dominant priority.

A well-intentioned person can still behave in ways that erode trust when the system repeatedly rewards those choices.

That is why codes of conduct and value statements, while useful, are not sufficient. They tell people how the organization would like to understand itself. Incentives show people how the organization actually operates under pressure.

Pressure is revealing.

When deadlines tighten, which review is shortened?

When revenue is at stake, which concern becomes acceptable?

When accountability would be costly, who is asked to compromise?

The answers do not need to be perfect. Trustworthiness does not require a system that never makes tradeoffs or causes harm.

It requires a system that does not hide the tradeoff behind the promise.

Trust needs more than transparency

Transparency is often offered as the primary solution to mistrust.

Make the policy public. Explain how the system works. Disclose the use of automation. Document the decision.

These are important steps, but transparency can become another form of performance if the information is inaccessible, incomplete, or disconnected from a person’s ability to act.

A system may disclose that an automated tool contributed to a decision without explaining what role it played. It may publish its standards without showing how those standards are applied. It may notify someone of an outcome without providing a credible way to question it.

Information has been provided. Power has not meaningfully changed.

For transparency to support trust, it needs to help people understand what happened and what they can do next.

That is why I think designing for trust requires at least four things: predictability, legibility, recourse, and incentive alignment.

Predictability does not mean that every outcome must be identical. It means people can reasonably anticipate how the system will behave and which principles will govern its decisions.

Legibility means that the factors, authority, and reasoning behind meaningful decisions are visible enough to evaluate.

Recourse means that when the system gets something wrong, affected people have a credible path to challenge the outcome, seek correction, or obtain repair.

Incentive alignment means the system does not consistently reward behavior that contradicts the values it asks people to believe.

None of these elements can guarantee trust. Nor should they.

People may reasonably examine the same evidence and reach different conclusions, particularly when their histories with similar institutions differ. Designing for trust does not mean engineering a particular emotional response.

It means creating conditions under which trust would be a reasonable judgment.

Trust is a verdict

The language of “building trust” can make trust sound like something an organization produces and gives to people.

I think the direction runs the other way.

Institutions build systems. They establish incentives. They distribute power. They decide what will be explained, what can be challenged, and what happens after failure.

People observe the evidence and decide what it means.

Trust is their verdict.

This is why designing for trust cannot begin with a communications strategy or a request for confidence. It begins with an examination of the system itself.

What does it reward?

What does it make predictable?

Whose experience counts as evidence?

Who can question a decision?

What happens when the system causes harm?

And when values come into conflict, which one wins in practice?

Misaligned incentives do not make trust impossible. Pretending that the misalignment does not exist makes trust much harder to earn.

A trustworthy system can acknowledge its competing priorities. It can make the resulting tradeoffs visible. It can create safeguards that remain meaningful when another incentive becomes more powerful. It can give people ways to respond when its promises and behavior diverge.

Designing for trust, then, is not about persuading people that a system deserves their confidence.

It is about doing the work required to become worthy of it.