Why Trust in AI Agents Will Be a Challenge

AI agents are not people

The age of AI Agentics is here … a multi-trillion-dollar opportunity.

— Jensen Huang, January 6, 2025

We believe that, in 2025, we may see the first AI agents “join the workforce” and materially change the output of companies.

— Sam Altman, January 6, 2025

Behind these glowing predictions on the imminent leading societal role of AI agents lies an important assumption—one that seems to be mostly invisible to purveyors of AI agents and associated hype. The assumption is that humans will trust AI agents to take over tasks that humans now perform themselves.

It’s not obvious that this assumption is justified, at least in the near future. Even if we assume that the long-term future of agents is bright, there is reason to believe—at least in the medium term—that trust will be an important challenge to widespread adoption of agents.

Headwinds for AI from trust issues matter because differences in the pace and manner of AI adoption can have huge impacts on society and the economy. Despite the frenzy of breathless reports on the massive impacts we will see from AI within a few years—both from those who want to build AI and those who want to block it—the reality appears to be more measured. For example, the paper ‘The simple macroeconomics of AI’, published last summer by leading MIT economist Daron Acemoglu, estimates that AI will increase global GDP between 1.1 to 1.6 percent over the next 10 years, and raise productivity by about 0.05 percent annually.

In a well-known example of failure of prediction of AI impact, Geoff Hinton said in 2016:

We should stop training radiologists now. It’s just completely obvious that within five years, deep learning is going to do better than radiologists.

In fact, the New York Times recently reported that the Mayo Clinic has increased radiologist staffing by 55% since Hinton’s statement, while improving their performance substantially using AI support.

In this world of massive uncertainty about the impact of AI, trust is central. The challenge of trust in AI agents can be deconstructed to three central issues:

• AI agents are not people.

• People trust credibility and expertise.

• Digital trust is technically challenging.

Let’s take a look at each of these.

AI agents are not people

It’s important to define what we mean by an AI ‘agent’. Amazon Web Services offers a good definition:

An artificial intelligence (AI) agent is a software program that can interact with its environment, collect data, and use the data to perform self-determined tasks to meet predetermined goals. Humans set goals, but an AI agent independently chooses the best actions it needs to perform to achieve those goals [my emphasis].

Unfortunately, the term ‘agent’ has been hijacked by the AI hype cycle, as NYU computer science professor Ernie Davis rants in recent commentary about Google DeepMind’s remarkable AlphaEvolve framework, explaining both that AlphaEvolve “is in no sense an agent” and that ‘agent’ is not a new term in the AI field¹.

Focusing on AI agents that do “perform self-determined tasks to meet predetermined goals”, let’s state the obvious: although such agents can perform tasks on behalf of people, they are not people. They are in fact not at all similar to people—in at least three important ways:

• AI agents perform narrow, specifically-defined tasks. This constraint will gradually relax over time, and some think it already has started to do so: Azeem Azhar recently published a post here on Substack ‘Claude 4 — The first universal assistant?’ Realistically, though, there appears to be no medium-term prospect of AI agents that autonomously handle a broad range of tasks.

• AI agents and other models fail in distinctive and unpredictable ways, including because their behavior is usually stochastic. A key aspect of this is the tendency of large language models to confidently hallucinate²—for which they can be described as ‘plausibility monsters’—which still has no broad solution. Humans also frequently fail, but not usually in this way. (On the other hand, AI agents avoid various human failings, such as becoming tired or intoxicated.)

• Probably most important, AI agents lack key attributes and needs of humans, including consciousness and emotions; needs for food, water, sleep, protection from the elements, and companionship; and the desire for sex.

So in evaluating Sam Altman’s claim that AI agents will “join the workforce” (even giving him the benefit of doubt that this will happen fairly rapidly), we should recognize that agents will not become workers in the same way that human are, even where they take human jobs. This will remain the case for the foreseeable future, even where agents are embodied in humanoid robots that have the general physical form of people.

The human-like aspect of humanoid robots is intended to lead humans to believe that they are like humans. Humans have a long history of being convinced by the humanity of AIs, ranging from psychotherapist ELIZA in the 1960s to a growing trend of AI romantic ‘relationships’. But to whatever extent we emotionally believe AIs are like people, that does not make them so. And for humanoid robots, trust issues are also legion, as the MIT Technology Review explored late last year in ‘Will we ever trust robots?’

People trust credibility and expertise

So what is it that will lead humans to trust non-human AI agents?

A solid hypothesis is that the bases for trust in AI agents will be similar to those that apply to trust between humans. It is hard to see how it could be fundamentally otherwise, because the instinct to trust (or not) has evolved over millennia. On the other hand, sellers and deployers of AI agents will certainly develop tricks that encourage us to trust the agents, whether or not that trust is justified from a traditional perspective.

Human trust is complicated. An excellent 2023 study ‘How and why humans trust: A meta-analysis and elaborated model’ from a group of US academics breaks it down into factors involving the trustor (the person who trusts), the trustee (the person who is trusted) and contextual factors. With respect to characteristics of the trustee—which is most relevant to considering what AI agent characteristics will lead to trust—the study identifies strongest correlations with transparency (r = 0.48), expertise (r = 0.41), reliability (r = 0.32), performance (r = 0.28) and reputation (r = 0.27).

Simplifying this analysis, people trust credibility (i.e. transparency, reliability and reputation) and expertise³ (i.e. both perceived expertise and expertise demonstrated through performance)⁴.

A recent experience with PlaylistBuilder, the AI-driven YouTube curation application that my start-up is building and commercializing, starkly illustrated for me the central roles of credibility and expertise in trust. This experience involves a tale of two marketing campaigns. Paraphrasing Dickens, one was the among the best of campaigns, the other … less successful.

We have focused PlaylistBuilder marketing on LinkedIn, which is highly content-driven and where I have a strong personal network. Usually, I connect on LinkedIn with people whom I meet in person, and interesting and credible people who reach out to me. But recently, I have reached out pro-actively to 2nd-degree connections (people with whom I share at least one common connection) who have academic roles, seeking dialogue about AI and edtech.

This effort has been tremendously successful. About 50% of those to whom I reach out have agreed to connect with me. Our marketing advisors tell me they rarely see this type of campaign yielding above 30% success, and have never seen results above 40%. I put this success down to various factors, including (i) my many years of work on AI and edtech, (ii) the trust-building effect of 2nd-degree connections and (iii) that my outreach has been entirely about content (and not sales). All of these factors build credibility, and the first factor demonstrates expertise.

The other campaign has focused on bringing traffic to PlaylistBuilder, by inviting people to try the service, which is currently available for free. The campaign has had success in bringing thousands of people to our website, but (to our disappointment) many fewer have chosen to sign in to the PlaylistBuilder service—with by far the best response from people with whom I am already connected.

Our hypothesis is that PlaylistBuilder’s requirement to ‘Sign in with Google’—which is needed because we help people better use their YouTube accounts—can be dissuasive for people who don’t know PlaylistBuilder or the team behind it. Indeed, it’s reasonable for people to hesitate to trust unfamiliar web services with access to their Google accounts (certainly, that’s how I feel about my own Google account). Apparently, we have not yet established sufficient credibility and expertise to convince large numbers of people to trust and connect with us via Google.

We are working hard to address this, through a variety of measures to steadily build credibility for PlaylistBuilder and demonstrate our expertise. We are also considering a simplified version of PlaylistBuilder that people can try for free without log-in.

This tale of two marketing campaigns contains a broader message: most AI-based applications, including agents, face the same challenge of building trust through credibility and expertise.

Digital trust is technically challenging

The third component of the trust challenge for AI agents is that building trust—through credibility and expertise—between humans and non-human agents, over digital networks, is technically challenging.

I recently wrote about general issues of AI trust in my post ‘Trust in a World of Pervasive AI’, where I focused on a technical definition of trust:

firm and evidence-supported belief in the ability and reliability of an AI system to meet defined requirements.

This definition is closely aligned to the focus on credibility and expertise that I set out in the current post—reliability aligns with credibility and ability with expertise—but with a focus on technical requirements and evidence that they are met. I have since posted a couple of notes with further resources on technical aspects of AI trust.

There is plenty of evidence for the point that technical trust is challenging and complex.

In the 1990s, as the Internet hit its growth spurt towards the dotcom boom, the leading standard for digital trust was public key infrastructure (PKI) based around the X.509 standard. PKI (with which I was deeply involved as a lawyer) was a very powerful framework, but doomed by extreme complexity. PKI remains very important in some applications, most visibly the SSL/TLS protocols used to secure HTTPS communications for the World Wide Web and other applications. However, as a solution for plug and play security in a wide variety of applications—which it was marketed as in the 1990s—PKI has been largely a failure.

Self-sovereign identity (SSI) is an emerging blockchain-based solution that could play a similar role for digital trust to that which was intended for PKI. SSI is a key part of Web3, with its core concept of decentralized trust; and SSI has the significant advantage that its key data components—decentralized identifiers (DIDs) and verifiable credentials (VCs)—can be stored on a public blockchain, eliminating a main complexity of PKI that a hierarchical chain of certificate authorities is typically required to store X.509 certificates. A couple of years ago, I was optimistic that this advantage would help SSI to succeed where PKI did not, but unfortunately SSI seems to be descending into the same morass of complexity that plagued PKI.

Owing to the lack of a global digital authentication/trust standard like PKI or SSI, it often remains the case that “On the Internet, no one knows you’re a dog”, or a bot, or an AI agent. To make AI agents viable at scale, we must solve this problem, ideally with some kind of global identity standard that allows seamless and secure interactions between humans and agents, and among agents. This will not be easy to achieve.

Building trust in AI agents

Given the huge potential for AI agents, I am confident that solutions to these challenges will emerge. But it is not currently obvious what the solutions will be, so for the time being we should expect trust issues to blow at least moderately strong headwinds for AI agents.

While the solutions to these trust issues are not yet visible, there are some clear signposts from the three trust issues discussed above:

• We must recognize that AI agents are not people, and design a trust infrastructure that fits their distinctive characteristics.

• At least for interactions between humans and AI agents, solutions should take account of the key trust dimensions of credibility and expertise. (For agent-to-agent interactions, trust dimensions may be different and/or more technical, but to the extent that agent-to-agent processes serve human ends, credibility and expertise will still be relevant.)

• It will be crucial to develop scalable technical solutions for trust that are widely accepted and not overly complex.

Across these considerations, we must recognize that trust is ultimately local, because credibility and expertise cannot be generalized, and depend upon context in a nearly infinite variety of situations. I have recently written extensively about the importance of local solutions for both global and local challenges.

As I have also written recently, there is a huge business opportunity to solve the important challenges of AI trust. I am actively looking for opportunities to pursue this opportunity, and I would love to discuss with those who are interested.

1. Davis’ rant is worth reproducing here: “AlphaEvolve is in no sense an agent. Here I have to rant. An agent is something that acts. A robot or a self-driving car is an agent; they do physical things. A commercial web pages that sells shoes is an agent; it places a charge on your credit card and arranges for a pair of shoes to be mailed to you. A thermostat is an agent; it turns the furnace on and off. AlphaEvolve is no more an agent than a desk calculator or the C compiler. The word “agent” had a fairly specific and useful meaning in AI research from the early 1980s until a few months ago; then it was taken up as a hot buzzword by Google, OpenAI, and everyone else; now it has become completely vacuous. End rant.”

2. The image for this post was produced by Gemini 2.5 Flash in response to the prompt “Please produce an image of an AI agent struggling with trust issues”. I love the hallucinated caption, with one intelligible word ‘trust’.

3. I spoke at length about expertise on a recent episode of The Sales Scoop podcast.

4. Here, I categorize reliability (doing what is expected, when it is expected) as an element of credibility and performance (delivering to a high standard) as an element of expertise. But there is overlap of these terms.