What Leaders Need to Know About Explainable AI

Explainable AI is a set of tools and methods that helps people understand how and why an AI model reaches a specific decision or prediction. For business leaders, that definition is quickly becoming practical. AI now shows up in back office operations and in the reports and analysis people rely on to make decisions, and each of those outputs eventually lands in front of someone who has to decide whether to trust it. For most organizations, explainable AI for business comes down to that trust and the controls that support it.

Transparency and Explainability Go Together

As AI becomes more common in business settings, two things are getting more important. The first is transparency around when and where AI is used. Businesses build trust with their employees and their customers, and keeping that trust means being open about how AI plays a role in the work they do and the decisions they make. The NIST AI risk management framework treats transparency and explainability as separate characteristics of trustworthy AI that support each other, which matches how they work in practice.

The second is explainability, and transparency tends to require it. That makes explainability something to plan for as a requirement whenever AI becomes part of a process. Once an organization says it uses AI in a process, the next question is how and why. Explainability means being able to describe where and how AI is used and giving as much insight as possible into why it produced a particular output. To some extent, nobody can fully explain why an AI model returns the result it does, and that is a hard thing for some people to accept.

Why This Gets Hard in Audit and Financial Reporting

The challenge is sharpest in the audit space. Auditing standards and internal controls can push an organization toward wanting to explain everything a system does. When that isn’t possible, it raises a difficult question about how you justify a number on the financial statements if you can’t explain how that number got there. That’s why explainability needs to be part of the thinking from the outset, as an organization decides where AI will play a role in its controls and financial reporting.

People Can’t Always Explain Their Judgment Either

People can’t always explain how they reach the decisions and conclusions they make, because a lot of judgment goes into that work. That makes the comparison to people useful. Many accountants are used to software that returns the same output every time, so AI takes a different mental model. It works more like hiring someone. If you give a new hire the same task more than once, the answer probably won’t come back exactly the same each time, and their work improves as they get more training and information. As teams get more comfortable with AI and the models keep getting better, the outputs and their impact on the organization improve too. You still review what comes back before you make a decision or send something out the door, the same way you would with a person.

When an AI tool produces an output, whether that’s a narrative or a number, get as much of an explanation from it as you can. Then build human review and approval processes that verify any claims and confirm that anything stated or used as an input into a decision is grounded in facts and can be backed up. That holds regardless of how the AI came to its conclusion. There needs to be some reconciliation or justification around the output either way.

Explainability ends up being a combination of what you can learn from the model itself and how comfortable the organization is relying on what it generates with AI as part of the process. A lot of that comfort comes from the controls the organization puts around those outputs.

There Is No Gold Standard Yet

When leaders think about how much explainability they need, it helps to remember how early this is. Businesses are still in the early stages of truly embedding AI into their processes, and even with growing adoption, it’s a new technology compared with the long history of business operations. There aren’t gold standards or practical guidebooks yet for how organizations should build explainability into their systems and the way they use AI.

The AI companies are still learning too. Anthropic CEO Dario Amodei has written that even his own researchers can’t pinpoint why a generative AI model makes the specific choices it does when summarizing a financial document, including the occasional mistake. Most organizations are probably explaining less than would be ideal, simply because the ability to do more isn’t there yet. 

Where a full explanation isn’t possible, transparency carries more of the weight. Pair the right level of controls for the risk involved with a clear statement about how AI is being used and what risks come with it. Then explain how the organization got comfortable with those risks. That gives the people using the information what they need to make their own determination about how much weight to put on it.

What Leaders Need to Know About Explainable AI for Business

Some people won’t start with AI at all because they don’t trust anything it produces, and others use it without looking at what it gives them, which is how low-quality work slips through. The right place sits somewhere in between. The main takeaway is to keep moving forward with AI even when its outputs are hard to explain, and to let the level of explainability guide how you design reviews and controls and how transparent you are about using it. James Moore Digital helps organizations decide where AI and automation belong in their workflows and what reviews and controls should surround them. If you’re working through how much explanation your AI-assisted reporting needs, reach out to James Moore Digital to talk it through.

 

All content provided in this article is for informational purposes only. Matters discussed in this article are subject to change. For up-to-date information on this subject please contact a James Moore professional. James Moore will not be held responsible for any claim, loss, damage or inconvenience caused as a result of any information within these pages or any information accessed through this site.