In nearly every AI strategy I read the same demand comes up. Systems need to become more explainable. Companies need to understand how their models arrive at a decision, because transparency is supposed to be what makes people trust the result. For me, this is not optional: without explainability, I do not consider AI use responsible.
Explainable and transparent AI is, in my view, an important demand.
Then the same strategies go on to recommend ChatGPT, Claude, DALL-E and Midjourney, without a flicker of hesitation. All four run on large language models. Nobody at OpenAI can explain why ChatGPT gives one particular answer to one particular question, not even in rough outline. The models have billions of parameters, and no person can track how those parameters interact to produce a given output. That is what makes them black boxes. I mean the term descriptively, the way an engineer would use it, not as a complaint.
A prompt goes in, an output appears immediately. Change one unimportant word in the prompt and the result can come out completely different. Something happens in between that no one can fully explain.
My first thought was whether this reflects ignorance or intent. I take it to be intent. A company markets its tool as the perfect all-purpose solution for everyone and everything, and explains nothing about how it actually works. The gap between the claim and the disclosure needs a justification, and trade secrecy supplies it. As an outsider I got used to this a long time ago. I use tools every day that I do not understand in the slightest, and so does everyone else. I do not know in detail how my phone works either, and I use it anyway.
But my phone does not make substantial decisions.
That is the line worth drawing. Understanding a tool and trusting the decisions it makes are two different problems. With the phone, I trust the result because I can check it against the world myself. I can tell for myself whether a text reads well or a photo looks right. I am the corrective, and my own judgement is enough to catch what goes wrong. With a decision system, I trust the result because I have no other way to check it. Say an AI tool tells an employer that one candidate is qualified and another is not. Sophisticated methods sit behind that verdict, and the verdict itself may well be sound. But who actually verifies it.
The strategy documents move between these two worlds without ever naming the difference. They recommend ChatGPT for writing and call it productivity, then turn around and recommend AI-based analytics for hiring, lending and other business decisions, calling it transformation instead. Explainability stays a slogan for the strategy documents. Nobody presses the providers to open up the actual mechanism, because the mechanism is proprietary, and the arrangement goes largely unquestioned.
What bothers me most about this is how casual it is. The black box itself barely comes up in the conversation, though it urgently should. What matters is whether you notice you are deciding on grounds you cannot trace yourself.