New Tool Reveals How AI Makes Decisions

Large language models such as ChatGPT tend to make things up. A new approach now allows the systems to explain their responses—at least partially.

Most AI programs function like a “black box.” “We know exactly what a model does but not why it has now specifically recognized that a picture shows a cat,” said computer scientist Kristian Kersting of the Technical University of Darmstadt in Germany to the German-language newspaper Handelsblatt. That dilemma prompted Kersting—along with computer scientists Patrick Schramowski of the Technical University of Darmstadt and Björn Deiseroth, Mayukh Deb and Samuel Weinbach, all at the Heidelberg, Germany–based AI company Aleph Alpha—to introduce an algorithm called AtMan earlier this year. AtMan allows large AI systems such as ChatGPT, Dall-E and Midjourney to finally explain their outputs.

“And it seems like AtMan can do even more,” says Deiseroth, who also helped develop the algorithm. “You could use the explanations from AtMan specifically to improve AI models.” Past work has already shown that smaller AI systems produce better results when trained to provide good reasoning. Now it remains to be investigated whether the same is true for AtMan and large transformer models. “But we still need to check that,” Deiseroth says.

https://www.scientificamerican.com/article/new-tool-reveals-how-ai-makes-decisions/