Are You Being Unfair to LLMs?

hype surrounding AI, some ill-informed ideas about the nature of LLM intelligence are floating around, and I’d like to address some of these. I will provide sources—most of them preprints—and welcome your thoughts on the matter.

Why do I think this topic matters? First, I feel we are creating a new intelligence that in many ways competes with us. Therefore, we should aim to judge it fairly. Second, the topic of AI is deeply introspective. It raises questions about our thinking processes, our uniqueness, and our feelings of superiority over other beings.

Millière and Buckner write [1]:

In particular, we need to understand what LLMs represent about the sentences they produce—and the world those sentences are about. Such an understanding cannot be reached through armchair speculation alone; it calls for careful empirical investigation.

LLMs are more than prediction machines

Deep neural networks can form complex structures, with linear-nonlinear paths. Neurons can take on multiple functions in superpositions [2]. Further, LLMs build internal world models and mind maps of the context they analyze [3]. Accordingly, they are not just prediction machines for the next word. Their internal activations think ahead to the end of a statement—they have a rudimentary plan in mind [4].

However, all of these capabilities depend on the size and nature of a model, so they may vary, especially in specific contexts. These general capabilities are an active field of research and are probably more similar to the human thought process than to a spellchecker’s algorithm (if you need to pick one of the two).

LLMs show signs of creativity

When faced with new tasks, LLMs do more than just regurgitate memorized content. Rather, they can produce their own answers [5]. Wang et al. analyzed the relation of a model’s output to the Pile dataset and found that larger models advance both in recalling facts and at creating more novel content.

Yet Salvatore Raieli recently reported on TDS that LLMs are not creative. The quoted studies largely focused on ChatGPT-3. In contrast, Guzik, Erike & Byrge found that GPT-4 is in the top percentile of human creativity [6]. Hubert et al. agree with this conclusion [7]. This applies to originality, fluency, and flexibility. Generating new ideas that are unlike anything seen in the model’s training data may be another matter; this is where exceptional humans may still be at an advantage.

Either way, there is too much debate to dismiss these indications entirely. To learn more about the general topic, you can look up computational creativity.

LLMs have a concept of emotion

LLMs can analyze emotional context and write in different styles and emotional tones. This suggests that they possess internal associations and activations representing emotion. Indeed, there is such correlational evidence: One can probe the activations of their neural networks for certain emotions and even artificially induce them with steering vectors [8]. (One way to identify these steering vectors is to determine the contrastive activations when the model is processing statements with an opposite attribute, e.g., sadness vs. happiness.)

Accordingly, the concept of emotional attributes and their possible relation to internal world models seems to fall within the scope of what LLM architectures can represent. There is a relation between the emotional representation and the subsequent reasoning, i.e., the world as the LLM understands it.

Furthermore, emotional representations are localized to certain areas of the model, and many intuitive assumptions that apply to humans can also be observed in LLMs—even psychological and cognitive frameworks may apply [9].

Note that the above statements do not imply phenomenology, that is, that LLMs have a subjective experience.

Yes, LLMs don’t learn (post-training)

LLMs are neural networks with static weights. When we are chatting with an LLM chatbot, we are interacting with a model that does not change, and only learns in-context of the ongoing chat. This means it can pull additional data from the web or from a database, process our inputs, etc. But its nature, built-in knowledge, skills, and biases remain unchanged.

Beyond mere long-term memory systems that provide additional in-context data to static LLMs, future approaches could be self-modifying by adapting the core LLM’s weights. This can be achieved by continually pretraining with new data or by continually fine-tuning and overlaying additional weights [10].

Many alternative neural network architectures and adaptation approaches are being explored to efficiently implement continuous-learning systems [11]. These systems exist; they are just not reliable and economical yet.

Future development

Let’s not forget that the AI systems we are currently seeing are very new. “It’s not good at X” is a statement that may quickly become invalid. Furthermore, we are usually judging the low-priced consumer products, not the top models that are too expensive to run, unpopular, or still kept behind locked doors. Much of the last year and a half of LLM development has focused on creating cheaper, easier-to-scale models for consumers, not just smarter, higher-priced ones.

While computers may lack originality in some areas, they excel at quickly trying different options. And now, LLMs can judge themselves. When we lack an intuitive answer while being creative, aren’t we doing the same thing—cycling through thoughts and picking the best? The inherent creativity (or whatever you want to call it) of LLMs, coupled with the ability to rapidly iterate through ideas, is already benefiting scientific research. See my previous article on AlphaEvolve for an example.

Weaknesses such as hallucinations, biases, and jailbreaks that confuse LLMs and circumvent their safeguards, as well as safety and reliability issues, are still pervasive. Nevertheless, these systems are so powerful that myriad applications and improvements are possible. LLMs also do not have to be used in isolation. When combined with additional, traditional approaches, some shortcomings may be mitigated or become irrelevant. For instance, LLMs can generate realistic training data for traditional AI systems that are subsequently used in industrial automation. Even if development were to slow down, I believe that there are decades of benefits to be explored, from drug research to education.

LLMs are just algorithms. Or are they?

Many researchers are now finding similarities between human thinking processes and LLM information processing (e.g., [12]). It has long been accepted that CNNs can be likened to the layers in the human visual cortex [13], but now we are talking about the neocortex [14, 15]! Don’t get me wrong; there are also clear differences. Nevertheless, the capability explosion of LLMs cannot be denied, and our claims of uniqueness don’t seem to hold up well.

The question now is where this will lead, and where the limits are—at what point must we discuss consciousness? Reputable thought leaders like Geoffrey Hinton and Douglas Hofstadter have begun to appreciate the possibility of consciousness in AI in light of recent LLM breakthroughs [16, 17]. Others, like Yann LeCun, are doubtful [18].

Professor James F. O’Brien shared his thoughts on the topic of LLM sentience last year on TDS, and asked:

Will we have a way to test for sentience? If so, how will it work and what should we do if the result comes out positive?

Moving on

We should be careful when ascribing human traits to machines—anthropomorphism happens all too easily. However, it is also easy to dismiss other beings. We have seen this happen too often with animals.

Therefore, regardless of whether current LLMs turn out to be creative, possess world models, or are sentient, we might want to refrain from belittling them. The next generation of AI could be all three [19].

What do you think?

References

1. Millière, Raphaël, and Cameron Buckner, A Philosophical Introduction to Language Models — Part I: Continuity With Classic Debates (2024), arXiv.2401.03910
2. Elhage, Nelson, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, et al., Toy Models of Superposition (2022), arXiv:2209.10652v1
3. Kenneth Li, Do Large Language Models learn world models or just surface statistics? (2023), The Gradient
4. Lindsey, et al., On the Biology of a Large Language Model (2025), Transformer Circuits
5. Wang, Xinyi, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang, Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (2025), arXiv:2407.14985
6. Guzik, Erik & Byrge, Christian & Gilde, Christian, The Originality of Machines: AI Takes the Torrance Test (2023), Journal of Creativity
7. Hubert, K.F., Awa, K.N. & Zabelina, D.L, The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks (2024), Sci Rep 14, 3440
8. Turner, Alexander Matt, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid, Activation Addition: Steering Language Models Without Optimization. (2023), arXiv:2308.10248v3
9. Tak, Ala N., Amin Banayeeanzade, Anahita Bolourani, Mina Kian, Robin Jia, and Jonathan Gratch, Mechanistic Interpretability of Emotion Inference in Large Language Models (2025), arXiv:2502.05489
10. Albert, Paul, Frederic Z. Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad, RandLoRA: Full-Rank Parameter-Efficient Fine-Tuning of Large Models (2025), arXiv:2502.00987
11. Shi, Haizhou, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang, Continual Learning of Large Language Models: A Comprehensive Survey (2024), arXiv:2404.16789
12. Goldstein, A., Wang, H., Niekerken, L. et al., A unified acoustic-to-speech-to-language embedding space captures the neural basis of natural language processing in everyday conversations (2025), Nat Hum Behav 9, 1041–1055
13. Yamins, Daniel L. K., Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo, Performance-Optimized Hierarchical Models Predict Neural Responses in Higher Visual Cortex (2014), Proceedings of the National Academy of Sciences of the United States of America 111(23): 8619–24
14. Granier, Arno, and Walter Senn, Multihead Self-Attention in Cortico-Thalamic Circuits (2025), arXiv:2504.06354
15. Han, Danny Dongyeop, Yunju Cho, Jiook Cha, and Jay-Yoon Lee, Mind the Gap: Aligning the Brain with Language Models Requires a Nonlinear and Multimodal Approach (2025), arXiv:2502.12771
16. https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/
17. https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-hofstadter-changes-his-mind-on-deep-learning-and-ai
18. Yann LeCun, A Path Towards Autonomous Machine Intelligence (2022), OpenReview
19. Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Constant, George Deane, et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023), arXiv: 2308.08708