The illusion of conversation. From the manipulation of language to the manipulation of the human
Chair(s): Francesco Striano (Università di Torino, Italy)
This panel will critically examine the impact of Large Language Models (LLMs) as part of the intimate technological revolution, analysing how these technologies infiltrate our everyday communication and the fabric of interpersonal trust, culminating in an analysis of the implications for political manipulation. Starting with a technical analysis of the capabilities of LLMs, it will explore how these technologies, while not endowed with authentic intention or understanding, can influence our beliefs and interactions with the digital world.
The introductory talk will analyse the generalisation capabilities of LLMs and examine how these models are able to produce coherent and contextually relevant texts. It will discuss whether their ability to produce new content is the result of genuine abstraction or mere storage and reorganisation of data. This will raise the fundamental philosophical question of whether language models have a true understanding of language or solely reproduce patterns.
The second talk will examine the nature of LLMs through the lens of speech act theory. It will be argued that despite their ability to produce locutionary correct linguistic utterances, LLMs completely lack the illocutionary component of speech acts, namely intention. However, although LLMs do not possess intentions of their own, they produce perlocutionary effects that influence the user’s reactions and decisions, leading to a projection of intention on the part of the user. This illusion of understanding supports the notion of LLMs as “conversational zombies”, illustrating how said technologies influence users' emotions and decisions, shaping the field of our intimate relationship with technology.
The following talk will argue that trust in the reliability and trustworthiness of LLMs is based on a double conceptual fallacy: on the one hand, one extends to them a judgement about reliability that is inherent to non-probabilistic linear technologies, and on the other hand, one considers them “trustworthy” as if they had intentions. It will be discussed that this trust is misplaced, however, as LLMs are not (re)producers of facts, but rather producers of stories.
The concluding talk will focus on the risks of manipulation that arise in political communication through the use of Generative Artificial Intelligences (GenAIs). It will discuss how GenAI tools, such as LLM and AI-generated content, represent a qualitative shift from traditional forms of digital manipulation. The ability to generate realistic content and simulate human interactions increases the risk of epistemic confusion and reduced trust in democratic processes. It will be argued that manipulation no longer occurs only at the level of content dissemination, but also through interaction, which has a greater impact on belief formation. Microtargeting strategies and the use of social bots will be analysed as forms of manipulation enhanced by GenAI.
Each presentation will last approximately 15 minutes and will be combined into a single lecture, leaving about 30 minutes for discussion afterwards. One of the panel members will be in charge of moderation.
Presentations of the Symposium
Understanding generalization in large language models
Alessio Miaschi
Cnr-Istituto di Linguistica Computazionale “Antonio Zampolli”
The advent of recent Large Language Models (LLMs) has revolutionized the landscape of Computational Linguistics and, more broadly, Artificial Intelligence. These models, built on the transformative architecture of Transformers, have introduced unprecedented advancements in understanding, processing, and generating human language. Transformer-based Language Models have demonstrated remarkable capabilities in solving a wide range of tasks and practical applications, spanning from machine translation and text summarization to conversational agents and sentiment analysis. Beyond task-specific applications, these models exhibit an extraordinary ability to generate coherent and contextually relevant text, underscoring their potential to capture complex linguistic structures and fine semantic nuances with high precision and accuracy.
In this context, recent years have seen an increasing focus on studying and evaluating the generalization abilities of such models. On one hand, numerous studies highlight these models’ ability to discover regularities that generalize to unseen data (Lotfi et al., 2024), thus allowing them to produce novel content across different contexts, domains, and use cases. On the other hand, many works point out that these generalization capabilities often stem from memorization phenomena rather than true abstraction or reasoning. Evidence for this has emerged e.g. from studies investigating data contamination in the evaluation of Language Model capabilities (Deng et al., 2024). Antoniades et al. (2024) pointed out that LLMs tend to memorize more when engaged in simpler, knowledge-intensive tasks, while they generalize more in harder, reasoning-intensive tasks. A fundamental challenge underlying this debate is the lack of consensus on what constitutes good generalization, the different types of generalization that exist, how they should be evaluated, and which should be prioritized in varying contexts and applications (Hupkes et al., 2023).
This talk will provide an overview of the latest developments in research on understanding and analyzing the generalization processes of state-of-the-art Language Models. In particular, we will focus on a study evaluating the lexical proficiency of these models, emphasizing their generalization abilities in tasks involving the generation, definition, and contextual use of lexicalized words, neologisms, and nonce words. Findings reveal that LMs are capable of learning approximations of word formation rules, rather than relying solely on memorization, thus demonstrating signs of generalization and the ability to generate plausible and contextually appropriate novel words.
Bibliography
Sanae Lotfi, Marc Anton Finzi, Yilun Kuang, Tim G. J. Rudner, Micah Goldblum, Andrew Gordon Wilson (2024). Non-Vacuous Generalization Bounds for Large Language Models.
Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, Arman Cohan. Unveiling the Spectrum of Data Contamination in Language Model: A Survey from Detection to Remediation. In Findings of the Association for Computational Linguistics: ACL 2024.
Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, William Yang Wang (2024). Generalization vs. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data. In Proceedings of the ICML 2024 Workshop on Foundation Models in the Wild.
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell & Zhijing Jin. A taxonomy and review of generalization research in NLP. Nature Machine Intelligence, volume 5, pages 1161–1174 (2023).
Large language models are conversational zombies. Chatbots and speech acts: how to (not) do things with words
Laura Gorrieri
Università di Torino
Transformer models such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) have revolutionised NLP (Natural Language Processing), achieving unprecedented results across different tasks such as translation, summarization, and dialogue generation. These models leverage innovative mechanisms like attention and task transferability, enabling applications such as chatbots to produce coherent, contextually appropriate conversations on virtually any topic. Despite their remarkable ability to generate human-like text, questions persist regarding their capacity to perform genuine speech acts. Specifically, can these models truly "do things with words" or do they simulate linguistic competence?
This paper explores the limitations of Transformer-based Large Language Models (LLMs) through the lens of speech act theory, a framework introduced by Austin (1962) and further developed by Searle (1969). The idea at the core of this theory is that language is an action, and therefore when one speaks they are at the same time doing things. Austin splits speech acts into three levels: the locutionary act (producing syntactically and grammatically correct sentences), the illocutionary act (what the speaker intends to do with their words, for example making a promise or giving an order), and the perlocutionary act (the effect elicited on the listener). LLMs nowadays master the locutionary act, generating grammatically accurate and semantically coherent text, and often achieve a perlocutionary response, influencing user reactions or decisions. However, this paper argues that LLMs fundamentally lack the capacity for illocutionary acts due to their absence of intention, a key component of speech acts.
Intention is central to illocutionary acts, as it is needed to convey illocutionary force to one’s words. For instance, saying "I am so sorry that didn’t work out" is not merely expressive an utterance; it embeds an intention to empathize with the listener, an act rooted in agency and self-directed goals. In contrast, LLMs operate as statistical models, selecting words based on probability distributions rather than intentions. They mimic intention but do not possess it. Gubelmann (2024) emphasizes this distinction, likening chatbot-generated responses to a tortoise accidentally forming words in the sand - the words may appear purposeful, but no intention exists behind them.
However, LLMs achieve a unique phenomenon: the perlocutionary effect they generate often aligns with the perceived illocutionary force, leading users to project intention onto the chatbot. This phenomenon is described as “conversational zombies,” paralleling philosophical zombies that mimic human behaviour without consciousness. LLMs emulate illocutionary acts convincingly enough to influence user emotions, decisions, and even legal outcomes, as demonstrated in the Air Canada case, where a chatbot’s response had binding legal implications.
This paper underscores the dual nature of chatbot interactions: while lacking genuine intention, LLMs shape real-world outcomes through their perlocutionary impact. Their capacity to mimic illocutionary force raises ethical and practical questions about their deployment in many domains. As chatbots become increasingly integrated into daily life, understanding their limitations and the active role of users in ascribing meaning becomes essential.
Bibliography
Austin, J. L. (1962). How to Do Things with Words (M. Sbisá & J. O. Urmson, Eds.). Clarendon Press.
Gubelmann, R. (2024). Large language models, agency, and why speech acts are beyond them (for now) – a Kantian-cum-pragmatist case. Philosophy & Technology, 37(1). https://doi.org/10.1007/s13347-024-00696-1
Searle, J. R. (1969). Speech Acts: An Essay in the Philosophy of Language (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139173438
The double LLM trust fallacy
Francesco Striano
Università di Torino
In today’s context of the growing popularity of generative artificial intelligence, Large Language Models (LLMs) represent an important technological and cultural phenomenon. However, the trust placed in these technologies raises critical questions, especially regarding their reliability and their alleged moral or intentional “trustworthiness.” This talk will explore the conceptual foundations of trust in LLMs. It will be argued that it rests on a double misunderstanding: an improper superimposition of the notion of reliability typical of linear and non-probabilistic technologies and a false attribution of “trustworthiness” as if LLMs possessed intentions or motivations.
Traditionally, trust in digital technologies has been based on their ability to deliver reliable and predictable results. Traditional digital systems follow deterministic models where the input produces an output determined by precise and repeatable rules. This has led to an implicit extension of this reliability to new technologies such as LLMs. However, LLMs operate on a probabilistic basis and generate answers based on statistical models learnt from large amounts of text data. This means that, unlike deterministic technologies, the outputs of LLMs are neither always reproducible nor necessarily accurate, but rather have a probabilistic likelihood.
In parallel, there is a tendency to view LLMs as “trustworthy” in a more human sense, almost as if they were entities with intentions or morality. This perspective anthropomorphises the capabilities of machines, ascribing to them a form of “trustworthiness” more appropriate to a human agent than an algorithmic system. LLMs in fact, are not aware of the information they produce and have no commitment to truth or accuracy. They act as “story producers,” generating narratives that may be persuasive, but are not necessarily true or accurate.
These two conceptual fallacies lead to false confidence in LLMs. The perception of technical reliability clashes with the reality of their probabilistic nature, while the attribution of moral trustworthiness raises unrealistic expectations of what LLMs can offer. This situation raises important ethical and practical issues, particularly in contexts where decisions based on information generated by LLMs can have significant consequences.
The talk will commence by delineating the concepts of reliability and trust and their application to the relationship between humans and technology. It will then describe the undue over-extension of the perception of reliability from linear to probabilistic technologies. It will then discuss the undue attribution of intention and trustworthiness. Finally, the importance of providing users - and policymakers - with a deeper and more nuanced understanding of the capabilities and limitations of generative artificial intelligence will be emphasised.
Bibliography
de Fine Licht, K., Brülde, B. (2021). On Defining “Reliance” and “Trust”: Purposes, Conditions of Adequacy, and New Definitions. Philosophia, 49, 1981-2001. https://doi.org/10.1007/s11406-021-00339-1
Eberhard, L., Ruprechter, T., Helic, D. (2024). Large Language Models as Narrative-Driven Recommenders. arXiv preprint. https://doi.org/10.48550/arXiv.2410.13604
Gorrieri, L. (2024). Is ChatGPT Full of Bullshit?. Journal of Ethics and Emerging Technologies, 34(1). https://doi.org/10.55613/jeet.v34i1.149
Shionoya, Y. (2001). Trust as a Virtue. In: Shionoya, Y., Yagi, K. Competition, Trust, and Cooperation: A Comparative Study. Berlin-Heidelberg: Springer, 3-19. doi.org/10.1007/978-3-642-56836-7_1
Striano, F. (2024). The Vice of Transparency. A Virtue Ethics Account of Trust in Technology. Lessico di Etica Pubblica, 15(1) (forthcoming).
Taddeo, M. (2017). Trusting Digital Technologies Correctly. Minds and Machines, 27, 565-568. https://doi.org/10.1007/s11023-017-9450-5
Wang, P. J., Kreminski, M. (2024) Guiding and Diversifying LLM-Based Story Generation via Answer Set Programming. arXiv preprint. https://doi.org/10.48550/arXiv.2406.00554
Generative AI, political communication and manipulation: the role of epistemic agency
Maria Zanzotto
Università di Torino
The rise of generative AI (GenAI) systems presents important ethical, political, and epistemological challenges, particularly in political communication. This paper explores how GenAI tools, such as large language models (LLMs) and AI-generated content, reshape the landscape of digital manipulation compared to traditional machine learning algorithms. The starting point is the established framework of digital manipulation in social media platforms, notably exemplified by the Cambridge Analytica scandal, where algorithms distributed targeted political messages based on users' extracted data to influence voter behavior. This type of manipulation, described by Ienca (2023), focuses on the distribution of content, data extraction, and passive interaction, with human intentions behind the system’s design and usage as a necessary condition (referred to as the “intentionality” condition).
However, the introduction of GenAI technologies marks a qualitative shift. GenAI tools actively generate content that appears realistic and human-like, fostering interactions that blur the boundaries between human and machine communication. This indistinguishability, where users struggle to differentiate between AI-generated and authentic content or profiles, is unprecedented in scale and poses significant threats to epistemic agency (Coeckelbergh, 2023) - the capacity to form, control, and trust one’s beliefs. Unlike earlier AI systems that manipulated content distribution, GenAI tools engage users in human-like conversations, creating the illusion of authentic communication and leading to potential manipulation of beliefs.
A key challenge with LLMs is that users tend to anthropomorphize these systems, attributing mental states such as beliefs, intentions, or desires. However, these AI tools do not possess true understanding or agency; they merely produce outputs by predicting the next most probable word based on patterns in large datasets. This has led researchers to describe them as "stochastic parrots" (Bender et al., 2021) - powerful computational systems that reorganize existing data without genuine comprehension. The anthropomorphic style of chatbots, which often use phrases like “I think” or “I believe,” reinforces the illusion of intelligence and encourages users to apply what philosopher Daniel Dennett calls the intentional stance, wherein humans interpret behavior in terms of mental states. This illusion creates fertile ground for manipulation, particularly in political communication, where trust and authenticity are crucial.
The study identifies two key pathways of manipulation with GenAI: microtargeting, where AI-generated messages are tailored to individuals, and the use of social bots that simulate human interaction. These processes increase the risk of epistemic confusion, diminishing users' trust in digital environments and, by extension, in democratic processes. While Ienca’s framework highlights the intentions behind manipulation, it falls short in addressing the active role played by GenAI tools in fostering false beliefs through their anthropomorphic and probabilistic nature.
This research argues that GenAI-induced indistinguishability fundamentally impacts political communication, necessitating a revised framework to address new forms of digital manipulation. The manipulation of interaction, rather than mere content distribution, represents a stronger influence on belief formation and autonomy.
Bibliography
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922
Coeckelbergh, M. (2023). Democracy, epistemic agency, and AI: political epistemology in times of artificial intelligence. AI and Ethics, 3, 1341-1350. doi.org/10.1007/s43681-022-00239-4
Dennett, D. (2023, May 16). The problem with counterfeit people. The Atlantic. Retrieved August 2024, from https://www.theatlantic.com/technology/archive/2023/05/problem-counterfeit-people/674075/
Ienca, M. (2023). On Artificial Intelligence and Manipulation. Topoi, 42, 833-842. doi.org/10.1007/s11245-023-09940-3
|
