Philosophers and cognitive scientists inspired by the work of Vygotksy (1986) typically hold that complex thought results from the internalization of social interactions in development. More specifically, they propose that internalized speech facilitates various cognitive processes, including meta-cognition, reasoning and planning (Frankish, 2018; Kompa & Müller, 2022; Gauker, 2018). However, what exactly is internalized has not been spelled out in detail. In the literature on inner speech, it is usually held that what is internalized is a kind of dialogical structure (e.g. Fernyhough, 1996). In order to get a better handle on the structures involved, I present some ideas on discourse structure (Sanders et al., 1992; Thompson & Mann, 1987), and show how they can be applied to naturalistic data of overt private speech (Nelson, 1989). I will argue that these ideas allow us to describe 1) inner speech structure above the level of individual speech acts and 2) the function of individual inner speech utterances as part of a larger train of thought.