International perspectives on large-scale assessments shaping education and education systems: role, rewards, and future desiderates
Improving quality and effectiveness of education is one of the key objectives of the European Strategic Framework for Education and Training (ET 2020). Quality, however, needs to be continuously monitored and improved, which calls for effective quality assurance systems covering all education levels. An important element of quality assurance systems are educational assessments. These assessments give policymakers, researchers and practitioners the latest information in order to develop and to understand their local education context.
The assessment field has developed rapidly during the last twenty years and the amount of international and national assessments conducted have substantially increased. Implementation strategies for educational assessments and their conceptual frameworks are challenging tasks for all modern countries. However, we know often very little of other countries’ assessment policies and practices outside our own. While remedial actions are made and taken with attention on the local context, sometimes the deeper understanding of for instance the long term consequences or larger global influences are missing. Therefore, a more complex understanding of different educational systems, their assessment strategies, policies, practices and their connections is needed.
The international symposium presents comprehensive research on large-scale assessments in multiple countries (Austria, Finland, Germany, Sweden) as part of the continuously evolving monitoring systems. It brings together presentations providing an in-depth look on different strategies to use both national as well as international student assessments for countries’ educational monitoring strategies. Additionally, core research results from each country will be presented. Thereby, this symposium integrates distinct perspectives on role, rewards, and future desiderates of LSA and allows for joint discussion.
The first presentation presents the situation in Austria and implementation, results, and political reactions on LSA. Two of the issues reported for the country are the high rate of students who do not reach the baseline proficiency level and the relatively high correlation between family background and student outcomes.
The second presentation focusses on Germany looking on strategy, core results, consequences, and current state of monitoring student achievement. The systematic relationship between family background variables and student outcomes is one topic across various LSA here as well.
The third presentation investigates how international large-scale assessments shape education in Sweden. The country has a long involvement in student assessment in international large-scale assessments and some potential drawbacks of LSA are illustrated in the talk as well.
The fourth presentation moves the focus to Finland and the question how LSA shape Finnish assessment policies and practices. One special characteristic of the Finish case is the observed declining and segregation trend after very positive assessment results in earlier LSA.
The four talks are followed by an innovative discussion format: All four presenters are brought back on stage in order to discuss communalities and differences across countries and conclusions for further LSA development in a moderated discussion in which also the audience is being actively involved.
Living in a globalized world, it is important that we understand the context of others in order to reflect our own and also to justify possible actions in the area of education. This symposium brings together student assessment academics from across Europe to explore current questions around student monitoring. The four presentations show the complexity but also the connections around strategies and assessments, core topics and evidence explored on the basis of (inter)national LSA but also enablers and barriers of student monitoring across nations. Background information and core empirical results allow for an in-depth view into multiple nations’ assessments. The closing overall discussion will bring together the different perspectives and discuss future directions for research, policy, and practice.
Beiträge des Symposiums
Monitoring Student Achievement in Austria: Implementation, Results and Political Reactions
Large-scale assessments aim at obtaining information on students’ competences and therefore on the outcomes of the education system. Accompanying questionnaires on different levels provide background information to enrich and explain the results. The data collected are supposed to enable an evidence-based education policy and to serve schools as a basis for quality development.
The overarching goal of the symposium is to obtain insights into different European monitoring systems and how they affect school system governance. In accordance with this the objective of our research was to analyse the development of the implementation of large-scale assessments in Austria and the impact of the key results on political and reform debate. Therefore, we first summarise the development of systematically monitoring student achievement in Austria. The second part identifies and analyses the (recurring) key results of the national and international assessments, and the last part outlines how those results have been used in order to issue measures within the education system.
Measuring and monitoring the outcomes of the school system has been systematically established in Austria only over the past 20 years. The beginning of the change from an input-driven to an output-oriented paradigm and the establishment of an evidence-based educational policy can be traced to the introduction of PISA. Nowadays, Austria has an extended monitoring system based on international and national large-scale assessments of multiple age groups, which compose in combination a comprehensive system of data and feedback on different levels (Wiesner et al.2017).
Two recurring results from large-scale assessments in Austria have been identified in our analyses that elicited a lot of media-related political and scientific discussion: (1) A high rate of students who do not reach the baseline proficiency level. Depending on the assessed competence and on the age of the students, between 16 % and 45 % are low-performing students. (2) The association between social background and academic achievement is relatively high, with students from advantaged homes performing, on average, much better than those from disadvantaged homes. This has been shown for example by the fact, that 25 % of the variance in reading performance of 15- to 16-year olds can be explained by family background. Among OECD-/EU-counties this is one of the highest correlations between social background and achievement (Oberwimmer et al. 2016, pp. 178).
In general, regularly delivering data from international and national large-scale assessments reignited the discussion on the central issues and problems of the educational system. The quantity and quality of public debate has definitely improved. Furthermore, the development of a monitoring system led to a change in paradigm of input-driven to output-oriented policy-making as well as to the establishment of a paradigm of evidence-based policy. However, although some reforms were driven by discussions on assessment results, basing political decisions and reforms on the evidence of large-scale assessments has turned out to be extremely challenging – especially regarding the lack of equity in education. The discussion of this key result in conjunction with a significant feature of the Austrian school system, the early tracking at the age of ten years, exemplifies how difficult it is to implement far-reaching but necessary reforms on the system level based on evidence.
Monitoring Student Achievement in Germany: Strategy, Core Results, Consequences, and Current State
Monitoring students’ achievement is one core element for countries’ assurance and improvement of educational quality and effectiveness (Harju-Luukkainen, McElvany, & Stang, in press). International as well as national large-scale assessments (LSA) are an important basis for administrations’ and governments’ decisions on measures for quality and future strategies for their educational systems. (West)Germany started to participate in international student assessments early on but with limited scope. After a long phase of abstinence from international educational studies and disappointing results in newer LSA, the politics of educational monitoring in Germany were significantly changed towards a systematic overall approach in the last 20 years. The recent German perspective and assessment strategies based on the adjustment from input to output orientation are presented in three core documents agreed on by the Standing Conference of the Ministers of Education and Cultural Affairs (KMK) which includes ministries from all 16 federal states. In theses, four areas or tools were identified and agreed upon on whom educational monitoring in Germany is currently based on: (1) Participation in international large-scale assessments (PIRLS, PISA, TIMSS); (2) Evaluation and implementation of educational standards (Bildungsstandards), (3) Quality insurance on the school level, and (4) Publication of a comprehensive national report by the Federal Ministry of Education (BMBF) and all German states on the status of educational system every two years. Additionally, six thematic areas of particular interest were identified by the federal states in 2016, one of those being students’ heterogeneity (KMK, 2016).
In accordance with the overall aim of the symposium to bring together international perspectives on large-scale assessments shaping education and education systems the talk will provide (1) a short review of Germany’s participation in national and international assessments, (2) an overview of its current educational monitoring system, and (3) present core findings from LSA regarding the German education system focussing on evidence concerning the impact of family background variables such as socioeconomic or immigrant background, and (4) their consequences.
Methodologically the talk is based on literature review and analyses presented in LSA publications regarding PIRLS, PISA, and TIMSS, as well as ICCS, ICILS, and PIAAC regarding international and BT and NEPS concerning national studies.
Important findings are that Germany’s political strategy on educational monitoring led to a variety of regularly implemented LSA which provide insights in different areas of the German educational systems but at the same time many point to some shared topics. One of these is that the LSA results for Germany depict a close relationship between students’ socioeconomic background and their success in the education system. Another one is that students with an immigrant background achieve systematically lower educational results. Taking together the analyses regarding the various outcomes investigated in the different studies a complex image emerges. Consequences of LSA can be identified on different levels looking at educational policies (e.g., system, focus, funding), practice, and research.
Finally, Germany’s assessment policies, practices and results are discussed in regard to future desiderates, among other the currently little attention non-cognitive outcomes such as students’ well-being or teacher and school variables are receiving.
How International large-scale assessments shape education in Sweden
Sweden has a long involvement in student assessment in the international large-scale assessments (ILSA). Since the late 1950s when The International Association for the Evaluation of Educational Achievement (IEA) initiated, Sweden has been an active member of the ILSA community and participated almost all the student assessment surveys conducted by IEA. This proposal will cover a general introduction of the ILSA studies in Sweden, their benefits and drawbacks. It describes how ILSA has functioned as an external support, monitoring academic outcomes and educational equity over time – where the national assessment systems failed to provide valid information about the trend in academic achievement. Finally, we illustrate some potential drawbacks of ILSA.
As regards to the performance trend in the Swedish compulsory school, Sweden has faced a general decline in the past decades. However, in the most recent assessments, an increasing performance has been observed in mathematics and science in both the TIMSS 2015 and PISA 2015. PISA 2015 even showed an improvement in reading comprehension. In TIMSS Advanced 2015, mathematical results improved, compared to 2008, but the results in physics continued to deteriorate. Albeit the recent reversed trend in achievement results, the socioeconomic gap in achievements gets continuously intensified. Since the national evaluation system long were lacking an adequate evaluation system for monitoring socioeconomic gaps, ILSA’s provide important information which cannot be found elsewhere. This holds also true for the monitoring of student achievement: In Sweden, it has been observed that average school grade (GPA in ninth grade) has increased over the last thirty years – thus indicating the opposite trend compared to that showed by for example PISA data. One explanation for this is that grading systems changed over time and that grades are subject to inflation. In fact, Hanushek, Peterson and Woessmann (2012) analysed the ILSA achievement trend in 49 countries and found that Sweden being the country with the most severe decline. For the part of Sweden it would have been difficult to observe and evaluate how the knowledge of students in compulsory school was developing, without the information of ILSA.
Regarding the drawbacks of ILSA some argue they have unintended impact on various school-systems, i.e., they legitimize or delegitimize certain actions and reforms on weak evidence. ILSA results may also lead countries to copy strategies of others, albeit these strategies are not easily implemented. Borrowing and traveling ideas of educational policies are mechanisms of change that are subtle and difficult to chart and analyze, not least in terms of long-term effects. One reason for this is that what has been written on paper, does not imply change in actual teaching or achievement (e.g., Steiner-Khamsi & Stolpe, 2004). This issue was addressed by Johansson and Strietholt (2019) who investigated whether countries converge with respect to their patterns of knowledge over time, aiming at framing the larger question about a trend towards isomorphism in countries’ curricula. The results showed little evidence for a convergence at global level, however. Rather, similarities in culture and language seemed to have great impact on the knowledge patterns.
Large Scale International Assessments Shape Finnish Assessment Policies and Practices
International Large-Scale Assessments (ILSA) are gaining more momentum in Finland, which traditionally have had a low effects on the Finnish monitoring policies and practices. Traditionally the Finnish education system has been described as a system with lack of standardised measurement and control (see closer Vainikainen & Harju-Luukkainen, 2019). Here teachers have had high autonomy and the decision-making power have been at local governments, communities, and schools.
However, a declining trend in international large-scale assessments (PISA, TIMSS and PRILS) have been observed since 2006 (Hautamäki et al. 2013; Harju-Luukkainen et al. 2016; 2014; Harju-Luukkainen & McElvany, 2018). Recent studies have indicated that at the moment the possibility to select ones educational path in Finland depends on the family’s available social, cultural and economical capital. These possibilities (or lack of them) are inherited in Finland, like in many other countries (Kalalahti & Varjo, 2012). Also different resources of municipalities regulating the school specific resources might cause inequality in education as well between different areas (Kauppinen & Bernelius, 2013; Harju-Luukkainen et al., 2016). Further, the Finnish society has along with the other European countries faced many changes during the past decade. One major change has been the increasing number of students speaking languages other than the national ones. Here, the ILSA have given an alarming picture of the learning achievements of the students with a migrant background. For instance, Harju-Luukkainen et al. (2014) reported students’ performance by proficiency levels in mathematical literacy in PISA 2012. On the highest performance levels (levels 5 and 6) there were almost no students with a migrant background (varying between 0.4% to 2.4%). Conversely, 51.5% of first-generation migrant students were on the lowest performance levels (on level 1 and below it), as were 38.1% second-generation migrants. Further, some analysis (Harju-Luukkainen et al., 2016) indicate that teachers tend to adjust their grading practices in Finland to the general competence level of their class or school rather than strictly following the national curriculum guideline for student assessment. This may lead to local variations in terms of students’ school grades as indices of learning achievement and becomes evident when contrasted with an external, nation-wide reference point, such as their reading literacy performance in PISA 2009.
This all has led to more recent longitudinal studies and attempts to understand the observed declining and segregation trend. Now more interest has been turned towards international large-scale assessments and their explanatory power as well as towards policies and defining more specifically assessment practices. In this presentation I firstly take a closer look at the Finnish monitoring system including international large-scale assessments and their main results. Secondly, with the help of a literature review the aim is to give an overview of the recent policy implications of the country’s education system regarding the declining trend in international large-scale assessments. This presentation highlights the fact that the declining trend and the increasing segregation of the education system calls for a more detailed monitoring system of the country.