On 21 October 2022, Population Europe and the project ‘Development of Inequalities in Child Educational Achievement: A Six Country Study’ (DICE project) held the online discussion ‘Understanding and comparing inequalities in skills development’. The event was divided into two parts: first, two members of the DICE project – Anne Solaz, Senior Researcher at the French Institute for Demographic Studies (INED) and Thorsten Schneider, Professor of Sociology, Comparative analysis of contemporary societies, Leipzig University – presented project results. Second, there was a panel discussion with experts on education-related data collection: Francesco Avvisati, Education and Skills Analyst at the Organization for Economic Cooperation and Development (OECD), Neil Kaye, Research Fellow at CLOSER, UCL Social Research Institute, Gary Pollock, Professor of Sociology, Manchester Metropolitan University and Silke Schneider, Deputy Head of the team Questionnaire Design and Evaluation at the Survey Design and Methodology Unit, GESIS Leibniz Institute for the Social Sciences.
Presentations from Anne Solaz and Thorsten Schneider – Explaining differences in achievements among students in lower secondary education
Solaz and Schneider presented findings on the relevance of social origin, parental instability and the organisation of the lower secondary education systems on students’ achievements in England, France, Germany and the United States.
Concretely, Solaz stressed that while comparing the socioeconomic status (SES) gap in child achievement has been extensively done in different countries, where explaining the mechanisms behind is challenging particularly from a comparative perspective. In this meeting, she presented the results of a study (available soon here) looking at the role of family structure on the SES gap in reading and math scores in the previously mentioned four high-income countries. Her results indicate that any family arrangement that differs from a two-parent family (married and cohabiting parents) widens the SES gap in the US and UK. However, overall, family structure does not explain much of this gap: household income is the primary explanatory factor. Thanks to longitudinal data sources, her work also shows that a large part of the SES gap observed at the end of the school period already existed at the entry, stressing the importance of initial conditions for child development.
Thorsten Schneider’s added to the question – what explains SES gaps in child achievements? - by presenting results of a study he has conducted with colleagues in the framework of the DICE project. The study shows that while initial conditions are of key importance for students, achievement inequalities observed at the end of lower secondary school are not simply the result of characteristics already observed at the start of lower secondary education. According to his results, in England, France, Germany and the US, parental education has a direct effect on mathematical skills above and beyond earlier skills. He found that the role of school composition by students’ social background is rather low and very similar in strength in all four countries. What seems to be more relevant is the way the education system is organised: while Germany divides most children into different school types already at age 10 (external school tracking), the other three countries do not. Taking this difference into account, Schneider found that school tracking reinforces the effect of parental education on mathematical skills.
In terms of data, the two scholars pointed out that DICE’s work on ex-post harmonisation of data from different countries has been challenging. This concern, for example, the comparability of survey-specific achievement tests, the way specific information is collected (administrative data on the share of children receiving free lunch vs headmasters’ report on the share of students from a lower status family) or even the sampling design. They concluded their intervention by stressing that longitudinal studies with a comparable sampling design, cross-nationally validated and longitudinally linked test instruments and pre-harmonized survey questionnaires are needed.
Panel Discussion – Comparing educational trajectories – What are the challenges?
The presentations were followed by a panel discussion with leading education-related data experts. Discussions focused on the most challenging issues nowadays when comparing educational trajectories or educational outcomes of individuals in different countries.
For 15 years, Silke Schneider has worked with and on the International Standard Classification of Education (ISCED), which is the main tool for comparing educational programmes and qualifications across countries. She said that using this classification in cross-national surveys was a big challenge. She has been directly involved in the ISCED 2011 revision and tried to implement this classification in various surveys: ESS, EVS, SHARE and PIAAC Cycle 2. However, a straightforward implementation was not possible: none of these surveys use the official ISCED codes and, with the exception of PIAAC, mappings.
Here is why: Firstly, countries apply their criteria differently, which appears to be partly politically motivated. Secondly, ISCED does not make distinctions that are important for measuring specific types of inequalities, which also differ across countries. For example, while Germany classified the selectivity of schools at the lower secondary schools as general, other countries determine selectivity based on economic, rather than academic, measures. This variability is not accounted for at ISCED. For those reasons, Schneider ended up adapting the coding system itself for the different surveys, to improve the differentiation of specific ISCED categories, while making sure that the official ISCED codes can still be derived.
Francesco Avvisati mentioned that his background is in designing and analysing large-scale international assessments at the OECD, in particular PISA. In his position, he is constantly interacting with national education administrations, which have their own administrative data. Concerning large-scale assessment data, they are useless to compare individual trajectories as they are designed for group-level comparisons, which they do well. Administrative data are appropriate to reconstruct individual trajectories through the education system and in countries where this data is available – for instance in Italy and France – it is possible to merge this data with standardised tests students have done over their life course and have an almost perfect picture of individuals’ educational trajectories. On the one hand, he mentioned that we have large-scale assessment data at different points of educational careers. It is possible to contrast the end of primary education with the end of secondary education. However, there is no guarantee that group membership is held constant across surveys. In addition, the questions that are asked may not be the same across the different surveys. On the other hand, if we rely on register data and education trajectories reconstructed from them, we have to deal with the fact that this data is not designed to compare countries, which is certainly a challenge for analysts using register data for comparison purposes.
Gary Pollock highlighted that properly accounting for national contexts in education and policy is the largest challenge when aiming to use survey data to compare countries. Countries have different legal frameworks within which education takes place, different ages at which children are eligible for a particular school year, different regimes of assessments, different frequencies of testing and different types of testing at different levels. Pollock suggested that possible solutions for this challenge are (1) to do good and solid national analyses and then do a meta-analysis of different national analyses; (2) to develop comparable datasets of survey data; (3) to explore ways to make national register data that are not comparable, more comparable in the future. For that to be possible, he suggested that more should be done by organisations like the OECD to encourage greater levels of comparability in register data. His work at the moment is to create a harmonised cross-national survey across Europe, where the team will try to iron out these differences as best as they can.
Finally, Neil Kaye started his contribution by mentioning his different areas of work: he is currently involved in a cross-national European project looking at early school dropout and all issues around the comparability of the data collection and analysis that it entails. He is also a user of secondary data – PISA and PIAAC. In his role at CLOSER, he works on data harmonisation across different cohort studies within the UK. The extent to which data is comparable and the external validity of data are the main challenges for him. Properly considering contexts is also very important for his work. For instance, educational systems have historically grown organically in each country, meaning that they have resulted in systems not being focused on comparability. It is also important for him to consider the policy relevance of comparing countries, as comparisons may not be politically relevant in national contexts.
The event was concluded by a quick round of interventions where speakers and panellists shared their preferences in terms of data sources, which largely depend on research questions.