Zooming out and zooming in on student data: Developing teacher data literacy to enhance teaching and learning

Introduction

While teachers and schools have access to an increasing range of data, a challenge is using these data to support student learning outcomes. Over the past decade, expectations for teacher use of data as a basis for instructional decision making have increased (Pierce & Chick, 2011; Schildkamp & Poortman, 2015). Teacher assessment literacy, data-based/data-informed decision making, and data literacy have emerged as focuses for policy and professional development. Despite this, and increasing policy and commercial interest in evidence-based practice, international research consistently reports that many educators do not make effective use of the student data they collect (Kippers et al., 2018; Mandinach et al., 2015). Within New Zealand, the Education Review Office (ERO) has expressed concerns about current levels of teacher data literacy in New Zealand schools (Education Review Office, 2017). Pilot research by Peter et al. (2017) indicates that many schools collect limited or no school-wide data and that, even when they do, teachers and leaders often struggle to analyse and act on the data in the classroom.

Through a design-based research approach (Penuel et al., 2011) the Zooming Out and Zooming In project focused on the persistent problem of practice: how to optimise the use of standardised data for the purposes of improving mathematics learning and teaching and then scale out the data use approaches to the wider school learning community through the use of data coaching.

For the project, 13 teachers from six primary schools and one intermediate school from one Kāhui Ako came together to enhance their data literacy skills and explore the instructional potential of data from a widely used standardised mathematics assessment tool (Progressive Achievement Tests [PAT]: Mathematics: https://www. nzcer.org.nz/tests/pats). While there is no definitive definition of data literacy, it is generally considered to involve teachers establishing a purpose for collecting, analysing, and interpreting data, and using the insights gained to take instructional action as part of focused inquiry (Kippers et al., 2018; Mandinach & Gummer, 2016). The project teachers and researchers collaborated to develop a working definition of data literacy to anchor their investigations. Together they explored the nature of supports that enable teachers to “zoom in” and focus in depth on the detail of individual student PAT: Mathematics data. Teachers used the insights they gained from this process to inform their teaching with small groups of students. Finally, we “zoomed out” to consider possible implications of student achievement data when they were aggregated at the level of a class, a school, and across schools. These experiences, coupled with team discussions and readings, formed a basis for the project teachers’ work with colleagues as data coaches.

Research design

The project employed a design-based research approach (Penuel et al., 2011). This approach is distinguished by: (a) a focus on persistent problems of practice from multiple stakeholders’ perspectives; (b) a commitment to iterative, collaborative design; (c) a concern with developing theory related to both classroom learning and implementation through systematic inquiry; and (d) a concern with developing capacity for sustaining change in a system. How to make more productive use of readily available data was the problem of interest in our project. We explored this question through three main research focuses.

Research focus 1: Data literacy in action in the classroom

1. What kinds of insights and classroom actions do teachers plan for as a result of zooming in on student data at the individual student, subgroup of students, and their class levels?
2. What impact, if any, do teachers think their planned actions have on student learning and achievement of mathematics ideas?
3. What are students’ thoughts and experiences about being more involved in monitoring and progressing their own understanding through the use of assessment data?

Research focus 2: Data coaches as change agents

1. What does it mean to act as a data coach in a Kāhui Ako school?
2. What kinds of materials and protocols support teachers to develop and act effectively as data coaches for colleagues in their own schools, and what are the enablers and barriers for this role?

Research focus 3: Making sense of cross-school data

1. What insights are afforded when teachers collaborate to “zoom out” through analysing data from across their school/s?
2. What materials, protocols, and ways of working support critical and constructive data sharing, analysis, and interpretation within and across schools?

Teacher participants and school contexts

Teachers from a 16-school Kāhui Ako in Pukekohe were invited to participate in the project with the active consent of the Kāhui Ako governance group and their principals. Thirteen teachers from seven schools volunteered to take part in the first year. Ten of the teachers had over 10 years’ teaching experience, the others had taught for between 5 and 10 years. Around a third were the mathematics leaders in their school, a third were not, and the remaining third had previously been a mathematics leader in their school. Four of the schools were full primary schools (Years 0–8), two were contributing schools (Years 0–6), and one was an intermediate school (Years 7–8). Two of the schools were low decile, two were mid-, and three were high decile. The number of students ranged from 115 to 300 for the primary schools. The intermediate school had around 750 students. In Years 2 and 3, seven teachers from six of the Year 1 schools continued their participation in the project. Each teacher participant was involved actively in researcher-teacher workshops and each participated in their own teacher inquiry based on data from their own class. Thereafter, each teacher took on the role of coach for teacher/s in their own school. The coachees the project worked with did not participate in research workshops but they, along with the project teachers and the students and parents in the project teachers’ classes, gave informed consent each year of the project.

Research activities

Researcher-teacher workshops

A series of researcher-teacher workshop meetings were held in 2019, 2020, and 2021 for collaborative planning, teacher presentation of their inquiries, discussion of findings, and readings; two per term for the first three terms, and one in the final term in 2019 and 2020. In 2021, the team met five times over terms 1–3. Teacher inquiries and research workshops in 2020 and 2021 were impacted by COVID-19 lockdowns, with some meetings via Zoom. Over the course of the workshops, the researchers introduced a range of ideas (e.g., definitions of data literacy), resources (e.g., a Data Conversation Protocol [DCP]), readings (e.g., on coaching), and discussion prompts (e.g., PMI chart on the value of collaboratively zooming up and down levels of analysis of PAT data from item–student–class–syndicate/school–cross school–national). All workshops were audio-recorded and field notes taken. Teacher PowerPoint presentations on the results of their inquiries and their coaching experiences were collected as were any materials produced during the meetings. Audio recordings from teacher meetings were selectively transcribed. Data collected from workshops were used to address Research focuses 1 and 2 in 2019, and Research focuses 1, 2, and 3 in 2020 and 2021.

Introduction and use of a data conversation protocol

Throughout the project we used a “Data Conversation Protocol” (DCP) to scaffold our discussions about data and to plan and reflect on data-informed action (Research focuses 1–3) (Appendix A). The DCP was adapted from a discussion protocol developed by Dalton and Anderson (2016). The stepwise prompt structure echoes key constructs from assessment for learning literature (Black & Wiliam, 1998). The first “Here’s what?” prompt indicates the need for critical consideration of the source and scope of the data that would be acted upon. The second “So what?” prompt guides teachers to consider why students responded to particular assessment questions in the ways that they did. The “Now what?” prompt is intended to help teachers to look across their class data to identify which questions/concepts had posed the most challenge for their students, to select one for inquiry, and to plan an intervention. The “So then?” prompt, which we added to the Dalton and Anderson (2016) discussion protocol, was designed to guide teachers to reflect on student responses to their teaching actions and consider next steps.

Teacher inquiries

Each year, all teacher participants designed data-informed teaching inquiries into the mathematics achievement data of their own classes (Research focus 1). They used the DCP to interrogate the data they had gathered and then designed focused interventions for groups of priority students based on the results of their analysis. The teachers reported the outcomes of their inquiries to the project team. Teachers drew on what they had learnt from their data-informed inquiries to develop school-specific plans and protocols for coaching colleagues.

Colleague coaching

During 2020 and 2021, each participant worked with one or more teachers within their own school as a data coach (Research focus 2). In general, coachees were volunteers, although in some schools they were assigned, which posed a challenge in terms of gaining buy-in, trust, and confidence. The coach met with the coachee/s a number of times throughout the inquiry cycle. Considerable flexibility was afforded to teacher participants so they could plan coaching that would work best in their particular school context. The teacher coaches worked with their coachees to:

• discuss definitions for data literacy and why being data literate might be important
• familiarise coachee/s with the DCP
• link the DCP to action steps within the coachee teacher inquiries
• support the coachee/s to unpack/analyse their class data and develop strategies for targeted instructional action (using the Here’s what?, So what?, Now what? steps in the DCP)
• support the targeted action by modelling, co-teaching, or observing coachees
• reflect on their instructional action and its impact on student learning, including the development of student agency.

Teacher-composed narrative case studies about data coaching

Teachers reported the outcomes of their coaching and their coachee inquiries via PowerPoint presentations at project workshops. In addition, project teachers wrote case studies of an aspect of their experience of data coaching. Each case was the story of an event that they felt was particularly striking and included a message they thought peers would benefit from (Research focus 2). Teachers then partnered with a project teacher from another school to provide feedback on each other’s case to ensure that both cases were likely to be understood by colleagues who might be interested, but not experienced, in data coaching. The teachers composed video case studies in response to a researcher-teacher discussion during a workshop about effective methods and modes for communicating findings with a wider audience. Based on their written cases, the teachers wrote scripts, practised, and recorded their video cases in an afternoon workshop (Teacher-researcher workshop, September 2020). The videos were a brief snapshot of no more than 1 minute in length that communicated a key message about the coaching relationship (Research focus 2).

Reflective interviews

Teachers participated in one-to-one end-of-year reflective interviews each year (Research focuses 1, 2, and 3). As we regularly discussed the project research questions as a team, interview questions were based on these. Interviews lasted from 20 to 50 minutes.

Online surveys

Online surveys that were an adaptation of Wayman et al. (2009) were used to gather participant demographic information, the timing of data collection, and the nature of the assessment tools used by teachers and schools (including PAT) in February 2019, and again in 2021 (Research focuses 1 and 3).

Data were collated and analysed thematically (Braun & Clarke, 2006), with this process framed by the research questions. Findings are presented in terms of the three research focuses.

Findings

Research focus 1: Data literacy in action in the classroom

The first focus for the project was on the nature of teacher data literacy in action within classroom instruction. The aim was to identify the kinds of insights and classroom actions teachers might plan for and enact as a result of zooming in on student data at the individual student, subgroup of student, and class levels. Within this focus we also examined the impact that teachers perceived their actions had on student learning and achievement and student experience of these actions. This section reports on the five key findings for this research focus.

The importance of collaborative scoping of the nature and intent of “data literacy”

We established that our teachers had a variety of understandings for the purpose of data use (see also

Peter et al., 2017) and so time was needed to come to a taken-as-shared understanding of the intent of “data literacy”. Initial discussion of the scope and purpose of data literacy was effectively scaffolded through teacher analysis and critique of a small number of definitions (two to three) drawn from literature. The following definition featured prominently in our discussion:

Data literacy for teaching is the ability to transform information into actionable instructional knowledge and practices by collecting, analyzing, and interpreting all types of data (assessment, school climate, behavioral, snapshot, longitudinal, moment-to-moment, etc.) to help determine instructional steps. It combines an understanding of data with standards, disciplinary knowledge and practices, curricular knowledge, pedagogical content knowledge, and an understanding of how children learn. (Mandinach & Gummer, 2016, p. 367)

Taking time to craft and refine our own working definition allowed us to clarify and confirm that the focus was on informing instructional decision making and action, which all the teachers agreed had received limited attention in relation to PAT data use prior to the study. Over several researcher-teacher workshops in the first year, the teachers and researchers together developed the following description for teacher data literacy as a cyclic inquiry process:

Data literacy involves collecting, gathering data, analysing and understanding it, and then using this understanding to take action. It includes the knowledge needed to decide if data is worthwhile and/or valid and the ability to share information to different groups (children, other teachers, principal, boards of trustees

[school governance board] etc.). (Shared understanding developed at researcher-teacher workshops, 6 March, 10 March, 15 June 2019)

A distinctive feature of this definition, one in line with Aotearoa New Zealand understandings of the role of parents and whānau in assessment (Ministry of Education, 2007, 2011), is that data literacy needs to encompass the capacity for teachers to communicate data-informed insights to students and to those outside the classroom.

We found it was important to focus on the meaning of “data” within this statement. Teachers were emphatic that to develop a holistic understanding of students’ learning needs and achievements they needed to access data from a range of sources (i.e., more than school academic achievement data), and to analyse these data at different levels of aggregation:

For us ‘data’ is a wide range of information including student learning conversations, perceptions, observations, and products of learning, school processes, student demographics (after Bernhardt, 2018) and includes different levels of aggregation. (Shared understanding developed at researcher-teacher workshop, June, 2019)

While this understanding framed the project, we focused on PAT data as these standardised data were able to provide a common ground for our discussions.

Using a DCP slows down and deepens data analysis to inform action

Teachers reported the DCP was effective in slowing down their examination, analysis, and interpretation of student data, with this in turn leading them to taking more focused instructional action. The first “Here’s what?” prompt was particularly important in encouraging teachers to pause and ask themselves if and in what ways the data they were considering were robust/trustworthy evidence of student learning given their knowledge of a particular student. It encouraged teachers to focus on the “whole child” and take into account their wider knowledge of a student—something they viewed as important. Overall, teachers found their use of the DCP shifted their focus to in-depth sense-making as described by Teacher 2:

Having the time and taking the time to go deep—I was surprised—and this (the Protocol) made us go into this. I took the time whereas I’ve never really taken the time to go as deep. (Teacher 2 interview, 2019)

In our study, the “So what?” also led to teachers taking careful note of which distractors or known alternative conceptions students selected as their PAT responses, with student selections serving as a source of insight into their reasoning. The “Now what?” prompt helped teachers to stand back and identify patterns in student question and item level responses as part of planning short, focused teaching interventions with a small group of target students (usually between four and six):

I do not believe that Student A is able to partition numbers for either addition or subtraction. Student A needs lots of work just seeing how numbers are made up, such as 4 and 6 make 10. This will help them to become more confident when working with numbers. (Teacher 2 inquiry presentation, 2019)

Looking closely at the data I was able to see beyond the fact that these students couldn’t calculate a time difference—they couldn’t actually read an analogue clock. I was able to plan focused lessons to meet students’ needs first on teaching how to read an analogue clock. (Teacher 5 inquiry presentation, 2019)

The “So then?” prompt was important in directing teachers’ attention to student responses and to their instructional action:

It helped me to target specifics rather than, ‘Oh well try this, then we’ll try this’. I really slowed down in my inquiry and started to think about the things I’m trialling, I’m not just trialling for trial’s sake, I’m thinking about ‘Where is this going to end up at the end?’ (Teacher 4 interview, 2019)

These quotes from different teachers about their inquiries demonstrate how the DCP supported teachers to explore subtle variations in students’ understandings and to plan data-informed actions. Teachers were able to reflect on the impact of their actions to plan further steps.

Data analysis needs to probe root causes in light of horizon knowledge

The notion of “root causes” and of “horizon knowledge” emerged as central to the teachers’ analytical discussions. Claudet (2020) points out that, if root causes are not identified, time may be spent on symptoms with limited long-term impact. When considering their interpretation of and action on data, teachers took time to consider what might be the “root causes” of student responses; that is, they sought to probe beyond readily definable surface-level “symptoms”. However, as the teachers were well aware, identification of and action on root causes relies on them having in-depth content and pedagogical content knowledge (Shulman, 1987). PAT distractors that were based on common misunderstandings were helpful in this regard and teacher discussion of root causes associated with these often involved follow-up reading. In their analysis, teachers also employed what Ball (1993) refers to as horizon knowledge. Horizon knowledge includes the content and pedagogical content knowledge needed to understand the significance of “what comes before and after” in connection to a mathematical idea. This is important because it shapes teacher decision making and a teacher’s choices can anticipate or undermine later development. Here is a typical quote illustrating teachers’ concern to consider the reasons for the data more deeply:

What seems really obvious, for example in the PAT ‘A half plus a half’, and making the assumption that they know that but then half of them got it wrong. So what does that mean for what we are doing? What is the issue there? It’s not just knowing a half plus a half. Do they not understand the concept of fractions, the whole concept of digging down deeper or zooming or whatever you like to call it …? Uncover that assumption, that misconception that we thought they were OK with, and then the question is what we do about it. (End-of-year teacher interview, 2019)

Combined, teachers’ focus on root causes and horizon knowledge focused attention on the past, present, and future for student learning: What ideas and/or strategies might students have developed over time that were being expressed in the present, and what were the implications for the future of present ideas and/ or strategies? Seen this way, understandings of data literacy need to encompass consideration of teacher actions in the present in light of both the past and the future. Within the project, teachers’ timely, strategic, and intentional use of relevant elements of a robust standardised data set attributed value to data that might have been viewed as irrelevant/out of date if teachers had probed only to surface-level features rather than to root causes and implications going forward. This approach allowed teachers to optimise the value of data that in the past would have been collected but might have been considered only once and/or by the school leadership alone.

One impact of data analysis is to shift teacher focus from gap filling to front footing

Across the 3 years of the project, the teachers were clear that they did not want their DCP-informed teaching to become narrow “gap filling”. Instead, teachers aimed to use their targeted interventions to fast track or front foot student learning. Their PAT data-informed teaching sessions were strategically timed and designed to engage the small group of selected/target students in concepts and ideas they would soon encounter as part of whole-class lessons. Teachers identified a range of positive impacts on students’ attitudes and knowledge—which parents also commented on—when they moved from a focus on intervening to gap fill/ remediate to intervening to front load or front foot student learning.

The teachers’ interventions shared a number of characteristics that they considered contributed to their success. First, the interventions involved a small group of target students with the same alternative conceptions. Important to the positioning of the interventions, group composition varied depending on the topic and, on occasion, the group composition was not what might have been expected, in that some apparently maths-capable students had areas where their understanding was not robust. Second, the sessions were short, very focused, and interactive. For example, teachers made use of maths games and hands-on materials. Third, the focus of the teaching addressed student misconceptions but in a way also scaffolded their understanding of ideas they would meet in class mathematics lessons in the days ahead.

Teachers reported that their intervention students, who tended to be those who struggled in maths, gained confidence from realising that they could understand an idea that had previously seemed inaccessible. Their reports on student responses to their interventions highlighted their delight in the development of student understanding, confidence, and willingness to contribute their ideas in class, as illustrated in this teacher’s reflection early in the project:

I analyse data about 10 times better than I ever did … But actually, going deeper into the testing, analysing questions, analysing patterns within those questions, and then narrowing it down to groups, individual children, that has been so powerful. And the data when we did GloSS mid-year, I saw a big increase with the children in the target group. Just done PATs now and 3 of the 5 in that group have gone up 2 stanines which is huge. They were [stanine] 4 and now a confident 6. (Teacher-researcher workshop, September 2019)

Towards the end of the project, teachers reiterated that student self-efficacy was enhanced subsequent to students understanding an idea through a “targeted fast track intervention” (Teacher-researcher workshop, August 2021). When asked about their experiences of the intervention by a project teacher, student comments included that, in the small groups, “I was not nervous about the answer” and “I didn’t get embarrassed if I didn’t get it” (Teacher-researcher workshop, August 2021). In many cases, teachers thought the shift in efficacy flowed through to other maths topics and even other learning areas, meaning the intervention had a longer range and term ripple effect (Teacher-researcher workshop, September 2019, August 2021). Other observations were that students were happier in their work and more willing to attempt more complex problems. One group of students, interviewed 3 years after the TLRI small-group teaching, recalled the 2019 intervention, saying they had liked the small-group work and an interactive racing game. However, while this kind of intervention was seen as valuable, teachers pointed out that the extra time involved might limit student access to other activities/ideas as the target students were often in need of additional assistance in many curriculum areas.

Some teachers found they were able to actively engage parents and whānau in their interventions by regularly texting parents about progress and sending photos of students’ work. As parents understood and learnt more about the targeted interventions, they reported they felt more confident to support their children themselves. Some parents spontaneously reported that there had been a distinct shift in their child’s attitude towards maths to the extent they shared their learning at home. One teacher said, “The parents have said, ‘Wow, my kid loves maths!’ So that has been very positive. And that drives you as a teacher to spend the extra time analysing your data. And really look for next steps” (Teacher-researcher workshop, September 2019).

Using a DCP supports discussion and collaborative analysis

Beyond its value in supporting teachers to take time to analyse their student data, plan, and reflect on the impact of their pedagogical actions, a key finding was that the DCP provided a framework and takenas-shared language for collaborative analysis of data and development of possible actions. It served this purpose amongst the project team and when project teachers moved to discuss data and data-informed action with the colleagues they were working with as data coaches. The teachers were enthusiastic about having an opportunity to discuss the analysis of their own and other teachers’ data with like-minded colleagues. Discussions led to the recognition that there were areas of common concern across the project schools: “It wasn’t just our kids” (Project meeting, September 2019). That is, the concern was not unique to their school and because some of the teachers taught students in the same year, the concern was also not unique to their class. Nor was it always associated with one school year/PAT level. This prompted the teachers to consider if and how particular earlier teaching approaches might have contributed to a later common misapprehension. One such example was the well-known misconception about the meaning of the equals sign where students understand the equals symbol as “find an answer” rather than “the same as”. This manifested in difficulties finding answers to questions such as: 5 + 3 = __ + 4. Another was to do with double digit subtraction where some of the teachers came to appreciate the implication of teaching decomposition in addition: students had generalised the strategy to subtraction, not realising subtraction is not commutative. Their analysis of student responses to double digit subtraction had a powerful impact on teachers with this deriving from a combination of collaborative analysis that focused on root causes with what knowledge was on the horizon in mind.

Research focus 2: Data coaches as change agents

The development of teachers as data coaches of their colleagues was the second focus for our research. In particular, we wanted to build knowledge about what it means to act as a data coach in a Kāhui Ako school. Data coaching has gained traction recently, largely in the United States, often with coaching provided by external coaches (Huguet et al., 2014; Love et al., 2008). Collegial coaching as it was envisioned in this study has received limited attention in New Zealand. However, we anticipated it would be an effective way of building capacity in data literacy within a school because the coach and coachee would come to the process with a common understanding of the nuances of the context for teaching and learning. In practice, coaching proceeded in different ways in different settings as project teachers acted as colleague coaches, co-developing plans for teacher actions that were attuned to their coachee’s interests and best matched their student cohorts (Edwards et al., under review). In this way, the coaching was different from professional learning and development (PLD) approaches teachers often experience as it was targeted and personalised. This sets out five findings in relation to Research focus 2.

The focus of data literacy coaching is on using data to improve learning and not to judge coachees

The project teachers as coaches reiterated the need to be clear with their coachee that the focus was not on them as a teacher. The data coaching process is known to rely on high levels of trust between coaches and coachees (Lasater et al., 2020; Lowenhaupt et al., 2014) and teachers as coaches found it was important that both they and their coachee trusted that the investigation into student data was to benefit teaching interventions and not to attribute blame or to point out teacher weakness. A teacher explained the process she followed:

First, you need to build a relationship with them, understand their resistance, be transparent with the reasons behind the coaching and show that we can all be vulnerable when it comes to learning. Build trust! (Teacher case study, Teacher 4)

In order to approach co-viewing and co-analysis of data positively and productively, coachees needed to appreciate that if students were found to be achieving at low levels there was no accusation of poor teaching; the focus was on what the basis of student reasoning was and what could be done differently. Collaborative drilling down on data was aimed at understanding student thinking processes and asking how and why teaching or assessment actions might have led to a particular student response. The DCP proved to be a useful framework for interrogating assessment data at item level to see if root causes for student responses could be determined.

Effective data coaches position themselves as learners who are on the same level as coachees

The project teachers as colleague coaches were emphatic they were “on the same level” or “just teachers helping teachers”. They were “not always having all the answers” but they were learning together with their coachees (Researcher-teacher workshops, 2019, 2020, 2021). They explained that this was important because this meant they were able, to some extent, to circumvent issues of school-based organisational power: “Not being in the senior leadership team puts the power balance in a different place.” A typical comment was:

Because I don’t have any official leadership role in the school, they feel quite comfortable coming to me. They’ll come and ask me, ‘What do I do about this?’ I’m not their boss in any shape or form, I’m there to help them. (Researcher-teacher workshop, March 2020)

Teachers revisited this point over a number of workshops, concluding that coaches not being in a position of power meant they were approachable which enhanced the likelihood of open dialogue where coach and coachee shared data and ideas without fear of being judged.

Data coaches’ credibility and confidence is grounded in personal experience and public commitment

After initial nervousness about working with and coaching colleagues, the project teachers came to appreciate and draw on the authority of the learning and experiences they had developed during project workshop sessions and through their own inquiries using the DCP. During a workshop discussion, the teachers agreed that:

Teachers need to be data literate in order to use data effectively. [We understand] the value of having a shared definition and culture of data use and that has driven everything we have done. But this has not come easily—we have had to work at it. Don’t assume that everyone knows how to gather and use data: ‘You’re a teacher so you know how to use PATs.’ [Teachers] need to be educated on how to do this. (Researcherteacher workshop, June 2021)

The teachers also considered they benefited from their demonstrated public commitment to the value of data use through being part of a research project, which offered additional status to them in their coaching role. Teachers sharing their coaching experiences and crafting case studies of their coaching practice enriched their experience and knowledge base. The resources, tools, and strategies used and developed by the project researchers and teachers together (such as readings, collectively developed definitions, the DCP, teacher-composed case studies, workshop on the Mathematics Learning Progression Framework [Ministry of Education, 2019]) were also important sources of their authority and self-confidence. Group analysis of across-school Kāhui Ako PAT data also expanded the knowledge base the coaches had to draw on to inform their work in their own schools. For example, common areas of weakness or misconceptions across schools were identified and could be discussed by the project team as a whole before becoming targeted as areas for action within a coaching relationship.

Barriers and enablers to data coaching are context and relationship dependent

Table 1 provides a summary of the enablers and barriers of working as a data coach, as the data coaches identified them, during a workshop (24 March 2020) and distilled from individual reflective interviews and teachers’ written case studies.

Building up from individual to small-group colleague coaching

Following their first coaching experience, a number of coaches worked to extend the development of data literacy at the whole-school level with this shift in focus attributable to their increased confidence and knowledge. This finding suggests there is merit in a staged introduction of data coaching, where teachers first gain experience in a like-minded group then work with a small number of teachers in their own school before considering a wider initiative. In this study, in one school, a teacher as data coach moved to work with six teachers after the PAT: Mathematics data became available in the second year. She showed teachers the process she followed and discussed the data and implications for teaching with them. In a second school, after consultation with senior management, the coach planned for a whole-staff data literacy PLD focus. Unfortunately, this was delayed because of COVID-19 lockdown. The flexibility that coaches have had to make decisions that best suit the needs of their school community is an important feature of these developments, one that would need to be taken into account in taking an initiative such as this to scale.

Research focus 3: Making sense of cross-school data

Identifying how to optimise data use for mathematics teaching and learning purposes through a combination of zooming in and out on data at the level of the individual student, class, school, and cluster of schools was the third focus of our project. Specifically, we wanted to find out what insights were afforded when teachers collaborated to “zoom in” and to “zoom out” through analysing data from across their school/s. In this section, we explain the four key findings from Research focus 3.

Zooming between individual and class analysis supports planning for instruction

Moving between class and individual student data allows teachers to maintain an overview of the class whilst keeping the needs and strengths of individuals/groups in mind during planning and teaching. In terms of their classroom practice, teachers noted that processes of “zooming in and out” from individual student data to class, and school and national-level data were useful in a number of ways which depended on teachers’ agendas at the time. For example, the purpose of moving between individual student- and class-level data was different at the beginning of the year than the year end. Typically, at the start of the year, teachers would glance at a PAT report and use scale scores “to give us a rough idea” or “look for high flyers” (Researcherteacher workshop, March 2019). Teachers would next focus on mathematical strands and student knowledge gaps to group students and plan for teaching, before looking deeper for misconceptions at the individual item report level “to paint a picture and set us up to start teaching” (Researcher-teacher workshop, August 2019). The item report was also used to identify individual learning needs; for example, “Why are they there? Do they need a reader?” (Researcher-teacher workshop, May 2019). Teachers would revisit the class-level data and scale score reports throughout the year to check for surprises, strengths, and weaknesses within their class, with their focus depending on the next class learning focus. Teachers spent more time with the list report towards the end of the year as they looked for progress. The list report is a summary of results for a group of students who have taken the same test in which each individual’s results are provided as well as summary statistics for the whole group.

Cross-school/s data analysis assists in shifting the focus from students/schools to instruction

We found that collaborative/cross-school examination of individual data at the question/item response level supported a shift from consideration of the student and/or school as possibly deficit to an interrogation of teaching approaches. For example, one of the project teachers completed a Master’s study examining the patterns of achievement across all the classes in her school across 5 years (Jolly, 2019). This provided insight into the sustained nature of some misconceptions such as that associated with double digit subtraction. It also highlighted areas where the school’s performance was inconsistent with national data thereby supporting reflection on where practice was effective and where more attention was needed. Similarly, a pair of data coaches working in another school found that zooming in and out on school data helped them to more effectively understand the patterns of student achievement in their school, which they were then able to present to the school leadership:

We had ‘zoomed in’ on our own classroom data; developed and worked through our individual focuses with our target groups in our own classrooms … This then allowed us to ‘zoom out’ to the next level, looking at school wide data, looking for trends and really mining the school wide data. We ‘zoomed back in’ on specific areas and put a presentation together with our findings for the senior management team. (Researcherteacher workshop, March 2021)

In another example, teachers noted that, when data over the school years were collated, there was a drop in achievement between Years 6 and 7, with this drop in achievement mirrored in national PAT data. Initial conversations revolved around the idea that this was due to school transition but this was contradicted when project teachers from the Years 1–8 schools pointed out this pattern was also evident in their data. This prompted teachers to check national data and to think again about what the root causes might be. In 2021, as part of a Kāhui Ako-wide initiative spurred by the project findings, collation of PAT across all Kāhui Ako schools allowed for the identification of topics that were posing a challenge across the school community. The results of this work are beyond the scope of this TLRI project.

Zooming across class–school–national data can lead to a sense of collective responsibility

In our study, the analysis of data at the class, school, and across-schools levels led to a sense of collective responsibility for student learning that was, as above, anchored on the interrogation of instructional practice. Teachers were interested in how their class compared to others in the school as a way of taking collective responsibility for student achievement. Teachers in small schools were interested in data from other schools as a means of better understanding their data, their curriculum, and their teaching practices. Teachers in small schools were conscious of the limitations imposed by having a small sample size for data at many/ most of their year levels in relation to variability of the student cohort across calendar years. Looking at the national-level data, teachers were interested in how their students/syndicate/school compared as a benchmarking exercise and also in data trends over time in the sense of, “Are we progressing in areas of concern?” For teachers, the ability to compare school data distributions alongside Kāhui Ako and national data distributions supported consideration of similarities and variations in student collective achievement. They were able to collaboratively consider how their practices might underpin patterns and variations and how these patterns could be used to focus and facilitate developments and changes in teaching practice. In line with their critical stance towards data literacy as a process for informing and guiding action, the teachers were emphatic that there was a need to ask, “Are we happy with the ‘average’? What does this mean in terms of what our students do and do not know?” “What does it mean for our aspirations for them?” (Researcherteacher workshop, September 2020).

Zooming in and out requires individual and institutional relational trust and standardised data

Teachers and school pooling data for public analysis relies on a level of trust within the group that the goal of the process is to inform teaching and enhance learning and not to make pejorative judgements about teachers and/or schools (Gerzon, 2015; Schildkamp & Poortman, 2015; Schildkamp et al., 2017). Through the project we also affirmed that before data can be pooled it needs to be standardised in some way. In our case, we used PAT data as PAT was used by most schools, and, as the project progressed, all schools agreed to use PAT: Mathematics. PAT data have the advantage that they are digital and so can readily be collated across classes in one and multiple schools. Collated distributions could also be compared with national profile data. Attempts to use collated data to inform action highlighted the need for shared data to be complete and reliable so that all schools had a similar grounding to analyse, understand, and move forward with their data.

Implications for practice

For individual teachers

• In order to make best use of the data they collect from students, teachers need to be allocated time and support to develop data literacy skills.
• A tool like the project DCP can assist teachers to slow down and question: How were the data generated? What could it mean in terms of student thinking and prior experience of different teaching strategies? What might be the root cause of student responses? What is the horizon for student development?
• Teachers need to consider how they can plan for front footing for students in their classes—anticipating what students need to know to learn upcoming material and supporting them in small groups to learn this and a bit beyond this in order to support student self-efficacy and understanding. In planning for action, the notion of horizon knowledge can help teachers take a longer-term view of students’ learning pathways.

Data coaching to build capacity and culture

• For collegial data coaching to be successful, school leaders need to be supportive and clear that their focus is on improvement.
• In order for data-literate teachers to act as effective, in-school data coaches for their colleagues they need to be able to develop a coach/coachee relationship with high levels of trust with a shared focus on improvement, not judgement.
• Data coaches need access to tools and resources to enable productive coaching—these can include evidence from their own inquiries as exemplars, the project DCP, and knowledge of progression frameworks.

For schools/Kāhui Ako: For collaborative analysis and action

• School leaders and teachers within a community share responsibility for student learning across their school and Kāhui Ako. It is important that trust is built within all layers of a Kāhui Ako and that active support is given to the development of a school data-use culture.
• Teacher agreement about the tools used (including standardised assessment data) and consistency of practice (including moderation) is needed to generate data that are trustworthy and can be meaningfully collated, analysed, and acted on within and across communities of schools.
• There is value in meeting across year levels and schools to share and discuss data, possible meanings and implications of data, and action on data.

Cross-school meetings to discuss data and actions

• Teachers valued having dedicated time to come together from different schools to co-construct definitions for key constructs, analyse research articles, and share the nature and outcomes of their work with students. A key issue raised by the teachers in the last year of the project was how they could sustain their collective and collaborative work. This challenge was not resolved.

Conclusion

There is an increasing imperative for teachers to be data literate but the intent of data literacy is contested. There is merit in teachers spending time developing an agreed definition, ideally focusing on instructional improvement. A data conversation protocol such as that used in this study is an effective and efficient way of supporting teachers to collaboratively analyse and act on data aggregated at the level of individual, small group, class, school, and across school. With support and tools such as a data conversation protocol, teachers can productively coach their colleagues in data use for instructional purposes. While in-depth analysis helps surface root causes of student misconceptions, the analysis of data collated at the level of school/schools directs attention to pedagogical practice. By moving between these two focuses, teachers can keep the needs of students and their pedagogical practice in mind.

Project team

Dr Frances Edwards is a Senior Lecturer at the University of Waikato. Her research focuses on assessment and teacher professional learning.
Email: frances.edwards@waikato.ac.nz

Professor Bronwen Cowie is Associate Dean Research, Division of Education, University of Waikato. Her research focuses on assessment / assessment for learning, science education and initial teacher education.

Dr. Suzanne Trask was a researcher at the University of Waikato during the TLRI, she is now at the University of Auckland. Her research focuses on science and science health education and assessment.

Project partners: Acknowledgement

Our partners were primary and intermediate teachers, teaching a range of year levels from the Pukekohe Kāhui Ako. We wish to acknowledge their commitment to the project and the knowledge and expertise they brought to our collective thinking. We also wish to thank the Kāhui Ako lead principals and the principals for all the participating schools for their ongoing support.

Nicola Gibson, Diana Jolly, Kim Woolliams, and four other teachers from Pukekohe Intermediate School

Michelle Wallis and Libby Bickley-Barry from Puni School

Lisa Goldsack (Buckland School)

Melinda Bolton (Paerata School)

Lyndal Fonokalafi (Pukekohe East School)

And four teachers from three other Kāhui Ako schools.

References

Ball, D. (1993). With an eye on the mathematical horizon: Dilemmas of teaching elementary school mathematics. The Elementary School Journal, 93(4), 373–397.

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5(1), 7–74.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org /10.1191/1478088706qp063oa

Claudet, J. G. (2020). Using design research thinking and data-teaming processes to transform educators’ professional practice: A school improvement case study. International Journal of Education and Social Science, 7(1), 17–41.

Cowie, B., Edwards, F., & Trask, S. A. (2021). Explicating the value of standardized educational achievement data and a protocol for collaborative analysis of this data. Frontiers in Education, 6. https://doi.org/10.3389/feduc.2021.619319

Dalton, J., & Anderson, D. (2016). Learning talk: Important conversations at work. Hands On Educational Consultancy.

Education Review Office (ERO). (2017, January 17). Communities of learning | Kāhui Ako: Collaboration to improve learner outcomes. http://www.ero.govt.nz/publications/communities-of-learning-kahui-ako-collaboration-to-improve-learner-outcomes/

Edwards, F., Cowie, B., & Trask, S. (under review). Using colleague coaching to develop teacher data literacy.

Gerzon, N. (2015). Structuring professional learning to develop a culture of data use: Aligning knowledge from the field and research findings. Teachers College Record, 117(4), 1–28.

Huguet, A., Marsh, J. A., & Farrell, C. C. (2014). Building teachers’ data-use capacity: Insights from strong and developing coaches. Education Policy Analysis Archives/Archivos Analíticos de Políticas Educativas, 22, 1–28.

Jolly, D. (2019). Insights from a deeper analysis of year 7 PAT mathematics scores. [Unpublished Special Topic Report]. University of Waikato.

Kippers, W. B., Poortman, C. L., Schildkamp, K., & Visscher, A. J. (2018). Data literacy: What do educators learn and struggle with during a data use intervention? Studies in Educational Evaluation, 56, 21–31. https://doi.org/10.1016/j.stueduc.2017.11.001

Lasater, K., Albiladi, W. S., Davis, W. S., & Bengtson, E. (2020). The data culture continuum: An examination of school data cultures. Educational Administration Quarterly, 56(4), 533–569.

Love, N., Stiles, K. E., Mundry, S., & DiRanna, K. (2008). A data coach’s guide to improving learning for all students: Unleashing the power of collaborative inquiry. Corwin.

Lowenhaupt, R., McKinney, S., & Reeves, T. (2014). Coaching in context: The role of relationships in the work of three literacy coaches. Professional Development in Education, 40(5), 740–757. https://doi.org/10.1080/19415257.2013.847475

Mandinach, E. B., Friedman, J. M., & Gummer, E. S. (2015). How can schools of education help to build educators’ capacity to use data? A systemic view of the issue. Teachers College Record, 117(4), 1–50.

Mandinach, E. B., & Gummer, E. S. (2016). What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions. Teaching and Teacher Education, 60, 366–376. https://doi.org/10.1016/j.tate.2016.07.011

Ministry of Education. (2007). The New Zealand curriculum for English-medium teaching and learning in years 1-13. Wellington, New Zealand: Learning Media.

Ministry of Education. (2011). Ministry of Education position paper: Assessment [schooling sector]. Learning Media.

Ministry of Education. (2019). Curriculum progress tools: The learning progression frameworks. https://curriculumprogresstools. education.govt.nz/lpf-tool/

Penuel, W., Fishman, B., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331–337. https://doi.org/10.3102/0013189X11421826

Peter, M., Cowie, B., Edwards, F., Eyers, G., & Adam, A. (2017). Beyond pretty charts: From student data to pedagogical action. http:// bit.ly/2q3he1R

Pierce, R., & Chick, H. (2011). Teachers’ intentions to use national literacy and numeracy assessment data: A pilot study. Australian Education Researcher, 38, 433–447. https://doi.org/10.1007/s13384-011-0040-x

Schildkamp, K., & Poortman, C. L. (2015). Factors influencing the functioning of data teams. Teachers College Record, 117(4), 1–42. Schildkamp, K., Poortman, C., Luyten, H., & Ebbeler, J. (2017). Factors promoting and hindering data-based decision making in schools. School Effectiveness and School Improvement, 28(2), 242–258. https://doi.org/10.1080/09243453.2016.1256901

Shulman, L. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1–22.

Wayman, J. C., Cho, V., & Shaw, S. (2009). Survey of educator data use. Unpublished document.

Appendix A

Data conversation protocol (Cowie et al., 2021, p. 6)