The focus of this TLRI project was on sustaining gains in reading comprehension made through TLRIfunded interventions in two clusters of schools in South Auckland. The aim was to develop a model for sustaining effective teaching and school practices so that student achievement continued to improve once the interventions ended. This involved identifying and explaining the conditions that enabled schools to continue improving achievement; explaining how the conditions interrelated; and how these relationships resulted in differing patterns of achievement after the intervention.
Research questions
- Can two clusters decile 1 schools with mainly Māori and Pasifika students sustain student achievement gains one year after their participation in TLRI reading comprehension interventions? Sustainability was judged as having sustained gains at the same rate as during the interventions. An associated analysis was whether the gains in achievement were sufficient to reach parity with national expectations.
- What were the practices associated with sustained improvements in achievement?
Method
Thirteen decile 1 schools from two clusters participated in the study. Across the clusters and over time, we collected information on about 7950 students with slightly more males than females (49.47 percent male, 49.28 percent male, and 1.25 percent unknown). The four main ethnicities were Samoan (34.11 percent), New Zealand Māori (19.83 percent), Tongan (19.04 percent) and Cook Island Māori (16.02 percent). Between 12 percent and 25 percent of students were absent or transient at each testing point. Approximately 120 teachers and 29 school leaders were involved.
A range of data collection measures was used in this study. After the reading comprehension interventions, achievement data from the STAR tests were collected from students in Year levels 4 to 9 at the beginning and end of the academic year, and the beginning of the following year. Interviews were conducted with all school leaders, the developers of the interventions, external facilitators working in the cluster (e.g., Team Solutions) and relevant Ministry of Education staff. Teachers completed surveys on pedagogical content knowledge and leadership; leaders completed the leadership surveys only. We conducted observations of school and cluster meetings where student achievement data were discussed. We also examined relevant school documents (e.g., strategic plans, annual plans).
The achievement data were primarily analysed using Hierarchical Linear Modelling (HLM), a statistical technique which enabled the research team to identify the amount of gain made during the interventions and the amount of gain made one year after the interventions, and to check what demographic factors influenced the achievement trends. To examine what school practices were associated with sustainability, we examined all the data sources (interviews, surveys, documents) to search for common themes using an analyst who was not aware of our theory for sustainability, and then matched the themes to our theory of what would sustain achievement.
What counts as sustainability?
Sustainability in our view is a process of organisational learning through inquiry and knowledge-building cycles to improve outcomes already achieved (Lai, McNaughton, Timperley, & Hsiao, 2009). As such, our theory for sustainability is an ongoing process that follows on from initial improvements made. The focus on valued outcomes for students in our definition comes from making a distinction between processes designed to achieve particular student outcomes and the outcomes themselves. Rather than assuming outcomes for students will follow from particular processes or programmes, judgements about sustainability should be based on the evidence of continued improvement in outcomes (Levin, 2008).
The components for ongoing sustainability in our view are (Lai, Timperley, & McNaughton, in press):
- Developing a cycle of inquiry and knowledge building that allows the school to learn using evidence the effectiveness of its practices, what it needs to do next and what it needs to stop doing. Part of developing a cycle of inquiry involves identifying the school practices and processes that are essential to maintaining and creating ongoing improvement, and having in place systems and processes to identify new challenges and how they will be acted on.
- Embedding the cycle of inquiry and the practices and processes that are essential to maintaining and creating ongoing improvement in schools’ “core business” as part of a coherent instructional programme. By “embedding” we mean that the practices, processes and cycle of inquiry become a taken-for-granted feature of the schools, and becomes part of the schools’ norms, structures, practices and culture (Datnow, 2005). Coherence in this context means that schools develop a set of interrelated programmes for students and staff that are guided by a common framework for curriculum, instruction, assessment and learning climate and that are pursued over a sustained period (Newmann, Smith, Allensworth, & Bryk, 2001).
- Creating interdependence with others (e.g., teachers within a school, other schools). By this we mean developing partnerships with experts within and outside the school (e.g., other schools, researchers, professional developers) to support the school in sustaining its improvements in student achievement. This usually involves a vehicle to systematically access and test knowledge that the school needs in order to continue improving outcomes, such as professional learning communities. Interdependence needs to be managed so that schools do not become overly dependent on external experts.
Was achievement sustained after the interventions?
The data showed that schools were able to accelerate achievement at the same rate as during the interventions (see Figure1). One year after the interventions, clusters gained an average of between 0.42 (Cluster 1) and 0.55 (Cluster 2) stanine in addition to expected gain; that is, they gained about four to five months in addition to the expected gain.
There were no ethnic group differences—all ethnic groups gained at similar rates. There were some differences in the amount of gain made for the following groups of students.
- In both clusters, students who started the year with lower achievement levels (at stanines 1 to 3) made greater rates of gain (up to one year in addition to expected progress in one year level) than students who started the year with higher achievement levels (at stanines 4 to 9).
- There were gender differences in one cluster only. In that cluster, males made more gains than females during the school year, but had greater losses in achievement between academic years (i.e., had lower scores at the start of the year than at the end of the previous year).
Figure 1 Accelerations in achievement made each year during the interventions (Intervention) and after the interventions (Sustainability) based on HLM
Despite the gains, clusters were on average not yet at national expectations, although some schools in both clusters had achievement levels close to national expectations. This was due in part to large drops in achievement from the end of the year to the beginning of the next year (i.e., summer holidays), and the fact that the majority of students in this project had not participated fully in the previous TLRI interventions and started this project, on average, 1 ½ years behind national averages.
What were the practices related to continued improvements in achievement?
- Organisational learning through inquiry and knowledge building cycles.
The main school practice associated with the sustained gains in achievement was ongoing inquiry and knowledge building (solving problems arising from teaching and learning). This involved teachers and school leaders regularly analysing and using achievement data to tailor teaching practices to students’ needs, and monitoring the changes of their teaching practices on student achievement.
- Embedding inquiry in schools’ core business.
The inquiry practices undertaken collaboratively at all levels of the school (e.g., inquiry between teachers, inquiry by senior managers, inquiry as a whole staff) were embedded into what the school normally did. All schools used their staff, syndicate and/or team meetings to analyse the data and change their teaching practices, and the changed teaching practices became part of the normal school and teaching programmes. Some schools put the inquiry process into their teacher appraisals (i.e., the appraisal goal was to demonstrate inquiry). Both clusters made inquiry a central part of their induction “curriculum” for new teachers.
School leaders also minimised other projects that might detract from their core literacy goals. Potential projects were not taken on board if they were perceived to be in conflict with the previous literacy interventions and in general few professional development opportunities outside of the cluster were undertaken.
- Interdependence.
After the interventions, schools continued to be interdependent with other schools in the cluster and with other experts. There were many formal opportunities for teachers to learn from other teachers and leaders in other schools, and from external experts such as researchers and professional developers. The clusters organised interschool teacher conferences where teachers inquired into an aspect of teaching and shared the findings with other teachers. Researchers and professional developers further supported the schools in collecting, analysing and discussing cluster achievement data, and all schools used some form of external expertise for professional development of their teachers. There were also informal networks, such as the leaders contacting the original interventions designers on an ad hoc basis for advice. In this sense, sustainability was not about “schools doing it alone”, but schools being strategic about whom they needed to call to support them and when such support was needed.
What might clusters do better?
- Schools need to identify specific student learning needs from the data, rather than discuss the data in a generic “show-and-tell” manner (e.g., talking about the number of students at stanine 3 without examining what the learning needs of students at stanine 3 might be). Identifying specific student learning needs increases the likelihood of sustaining achievement because the teachers would know precisely what the students’ learning needs are, and would be able to tailor their teaching practices to address the identified needs.
- Schools can enhance the effectiveness of their inquiry by developing greater pedagogical content knowledge; that is, the day-to-day knowledge of how students understand and misunderstand their subjects; how to diagnose and anticipate such misunderstandings and how to deal with them when they arise (Darling-Hammond & Bransford, 2005). Pedagogical content knowledge requires deep knowledge of the domain or the content that is being examined, for example, the knowledge of how texts work. Effective inquiry relies on having appropriate pedagogical content knowledge to identify the student learning needs and to determine the most effective strategies to address those needs.
Conclusion
Clusters continued to make gains in achievement at the same rate as during the interventions, despite the fact that they were no longer in the reading comprehension interventions and were no longer receiving any intensive professional learning directed at the original interventions’ focus. Clusters gained about four to five months in achievement in addition to the expected gain. The main school-based practice associated with sustainability was organisational learning through inquiry and knowledgebuilding cycles. Schools embedded the inquiry cycles into their normal school routines and were interdependent with other educators.
Despite the continued gains in achievement, neither cluster reached national expectations, although there were some schools in both clusters that had achievement levels close to national expectations. Examining specific students’ needs (rather than generic needs) from the achievement data, and increasing the level of teacher and school leader pedagogical content knowledge might increase achievement further.
References
Darling-Hammond, L., & Bransford, J. (2005). Preparing teachers for a changing world: What teachers should be able to learn and be able to do. uSA: John Wiley & Sons.
Datnow, A. (2005). The sustainability of comprehensive school reform models in changing district and state contexts. Educational Administration Quarterly, 41(1), 121-153.
Lai, M. K., McNaughton, S., Timperley, H., & Hsiao, S. (2009). Sustaining continued acceleration in reading comprehension achievement following an intervention. Educational Assessment, Evaluation and Accountability, 21(1), 81-100.
Lai, M. K., Timperley, H., & McNaughton, S. (in press). Theories for improvement and sustainability. In H. Timperley & J. Parr (Eds.), Weaving evidence, inquiry and standards to build better schools. Wellington: NZCER Press.
Levin, B. (2008, January). Sustainable large-scale change: Implications from the Ontario experience. Paper presented at the International Congress on School Effectiveness and Improvement (ICSEI) annual conference, Auckland, New Zealand.
Newmann, F. M., Smith, B. A., Allensworth, E., & Bryk, T. (2001). Instructional program coherence: What it is and why it should guide school improvement policy. Educational Evaluation and Policy Analysis, 23(4), 297-321.
Acknowledgements
We wish to acknowledge the professional expertise of the teachers and leaders in the schools. The achievements described here derive from their hard work.
We wish to acknowledge members of our research team (Angela McNicholl, Sophie Kercher, Sasha Farry, Alaisa Pritchard), the practitioner-researchers involved in this project (Karen Mose, Alison Hall, Nicky Knight and Colleen gray), and colleagues from the Ministry of Education, particularly Susan Warren. We would also like to acknowledge additional contributions to this project from other staff members of the Woolf Fisher Research Centre including Althea Leonard.
The Woolf Fisher Research Centre is a centre of the university of Auckland, supported by Auckland UniServices Limited, and receives funding and support from the Woolf Fisher Trust and The University of Auckland. of Minds (Learning Media, 2002). His current research is focused on properties of quality teaching of literacy and language in the context of research-based interventions with urban schools
Authors
Mei Kuin Lai (PhD) is the Associate Director at the Woolf Fisher Research Centre (university of Auckland), and Project Leader of the TLRI project on Sustainability. The Centre has a national and international reputation for excellence in research on teaching, learning and development in culturally and linguistically diverse communities. Mei is passionate about research that makes a difference to teaching and learning, and has research and teaching interests in sustainable research and development interventions focused on collaborative inquiry. She is currently (or has been) project leader or co-project leader of literacy interventions and evaluations in secondary and primary schools, including the recent national evaluation of schooling improvement initiatives. Her recent book is Practitioner Research for Educators (Corwin Press, 2006).
Stuart McNaughton (PhD) is the Director at the Woolf Fisher Research Centre and Professor of Education (School of Arts, Languages and Literacies, university of Auckland). He is the Research Associate on the project. Stuart has research and teaching interests in developmental and educational psychology with a focus on the development of language and literacy, and processes of education, socialisation and culture. His publications include Being Skilled: The Socialisation of Learning to Read (Methuen, 1987); Patterns of Emergent Literacy: Processes of Development and Transition (Oxford University Press, 1995); and Meeting of literacy and language in the context of research-based interventions with urban schools.
Helen Timperley (PhD) is Professor of Education (School of Teaching, Language and Development, university of Auckland) and Research Associate on the project. Her research interests are in policy, research and promoting professional learning. Her recent books include Evidence-based Conversations to Improve Educational Practices (Kluwer/Springer Academic Publishers, 2008), Using Evidence in Teaching Practice: Implications for Professional Learning (Hodder Moa Beckett, 2004) and Unlocking Formative Assessment: Practical Strategies for Enhancing Students’ Learning in the Primary and Intermediate Classroom (Hodder Moa Beckett, 2003).
Selena Hsiao (MSc) is the Senior Statistician at the Woolf Fisher Research Centre (University of Auckland), and lead statistician on the project. She is the lead statistician on various projects in the Centre specialising in statistical modelling to evaluate growth in literacy achievement. Her research interests are in complex statistical modelling, statistical programming such as hierarchical linear modelling using SAS and R languages, and innovative approaches to presentations of data through graphical and object-oriented programming.
