Enhanced teaching and learning of comprehension in Years 4-9 in Seven Mangere Schools

Executive Summary

The schools of South Auckland which have high proportions of Māori and Pasifika students have long been described by researchers as sites for low achievement, particularly in literacy (e.g., Ramsay, Sneddon, Grenfell & Ford, 1981). However, recent evidence suggests that the disparities between Māori and Pasifika students and other students in reading accuracy have been reduced, and that there has been a substantial reduction in the proportions of students in the lowest bands of achievement. Despite this, the evidence also suggests that at Year 4 and Year 9, the disparities in reading comprehension have continued, if not increased (Crooks & Flockton, 2005).

A research and development programme, conducted as a collaborative partnership between researchers, schools and the Ministry of Education, was designed to test several questions about achievement in seven decile 1 schools in South Auckland. These questions were:

• Can a research-practice collaboration develop cluster-wide and school based professional learning communities that are able to critically analyse and problem solve issues of instructional effectiveness, thereby developing more effective instruction that has a powerful educationally significant impact on Māori and Pasifika children’s comprehension at Years 4–9 in decile 1 schools?
• Can a set of effective instructional activities be identified that are able to be used by teachers to enhance the teaching of comprehension for Māori and Pasifika children in Years 4–9 in decile 1 schools?

In addition, there was a specific question about Samoan students and achievement in Samoan bilingual classrooms:

• Can the research and development programme contribute to more effective instruction for Samoan students in Samoan bilingual classes?

These questions were based on a set of hypotheses about the nature of effective instruction for reading comprehension, and the nature of effective school-based interventions. There were two main hypotheses: first, that more effective teaching could be developed through a professional learning community that has a continuing process of critical discussion and problem solving, based on evidence (Robinson & Lai, 2006); and secondly, that effective instruction would include a range of attributes, such as explicit teaching of strategies, and deliberate teaching of vocabulary (Pressley, 2002), but that these would need to be contextualized to the specific needs created by past histories of schooling and contemporary profiles.

The research and development programme was conducted over three years with up to 70 teachers and, in different years, between 1200 and 1900 students, over 90 percent of whom were Pasifika or Māori students. In the six Samoan bilingual classes from two schools, there were between 140 and 169 students across different years. A quasi-experimental design was employed to examine relationships between the programme and the outcomes over three years. The robustness of the design was enhanced by features such as a comparison with an untreated cluster of similar schools, and checks on subject attrition. Repeated measures of student achievement at the beginning and the end of each year, and a final measure at the beginning of the fourth year, form the basis of the design which, among other things, examines rates of gain against predicted patterns of growth generated from a baseline.

An initial step involved collecting baseline ‘profiles’ of achievement, using the standardised assessments of reading comprehension from PAT ( Reid & Elley, 1991), and of a range of aspects of reading comprehension, including decoding provided by STAR (Elley, 2001). It also involved collecting baseline profiles of classroom instruction, using systematic observations in classrooms. Together these baselines provided detailed evidence about strengths and weaknesses in the students’ reading comprehension, which were able to be mapped on to patterns of instruction in the classroom. For example, it showed that low decoding levels were generally not a problem; rather, it was patterns of checking and detecting threats to meaning in paragraph comprehension, and size and knowledge of vocabulary, that were posing difficulties. An unpredicted finding was that while high rates of explicit strategy instruction occurred, students were focused on the strategies as ends in themselves, and often resorted to guessing. Classroom observations showed a low incidence of teachers or students monitoring and checking strategies, and low rates of identifying and elaborating meanings of low frequency words, unusual uses of common words, or idiomatic uses.

The first phase included systematic feedback and analysis and problem solving at cluster, school and classroom levels, using the profiles as evidence. This process occurred each year thereafter. A second phase added targeted professional development, based on the evidence in the first phase, with all the Year 4–9 teachers. The third phase involved planned sustainability of the professional learning communities, with teacher designed projects and a cluster led conference.

At baseline, students were on average at stanine 3.1, approximately two years below expected levels, and this was generally the case, with some variation across year levels and across schools. To test the impact of the programme, a number of different analyses were made using longitudinal cohorts, comparisons with baseline projections, and total school population changes.

Analysis of achievement for longitudinal cohorts showed that by the end of the project, the average student now scored in the average band of achievement (stanine 4.21). The overall effect size for gains in stanines was 0.62. Māori students’ achievement accelerated at similar rates to those of the other ethnic groups participating in the project, so that by the end of the project, the average Māori student scored within the average band (mean =4.73), with one cohort of Māori students (Year 4) scoring above the national expected average at stanine 5.29. Males and females made similar rates of progress over the three years in the intervention, but female students, on average, started with higher levels of achievement than male students. On average, students in each school made accelerated gains in achievement from the beginning to the end of the project.

Analyses using the design format showed that after two years and after three years, students had statistically significantly higher achievement than baseline comparison groups (effect sizes ranged between 0.31 to 0.59), and were achieving statistically significantly higher than a comparison cluster of schools (effect sizes ranged between 0.33 and 0.61.)

When total school populations were analysed (which included new students entering and students leaving), a similar picture to that of the previous analyses emerged. The overall level of achievement showed a variable but increasing trend over time, so that by the end of the intervention, the average stanine for 1700 students at 7 schools was 3.61. A range of gains were made between schools and within schools across the three phases. Several factors were suggested as contributing to these differences in gains, including degree of participation by schools and teachers, and aspects of curriculum planning.

Observations of classroom instruction were carried out systematically in both the first and the second years. Significant changes in types of teacher and student exchanges relating to the focus of the intervention were linked to the pattern of the gains over two years in the component tests. Further case studies of teachers showed that a high gain teacher more often directed students’ awareness to the requirements of activities, clarified her high expectations, pushed her students with complex tasks, introduced more complex and less familiar language including idiomatic uses, created a classroom community that enjoyed the use and study of oral and written language, exposed students regularly to rich and varied texts, and was able to incorporate student cultural and linguistic resources, as well as clarifying areas of confusion.

The analyses of students in Samoan bilingual classrooms showed that the programme was effective in those classes too. Gains by students in the bilingual classrooms were at least as high as the gains by Samoan students in the mainstream classrooms, and in three of the year levels, they were noticeably higher. Students in bilingual classrooms were significantly lower in English reading achievement in Year 4 and Year 5, but from Year 6 onwards, their achievement levels in English were similar. Overall, cohorts made 1.13 stanine gain in two years; for four cohorts, this was a higher rate of gain than for Samoan students in mainstream classes. Gains in these classrooms could also be linked with the degree of participation by schools and teachers.

We concluded that it is possible to develop more effective teaching that impacts directly on the reading comprehension achievement of Year 4–9 children. The level of gains overall were in the order of one year’s gain in addition to nationally expected progress over three years. When these gains are considered in terms of the history of schooling in South Auckland, the educational significance of the gains, and the international literature of schooling improvement, they are seen to be substantial. Even when results for all the students present from the beginning to the end are considered, including those who subsequently left and those who subsequently entered the school, either from earlier levels or as new students from other schools, the levels of achievement at the schools have increased considerably. Given the quasi-experimental design with its additional strengths, these gains can be attributed with some confidence to the effects of the three phase model adopted by the research and development programme.

The analyses suggest that thinking about and critically discussing the evidence at a classroom, school and cluster level led to a significant part of the overall gains in achievement and that the professional learning communities had the capacity to use the evidence to make changes to existing practices. This is likely to be dependent on external support, in the form of collaborative research-practice-policy partnerships (e.g., Robinson & Lai, 2006). We need to consider how to foster such partnerships, in terms of both the kinds of partnerships being developed, and the infrastructure to support the development and sustainability of such partnerships.

The analyses of instruction show that specific aspects of instruction changed, including the focus on checking and detecting threats to gaining meaning in texts and boosting vocabulary acquisition, consistent with the focus of the programme and consistent with the gains that were made. But they indicated the need for caution in making assumptions about instructional and learning needs from the existing literature alone. They also indicated that effective instruction needs to be designed to fit the context-specific needs created by past histories of schooling and contemporary profiles. Interestingly, gains on the decoding test also increased to about the same degree as gains in other areas, despite not being a direct target of the intervention.

The educational intervention also impacted on Samoan students’ achievement in bilingual classrooms, demonstrating that Samoan students in bilingual classes can develop literacy in English to levels similar to those of other Samoan students who are not in bilingual classes. The evidence also shows that developmental changes in English comprehension come to reach mainstream levels by around Year 6, but that this rate of change may be modifiable too. It is important to see these results in a wider developmental and educational context, involving bilingual and biliteracy development in these classes.

1. Introduction

Yesterday was too late?

In 1981, Peter Ramsay (Ramsey, Snedden, Grenfell & Ford, 1981) and his colleagues at the University of Waikato completed a study of the schools in South Auckland. The title of their report was Tomorrow may be too late. They argued that there was an impending crisis created by “educational disadvantage suffered by most school-aged students in Mangere and Otara” who were “achieving well below their chronological age” (p. 41). They concluded with “a plea for urgency as the needs of the children of Mangere and Otara are very pressing. Tomorrow may be too late!” (p. v).

The gap in achievement between Mäori and non-Mäori children in mainstream schools is not a recent phenomenon. Earlier reports, such as the Currie (1967) and Hunn (1960) reports on education in the 1950s, had identified this difference as important and as urgently in need of a solution (see also Openshaw, Lee & Lee, 1993). The long standing on the “problem” for Mäori students is important to note, because some commentaries suggest it is relatively recent, and can be linked to changes in methods of teaching reading and writing which began in the 1960s (Awatere Huata, 2002; Nicholson, 2000).

Yet the historical picture is not entirely bleak. There is evidence that in the colonial period, there were times when Mäori children outperformed other children in some schools. Some evidence for this can be found in the Survey of Native Schools for 1930 (Education Gazette, 1 December 1930, Department of Education, 1930; see also McNaughton, 2000).

The sense of crisis that Ramsay expressed for the sake of children, communities and families is also present in reports from other countries (Snow, Burns & Griffen, 1998). The need is identified for communities who have, relative to the mainstream communities, less economic and political power, whose children are considered to be “minorities”. But there has been little evidence that the crisis is able to be solved in schools. In the United States, Borman (2005) shows that national reforms to boost the achievement of children in low performing schools serving the poorest communities have produced small gains in the short term (of the order of effect sizes of less than 0.20), but that after seven years, in those few schools that sustain reforms over a long period, the effects increase (estimated to be around effect sizes of 0.50). When considered across the country, while some achievement gains have occurred, they have typically been low and need to be accumulated over long periods of time.

At a more specific level, some studies from the United States have shown that clusters of schools serving ‘minority’ children have been able to make a substantial difference to the achievement of children. In one set of studies (Taylor, Pearson, Petersen & Rodriguez, 2005), researchers intervened in high poverty schools with carefully designed professional development research and development. They too found small cumulative gains across two years. This study and others pointed to important school level factors that must be in place in order for all children to achieve at high levels in reading. Summarising these, Taylor et al. (2005) noted six key elements: improved student learning; strong building leadership; strong staff collaboration; ongoing professional development; sharing student assessment data; and reaching out to parents. In these studies, there is evidence that achievement can be effected, and in the case of studies such as Taylor et al., that small gains over two years could be attributed to these characteristics.

The days after Ramsay’s tomorrow

Where does such offshore evidence leave the schools of South Auckland, which, according to Ramsay, had already received substantial additional resources by the early 1980s? There is little evidence that Ramsay’s concern led to immediate changes. The evidence from both national and international comparisons suggests that by the beginning of the 1990s, the children in decile 1 schools, and more generally children who were Mäori and Pasifika, were still not achieving as well as non-Mäori and non-Pasifika children in reading comprehension. The reading comprehension comparisons across 32 countries in the International Association for Evaluation of Educational Achievement study (see http://www.iea.nl/reading_literacy.html) provided stark evidence of what came to be called a “long tail” in the distribution of achievement. The problem was that while in general New Zealand continued to have high average achievement, and the best students in New Zealand were superior to other students in the world, Mäori and Pasifika children were over-represented in the “long tail” (Elley, 1992; Wagemaker, 1992).

In New Zealand, the recognition of the distribution problem, as well as other research developments, has had an effect. Reports by a Literacy Task Force (1999) and a Literacy Experts Group (1999) contributed to a national policy shift, which was implemented in the National Literacy and Numeracy strategy. The policy shift promoted concerted professional development and research practice development which was focused on Years 1–4 and Mäori and Pasifika children, especially those in decile 1 schools.

Associated with this policy and practice shift, there is now evidence from the national educational monitoring project (NEMP) and renorming exercises that changes in the targeted areas have occurred (Elley, 2005). The news is positive for the early stages of literacy instruction. From NEMP, the one area in literacy achievement where there are clear changes is in reading decoding, both accuracy and fluency (Flockton, & Crooks, 2001). Their second cycle of assessments of reading showed that the percentages of children reading below age level in reading accuracy at Year 4 had reduced markedly from 1996 to 2000, from around 20 percent to around 13 percent.

Little improvement occurred for Year 8 children in oral reading (Flockton & Crooks, 2001). A recent renorming of standardised assessments at Year 1 (6 years) conducted in 2000 also suggests that knowledge of letters and sounds has improved (Clay, 2002).

These increases in oral reading accuracy were found to have been maintained in the third (2004) cycle of assessments at Year 4. Further notable increases in accuracy were found for the Year 8 children, with only around 11 percent at both year levels now reading below age level (Crooks & Flockton, 2005). The breakdown of gains in 2000 and 2004 suggest that reading accuracy had improved at similar rates at Year 4 for both Mäori and Pakeha children (Flockton, 2003). But by 2004, the analyses showed substantial reduction at Year 4 in the gap between Pakeha and Mäori students (see further comment nemp.otago.ac.nz/forum_comment/2004).

Research based interventions using experimental designs have shown that the gaps at this early stage can be reduced considerably. We also know many of the characteristics of effective teaching at that early stage. For example, in the “Picking up the Pace” research with Mäori and Pasifika children in decile 1 schools in Mangere and Otara, their typical achievement was two stanines^[1] below average levels in areas of decoding after a year at school (Phillips, McNaughton, & MacDonald, 2004). A research based intervention used professional development with teachers and teacher leaders to increase effectiveness in areas of reading and writing, including specific phonics instruction. Where teaching approaches were fine-tuned to solve children’s confusions and to make the purpose of classroom activities more obvious, and higher expectations about achievement were developed through evidence based analyses of progress, the children’s achievement was raised to close to the national distribution (see Phillips, McNaughton, & MacDonald, 2004). In some areas, such as alphabet knowledge, their progress was as good as or better than typical progress; in others, e.g. progress through text levels, they closely approximated typical progress; but in one area, generalized word recognition, they were still noticeably below average levels.

“Tomorrow” is still the same for reading comprehension

These indicators of progress are cause for some celebration, given the urgency signalled in Ramsay’s report, and the seemingly intractable nature of the teaching difficulty over decades. But the news has not all been good. For reading comprehension, little appeared to have changed for Mäori and Pasifika children in low decile schools over the period in which the decoding changes occurred, as we will show below. The NEMP data indicate increases in levels of comprehension in Year 4 from 1996–2000, but the breakdown of the achievement patterns suggests a substantially wider disparity between Mäori and non-Mäori in comprehension both at Year 4 and at Year 8. Furthermore, for children in low decile schools, gaps in comprehension increased both at Year 4 and at Year 8 (Flockton, 2003).

In the third cycle of assessments in 2004, the gains in oral reading accuracy were not matched by similar gains in reading comprehension for the total group of students at either Year 4 or Year 8. The detailed comparisons suggest that the gaps in oral reading accuracy between Mäori and Pasifika students and Pakeha students which had closed between 1996 and 2000 reduced further in 2006. But this was not matched in comprehension (Crooks & Flockton, 2005). Commentaries on this 2004 report note that Mäori children performed well in decoding, but there were large differences in favour of Pakeha in aspects of comprehension (nemp.otago.ac/forum_comment/2004). These differences were apparent for Pasifika children too, and they were apparent for decile 1-3 schools when compared with other decile groups (Crooks & Flockton, 2005).

This is true also of at least some of the schools in South Auckland. When we completed what we describe further in this report as a baseline profile of a cluster of schools in Mangere, we found that across schools and year levels, achievement in reading comprehension was relatively flat at around stanine 3, and something like two years below what would be expected as average progress nationally (Lai, McNaughton, MacDonald, & Farry, 2004). We have since repeated this finding with a cluster of Otara schools. They too were, on average, around stanine 3 across year levels and across schools (Lai, McNaughton, MacDonald, Amituanai-Toloa & Farry, 2006).

What we now know is that even if we achieve a dramatic change in teaching early reading, it does not necessarily mean that the gap reduces further on up the system. Experimental demonstrations specifically targeting the teaching of phonics also tend to show very limited transfer to comprehension (Paris, 2005). Recent national data from the AsTTle project across multiple dimensions of reading comprehension confirm the NEMP picture of large differences between Mäori and Pasifika children which are stable across decile levels, despite significant trends of higher achievement from lower to higher decile level schools (Hattie, 2002).

These comparisons need to be treated with an important qualification. The broad description of these disparities can mask important aspects of the literacy development and achievement of children in so-called “minority” groups. The conventional indicators of school literacy represent some of what children learn about literacy. But children who are in communities which have low employment, low incomes, and minority cultural and language status have engaged in a range of literacy and language activities, some of which might be quite different from mainstream children. Their knowledge, therefore, may not be well represented in tests of conventional literacy practices, especially at the beginning of schooling (McNaughton, 1999; Snow, Burns, & Griffen, 1998) and as they move into the middle school levels.

Here, it is important to note that there is an urgent challenge which has strategic importance to all of New Zealand. Students need greater ranges and levels of knowledge and skills for post secondary school study and for employment. Education is increasingly important to the success of both individuals and nations (Darling-Hammond & Bransford, 2005). Over-representation of particular groups in low achievement bands is not acceptable at individual, community or national levels, no matter what the proportion of the population. It is a pressing matter of cultural, political, constitutional (Treaty of Waitangi), ethical, economic and educational significance that we develop more effective forms of instruction for these students. It is worth noting that by 2021, Mäori children will comprise 28 percent and Pasifika children 11 percent of all the under-15-yearolds in New Zealand (Statistics New Zealand, 2002). In Mangere and Otara schools, children from these communities already make up over 90 percent of many school rolls.

There is an additional dimension to that challenge. Many of the children in the South Auckland schools have a language other than English as their home language. Yet language development for these children is not well understood. In the context of bilingual instruction, for example, and the relationships between development in two languages and two systems of literacy, little is known about biliteracy development and relationships with literacy and literacy instruction (Sweet & Snow, 2003).

The research we report here has two foci. There is a main study of seven schools, their teachers and their students. Within this, there is a second study specifically of teachers and students in Samoan bilingual classrooms. This is the first time, to our knowledge, that there has been a specific research based intervention with Samoan students in bilingual programmes which looks at their achievement.

Twenty-five years after the Ramsay report, we can report in this study important gains in reading comprehension in Year 4–9 students in decile 1 schools in Mangere. This report describes the science of these changes and documents the research and development programme that have taken place. However, the science is closely bound up with a policy context of associated changes in practices. It is likely that without the policy context, the science involved in developing more effective instruction would have achieved less. The results reported here need to be considered with this policy context in mind (Annan & Robinson, 2005).

In addition, the research is located in a particular historical context of school-based interventions. One is the landmark study “Picking up the Pace” (Phillips, McNaughton & MacDonald, 2001). As noted above, this focused on instruction in the first year, and set out to examine the separate and combined effects on children’s achievement of providing co-ordinated professional development to teachers in early childhood settings, and to teachers of children in their first year of schooling. Since the success of that project, which was completed in 2001, further professional development for Year 1 teachers has occurred, based on the practices identified in the research, and the programme in some schools has been extended through to Year 3.

That study and its further development were part of a much broader project initiative, “Strengthening Education in Mangere and Otara” (SEMO), which aimed to raise the achievement levels of children in these two areas. SEMO’s general aim was to strengthen schools in the area and to enhance children’s learning opportunities, particularly in literacy, by enhancing the work of early childhood and primary teachers who were providing literacy programmes. SEMO was succeeded by a further policy and practice development in Mangere and Otara, “Analysis and Use of Student Achievement Data” (AUSAD). This project is located within that government-funded school improvement initiative. The goal of AUSAD is to offer high quality learning environments to raise achievement. This is done by using student achievement information to inquire into the nature of the under-achievement, to test competing explanations of its cause, and to monitor the impact of teachers’ decisions about how to intervene. In short, the focus is on developing the inquiry skills of teachers to improve school practices and student learning outcomes. The initiative comprises a number of interventions focusing on improving literacy and numeracy achievement (e.g., the “Third Chance” programme aimed at improving literacy in Years 1–3).

Reading comprehension

Recent commentaries identify a major theoretical challenge facing literacy instruction. Now that some of the pressing issues in beginning reading instruction (but by no means all) have been resolved, the challenge concerns the teaching of reading comprehension. Higher levels of reading comprehension and related areas of critical thinking are central to the purposes of contemporary schooling, and are part of the education priorities and key competencies that have been set for New Zealand education (Ministry of Education, 2005). But there is a critical need for research into instruction that enhances comprehension, and into interventions that enable schools to teach comprehension effectively. The most recent reviews of relationships between research and practice note that overall evidence of teacher effectiveness is limited, and that research has not impacted greatly on effective comprehension instruction (see Block & Pressley, 2002). Similarly, the RAND reading study group, which was set up in 1999 by the US Department of Education’s Office of Educational Research and Improvement to identify the most pressing needs for research in teaching reading, has concluded:

We have made enormous progress over the last 25 years in understanding how to teach aspects of reading. We know about the role of phonological awareness in cracking the alphabetic code, the value of explicit instruction in sound–letter relationships, and the importance of reading practice in producing fluency.… The fruits of that progress will be lost unless we also attend to issues of comprehension. Comprehension is, after all, the point of reading. (Sweet & Snow, 2003, p. xii)

The challenges to teaching effectively have been identified (Pressley, 2002; Sweet & Snow, 2003). One is the need to build on the gains made in research about instructional practices for beginning literacy. A second is to do with knowledge transfer, a failure to turn all that we know about comprehension and comprehension instruction into generally more effective teaching.

These needs are particularly significant for schools serving culturally and linguistically diverse populations in low income areas (Garcia, 2003).

As noted above, on average, students in the middle years of school in New Zealand have high levels of reading comprehension, judged by international comparisons; however, there are large disparities within the distribution of achievement. These are between children from both Mäori and Pasifika communities in urban schools with the lowest employment and income levels, and other children (Alton-Lee, 2004). These findings highlight the need for instructional approaches that enable teachers to develop, use and sustain effective teaching of reading comprehension with culturally and linguistically diverse students. For Pressley (2002), this challenge represents an application problem. We know a lot about what students need to be able to do, which includes such things as regulating strategy use, and we know a lot about specific instructional effects, such as the need for explicit strategy instruction. What he claims we have failed to do is translate that knowledge into widespread usage with known effects. While Sweet and Snow echo this claim in their RAND summary of reading comprehension instruction, they also argue that there is yet more to be known about specific teaching and learning relationships, especially in the context of diverse readers, diverse text types and diverse instructional contexts (Sweet & Snow, 2003).

Generally, there is considerable consensus around what students need to learn, and what effective teaching looks like. In order to comprehend written text, a reader needs to be able to decode accurately and fluently, and to have a wide and appropriate vocabulary, as well as appropriate and expanding topic and world knowledge, active comprehension strategies, and active monitoring and fix up strategies (Block & Pressley, 2002; Pressley, 2002). So it follows that children who are making relatively low progress may have difficulties in one or more of these areas. The consensus around teaching effectively identifies attributes of both content (curriculum) and process (Taylor et al., 2005). For the middle grades, these include instructional processes in which goals are made clear, and which involve both coaching and inquiry styles that engage students in higher level thinking skills. Effective instruction also provides direct and explicit instruction for skills and strategies for comprehension. Effective teaching actively engages students in a great deal of actual reading and writing, and instructs in ways which enable expertise to be generalisable and through which students come to be able to self regulate independently.

In addition, researchers have also identified the teacher’s role in building students’ sense of self efficacy and, more generally, motivation (Guthrie & Wigfield, 2000). Quantitative and qualitative aspects of teaching convey expectations about students’ ability which affect their levels of engagement and sense of being in control. These include such things as text selection. Culturally and linguistically diverse students seem to be especially likely to encounter teaching which conveys low expectations (Dyson, 1999). There are a number of studies in schooling improvement which have shown how these can be changed. In general, changes to beliefs about students and more evidence based decisions about instruction are both implicated, often in the context of school wide or even cluster wide initiatives (Bishop, 2004; Phillips et. al., 2004; Taylor et. al., 2005).

Just as with the components of reading comprehension, it follows that low progress could be associated with teaching needs in one or more of these areas. Out of this array of teaching and learning needs, those for students and teachers in any particular instructional context will have a context specific profile. While our research-based knowledge shows that there are well established relationships, the patterns of these relationships in specific contexts may vary. A simple example might be whether the groups of students who make relatively low progress in a particular context, such as a cluster of similar schools serving similar communities, have difficulties associated with decoding, or with use of strategies, or both, and how the teaching that occurs in those schools is related to those difficulties.

Several hypotheses are possible for the low levels of reading comprehension which are tested in the following research. One is that children’s comprehension levels are low because of low levels of accurate and fluent decoding (Tan & Nicholson, 1997). A second is that children may have learned a limited set of strategies; for example, they may be able to recall well, but are weaker in more complex strategies for drawing inferences, synthesising and evaluation; or they may not have been taught well enough to control and regulate the use of strategies (Pressley, 2002). Other possible contributing reasons might be more to do with language: that is, children’s vocabulary may be insufficient for the texts used in classroom tasks (Biemiller, 1999); or they may be less familiar with text genres. Well known patterns of “Matthew effects” may be present in classrooms, where culturally and linguistically diverse children receive more fragmented instruction focused on decoding or relatively simple forms of comprehending, or receive relatively less dense instruction, all of which compounds low progress (McNaughton, 2002; Stanovich, West, Cunningham, Cipielewski, & Siddiqui, 1996). There is also a set of possible hypotheses around whether the texts, instructional activities and the pedagogy of the classroom enable cultural and linguistic expertise to be incorporated into and built on in classrooms (Lee, 2000; McNaughton, 2002). But each of these needs to be checked against the patterns of instruction in the classrooms in order for the relationships to be tested.

This approach, which focuses on the need to understand specific profiles, has an implication for meeting the challenges posed by Pressley (2002) and Sweet and Snow (2003). Rather than test in an ad hoc way the significance of certain teaching and learning relationships, what we did in the study was to test a package of targeted relationships. These are relationships initially identified through a process of profiling both learning needs and patterns of existing instruction. The analysis is aimed at adding further to our research-based knowledge of relationships between teaching and learning in specific contexts, and thereby contributing to the research and application challenges signalled by Pressley (2002) and Sweet and Snow (2003).

We assume in this profiling that while much is known, there are still some areas where we need more knowledge and analysis. This need is pressing in the context of cultural and linguistic diversity. An example in our contexts is the role of activation and deployment of background knowledge. A theoretical argument is often made that instruction needs to incorporate more of the cultural and linguistic resources that minority children bring to classrooms (McNaughton, 2002). But complementing this is another argument: that students need to develop more awareness of the requirements of classroom activities, including the relationships between current resources and classroom activities (McNaughton, 2002). While the general hypothesis of the significance of background knowledge is well demonstrated in controlled studies of reading comprehension (Pressley, 2002), the particular idea of teachers being able to incorporate this, and balancing it with enhancing awareness of classroom requirements, has not been well tested.

In the following study we draw on known properties of effective comprehension and on known relationships between types of instruction and learning outcomes. But we apply this knowledge in an intervention context. Within that context, we test the significance of the assumed relationships and features of teaching and learning. Because the context includes substantial numbers of children for whom English is a second language and who come from diverse cultural backgrounds, this is also a context for discovering new learning needs and new relationships between teaching and learning.

Professional learning communities and critical analysis of evidence

A previous study, focused on literacy achievement over the transition to school, demonstrated substantial gains across a cluster of 12 decile 1 urban schools with primarily Mäori and Pasifika students (Phillips, McNaughton & MacDonald, 2001). Among other things, the programme involved intensive collection and analysis of achievement data within schools and across a group of schools. Instructional approaches were modified to impact more strongly on increasing student engagement and teaching effectiveness around agreed goals. Team leaders within schools led professional communities. While the initial development took place within schools over six months, the programme has now been in place in schools for several years. Follow-up research has indicated that those schools which maintained and built on these processes through a professional learning community focused on teaching and learning have increased student achievement over time (Timperley, Phillips & Wiseman, 2003).

The features of these learning communities appear similar to those described by Newman, Smith, Allensworth and Bryk (2001). They identify high “instructional programme coherence”, a necessary condition for improvements in student achievement that are more likely to be sustained over time. These authors define high instruction coherence as “a set of interrelated programmes for students and staff that are guided by a common framework for curriculum, instruction, assessment, and learning climate and that are pursued over a sustained period” (p. 229). The elements suggested which are crucial to high instructional programme coherence can be identified in the Phillips, McNaughton and MacDonald (2004) programme. They include a common instructional framework for teaching literacy across all schools involved in the programme; teachers working together to implement the common programme over a sustained period of time; and assessments which are common across time. Both the New Zealand programme and the high programme coherence schools in the USA rely on long-term partnerships between schools and external support organisations, the development of a common framework for literacy diagnosis which every teacher has to implement, expected collaboration between teachers, joint decisionmaking around assessments to use, and similar factors.

Underlying many of the features of schools with high programme coherence is the use of evidence to guide and evaluate teaching practices. For example, the aim of AUSAD was for practitioners to use student achievement data to inform practice. This has led directly to planning how to design classroom programmes that specifically meet the needs of students in these diverse urban schools. The partnership has responded to the increasing calls for greater understanding of the teaching and learning of comprehension to inform practice in New Zealand (e.g. Literacy Task Force, 1999; Learning Media, 2003) and internationally (Pressley, 2002).

Similarly, critical analysis of student data is identified as significant in school and teaching effectiveness research (e.g. Hawley & Valli, 1999; Robinson & Lai, 2006). In their literature review on effective professional development, Hawley and Valli (1999) identify critical analysis as a more effective form of professional development than traditional workshop models. The collection, analysis and discussion of evidence was present in the schools maintaining gains in the Phillips et al. (2004) programme (Timperley et al., 2003).

A general question that arises is how much the critical analysis process contributes to the student changes in successful programmes. In the research and development programme reported here, the question concerns its contribution to the development of more effective teaching of reading comprehension in schools serving culturally and linguistically diverse students in low income communities. The collection, analysis and discussion process took place in the context of collective analytic and problem solving skills, where teachers collaborated with researchers and professional developers to co-construct the professional development. It is important to note here our assumption that professional expertise was distributed within and across schools, and that teachers would be able to contribute as co-participants in a research-based collaboration (McNaughton, 2002). The issue of how teachers are viewed is particularly salient in the New Zealand context, as recent research syntheses show that school effects are consistently smaller than teacher/class level effects. These latter effects can account for up to 60 percent of the variance in student achievement, depending on the subject areas, level of schooling and outcome of interest, as estimated by Alton-Lee (2004).

This sort of collective problem solving represents one way of balancing two tensions identified in effective educational interventions (Coburn, 2003; Newman et al., 2001). One tension is around the issue of guaranteeing fidelity by adhering to a set of instructional procedures used in wellresearched interventions, versus developing procedures which are derived from context specific problem solving, but may have a less well known research intervention base. A related tension is between importing a set of procedures, in a way which risks undermining local autonomy and efficacy, and a more collaborative development of common procedures, which risks losing instructional coherence. It seems to us that it is possible to construct fidelity to a common programme which has been strongly contextualised by developing a highly focused collaborative context. There is research evidence that suggests approaches in which professional development focuses on joint problem solving around agreed evidence, such as student achievement outcomes, is more likely than predetermined programmes to result in sustainable improvements in student achievement, particularly in reading comprehension (Coburn, 2003; Hawley & Valli, 1999; Timperley, Phillips & Wiseman, 2003).

Evidence is critical to the processes of developing a professional learning community capable of solving the instructional problems associated with more effective teaching. Systematic assessment for formative and diagnostic purposes is essential in order to avoid the problems we have found before, where educators assume that children need a particular programme or approach, but close inspection of the children’s profiles shows that they already have the skills targeted in those approaches (McNaughton, Phillips, & MacDonald, 2003). The significance of collecting and analysing data, rather than making assumptions about what children need (and what instruction should look like), was recently underscored by Buly and Valencia (2002). Policy makers in the State of Washington had mandated programmes without actually analysing profiles of low progress students, identified by test scores from fourth grade National Assessment of Educational Progress (NAEP) scores. The assumption underlying policies and interventions was that poor performance reflected students’ difficulties with more basic decoding abilities. Yet there was little data about this assumption, and little evidence to show that focusing on such skills would improve comprehension at fourth grade. Using a broad band of measures, Buly and Valencia identified five groups of low progress readers, some of whom did indeed have limited fluency and accuracy in decoding. However, mandating phonics instruction for all students who fell below the proficiency levels had missed the needs of the majority of students, whose decoding was strong, but who struggled with comprehension or language requirements for the tests. This finding highlights the need for research-based applications of best practice, based on analyses of student needs. One particular need that has been identified in other countries is for more effective teaching of reading comprehension than has typically been the case (Sweet & Snow, 2003).

The issue of sustainability

Developmental sustainability

A major challenge has been created by the advances made in schooling improvement and increasing instructional effectiveness through professional development. This is the issue of sustainability (Coburn, 2003). For the following research and development programme, sustainability has two meanings. The immediate concern facing the schools in South Auckland has been the need to build further progress in literacy, adding to the more effective instruction in the early years. This inevitably means considering the quality of the teaching and learning of comprehension (Sweet & Snow, 2003). The issue in the decile 1 schools is that the subsequent instructional conditions set channels for further development, and if the channels are constructed for relatively ‘low’ gradients of progress, this creates a need for further intervention. Unfortunately, as we have already noted and describe further below, the available evidence shows that despite the gains in decoding, there were still wide and possibly increasing disparities in achievement on comprehension tasks for Mäori and Pasifika children, particularly in low decile schools (Flockton & Crooks, 2001; Hattie, 2002; Lai, McNaughton, MacDonald & Farry, 2004).

The reason for needing to deliberately build this sustainability resides in the developmental relationships between decoding and comprehension. Logically, there are relationships such as the one identified by Tan and Nicholson, (1997), who showed that poor decoding was associated with poor comprehension. It makes perfect sense, that if you can’t get the words off the page, you can’t comprehend. The problem is that the corollary doesn’t apply—decoding may be a necessary condition, but it is not a sufficient condition. So being a better decoder does not automatically make you a better comprehender.

The developmental reason for this can be found in Paris’s (2005) multiple components model of literacy development, or Whitehurst and Lonigan’s (2001) ‘inside outside’ model of the strands of literacy development. Each of these explains that there are different developmental patterns associated with acquisition for components such as items, and for language meaning and uses, and they are somewhat independent. This accounts for the phenomenon of rapid, accurate decoders who are not able to comprehend, which is described by professional educators and researchers (McNaughton, Lai, MacDonald & Farry, 2004). There is another developmental reason. Inoculation models do not apply to most phenomena in teaching and learning; just because you know and can do some stuff this year doesn’t mean that you automatically make further gains next year. It depends at least in part on whether the teacher you meet effectively enables you to build on to and extend your learning. Fluent, accurate decoding is a necessary but not a sufficient condition for developing further comprehension skills (Block & Pressley, 2002; Sweet & Snow, 2003).

Sustainability of effective professional learning community

There is a second meaning for sustainability. We now need to know which properties of teaching practices in schools enable success to be sustained with new cohorts of students and new groups of teachers joining schools (Timperley, 2003). Although effective practices may be able to be identified, this is an additional challenge. Sustaining high quality intervention, it now seems, is dependent on the degree to which a professional learning community is able to develop (Coburn, 2003; Toole & Seashore, 2002). Such a community can effectively change teacher beliefs and practices (Annan, Lai, & Robinson, 2003; Hawley & Valli, 1999; Timperley & Robinson, 2001).

Several critical features of a collaboration between teachers and researchers are predicted to contribute to such a community developing (Coburn, 2003; Toole & Seashore, 2002; Robinson & Lai, 2006). One is the need for the community’s shared ideas, beliefs and goals to be theoretically rich. This shared knowledge is about the target domain (in this case, comprehension); but it also entails detailed understanding of the nature of teaching and learning related to that domain (Coburn, 2003). Yet a further area of belief that has emerged as very significant in the achievement of linguistically and culturally diverse students in general, and indigenous and minority children in particular, is the expectations that teachers have about children and their learning (Bishop, 2004; Delpit, 2003; Timperley, 2003).

Being theoretically rich requires consideration not only of researchers’ theories, but also of practitioners’ theories, and of adjudication between them. Robinson & Lai (2006) provide a framework by which different theories can be negotiated, using four standards of theory evaluation. These standards are accuracy (empirical claims about practice are well founded in evidence); effectiveness (theories meet the goals and values of those who hold them); coherence (competing theories from outside perspectives are considered); and improvability (theories and solutions can be adapted to meet changing needs, or to incorporate new goals, values and contextual constraints).

This means that a second feature of an effective learning community, already identified above, is that their goals and practices for an intervention are based on evidence. That evidence should draw on close descriptions of children’s learning as well as descriptions of patterns of teaching. Systematic data on both learning and teaching would need to be collected and analysed together. This assessment data would need to be broad based, in order to understand the children’s patterns of strengths and weaknesses, to provide a basis for informed decisions about teaching, and to clarify and test hypotheses about how to develop effective and sustainable practices (McNaughton, Phillips & MacDonald, 2003). This means that the evidence needs to include information about instruction and teaching practices.

However, what is also crucial is the validity of the inferences drawn, or claims made, about that evidence (Robinson & Lai, 2006). The case reported in Buly & Valencia (2002), for example, shows how inappropriate inferences drawn from the data can result in interventions that are mismatched to students’ learning needs. Robinson & Lai (2006) suggest that all inferences be treated as competing theories and evaluated.

So a further required feature is an analytic attitude to the collection and use of evidence. One part of this is that a research framework needs to be designed to show whether and how planned interventions do in fact impact on teaching and learning, enabling the community to know how effective interventions are in meeting its goals. The research framework adopted by the community needs therefore to be staged so that the effect of interventions can be determined. The design part of this is by no means simple, especially when considered in the context of recent debates about what counts as appropriate research evidence (McCall & Green, 2004; McNaughton & MacDonald, 2004).

Another part of the analytic attitude is critical reflection on practice, rather than a comfortable collaboration in which ideas are simply shared (Annan, Lai & Robinson, 2003; Ball & Cohen, 1999; Toole & Seashore, 2002). Recent New Zealand research indicates that collaborations which incorporate critical reflection have been linked to improved student achievement (Phillips et al., 2004; Timperley, 2003) and to changed teacher perceptions (Timperley & Robinson, 2001).

A final feature is that the researchers’ and teachers’ ideas and practices need to be culturally located. We mean by this that the ideas and practices that are developed and tested need to entail an understanding of children’s language and literacy practices, as these reflect children’s local and global cultural identities. Importantly, this means knowing how these practices relate (or do not relate) to classroom practices (New London Group, 1996).

The main research project

This project is a result of a three year research and development partnership between the Ministry of Education Schooling Improvement Initiative AUSAD, the seven schools in the Mangere cluster, and the Woolf Fisher Research Centre at the University of Auckland.^[2] The representatives from the seven schools formed a Senior Assessment Team (SAT) to work with researchers, the Ministry of Education and the Initiative leaders on developing an intervention to raise student achievement.

The collaboration involved an innovative approach to research practice partnerships. The purpose was to determine the extent of the challenges for effective teaching of comprehension, and to create better teaching methods to meet those challenges. As part of this, a cluster wide intervention for all teachers teaching classes at Years 4–8 (and in one school, Year 9 also) in the seven schools took place. This required extensive school-based professional development, as well as systematic collection of achievement data and classroom observations within a rigorous research design. The research-based intervention was designed to test both the discrete components of effective teaching in school-wide implementation, and the model developed for a research-school practice partnership.

Embedded in the overall programme was a Samoan bilingual study. This involved two of the schools, with 177 Samoan children in six classes in Years 4–8. While the overall project’s main purpose was to enhance the comprehension achievement of all children, the Samoan bilingual study was an attempt to address how Samoan students and Samoan teachers learn and teach comprehension, given that they can speak and understand two languages (Samoan and English).

Research questions

This study: aims and research questions

This study aimed to raise the achievement in reading comprehension of students in seven Mangere schools, through a planned and sequenced research based collaboration. The study addresses several areas of strategic importance to New Zealand, as noted above.

The study also addresses specific theoretical questions. These are to do with the development of reading comprehension; effective instruction for reading comprehension; the development and role of professional learning communities; the role of (contextualised) evidence in planned interventions; and the nature of effective research collaborations with schools. The specific research questions were:

• Can a research-practice collaboration with seven decile 1 schools develop cluster-wide and school based professional learning communities that are able to critically analyse and problem solve issues of instructional effectiveness, thereby developing more effective instruction that has a powerful educationally significant impact on Mäori and Pasifika children’s comprehension at Years 4–9?
• Can a set of effective instructional activities be identified that are able to be used by teachers to enhance the teaching of comprehension for Mäori and Pasifika children in Years 5–8 in decile 1 schools?

A general hypothesis derived from these areas is that:

instructional approaches to reading comprehension present in the cluster of schools could be fine tuned to be more effective in enhancing achievement through a research practice collaboration, and the development of professional learning communities, using contextualised evidence of teaching and learning.

The research base for each of these areas is outlined in the following sections.

Samoan bilingual study

• The general aim of the Samoan bilingual study was to test the more general assumption that a major reason for lower than expected achievement for Samoan students on comprehension tests in schools was less than effective teaching.

What this report covers

This report describes the results of the research and development programme in action, as researchers and practitioners developed communities to meet the challenge of building more effective instruction for reading comprehension in linguistically and culturally diverse urban schools. The design methodology and frameworks for the interventions are described in Chapter 2. Chapter 3 describes the results of these interventions for the overall three year research and development partnership between schools and researchers. Results for the study of students in Samoan bilingual classes follows in Chapter 4. In the final chapter, results are summarised and discussed.

2. Methods

The overall partnership involved schools in the Ministry of Education Mangere Analysis and Use of Student Achievement Data (AUSAD) school improvement initiative, the initiative leaders, the Woolf Fisher Research Centre (University of Auckland) and Ministry of Education representatives.

Main study participants

Schools

The study involved seven decile 1 Mangere schools. Two of these schools are contributing schools (Year 1–Year 6); three are full primary schools (Year 1–Year 8); one is an intermediate school (Year 7–Year 8); and one is a middle school (Year 7–Year 9). The schools ranged in size from 292 students to 593 students.

Students

In the following study, we report on several overlapping groups of students. The first group consists of all the students present at the beginning of the three year study (Baseline sample). The second consists of three cohorts of students, initially from Year 4, Year 5, and Year 6, who were followed longitudinally for three years. The third group consists of all students who were present at the beginning and at the end of each year.

Overall baseline samples
Baseline data (February 2003) were collected from 1216 students in six of the schools (one school who joined the partnership was unable to participate in the first round of data collection) at the following levels: Year 4 (mean age 8 years, n=205); Year 5 (mean age 9 years, n=208); Year 6 (mean age 10 years, n=265); Year 7 (mean age 11 years, n=267); and Year 8 (mean age 12 years, n=271). The total group consisted of equal proportions of males and females from 14 ethnic groups. Four main groups made up 87 percent of the sample. These groups were Samoan (33%), Mäori (20%), Tongan (19%) and Cook Island (15%). Approximately half the children had a home language other than English.

Longitudinal cohorts
Several cohorts of students were followed longitudinally from Time 1 to Time 6; these were those students who were present at all 6 time points, a total of 238 students. There were three cohorts. Cohort 1 (n=114), those students who were Year at Time 1; Cohort 2 (n=56), those students who were Year 5 at Time 1; and Cohort 3 (n=68), those students who were Year 6 at Time 1. These students were a subset of the students included in the baseline sample.

Overall group year by year
A third group of students were those present at the beginning and end of each year. In Year 1 (2003), there were n=1216; in Year 2 (2004), there were n=1683; and in Year 3, there were n= 1619. All of the students who were in the longitudinal cohort group were part of these groups, but these groups also included students who were present for only a single year, including Year 7 and Year 8 students, new Year 4 students (in the second and third year), new students arriving at the school and staying at least a year, and students who were present for a year only.

Teachers

Around 70 teachers were involved in each year of the project, including literacy leaders. Characteristics of the teachers varied somewhat from year to year, but in general, around twothirds had five or more years of experience, and 10 percent were beginning teachers. A total of 11 percent were in bilingual classes (including Samoan, Tongan and Mäori bilingual classes). In the second year, 25 of the teachers (a third) were Pasifika or Mäori.

Samoan bilingual study participants

Students

Samoan bilingual baseline samples
Within the overall study, Year 4–Year 8 students from six Samoan bilingual classrooms were involved. There were between 24 and 30 students enrolled in each of the classes across two years A cross section at baseline of students at different school years, from Year 4 (aged 8) to Year 8 (aged 13), was assessed at the beginning of the school year in 2003. This cross-sectional group provided a baseline against which cohorts in the classrooms of the teachers receiving the professional development could be compared. The six Samoan bilingual classes operated in two schools (School A and School B) that were involved in the larger project. School A had three bilingual classrooms: a Year 7 classroom (12-year-olds), a Year 7/8 composite classroom (12- and 13-year-olds), and a Year 8 classroom (13-year-olds). These classrooms were in close proximity to each other, being located in one school building. School B had three bilingual classrooms also: two composite classes: a Year 4/5 (8- and 9-year-olds), and a Year 5/6 (9- and 10-year-olds) were housed in one part of the school, and the other, a Year 7/8 classroom, was housed in another part.

Samoan bilingual longitudinal cohorts
A second group involved several longitudinal cohorts and was based on the original baseline students. From each year, students who were continuously present at each of four time points— the beginning (assessed in February) and end (assessed in November) of both the 2003 and 2004 school years—are identified and described. Cohorts at each year level, from Year 4 through to Year 8, were repeatedly measured over two years. In the second year, the Year 8 cohort moved on to secondary schools. Given the large movement into and out of the classrooms, this meant there were 140 students represented in the baseline group, but between 10 and 35 students at each age level in the longitudinal cohorts.

Samoan mainstream samples
For comparison purposes, Samoan students in mainstream classes (n = 62 classes) from all seven schools involved in the overall project were identified. These students came from the same communities, but because there were only 67 Samoan students across Time 1 and Time 4 in School A and School B, it was decided to include all Samoan students in mainstream classrooms in the wider study for comparisons. The number of mainstream students across two years ranged from 345 to 456. Two samples of students were also used. Again, a cross-sectional baseline of students from different school years, from Year 4 (aged 8) to Year 8 (aged 12) was assessed at the beginning of the school year 2003. In addition, longitudinal cohorts were identified. From each year, students who were continuously present at each of four time points—the beginning (assessed in February) and end (assessed in November) of both the 2003 and 2004 school years— were identified and described. Cohorts at each year level, from Year 4 through to Year 8, were repeatedly measured over two years. In the second year, the Year 8 cohort moved on to secondary schools. With the larger movement into and out of the classrooms, this meant that there were 345 students at Time 1 represented in the baseline group (and at subsequent times, the number of mainstream students were: 451 at Time 2; 456 at Time 3; and 422 at Time 4); but these were between 24 and 48 students at each age level in the longitudinal cohorts.

Teachers

The six teachers comprised five females (two from school A; three from school B) and one male (school A). Three teachers were originally from Samoa (Teacher 1, Teacher 2 and Teacher 6). The first two teachers had undergone teacher re-training in New Zealand, the other had not. The other three teachers were New Zealand trained (Teacher 3, Teacher 4 and Teacher 5). Half of the teachers were in the 25–35 age range, the other half in the 36–45 range. Teacher qualifications ranged from Diploma in Teaching (with one completing the Bachelor of Education degree), to Bachelor of Education (Teaching), and Bachelor of Teaching and Graduate Diploma in Teaching. Two held English for Students of Other Languages (ESOL) Diplomas.

School reading comprehension lessons

Observations were carried out as part of the intervention (see below), and they provided a general description of the programmes across phases through which the intervention was delivered. Generally the programme was similar across classes and schools, and similar to the general descriptions of the New Zealand teaching in the middle grades (Smith & Elley, 1994; Ministry of Education, 2006). A 10-15 minute whole class activity, which involved mostly introducing and sharing a text, often a narrative text, or reviewing the previous day’s work, was usually followed by a 30-40 minute guided reading session in small groups, led by the teacher using an instructional text. These included text study and analysis (such as study of plot or character in narrative texts and extracting and using information in informational texts), specific group or paired forms of instructional/guided reading (such as ‘reciprocal teaching’), and individual or group project work (such as developing taxonomies of categories introduced in science topics). Typically, the teacher worked with two groups over this time period and held conferences on the run with other groups.

Levels of engagement were generally high, with routines well established and many instances of teacher-student and student-student interactions. The general organisation meant that whole class activities occurred on 3-5 days per week and small group work with one or two groups often daily, so that each group had at least one session but up to three sessions with direct teacher guidance each week. However, the variation in frequency of contact with each group was quite marked between schools. When they were not with the teacher, groups did a range of activities. Some had developed to the point of being able to operate just with peer guidance in reciprocal teaching. In most classrooms, worksheets, sometimes related to the texts, were used; these contained questions about a text and often contained sentence, word or sub word studies.

Design

Rationale for the quasi-experimental design

At the core of the following analyses is a quasi-experimental design from which qualified judgements about possible causal relationships are made. While it has been argued that the gold standard for research into schooling improvement is a full experimental design, preferably involving randomised control and experimental groups over trials (McCall & Green, 2004), a quasi-experimental design was adopted for two major reasons. The first is the inapplicability of a randomised control group design for the particular circumstances of this project. The second is the usefulness of the quasi-experimental design format, given the applied circumstances.

Schools are open and dynamic systems. Day to day events change the properties of teaching and learning and the conditions for teaching and learning effectively. For example, in any one year teachers come and go, principals may change, the formula for funding might be altered, and new curriculum resources can be created. More directly, teachers and schools constantly share ideas, participation in professional conferences and seminars adds to the shared information, and new teachers bring new knowledge and experiences. Such inherent features of schools are compounded when the unit of analysis might be a cluster of schools who deliberately share resources, ideas and practices.

This ‘messiness’ poses tensions in a randomised experimental and control group design. On the one hand, the internal validity need is to control these sources of influence so that unknown effects do not eventuate which may bias or confound the demonstration of experimental effects. On the other hand, if schools are changed to reduce these influences so that, for example, there is no turnover in teaching staff, external validity is severely undermined because these conditions may now not be typical of schools in general.

It is of course possible to conceive of selecting sufficiently large numbers of teachers or schools to randomly assign. Then one assumes that the ‘messiness’ is distributed randomly. If the teachers and the schools in the total set are ‘the same’, then the error variance associated with this messiness is distributed evenly across experimental and control classrooms and schools. Leaving aside the challenges which large numbers of schools pose, a problem here is the assumption that we know what makes teachers and schools similar, and hence are able to be sure about the randomisation process. This is a questionable assumption to make. For example, in the current project the presence of bilingual classrooms in some schools, with different forms of bilingual provision, would create difficulties for random assignment as well as for comparability across teachers, let alone across schools. So what counts as an appropriate control is not necessarily known. There may also not be enough instances of different types of classrooms or schools even to attempt random assignment.

There is another difficulty: that of withholding treatment from the control group of schools. Just about any well resourced, planned intervention is likely to have an effect in education (Hattie, 1999). The act of deliberately withholding treatment, as required in control group designs, raises ethical concerns. Some researcher groups in the United States, also concerned for educational enhancement with schools serving poor and diverse communities, have deliberately adopted alternatives to randomised experimental and control group designs, because of ethical concerns for those settings not gaining access to the intervention (Pogrow, 1998; Taylor et al., 2001). Hattie (1999) proposed that the ethical difficulty could be overcome by comparing different interventions, thus not withholding potential benefits from any group. This is not always a workable solution, for example when the theoretical question is about the effects of a complex multi-component intervention that reformats existing teaching in a curriculum area, such as literacy instruction. Here there is no appropriate alternative intervention other than existing conditions. The American Psychological Association has detailed guidelines for conditions under which withholding treatment is justified. For example, if an intervention is shown to be effective, then it should be implemented in the control group. This route has similarities with the design proposed below.

The most damaging problem, however, is the underlying logic of experimental and control group designs. In these designs, the variation within each group (given the simple case of an experimental and a control group) is conceived as error variance and, when substantially present, is seen as problematic. The alternative design adopted below is based on a view of variability as inherent to human behaviour generally (see Sidman, 1960), and specifically present in applied settings (Risley & Wolf, 1973). It deliberately incorporates variability and the sources of the variability into the design. Questions about the characteristics and sources of variability are central to knowing about effective teaching and learning, and can be explored within the design. Such a design is more appropriate to the circumstances of building effectiveness over a period of time, given that the variability is an important property (Raudenbusch, 2005). Similarly, such designs are useful in the case of planning for sustainability with ongoing partnerships. In fact, longitudinal designs are recommended in which sources of variability are closely monitored and related to achievement data, such as levels of implementation, the establishment of professional learning communities, coherence of programme adherence, and consistency of leadership and programme focus over time (Coburn, 2003). These are all matters of concern in the research reported here.

Repeated measures of children’s achievement were collected in February 2003 (Time 1), November 2003 (Time 2), February 2004 (Time 3), November 2004 (Time 4), February 2005 (Time 5) and November 2005 (Time 6) as part of the quasi-experimental design (Phillips, McNaughton & MacDonald, 2004). One further time (February 2006) was added in the report to add to the evidence of sustained changes. The design uses single case logic within a developmental framework of cross-sectional and longitudinal data. The measures at Time 1 generated a cross section of achievement across year levels (Years 4–8), which provided a baseline forecast of what the expected trajectory of development would be if planned interventions had not occurred (Risley & Wolf, 1973). Successive stages of the intervention could then be compared with the baseline forecast. The first of these planned interventions was the analysis and discussion of data. The second was the development of instructional practices. The third was a phase in which sustainability was promoted. This design, which includes replication across cohorts, provides a high degree of both internal and external validity. The internal validity comes from the in-built testing of treatment effects described further below; the external validity comes from the systematic analysis across schools within the cluster.

The cross sectional baseline was established at Time 1 (February 2003). Students from that initial cross-section were then followed longitudinally and were re-tested at Time 2, 3, 4, 5 and 6, providing repeated measures over three school years. Two sorts of general analyses using repeated measures are possible. Analyses can be conducted within each year. These are essentially pre- and post-measures. But because they are able to be corrected for age through transformation into stanine scores (Elley, 2001), they provide an indicator of the impact of the three phases, against national distributions at similar times of the school year. However, a more robust analysis of relationships with achievement is provided using the repeated measures within the quasiexperimental design format. They show change over repeated intervals.

Good science requires replications (Sidman, 1960). In quasi experimental research, the need to systematically replicate effects and processes is heightened because of the reduced experimental control gained with the design. This need is specifically identified in discussions about alternatives to experimental randomized designs (Borko, 2004; Chatterji, 2005; Raudenbusch, 2005). For example, McCall and Green (2004) argue that in applied developmental contexts, evaluation of programme effects requires a variety of designs, including quasi-experimental, but our knowledge is dependent on systematic across site analyses. Replication across sites can add to our evaluation of programme effects, particularly when it is inappropriate or premature to conduct experimental randomized designs. Such systematic replication is also needed to determine issues of sustainability (Coburn, 2003) and scaling up (McDonald, Keesler, Kauffman, & Schneider, 2006). Coburn argues that the distribution and adoption of an innovation are significant only if its use can be sustained in original and subsequent schools.

In the design used with this cluster of schools, there were in-built replications across age levels and across schools within the quasi-experimental design format. These provide a series of tests of possible causal relationships. However, there are possible competing explanations for the conclusions of the cluster wide results which are difficult to counter with the quasi-experimental design. These are the well known threats to internal validity, two of which are particularly threatening in the design adopted here.

The first is that the immediate historical, cultural and social context for these schools and this particular cluster meant that an unknown combination of factors unique to this cluster and these schools determined the outcomes. Technically, this is partly an issue of ‘ambiguous temporal precedence’, and partly an issue of history and maturation effects (Shadish, Campbell & Cook, 2002). For example, it might be that the nature of students changed in ways that were not captured by the general descriptions of families and students. Or, given that the immediate history included a number of initiatives such as ECPL (Early Childhood Primary Links) and AUSAD (Annan, 1999), the schools were developing more effective ways of teaching anyway.

A second is that the students who are followed longitudinally and were continuously present over several data points were different in achievement terms from those students who were present only in the baseline, and subsequently left. It might be, for example, that the comparison groups contain students who were more transient and had lower achievement scores. Hence over time, as the cohort followed longitudinally is made up of just those students who are continuously and consistently at school, scores rise. Researchers such as Bruno & Isken (1996) report lower levels of achievement for some types of transient students. This is partly an issue of potential selection bias, and partly an issue of attrition (Shaddish et al., 2002). As the projected baseline bases its projections on the assumption that the students at baseline are similar to the cohort students, having a lower projected baseline may result in finding large improvements due to the design of the study, rather than to any real effects.

Other threats to internal validity, such as regression, testing and instrumentation, are handled by other aspects of the methods. For example, all students in all achievement bands were in the cohorts; similarly repeated testing occurred, but with instruments that were designed for the interval of repetition and with alternative forms. But given that there is debate within New Zealand about how influential school-based interventions focused on teaching practices can be in raising achievement (Nash & Prochnow, 2004; Tunmer, Chapman & Prochnow, 2004), and the significance of these counter-arguments to policy directions, it is important to add to the evidence of teacher effectiveness (Alton-Lee, 2003).

There are three ways of adding to the robustness of the design, in addition to the in-built replications, which meet the major threats. The first is to use as a comparison a similar cluster of schools that has not received the intervention. It was possible to identify such a cluster post hoc, and examine the baseline levels in these schools after a year of intervention, to check whether levels in the second cluster had changed significantly. The second cluster was similar in geographical location (neighbouring suburbs), in type (all decile 1 schools), in number of schools (n=7), in number of students (n=1161), in ethnic and gender mix (equal proportions of males and females from over 12 ethnic groups, the major groups being Samoan (37%), Mäori (22%), Cook Island (18%) and Tongan (15%)), in starting levels of achievement, and in prior history of interventions. The second cluster was measured exactly one year after the baseline was established in the first cluster reported here (Lai et al., 2006).

The two baselines are shown in Figures 1 and Figure 2 below. The comparison shows that the second cluster of schools had similar levels of achievement to the first cluster of schools. This was so even though there was a delay of a year and after an intervention had been in place for a year in the first cluster of schools, and, as we report below, achievement levels had risen in those schools. This comparison adds to the design conclusions by establishing that there was no general impact on similar neighbouring area decile 1 schools operating over the time period of the intervention.

Figure 1 Baseline (at Time 1) student achievement for Cluster 1 by year level

Figure 2 Baseline student achievement for Cluster 2 by year level (a second cluster delayed by a year)

A second way of adding to the robustness of the design is by checking the characteristics of students who are included in the cross-sectional analysis, but not included in the longitudinal analysis, because they were not present in subsequent repeated measures. This is done for the first year data by checking the achievement data for those students who were present at two time points (Time 1 and Time 2), versus those students who were present only in the cross-sectional baseline established at Time 1 (Time 1 only). The results of this checking are given in Table 1 for raw scores, and in Table 2 for stanines. The comparisons indicate that in each case, in all but one comparison, the two groups of students were not significantly different.

Table 1 Raw Score Means for Time 1 (Feb 03) by Year Level

	Time 1 only (Feb 03) a			Time 1 pre-post (Feb 03) b
	N	M	SD	N	M	SD	t value		ES
Year 4	34	16.26	7.57	205	16.90	6.82	0.50		0.09
Year 5	34	18.94	9.42	208	21.96	8.13	1.96		0.34
Year 6	30	23.60	9.58	265	24.09	8.87	0.28		0.05
Year 7	33	30.61	10.70	267	30.16	12.26	0.20		0.04
Year 8	34	32.68	13.18	271	37.41	13.11	1.99	*	0.36

^a Note: this group contains those students who sat test at Time 1 only

^b Note: this group contains those students who sat test at both Time 1 and Time 2

Table 2 Stanine Means for Time 1 (Feb 03) by Year Level

	Time 1 only (Feb 03) a			Time 1 pre-post (Feb 03) b
	N	M	SD	N	M	SD	t value		ES
Year 4	34	3.06	1.58	205	3.27	1.32	0.85		0.14
Year 5	34	2.88	1.81	208	3.52	1.52	2.22		0.38
Year 6	30	3.07	1.55	265	3.16	1.56	0.32		0.06
Year 7	33	2.85	1.15	267	2.84	1.31	0.56		0.01
Year 8	34	2.56	1.31	271	2.99	1.46	1.66	*	0.31

^a Note: this group contains those students who sat test at Time 1 only

^b Note: this group contains those students who sat test at both Time 1 and Time 2

These two additional checks add to the robustness of the design by demonstrating that the intervention can not easily be explained as arising from external and general effects on decile 1 schools in these suburbs, or the immediate histories of interventions and resourcing. In addition, they do not support the competing explanation that the students analysed in the longitudinal design were higher achievers anyway, and hence any ‘progress’ is simply their usual levels compared with all other students.

A third way to add to the robustness is provided by analyses which are not part of the design logic. One already mentioned is in the form of pre- and post-testing, using normalized scores. In addition, and as an extension to these analyses, analyses are conducted of the overall student group at each testing time, irrespective of their previous presence or subsequent absence. This allows us to check whether overall achievement levels in schools could be influenced, despite new cohorts of students entering during the three years.

Procedures

Interventions across phases

Phase 1 – Analysis of data, feedback and critical discussion
Current research on learning communities suggests that critical discussion and analysis of data may have an impact on effective practice that is independent of professional development programmes generally (Ball & Cohen, 1999; Timperley, 2003; Toole & Seashore, 2002). Theories about the needs for teaching and learning are developed through critical discussion, which is predicted to strengthen shared understanding and to inform current practices. In the current design we are able to examine the effects of this process prior to the planned second phase of the professional development.

Area-wide data was analysed by the school leaders and researchers in two meetings, then analysed by senior managers and senior teachers with each school using their specific school data. Additional sessions were conducted with support from one of the researchers (Mei Lai). This was the same procedure for the Samoan bilingual classrooms too, with support from another of the research group (Meaola Amituanai-Toloa).

The analysis, feedback and discussion process involved two key steps. Firstly, a close examination of students strengths and weaknesses and of current instruction to understand learning and teaching needs and secondly raising competing theories of the ‘problem’ and evaluating the evidence for these competing theories. This meant using standards of accuracy, coherence and improvability (Robinson & Lai, 2006). This process further ensured that the collaboration was a critical examination of practice and that valid inferences were drawn from the information. The feedback procedures with examples are described fully in Robinson and Lai (2006).

Phase 2 – Targeted professional development

General outline
Targeted professional development which took place in the second year consisted of 10 sessions over two terms, and was designed using research based examples and on known dimensions of effective teaching. The sessions were led by one of the researchers (Stuart McNaughton). Five groups of 10-15 teachers with literacy leaders from different schools attended these half day sessions, which occurred every two weeks from the middle of the first term 2004. The last session was held at the end of the year. The curriculum for the sessions used a mixture of theoretical and research-based ideas, as well as teacher investigation and exemplification from their own classrooms.

Specific sessions
The ten sessions were broken down in the following way. Session one introduced theoretical concepts of comprehension, and related these to the profiles of teaching and learning. A theoretical model was presented, drawing on Sweet and Snow (2003) and developmental analyses such as Whitehurst and Lonegan (2001). A task was set to examine individual classroom profiles of achievement, and how these mirrored or differed from school and cluster patterns. Each session from this point started with group discussion of the task that had been set, and sharing of resources relating to the topic. Session two focused on strategies, in particular the issues of checking for meaning, fixing up threats to meaning, and strategy use in texts. A task to increase the instructional focus on checking and fixing was set. The third session introduced theories and research relating to the role of vocabulary in comprehension. Readings used included Biemiller (1999), Pressley, (2000), and those which identified features of effective teaching of vocabulary. The task for this session was to design a simple study, carried out in the classroom, which looked at building vocabulary through teaching. Sessions four and five identified the significance of the density of instruction and repeated practice, with a particular focus on increasing access to rich texts, including electronic texts (Block & Pressley, 2002). The task mirrored this emphasis, with an analysis by the teacher of the range and types of books available in classrooms, and of engagement by different students. The sixth and seventh sessions introduced concepts of “incorporation” (of cultural and linguistic resources), and building students’ awareness of the requirements of classroom tasks and features of reading comprehension (from McNaughton, 2002). Tasks relating to observing and analysing these features of instructions were set. Sessions eight and nine used transcripts of the video classroom lessons to exemplify patterns of effective teaching in different settings, such as guided reading and shared reading, and developed the practice of examining and critiquing each other’s practices. The ninth session also had some specific topics which the groups had requested, such as the role of homework and teaching and learning in bilingual settings. Session nine also involved planning to create learning circles within schools, where colleagues observed in each other’s classrooms aspects of teaching such as building vocabulary, and discussed what these observations indicated about effectiveness. The final session reviewed these collaborative teaching and learning observations.

Phase 3 – Sustaining the intervention
The third phase was planned by the literacy leaders and researchers jointly. It involved four components. The collection, feedback on and critical discussion of achievement data continued. A second component was the continuation of the learning circles developed in the professional development phase. A third was the development and use of planned inductions into the focus and patterns of teaching and professional learning in the schools. The schools experienced staff turnover of differing degrees, but on average around a third of the staff changed from year to year. This component was designed to maintain and build on the focus with new staff. A fourth component was a teacher led conference. School teams developed action research projects, often with a pre- and post-testing component, to check various aspects of their programmes. The questions for these projects were generated by teams within schools. The researchers helped shape the questions and the processes for answering the questions. Two research meetings took place at each of six schools (the seventh had a change of principal and literacy leader and declined to develop projects, although staff attended the conference). Several of the research topics were about increasing vocabulary, both in language programmes and in instructional reading and writing programmes; others included increasing factual information in narrative writing (to build awareness of use of factual information); teaching of skimming and scanning in the reading programme; instructional strategies to increase the use of complex vocabulary in writing; the effects of using a new assessment tool for writing to inform teaching; redesigning homework to raise literacy levels; and the use of critical thinking programmes. In each case, the projects involved use of formal or informal assessments of student outcomes. A total of 11 projects were presented in power point format at a conference on a Saturday in the fourth term of the school year, attended by 90 percent of the teachers involved. Other professional colleagues, such as literacy advisors, also attended the conference.

Measures

Literacy measures in English

Initially data on reading comprehension were collected, using both the revised Progressive Achievement Tests (PAT) in Reading (reading comprehension section only) (Reid & Elley, 1991), and the Supplementary Tests of Achievement in Reading (STAR) (Elley, 2001). The tests were what schools had decided to use as a group to collect to measure reading comprehension, because they provided a recognised, standardised measure of reading comprehension which could be reliably compared across schools.

The revised PAT in Reading measures both factual and inferential comprehension of prose material in Years 4 to 9. Each prose passage consists of 100-300 words, and is followed by four or five multi-choice options. The prose passages are narrative, expository and descriptive, and different year levels complete different combinations of prose passages. The proportion of factual to inferential items per passage is approximately 50/50 in each year level.

STAR was designed to supplement the assessments that teachers make about students’ ‘close’ reading ability in Years 4 to 9 in New Zealand (Elley, 2001). The rationale behind STAR is the expectation that all students are to learn to read successfully at primary school. In other words, reading successfully at primary school means learning to read appropriate text fluently, independently, and with comprehension. The definition, according to the Literacy Task Force Report (1999), implies that students should also be equipped with reading skills thought to be central to reading programmes at each level of the primary school, although some of them (e.g. critical reading, gathering information) may be given greater emphasis at the upper levels (Elley, 2001).

Analyses over the first year revealed that the correlation between the two tests was .62 (P < .01). In the test manual, Elley (2001) reported correlations between 0.70 and 0.78 for Year 4 to 8 students. This, as Elley suggests, indicates that the tests measure similar but not identical facets of reading comprehension. Subsequently, schools focused on using the STAR data. The outcome data on reading comprehension reported here for the overall project are from across the six time points using STAR (Elley, 2001). These tests were designed for repeated measurement within and across years, are used by schools, and provide a recognised, standardised measure of reading comprehension which can be reliably compared across schools. In addition to these assessments, the schools used other reading measures for both diagnostic and summative purposes, and the baseline results for these are reported elsewhere (McNaughton et al., 2004).

STAR Sub-tests – Years 4–6
In Years 4–6, the STAR test consists of four sub-tests measuring word recognition (decode familiar words through identifying a word from a set of words that describe a familiar picture); sentence comprehension (complete sentences by selecting appropriate words); paragraph comprehension (replace words which have been deleted from the text in a Cloze format); and vocabulary range (find a simile for an underlined word). Only the paragraph comprehension subtest is not multi choice and consists of 20 items. 10 more than the rest of the sub-tests. In Years 7–8, students complete two more sub-tests in addition to the four sub-tests described above; the language of advertising (identify emotive words from a series of sentences); and reading different genres or styles of writing (select phrases in paragraphs of different genres which best fits the purpose and style of the writer). In Years 7–8, there are 12 items per sub-test, except for paragraph comprehension, which consists of 20 items. Both tests have high reliability and validity (Elley, 2001; Reid & Elley, 1991).

Sub-test 1: Word recognition
Word recognition assesses how well students can “decode words that are familiar in their spoken vocabulary” (e.g., umbrella, dinosaur, cemetery…). The test measures word recognition in the form of decoding of familiar words, through identification of a word from a set of words that describe a familiar picture. Ten pictures that are assumed to be familiar to students are in the subtest. Each picture has four words alongside it, one of which is the correct one. Students are asked to select the correct word that matches the picture. The words tested in STAR Test 4–6 are taken from Levels 6–8 of the NZCER Noun Frequency List (Elley & Croft, 1989), and are thus well within the range of most pupils’ spoken vocabulary. Evidence shows that, for the majority of pupils in the upper levels of the primary school in New Zealand, word recognition is a skill that has been well mastered, but many schools have a few pupils who will struggle with this task.

Sub-test 2: Sentence comprehension
Sentence comprehension assesses how well students can read for meaning. The prerequisite for this sub-test is that students are able to read a range of very short texts (sentences) well enough to complete them with an appropriate word. Students are to complete the 10 sentences by choosing, from four words, the word that best suits the sentence. This test assesses decoding skills, and the ability to use a range of sources to gain meaning. To some extent, it also reflects students’ mastery of the concepts of print, their vocabulary, and their ability to predict.

Sub-test 3: Paragraph comprehension
Paragraph comprehension assesses students’ reading comprehension by requiring them to replace words which have been deleted from the text (Cloze format). Using the context of the text as cues to meaning, students can find it easier to replace the missing words, given that they can comprehend the text. The sub-test shows how well pupils can apply the skills tested in sub-test 2 to longer texts, when more linguistic and knowledge cues can be called on from previous sentences. Unlike sub-tests 1 and 2, this test is not multi-choice. It does, however, consist of 20 items, 10 more than the other sub-tests. The students are required to fill in 20 blanks in three short paragraphs of prose (paragraph 1 = 6; 2 = 7; 3 = 7), using the context of the surrounding text as cues for meaning to assess skills.

Sub-test 4: Vocabulary range
The development of a good reading vocabulary is the main focus of this test, because it measures students’ knowledge of word meanings in context. Ten complete sentences are listed. One word in each sentence is in bold print and underlined. The students are required to circle one word from the four words under the sentence that means the same, or nearly the same, and is therefore close in meaning to the bold underlined word. The words included in this test are all taken from the New Zealand Oxford Primary School Dictionary of 30,000 words, and were selected after extensive trials had shown them to be of appropriate difficulty for the students in the relevant year groups.

STAR sub-tests – Years 7–9
In Years 7–9, students complete two more sub-tests in addition to the four sub-tests described above. These sub-tests are on the language of advertising (identify emotive words from a series of sentences) and reading different genres or styles of writing (select phrases in paragraphs of different genres which best fit the purpose and style of the writer). In Years 7–9, there are 12 items per sub-test, except for paragraph comprehension, which consists of 20 items. Both tests have high reliability and validity (Elley, 2001; Reid & Elley, 1991).

Sub-test 5: The language of advertising
This sub-test requires the students to identify emotive words which are typically used by advertisers when trying to attract consumers to buy. Students read a series of sentences and circle the one word that sounds appealing, but provides no information, e.g. “fabulous”, “gotta-go”, “cosy”. This skill is part of learning to be a critical reader, and is stressed in “English in the New Zealand Curriculum” for Years 7 and 8 (or Curriculum Levels 4 and 5).

Sub-test 6: Reading different genres or styles of writing
Pupils in the senior levels of primary school are expected to read with understanding various styles or genres of writing, both formal and informal. To assess this skill, pupils are given some paragraphs which represent a range of genres, and at particular points in each paragraph, they are asked to select the phrases which best fit the style and purpose of the writer. The genres represented include traditional fairy tales, business letters, informal letters, recipes, and computer manuals.

Reliability of STAR assessment

At the beginning of 2003, the Senior Assessment Team (SAT) developed an intra-school standardised process of administering the test and moderating the accuracy of teacher scoring. This involved standardising the week and time (morning) of testing, and creating a system of randomly checking a sample of teachers’ marking for accuracy of scoring. Accuracy of scoring was further checked by the data entry team from Woolf Fisher Research Centre during data entry and during analysis. The STAR and PAT were administered as part of schools’ normal assessment cycle at the beginning of the school year, and thereafter for STAR at the end of each year also (using the parallel form). At Time 1 (February 2003), a number of additional analyses took place. These involved analysing student scores on factual and inferential questions in the PAT, and analysing sub-test scores in STAR. In STAR, this also included qualitatively coding the types of errors students made on the Cloze passage, according to the types of errors reported in the STAR manual (Elley, 2001). Four raters were trained to code errors. These raters subsequently discussed how to code the errors, and collectively rated a sample of tests to determine reliability of coding. The coding was subsequently checked and inter-observer agreement on 10 percent of students’ sub-tests (across ages) was 90.5 percent.

Observations

Evidence about current classroom instruction came from systematic classroom observations carried out by the researchers. These were designed to provide a sample of how features of teaching and learning might map on to the achievement data. Our argument was that a fully effective teaching analysis needed to examine classroom instruction, otherwise assumptions about what was or was not being taught would be unchecked.

Observations at baseline (first year)

Phase 1 observations were carried out in 16 classrooms in seven schools from Years 4–5 through to Year 8 (including one bilingual Samoan classroom), selected to represent all schools and age levels within schools. In the second term, classroom instruction was observed for 40–60 minutes during the usually scheduled core reading session within which the teaching and learning of reading comprehension occurred. Class sizes generally ranged from 21–26. A combination of diary and audio recording of specific activities occurred. Discussions with teachers also provided an important level of professional reflection.

Observations at Time 3 and Time 4 (second year)

At the beginning of the second year (February 2004), 15 classrooms (including the six Samoan bilingual classrooms in two schools) were systematically observed; again, classrooms were nominated and selected to represent age levels. Video recordings were made of teacher-student and text interactions during the usually scheduled core reading session for comprehension. All of the whole group and at least one small group activity was recorded. The amount of time allocated to reading on a typical day ranged from 30 to 60 minutes (average 38.6 minutes, SD = 11.9 minutes). The observations were repeated at the end of the year (November) with 12 teachers, 9 of whom were videotaped at the beginning of the year, and were still teaching and available for a second videotaping. Discussions with the teacher were held again, and the observers also made notes on features of the general classroom programme.

Coding and reliability of observations

Systematic classroom observations of core literacy activities were coded, using the coding handbook developed by the research team.

Exchanges

Exchanges were the basic unit of analysis of the videotapes. An exchange was initiated by an utterance followed by a set of interactions on the same topic, involving comments, questions, directions, explanations or feedback between teacher/child or child/group. A minimal exchange would be one turn by a teacher or student. A change in topic during interactions, a return to text reading, or a new activity signalled the end of an exchange. Each exchange was coded as a specific type, using the following definitions:

• text-related;
• non-text-related;
• vocabulary elaboration;
• extended talk;
• checking and evaluating;
• incorporation;
• awareness;
• feedback.

Text-related exchanges and non-text-related exchanges (TR and NTR)

Text-related exchanges were exchanges that dealt specifically with the text at hand. Any comment or question related to the text came under this category. Non-text-related exchanges, on the other hand, were exchanges that were not related to the text, but were employed nevertheless to prompt students to answer comprehension questions that were otherwise difficult to respond to. An example of text-related-exchange:

	T:	What is the moonlight? If you know anything, don’t shout “I know” O le fesili la [The question is] “What is the moonlight? The next question will go to M. What is the moonlight. Ua na’o lo’u fia moe a… [I just wanna go sleep …]
	C:	(No response but child 1 interrupted)
	C:	A big circle in the air (laughter)
	T:	A big circle in the air is a bit …good imaginative …
	C:	Circle is shining… (one child says) the light’s shining down (another child says) shining (another child says) the light’s far away and it shines like a little but it’s real and it’s shining.
	T:	E sau mai fea le moonlight? O le a le fa’asamoa o le moonlight? [Where does the moonlight originate from? What is Samoan for ‘moonlight’?]
	C:	Masina [Moon]
	T”	O le masina [The moon]. O le masina e susulu i luga i o i le night time [The moon shines up there at]. O fea le mea e sau ai le masina? [Where does the moon come from?]
	C:	I le lagi [From the sky]
	T:	I fea? [Where?]
	C:	I le space [In space]
	T:	I le space [In space]. Manaia [Nice].
	C:	Mai Pluto [From Pluto]. E shine le la i le moon ae reflect mai i [The sun shines on the moon and it reflects it].
	T:	E scientific a? [It’s scientific isn’t it?] Lelei tele R [R is good].

(Teacher 4 Time 3)

Vocabulary elaboration

Exchanges that elaborated vocabulary in the text were coded into three (non-mutually exclusive) types. These were questions seeking elaboration (VEQ), elaboration of vocabulary by the teacher (VECT) and elaboration by looking up meanings in the dictionary (VECD). An example of vocabulary elaboration question and teacher comment:

Text word: catch

	T:	(reading) “They were always good friends and always share their catch” What do you mean by that? “Share their own catch”, can you say it in other words?
	C:	Divide them and halving each other
	T:	Divide them – good
	C:	When they get fish they share it
	T:	When they get fish they always share their …
	C:	Catch

And an example of teacher comment:

	T:	Or half the fish … What does it mean by the word “catch”? This time how is the word “catch” used in the story? We know that catch is a … what form of word?
	C:	A verb
	T:	A verb. Always we use ‘catch’ as a verb but this time it is used as a…?
	C:	(two students answer together). A noun
	T:	A noun. Very good. How do you describe it in your own words? What does it mean by catch this time? Because you say we use catch as a verb but this time it is used as a noun – so what are they?
	C:	Fish
	T:	Fish they ….
	C:	Catch
	C:	Catch and eats
	T:	Very good. The catch this time are the …
	All:	The fish
	T:	Refer to the  …
	All:	Fish

(both examples above from Teacher 1 Time 3)

Extended talk

Extended talk meant conversations sustained over several turns on a topic that allowed the teacher or child to develop further features of place, time, theme and concept. Exchanges that were limited to a synonym or brief comment on a word or phrase were not coded as extended talk. There were two types: extended talk by teacher (ETT), and extended talk by child or students (ETC). An example of extended talk:

	T:	Carry on (to one of them to read). “Finally mum …”
	All:	(some read together)
	T:	What feeling is showed or revealed in that paragraph?
	C13:	Anger
	T:	Do you think it’s anger? Why? Where does it say in the book?
	C14:	(read the sentence in the text) T: So who is angry here?
	All:	Mum
	T:	Why is she angry?
	C15:	Because she wouldn’t eat the porridge.
	T:	Because who wouldn’t eat the porridge?
	All:	Anna
	T:	What about Anna, how does she feel? What is the feeling revealed by the word “yukky”?
	C16:	Disgusted.
	T:	Disgusted. Very good word. Say that word.
	All:	Disgusted (again) Disgusted.
	T:	What does disgusted mean?
	C17:	Something that is gross.
	T:	Something that is?
	C17:	Gross. Not nice.
	T:	Not nice – yes. What about the word “dislike”, do you think it’s similar to disgusted?
	All:	Yes

(Teacher 6 Time 3)

Checking and evaluating

These are exchanges in which there is some explicit reference to checking and evaluating evidence. The reference could involve questions, directions, prompts, feedback or comments. It can be initiated by the teacher, a child or a group and involve the teacher, the child or a group. Three sub-categories were noted. Teacher checking (TC) is where the teacher makes reference to students to check the correctness of their responses by going back to the text to search for confirmations. Child checking (CC) is where the child checks the validity of the responses by verbalising what is found after the teacher prompts. The final subcategory involves the teacher and child checking (TCC) for the evidence together. An example of checking and evaluating:

	All:	(reading the question on the board). What is the ceremony called in Niue?
	C:	The hair cutting ceremony
	T:	Where do you get that answer
	C:	From the heading
	T:	Good, do you agree with that?
	All:	Yes

(Teacher 6 Time 3)

Incorporation

Incorporation means exchanges in which students’ knowledge, skills and other expertise are brought into an activity. It is a deliberate attempt by the teacher to make direct links between the text being read and the experiences of the student, through frequency of overt connection with topic events, and concepts that are familiar to the child—for example, when a child is prompted to talk about feelings, references to past events or activities, or being involved in the story. Furthermore, incorporation is also when the language of the child is incorporated. An example of incorporation:

	T:	The “Bush Supermarket” Do we have a bush supermarket in Mangere?
	C:	Yeah
	T:	Where?
	C:	Yeah, you know … (child was about to show where it is)
	C:	No. That’s not the bush
	T:	Do we have a bush supermarket in Mangere?
	All:	No (and others shook heads)
	T:	So why do you think it’s called the “Bush Supermarket”?
	C:	Cos birds go there
	T:	All birds go where? … to the supermarket?
	All:	To the bush
	T:	Alright. Good that sounds like a good idea.

(Teacher 3 Time 3)

Awareness

Awareness means exchanges that focus on the child’s awareness through teacher comments, questions, explanations or feedback which explicitly draws attention to the relevance of the child’s knowledge or reflection on knowledge, to the rules of participating, and to the purpose or ways of participating. The two types were awareness of strategy (AS), such as clarifying, predicting and summarising, and awareness of any other aspect of the task or child’s expertise (AVE). An example of awareness (AS):

	T:	First thing we need to do before we move on to the story is we need to know what the four strategies are for reciprocal reading.
	C:	Clarifying?
	T:	Clarifying
	C:	Questioning?
	T:	Questioning
	C:	Summarising
	T:	Summarising. Manaia. (Nice)

(Teacher 4 Time 3)

Feedback

Feedback is defined as teacher responses reliant on a student action or verbal contribution. Of the two subcategories, high feedback (FH) is any feedback that clarifies, elaborates on, or adds to the student’s statement or response. It includes teacher correction of a student’s incorrect answer or statement, or teacher response to a student’s utterance with a question. Low feedback (FL) is nondescriptive and provides no extra information for the student, other than correctness. An example of high feedback:

	T:	Tells us what the main points. Tells us what is gonna happen. That’s why when you see the movies … Who’s been to the movies? Ok hands down. They either have those brief (I’ve forgotten the name now) you know there’s a movie that’s coming out and they show you just a little bit about the movie. Sometimes it’s like “Oh yeah. I wanna see that!” Don’t know if you call it commercials. Maybe that’s your job tonight. Go home and find out what it’s called. Now don’t you just … I think it’s ‘trailer’. Just a brief description to show you what the movie is gonna be about. It’s just like the book. Just a brief description of what the story is about. (Bell went untimely – not fire but photographs).
		As you’re reading there are three things I’d like you to look at. Just go through and have a look for these three things I’m gonna … (writes the words on the board) “phrase”, “metaphor”, “simile” before we even look in our dictionaries to see what they mean, what is a phrase? Everyone’s ready to look but I want you to think. Can you predict? What is a phrase?

(Teacher 5 Time 3)

Development of the definitions and codes took place over several sessions in which three transcripts were randomly selected and coded by different members of the research team, until there was close to 100 percent agreement on the basic unit and on types of exchanges (all members of the team had to concur on presence or absence for an agreement to be scored). Subsequently, a further transcript was coded by each member independently, and the interobserver agreement calculated by the presence and absence of types of exchanges. The levels of inter-observer agreement ranged from 86 percent for the awareness categories to 100 percent on the text-related or non-text-related categories. One member of the team coded all observational data (including the Samoan bilingual data) used for the analyses presented here.

Data analysis

Reading comprehension achievement

The data were analysed in terms of patterns of achievement, using repeated measures and gain scores, as well as raw score shifts. In addition to the use of raw scores on subtests and stanines, the analysis of achievement patterns for the STAR assessments used distribution bands. The manual (Elley, 2001) groups stanines into 5 bands; stanine 9 (outstanding, 4 percent of students); stanine 7–8 (above average, 19 percent of students); stanine 4–6 (average, 54 percent of students); stanine 2–3 (below average, 19 percent of students; stanine 1 (low, 4 percent of students). These bands were used to judge educational significance. SPSS and Excel programmes were used to create a database where data from all testing periods could be recorded and analysed.

Testing the effectiveness of the interventions proceeded in a number of steps, using tests of statistical and educational significance. We compared means and distributions for the children in terms of both pre- and post-testing, and in terms of comparisons using the projected baselines. These comparisons use standard statistical procedures, such as t-tests. Two further steps determined the educational significance of the interventions. The first was based on an assessment of effect size of the educational intervention. Effect size (ES) is a name given to a family of indices that measure the magnitude of a treatment effect. Hattie (1999) describes a 1.0 effect size as an increase of one standard deviation, which usually represents advancing student achievement by about one year. To measure the magnitude of a treatment of effect in this study, Effect Size Cohen’s D was employed (Cohen, 1988).

A second means of judging educational significance uses the idea of risk. This represents the relative increase or decrease in the risk associated with literacy instruction in the decile 1 schools, compared with nationally expected outcomes (see Phillips, et al., 2001). In the present case, we analysed the relative risk of not being at national expectations at the beginning and the end of the three year study, using both longitudinal and total groups. A risk ratio represents the relative increase or decrease in the probability of a given outcome when one rather than another condition is obtained. In this case, the given outcome is expected progress in a range of literacy measures, as determined by the stanines. Following this analysis, Chi square comparisons of obtained frequencies of students’ in stanine bands were compared with expected frequencies in these stanine bands, using X² one sample case comparisons (Siegel, 1956).

Instruction

Two levels of analysis were carried out from the classroom observations. The first level was analysing changes in overall achievement and the text components from the pre- and post-testing, with changes in the categories of teacher instruction from the observations in the middle of the first year, and at the beginning and end of the second year. A second level involved both quantitative and qualitative analyses of two selected case studies of teachers from the second year. This enabled us to go beyond the frequency counts, using the transcript data and classroom records, to better understand the relationships with achievement and variability in achievement across classrooms.

Samoan bilingual study: three approaches to analysing teacher effectiveness

Three approaches to evaluating the effectiveness of teaching in the Samoan bilingual classrooms were developed. The first approach used the quasi-experimental design to demonstrate effects of the teaching, compared with baseline forecasts (Phillips et al., 2004).

The second approach to analysing effectiveness used the total group of students in the classrooms in the two different years. This approach examined teachers’ teaching with new combinations of students entering the classrooms in the second year. Essentially, this covers the dimension of effectiveness that has to do with sustainability (Coburn, 2003).

The third approach examined the outcomes of teaching Samoan students in bilingual classrooms, compared with teaching Samoan students in mainstream classrooms. This is essentially a comparison using the instructional context as the basis of comparison. All the teachers went through the professional development (although there were some variations in attendance, as noted below).

 

3. Results

Phase 1: baseline profile

The baseline (Time 1) results are presented in four sections: These focus on the general profile of reading comprehension, a content analysis of PAT and STAR, and gender and ethnic group breakdowns.

Achievement profile: general profile of reading comprehension

The stanine distributions of both tests, STAR and PAT, indicate that the average student experienced difficulty on these measures of reading comprehension. Figure 3 shows the stanine distribution in both tests across all year levels. The average student in both tests scored in the “below average” (stanine 2–3) band of achievement. For both PAT and STAR, the mean stanine was 3.1 (in the below average band), with over 60 percent of students scoring in the “low” (stanine 1) or “below average” (stanines 2–3) bands, compared with an expected percentage of 23 percent. Just over one third of students were in the “average” band (stanines 4–6), and less than 5 percent were in the “above average” or “superior” band (whereas 23 percent of students would usually be expected to be in these bands).

Figure 3 PAT and STAR stanine distribution across all year levels

Across the year levels, as Figure 3 shows, the pattern was the same in both tests, with the median in every year level at stanine 3. This is displayed graphically in Figures 4 and 5, in box and whisker plots. The range of achievement was large, from stanine 1 to 9 in the PAT and 1 to 8 in STAR. The relatively flat line in stanines across year levels indicates that under initial instructional conditions, children made about a year’s gain for each year at school, remaining at two stanines below national average across years.

Figure 4 Stanine distribution for PAT in Years 4–8

Figure 5 Stanine distribution for STAR in Years 4–8

PAT content analysis

The pattern of mean scores on factual and inferential questions on the PAT was very similar across all year levels (see Table 3). The maximum raw score for both factual items and inferential items was approximately 20 (Reid & Elley, 1991). This pattern suggests that students at different year levels experienced similar difficulties in answering factual and inferential items. This pattern is to be expected, as Elley (personal communication, October 22, 2004) reports that the pattern of achievement for the two types of questions is expected to be very similar in a large sample, although factual questions are assumed to be easier than inferential ones. (In the present study, there was a significant correlation between factual and inferential items (r = 0.61, p < .01)). However, in the present study, results for inferential questions were higher than for factual ones in some year levels, suggesting that the students in the sample have been instructed or have learned in some systematically different way from the majority mainstream approach (Elley, 2005).

Table 3 Means (and Standard Deviations) of Factual and Inferential Questions Across the Year Levels

Year level	N	Factual questions	Inferential questions
4	245	4.93 (2.80)	4.18 (2.36)
5	298	5.76 3.32	4.64 (2.76)
6	319	5.75 (3.04)	5.85 (2.76)
7	390	6.14 (2.82)	6.31 (2.67)
8	471	6.41 (3.10)	6.30 (3.12)
9	144	6.57 (3.30)	5.92 (2.51)
Total	1867	5.96 (3.08)	5.65 (2.90)

Content analysis on the STAR sub-tests

Analysis of the STAR sub-tests revealed consistent patterns across the sub-tests at each year level. Figures 6 and 7 show the average percentage obtained in each sub-test. At every year level, students scored highest on sub-test 1 (Word recognition) and lowest on sub-test 3 (paragraph comprehension), indicating that students in all year levels experienced more success in decoding words than in comprehending a paragraph. All the sub-tests of STAR were significantly correlated (p < .01). The sub-test results can be compared with percentages generated from national norms (see Table 4). A similar pattern of results can be seen in the percentages from the national norms.

Figure 6 Average percentages obtained in each sub-test (STAR) for Years 4–6

 

 

Figure 7 Average percentage obtained in each subtest (STAR) for Years 7–8

Table 4 Percentage Scores Based on National Norms for Each Sub-test of STAR for Years 4–8

Year Level	Subtest 1	Subtest 2	Subtest 3	Subtest 4	Subtest 5	Subtest 6
4	83	61	50.5	50
5	87	70	60	58
6	91	79	69.5	67
7	75	58.3	56	60	61.7	59.2
8	83.3	66.7	64.5	69.2	67.5	63.3

A series of paired t tests between sub-tests averaged across years revealed that at both age groupings, the means for sub-test 1 were significantly higher than the means for the other sub-tests (t values > 18.0; p < .001). Sub-test 3 means were significantly lower than those for each of the other sub-tests (t values > 12.0, p< .001). In addition, in Years 4–6, all sub-tests were significantly different from each other. In the older age group (Year 7 and Year 8), sub-tests 2 and 5, 2 and 6, and 5 and 6 were not significantly different (p > .05).

Error analysis of STAR sub-test 3 (paragraph comprehension – Cloze)

Errors on sub-test 3 (paragraph comprehension) were analysed according to the coding in the STAR manual, based on common errors made by students, such as “irregular plurals” and “tense problems”. However, nearly half (46 percent) of all possible errors across all year levels did not fit any of these categories. These errors appeared to be of two new types. One involved mistakes that sometimes made sense in the pre-sentence context, but resulted in nonsense or illogical sentences. The following examples show student responses in italics, and the correct response/s in brackets):

All they did (could) afford was a tiny room in a shoe (cottage/house/shop) in a village by a river.

He grabbed frantically, and felt his head (hand(s)/finger(s)) closing around the branch of a tree.

Suddenly, round a sharp bend in the head (road/path/track), he fell again, missed his self (footing/step) and plunged over the ugly (cliff/rock) face.

There were a few instances of a second type of error, where relatively close synonyms, not listed by the manual, were used:

It browses entirely on tall branches (trees/plants) especially the foliage of mimosa and acacia trees.

Ethnicity and gender

Analyses were also conducted by ethnicity and gender. For the PAT, the pattern of achievement was consistent across the major ethnic groups. The median stanine was 3, with half of the students scoring between stanine 2 and 4. In the STAR test, however, the pattern of scores differed slightly between NZ European students and students from the other ethnic groups. The median for NZ European students was stanine 4; the median for the other ethnic groups was stanine 3. The pattern of scores on STAR and PAT was also different for NZ European students, who in general obtained higher stanines in STAR. There were few gender differences between males and females in both PAT and STAR. The median for both males and females was stanine 3, with half the students falling between stanine 2 and 4.

Classroom instruction profile

At the beginning of the research programme, classroom instruction was viewed as an open problem, the critical aspects of which needed to be understood. Thus, while general principles informed what was to be observed, a predetermined category system was not used, and specific aspects of the resulting profile were developed in situ, in the form of hypotheses. In essence the approach had a relatively open-ended investigative purpose, but was theory driven. Some aspects, such as the needs for vocabulary instruction and the use of checking evidence, became working hypotheses from the first set of observations. Hence some limited quantitative estimates are possible, providing limited comparisons with the later systematic recording and analyses in the second year. The resulting instructional focus can be summarised as falling into four areas.

Vocabulary instruction
Across all classes, teachers were observed to identify and elaborate potentially new or unfamiliar vocabulary in both informational and narrative texts. Examples included topic-related words, such as ‘kelp’, as well as common and uncommon English words, such as ‘mysterious’ and ‘clairvoyant’. There were also instances where words in Mäori or a Pasifika language occurred in texts, such as ‘kete’ (Mäori word for flax kit). Less frequently, phrases and idiomatic or figurative uses of language were identified, such as ‘full of beans’ and ‘the sky boiled’.

Many of the instances of vocabulary interactions (estimated from the records to be around 50 percent) involved referring to and defining technical terms. Terms were used for linguistic categories at sub-word, word, sentence, paragraph and text levels. These exchanges often were associated with the teaching of strategies (see below) and the use of technical terms, such as ‘clarifying’, ‘predicting’, and ‘visualising’.

Estimates from the field notes suggested that 6–7 such instances, where the meaning of vocabulary (including technical terms) was specifically identified or elaborated on, occurred on average across the classrooms. This rate, together with other aspects of the verbatim records, suggested two issues. One was that the rate of these interactions occurring for any individual child may have been lower than what was needed. This hypothesis is suggested by a classroom record in which 25 interactions occurred in the course of a 20 minute session, averaging 1 interaction per student. The focus on any one particular word occurred once, and there was little evidence of repeated opportunities to use or elaborate. When asked, the teachers consistently identified vocabulary limits as constraining children’s comprehension.

The second issue, which became obvious in the observations of strategy teaching, was that meanings were seldom checked in ways that elaborated specific connotations in context. Many of the instances involved discussion of students’ ideas about meanings, often with the teacher accepting student contributions without critical appraisal. Few of the instances involved explicit instruction and modelling of how to check meanings within texts, or via a dictionary or a thesaurus (see below).

These issues were fed back to the lead teachers, who discussed them with their teachers in the form of a specific hypothesis that there was a need to increase the rate of vocabulary acquisition, especially non-technical language, and to do so in ways that gave access to multiple meanings and connotations. Research evidence to support the need to boost vocabulary through teacher guidance in elaborations and feedback (e.g., Biemiller, 2001) was identified during this process. Possible ways that were discussed included increased use of reading to small groups with carefully selected texts which provided variation in genre and topic, with planned rates of exposure to new vocabulary. In addition, language acquisition research, which noted how increased extended talk was associated with new vocabulary and with greater understanding of complex utterances, was introduced (e.g., Hart & Risley, 1995). Other strategies included increasing the actual time spent reading, either in groups or individually, because the records indicated a large amount of teacher discussion outside of texts.

Strategies supported by checking and evaluating threats to meaning
The deliberate reference to, teaching of and use of comprehension strategies was present in all classrooms. Specific use of some form of Reciprocal Teaching (Brown, 1997) was directly observed in four schools, and all schools reported some use of specific strategy instruction.

One issue with strategy instruction was suggested very early on in the observations. This was the limited use of text evidence to detect confusions or threats to meaning, or to check and corroborate meanings, and it was observed generally in strategy instruction. Additionally, it was observed as a limitation with vocabulary instruction (as noted above) and in the examples of incorporation (see below). There were few instances where the children were asked to provide evidence for their analyses, comments or elaborations (such as “How did you know?”).

This limited reference to texts to check understanding was especially noticeable with the use of predicting in the whole class and small group activities in which a text was shared, or introduced for some form of guided reading. In every such activity observed, ideas were generated with high engagement. However, explicit direction to check the evidence, in order to see if what was predicted was in fact supported in upcoming text (at sentence, between sentence and text levels), happened infrequently. There were only nine instances in the verbatim record in 16 hours. Across all classes, in all activities, predictions were often prompted, for word meanings, and for event outcomes and sequences, and without exception were accepted and supported (“Good prediction”, “That was clever”, “Could be”). Most of the dialogue observed was about generating ideas, not checking them, and the teachers’ responses were to accept and reinforce predicting. Similarly, asking for predictions often led to exchanges in which the students tried to figure out what the teacher was thinking about, rather than what evidence the text provided.

The hypothesis fed back to the teachers was that comprehension would be enhanced with more direct and explicit instruction and modelling of checking for evidence—for inferences, for meanings of words, for coherence, and so on—within sentences, between sentences, within a text and even across texts. There is some research evidence that this could be a problem in strategy instruction, leading to formulaic use of strategies in general, and to guessing, rather than to the appropriate use of texts to support inferences, to clarify meanings, to maintain coherence and to predict (Baker, 2002). Similarly, recent research reports have identified groups of children who have fast efficient decoding but low comprehension, and who thus have a high rate of errors termed ‘excessive elaborations’, which are essentially guesses (Dewitz & Dewitz, 2003).

Increased incorporation and awareness
Two complementary processes have been proposed as particularly important in effective teaching with culturally and linguistically diverse students (McNaughton, 2002). One of these is the use of students’ expertise in classroom activities. At one level, this involves capitalising on their event knowledge and interests through instruction, including the selection and matching of texts. At other more complex levels, this involves using familiar language forms, and even types of culturally based forms of teaching and learning. But complementing this process is instruction that increases students’ awareness of the relevance of their skills and knowledge and relationships to the goals and formats of classroom activities.

In each of these classrooms, there were instances where teachers incorporated their students’ event knowledge and language skills, drawing on their social and cultural identities. Examples included selection of texts with familiar sports topics, as well as local cultural events, such as formal ceremonies. More complex instances included close reading of local rap songs. In each of the recorded instances, there was a high degree of engagement and interest signalled by the complexity and appropriateness of students’ comments, and this was verified in incidental discussion with the children. However, the need to use more resources for children from different Pasifika communities and Mäori communities was mentioned by several teachers.

The complementary dimension, building the learner’s awareness in classroom activities, was more problematic. Several examples occurred in the observations of children where they did not know exactly what the task was that they were required to perform. To different degrees, these could be a vocabulary problem, or a grammatical complexity problem, or a problem of knowing what the tasks might fully entail. For example, in one classroom, a small group worked on extracting information on natural and artificial sources of light from a text on ‘Sources of light’. Incidental discussion revealed that each student in the group had little or only limited understanding of what ‘source’ meant. In another class, a group who was given the task ‘Find the word in the story so you can give the appropriate meaning’ were found to be unsure about what ‘appropriate’ definitions were. Associated with this was a relatively low rate of explaining (or checking) purposes.

The feedback with teachers focussed on the hypothesis that students’ learning would be improved if instruction enhanced students’ awareness of classroom goals and formats, and their knowledge and skills in relationship to these. Vehicles for this might include the information provided in contingent feedback, as well the setting of clear and consistent learning intentions (Hattie, 1999).

Increased exposure to texts and planned variation across texts
An instructional dimension common to each of the previous areas is the extent of practice within texts and planned variation in exposure across types of texts. Texts for topic study, and specifically for reading comprehension, were generally available in classrooms, but there were extremes. One classroom was filled with topic related texts of different genres. It was noticeable that the children were very familiar with selecting texts to extend current reading, and took home these texts. In one of the most engaged classrooms, something like a ‘book flood’ (Elley, 1991) operated. There were large numbers of resources for topics and for extended or extra reading. Special attention to the availability of texts for boys was a feature. Homework was used as a vehicle to increase the amount of reading in several classes. But these were exceptional, and teachers often commented on the need for the children to have exposure to a greater range of texts, both within classrooms and at home.

Other issues related to exposure to texts were observed at the level of classroom discourse. A common feature across classrooms was questioning in which the teacher tried to elicit a correct answer, but the interactions proceeded over several turns, and took on the feature of trying to guess what was in the teacher’s mind. In one Year 7/8 class, the teacher held a mini lesson on plurals and tense markers. The questioning about what you put ‘ing’ on went on for six questions before the teacher answered her own question.

The research evidence suggests that instruction for minority students may inadvertently reduce their engagement in cognitively complex tasks, and tasks that are critical to the long term development of reading comprehension (e.g., McNaughton, 2002). The identified risks therefore are around limited practice. The hypothesis developed with the teachers was that in each of the areas of concern, instructional ‘density’ could be increased. Testing this hypothesis would require attention to the general textual resources in classrooms, including electronic and internet based resources, to the link between home and schools, and to discourse features which increased rather than reduced engagement in text reading.

A critical feature of the focus in each area was the assumption that the general instructional approaches, such as guided reading and strategy instruction, would provide a platform for more effective instruction, which could be achieved by fine tuning specific aspects.

The observation records and teacher reflections were used, together with the achievement data, to develop a set of possible directions for increasing instructional effectiveness. In essence, these were emerging hypotheses about how instruction might be formatted to be more effective for reading comprehension. The hypotheses were about more effective instruction.

A major alternative hypothesis to those developed here (but not incompatible with them) is that the children had difficulties comprehending because of limits in their accuracy and fluency of decoding. Apart from small groups within classes who were having special instruction (such as Rainbow Reading), the teachers generally felt that accuracy and fluency was not a problem. They could refer to running records and sources of evidence for this. The pattern of results in the PAT and STAR are also generally consistent with this. The Word Recognition sub-test of the STAR had the highest scores, and some later passages on the PAT were done better than earlier passages. One school had examined the same hypothesis by comparing students on the word recognition PAT with the reading comprehension PAT. The former was consistently higher.

Longitudinal cohort analyses

The following analyses track the achievement of cohorts of students from the beginning to the end of the project (i.e. from Time 1: Term 1, 2003 to Time 6: Term 4, 2005). Analyses were conducted only with the same students who sat all six tests, to avoid any confounding effects from students with differential exposure to the programme. Because of the three-year timeframe with the focus on Years 4–9, this meant that the students available to be tracked were those who began with the project in Year 4 (Cohort 1), Year 5 (Cohort 2) and Year 6 (Cohort 3), who would be in Years 6, 7 and 8 respectively at the end of the project. The analyses also excluded two students who were retained in one year level.^>[3]

The achievement of other students not represented in this analysis can be found in the following section.

Overall gains in achievement across cohorts

There was a statistically significant overall acceleration in achievement from Time 1 to Time 6 of 0.97 stanine. This represents about one year’s progress, in addition to expected national progress over the three year period. Table 5 presents the mean stanine and raw scores for cohorts of students tracked across the three phases of the project.) By the end of the project, the average student scored in the “average band of achievement”. At the beginning of the project, the average student had scored in the “below average” band.

Table 5 Stanine and Raw Score Means by Cohort at Time 1 (Feb 03) and Time 6 (Nov 05)

All cohorts made statistically significant accelerations in achievement across the three years. The effect sizes for the age adjusted scores and raw scores are higher than reported in international studies on schooling improvement initiatives, which report effects of between 0.1 and 0.3^[4] (Annan & Robinson, 2005), although Borman reports on a small number of studies of school improvement which cumulatively, over more than seven years, achieved gains with effect sizes of around 0.5.

Figure 8 and Figure 9 present the overall changes for the total cohort in terms of the stanine distributions. Figure 8 displays the percentage of students in each stanine at the beginning (Time 1) and the end of the project (Time 6). Figure 9 shows the percentage of students in each of the achievement bands. Table 6 provides the mean percentages in each of these bands. There was a marked reduction in the percentage of students at the lower stanines (to 6 percent in the Low band and 26 percent in the Below Average band) and an increase in the percentage of students in the Average band (to 59 percent in stanine 4–6) and Above Average and Outstanding band (to 10 percent in stanine 7–9). The results therefore more closely approximate the national norms.

Figure 8 Stanine distribution at Time 1 (Term 1, 2003) and Time 6 (Term 4, 2005) against national norms

Figure 9 Percentage scoring at low, below average, average, above average and outstanding bands at Time 1 (Term 1, 2003) and Time 6 (Term 4, 2005) against national norms

Table 6 Mean Percentages of Students (and Numbers of Students) in Stanine Bands at Time 1 and Time 6 Compared with Expected Percentages of Students (and Numbers of Students)

	Low (Stanine 1)	Below Average (Stanine 2-3)	Average (Stanine 4-6)	Above Average (Stanine 7-8)	Outstanding (Stanine 9)
Expected	4	19	54	19	4
	(9.5)	(45.2)	(128.5)	(45.2)	(9.5)
Time 1	13	46	39	1	0
	(32)	(109)	(94)	(3)	(0)
Time 6	6	26	59	8	2
	(19)	(61)	(140)	(19)	(4)

However, the demonstration of the educationally significant shift of the local distribution to near national distribution should not mask the finding that still more students need to score within the above average and outstanding bands to fully match national norms. The observed stanine distributions at Time 1 (T1) and Time 6 (T6) can be compared with the expected (normal) distribution. Because of small cell sizes at T1 in the banded stanines, the band cells were combined into two cells: number of students in stanines 1–3 and in stanines 4–9. The distributions at T1 were significantly different (X² ₍₁₎ =176.5, p<.000). At T6, the distributions were not significantly different at p<.001, but were at .01 (X²₍₁₎ = 9.72, p<.01) (note p.01 = 9.21). With a stringent test (p=.001), the two cell comparison shows that the distributions are similar, but the less stringent test supports the conclusion that more gains are needed to better approximate the expected distribution. The shift from T1 to T6 can be expressed in terms of the risk of not being at average or above levels. The risk at T1 was 1.9 (i.e., the risk was almost double what would be expected). This reduced to a risk of 1.1 at T6, showing that there was by then only a small risk of not being at average or above average levels.

Gains in achievement across phases

The total group made statistically significant accelerations across each phase (see Table 7). On average, these were higher in Phase 1 (mean stanine gain=0.47) and Phase 3 (mean stanine gain=0.51) than in Phase 2 (mean stanine gain=0.35). Individual cohorts had somewhat variable patterns. The raw score results in Table 8 show that all cohorts made statistically significant gains in raw score points in each phase.

Table 7 Stanine Means by Cohort for Phases 1, 2, and 3 (Time 1–6)

Table 8 Raw Score Means by Cohort for Phases 1, 2, and 3 (Time 1–6)

Gain scores

When compared with nationally expected progress, 89 percent of students maintained or accelerated their achievement from the beginning to end of the project. The majority of students (60 percent) gained between one and five stanines, or maintained their stanines from the beginning of the project (29 percent). Figure 10 shows the gain scores from the beginning and end of the project.

Figure 10 Gains scores from Time 1 to 6 for longitudinal cohorts of students

The gain scores, when broken down by phase, show that in each phase most students maintained or accelerated achievement over the three years (78 percent, 83 percent and 86 percent respectively), with about half the students in each phase accelerating achievement (see Figure 11). There was also a trend towards a decreasing percentage of stanine losses by the end of the project, with fewer students not making expected progress for the year.

Figure 11 Percentage of loss, maintenance and acceleration across the three phases

The achievement of Mäori students

Mäori students’ achievement accelerated, like the other ethnic groups participating in the project (see Figure 12), gaining on average 1.1 stanines across the three years. By the end of the project, the average Mäori student scored within the “average” band (mean =4.73), which was only 0.27 below the expected national average of stanine 5 (see Table 9). Indeed, cohort 1 (Year 4) Mäori students at Time 6 achieved above the national expected average, at stanine 5.29. By contrast, at the beginning of the project, the average Mäori student scored in the “below average” band.

The intervention was designed from the profiles of the local students and their instruction. It contained elements that were designed to be both generic for the population of students, and to be personalised using cultural and linguistic resources. It appears that the fine tuning of instruction across the three phases of the research and development programme enabled this to happen.

Other ethnic groups combined made slightly under one stanine gain (0.95) across the three years. They scored within the “below average band” at the beginning of the project, and within the “average” band by the end, although their stanine scores were slightly further than those of the Mäori students from the expected national average of stanine 5 at the end of the project.

Figure 12 Mean achievement gain (in stanines) for Mäori students, compared with other ethnic groups combined

Table 9 Stanine Means by Cohort for Mäori Students and Other Ethnic Groups Combined, from Beginning to End of the Project

The achievement of males and females

Overall, both males and females accelerated significantly (see Table 10). By Time 6, female students on average scored in the average band of achievement, whereas at the beginning, they scored in the below average band. Male students scored slightly under the average band (mean = 3.97) by the end of the project.

Figure 13 shows that while both males and females made similar rates of progress over the three years in the intervention, female students, on average, started with higher levels of achievement than male students. Male students will need to accelerate their achievement more than female students if the gap between male and female students is to be closed.

Table 10 Stanine Means by Cohort for Gender — Phase 1, 2, and 3 (Time 1–6)

Figure 13 Stanine means by gender — Phase 1, 2, and 3 (Time 1–6)

 

School gains across the three phases

Four schools could be analysed using the longitudinal cohorts (they contained students in Years 4–6). Each school made statistically significant accelerations in achievement from the beginning to the end of the project, with effect sizes of between 1.15 and 0.39 (see Table 11). The largest gain was over two stanines for one school, from 3.27 to 5.48. By the end of the intervention, three schools were scoring in the average band of achievement, as opposed to the below average band of achievement at the beginning of the project.

The pattern across schools was one of increasing acceleration in achievement from Time 1 to Time 6, although there was variation across schools in the amount of gain, and in the pattern of losses and gains over the three phases (see Figure 14). For example, School C made consistent gains over the time points and a significant gain between Time 5 and 6. School F, by contrast, made a larger gain in Time 1 and 2, and maintained the gains they made from Time 2, with no further accelerations. Some schools had larger losses over the summer break than others (such as School E and school B), showing a drop in achievement between Time 4 (end of 2004) and Time 5 (beginning of 2005). However, this was not consistent across phases, as both schools did not show such large losses from the end of 2003 to the beginning of 2004.

Table 11 Stanine Means by Cohort for School — Phase 1, 2, and 3 (Time 1–6)

* p<.05
** p<.01
*** p<.001

Figure 14 Stanine means by school — Phases 1, 2 and 3 (Time 1–6)

Not all cohorts within and between schools made similar amounts of acceleration through the intervention (see Table 11). School F showed the widest disparity between cohorts, with one cohort having an effect size of 0.77, and the others having effect sizes of 0.17. Cohorts in other schools were more consistent, although there were still slight differences between them.

Design-based longitudinal and cross-sectional comparisons

In design terms, the previous analyses simply used pre- and post-measures, with no comparison against control groups or equivalents. Therefore, gains are not able to be systematically attributed to the intervention. To better test that attribution, gains were analysed, first as parallel comparisons with the projected means for each year established by the cross-sectional baseline, and secondly against a comparison group of similar schools (see the discussion of the design controls, p.20).

Cross-sectional analyses indicate that after one year of the intervention, all cohorts were statistically significantly higher than the projected baseline (see Table 12). This provides the initial design based evidence that the gains can be systematically attributed to the intervention.

Table 12 Mean Student Achievement In Comprehension (In Stanines) After One Year of Intervention, Against Cross-Sectional Baseline

		Cross Sectional Baseline	Cohorts after one year of intervention
		(Time 1, Feb 03)	(Time 3, Feb 04)	t value		ES
Year 4–5	Mean	3.42	3.96	3.12	*	0.37
	SD	1.57	1.35
	N	241	114
Year 5–6	Mean	3.15	3.84	3.04	**	0.44
	SD	1.55	1.62
	N	296	56
Year 6–7	Mean	2.83	3.26	2.51	*	0.34
	SD	1.29	1.22
	N	307	68

* p<.05
** p<.01
*** p<.001

Further analysis suggests that after two years of the intervention, all cohorts were not just statistically significantly higher than the projected baseline (see Table 13), but that the difference between their scores and those of students in the same year level two years previously had increased (see Tables 12 and 13). The effect sizes were now between 0.31 and 0.59, compared with 0.34 and 0.44 after one year of the intervention. This means that the intervention had a cumulative and positive effect on achievement. This is portrayed diagrammatically in Figure 15.

Table 13 Mean Student Achievement In Comprehension (In Stanines) After Two Years of Intervention Against Cross-Sectional Baseline

		Cross Sectional Baseline	Cohorts after two years of intervention
		(Time 1, Feb 03)	(Time 5, Feb 05)	t value		ES
Year 4–6	Mean	3.15	4.04	5.28	* **	0.59
	SD	1.55	1.45
	N	296	114
Year 5–7	Mean	2.83	3.23	2.12	*	0.31
	SD	1.29	1.29
	N	307	56
Year 6–8	Mean	2.95	3.53	3.04	**	0.42
	SD	1.45	1.29
	N	299	68

* p<.05
** p<.01
*** p<.001

Figure 15 Time 1-6 cohorts against 2003 baseline

We added two features to increase the robustness of the design. The first was to test the issue of subject selection bias. We showed that the students in the longitudinal cohorts did not generally differ from all students in terms of initial achievement levels. The second was to compare the baseline projections with a cross-sectional baseline from a similar cluster of schools after a year had elapsed, thereby controlling for general history and maturation and other associated factors. To follow this latter comparison up, and show that the baselines did not differ significantly, we have compared the outcomes of the intervention in the first cluster with the baseline of the second cluster (Tables 14 and 15).

It should be remembered that the second cluster had similar characteristics, but had not gone through the programme. In both clusters, there were seven decile 1 schools (including one intermediate) involved in a New Zealand Ministry of Education school improvement initiative; similar numbers of students in the baseline sample; similar proportions of males and females; and the same four major ethnic groups. The prior intervention histories of these initiatives were also similar. In both clusters, the same Ministry of Education school improvement initiative, Strengthening Education In Mangere and Otara, was implemented prior to the research (Annan, 1999). The intervention began with one year delay across the two clusters, and focused on the collection and analysis of data. Nonetheless, there were some differences between clusters, particularly in the establishment of professional learning communities, and in prior experience with analysing and interpreting the reasons for achievement patterns.

After one year of the intervention, all year level cohorts scored statistically significantly higher than the comparison cluster that had not experienced this intervention (see Table 14).

Table 14 Mean Student Achievement In Comprehension (In Stanines) After One Year of Intervention Against Cross-Sectional Comparison Cluster

		Cross sectional comparison cluster (Feb 04)	Mangere cohorts after one year of intervention (Time 3, Feb 04)	t value		ES
Year 5	Mean	3.39	3.96	3.42	**	0.40
	SD	1.53	1.35
	N	248	114
Year 6	Mean	3.32	3.84	2.28	*	0.33
	SD	1.51	1.62
	N	237	56
Year 7	Mean	2.71	3.26	3.43	**	0.45
	SD	1.22	1.22
	N	360	68

* p<.05
** p<.01
*** p<.001

After two years of the intervention, all year level cohorts scored significantly higher than the comparison cluster that had not experienced this intervention (see Table 14), and the difference between their scores and those of students in the same year level a year previously had increased (see Tables 14 and 15). The effect sizes were now between 0.41 and 0.61, compared with 0.33 and 0.45 after one year of the intervention. This provides further evidence that the intervention was systematically associated with improving student achievement.

Table 15 Mean Student Achievement In Comprehension (In Stanines) After Two Years of Intervention Against Cross-Sectional Comparison Cluster

		Cross sectional comparison cluster (Feb 04)	Mangere cohorts after two years of intervention (Time 5, Feb 05)	t value		ES
Year 6	Mean	3.32	4.04	4.21	***	0.49
	SD	1.51	1.45
	N	237	114
Year 7	Mean	2.71	3.23	2.95	**	0.41
	SD	1.22	1.29
	N	360	56
Year 8	Mean	2.75	3.53	4.60	***	0.61
	SD	1.26	1.29
	N	305	68

* p<.05
** p<.01
*** p<.001

Overall changes for total school populations year by year
A third way to analyse the achievement outcomes is to check the achievement of students across years, irrespective of presence or absence in any other year. This analysis answers the question of whether changes can be detected in how the school teaches all children, irrespective of membership in a continuously present cohort.

The following analyses, therefore, examine improvements in achievement from the beginning (Term 1) to end (Term 4) of an academic year in each of the three phases. Analyses for each phase were calculated only from the results for the students who sat both beginning and end of year tests in the academic year, to avoid any confounding effects from students with differential exposure to the programme. It should be noted that each phase built on the previous phase, and included processes that were part of that phase. Common to each phase was the analysis, feedback and discussion of evidence. Note that Phase 1 contains fewer students, because one school could not participate in the first round of data collection.

Overall gains in achievement
Table 16 includes all children present at both the beginning and end of each year of testing. The Term 1 and Term 4 comparisons show two things. The first is that each year, statistically significant gains were made from the beginning to the end of the year. This means that with the combination of continuing students, as well as new students, at each level, the effectiveness of the programme in accelerating student achievement in addition to expected national progress was sustained. Students made between 13 and 18 months’ worth of progress, approximately, for a year at school.^[5] Phase 1 was associated with the greatest accelerations in achievement, and Phase 2 with the smallest accelerations in achievement. (The reasons for this are discussed in the section on classroom gains in achievement.) The improvements in mean stanine are shown graphically in Figure 16. The second finding is that each year did not start at the initial level established in 2003, or finish at the levels in the first year, so that achievement levels tended to rise.

Table 16 Mean Stanine and Raw Score Comparing Term 1 and 4 in Each Phase

	Term 1	Term 4	Gain	T value	Effect size
Phase 1 (n=1216) Stanine	3.13 (1.45)	3.66 (1.64)	0.53	14.09***	0.34
Raw Score	26.82 (12.53)	34.20 (14.42)	7.39	31.07***	0.55
Phase 2 (n=1683) Stanine	3.51 (1.58)	3.61 (1.70)	0.10	3.62***	0.06
Raw Score	31.84 (14.77)	35.63 (13.85)	3.80	21.85***	0.26
Phase 3 (n=1619) Stanine	3.41 (1.58)	3.81 (1.73)	0.39	14.67***	0.24
Raw Score	30.69 (14.21)	36.71 (14.32)	6.02	34.13***	0.42

* p<.05
** p<.01
*** p<.001

Figure 16 Mean stanine for beginning (Term 1) to end (Term 4) of year in each Phase

Figure 17 shows the percentage of students in the achievement bands of low (stanine 1), below average (stanines 2–3), average (stanines 4–6), above average (stanines 7–8) and outstanding (stanine 9). In each phase, there was a trend towards the nationally expected percentage of students in each band by Term 4.

Figure 17 Percentage of students in stanine bands in each phase (Term 1 to Term 4) compared to national expectations

While there were statistically significant improvements in mean scores (stanine and raw scores), increasing trends towards the nationally expected distribution, and comparable effect sizes to successful international interventions, at the end of the project there were still fewer students in the average and above average bands than nationally expected. Schools in the intervention therefore still need to focus on improving student achievement further, through sustaining their improved teaching and inquiry practices, and developing new interventions to cater for the students who are now at and above average in their schools.

Year level gains in achievement
All year levels made statistically significant accelerations in achievement, in addition to expected national progress in Phases 1 and 3 (see Table 17). Phase 2 was associated with greater variability in achievement between year levels. In Phase 2, one year level (Year 6) did not make statistically significant accelerations in addition to expected national progress, and one year level (Year 8) made a significant loss in comparison to national expectations. However, in that phase, all year levels made statistically significant gain in achievement (as measured by raw scores) from the beginning to the end of the year. In other words, while on average students in those year levels gained in raw score points, this was insufficient (when adjusted to nationally expected progress for a year) to maintain and/or increase their stanine scores. The trends across year levels in each phase are shown graphically in Figure 18. Note that Year 5 is consistently within the average band by the end of the year, and Year 6 is within the average band by the end of the year in Phases 2 and 3.

Figure 18 Mean stanine (Term 1 and 4) in each phase by year level

Table 17 Mean Stanines and Raw Scores (and Standard Deviations) Across Year Levels for Phases 1, 2 and 3

School gains across the three phases
All schools made statistically significant accelerations in achievement in Phase 1, with all but one making significant accelerations in achievement in Phase 3 (see Table 18). The one school that made a significant loss, compared with national expectations, increased their mean raw score significantly, but this was insufficient (when adjusted to nationally expected progress for a year) to maintain their stanine scores. In Phase 2, there was greater variability across schools. Three out of seven schools made statistically significant accelerations in achievement, compared with national expectations, and one school made a significant loss. However, all schools made statistically significant gains in Phase 2, as measured by raw scores, but this was insufficient (when adjusted to nationally expected progress for a year) to increase their stanine scores.

A range of gains was made between schools and within schools across the three phases. This suggests that schools may have differentially benefited from the combination of processes associated with the three phases. There did not, however, appear to be a Matthew effect, where schools who were already succeeding gained more from the professional development (see Figure 19). This suggests that the impact of the professional development is mitigated somewhat by school characteristics. Some of the characteristics of schools that may have impacted on the achievement results are discussed in greater detail in the next section, on gains in achievement across classrooms.

Table 18 Mean Stanines and Raw Scores (and Standard Deviations) for Terms 1 and 4 for Each Phase by School

Figure 19 Mean stanine in each phase by school

Classroom gains across the three phases
The gains in achievement in each classroom provide some explanation for the differences in the three phases. The following figures (Figures 20, 21 and 22) show the gain scores in each classroom from Term 1 to 4 in Phases 1, 2 and 3 respectively. Phase 1 was associated with the most consistent gains across classrooms, with 88 percent (52 out of 59) classrooms making accelerations in achievement. By contrast, Phase 2 was associated with the least consistent gains between classrooms, with 58 percent (46 out of 79) classrooms making accelerations in achievement. Even so, over half the classrooms made accelerations in achievement. In Phase 3, the gains came between those of the other two phases, in that nearly three-quarters, 74 percent (58 out of 78) classrooms made accelerations in achievement.

Figure 20 Mean stanine gain score for classes in Phase 1

Figure 21 Mean stanine gain score for classes in Phase 2

Figure 22 Mean stanine gain score for classes in Phase 3

Further analyses indicated that two schools were the primary reasons for the higher percentage of stanine losses in Phases 2 and 3. The school that was missing in Phase 1, School G, accounted for most of the stanine losses in Phases 2 and 3, in that 45 percent of the classes in Phase 2 and 60 percent of classes in Phase 3 that made stanine losses were from that one school. Another school, School A, contributed to 24 percent of the stanine losses in Phase 2, although by Phase 3, only one of its classrooms made a stanine loss on average.

The reason for School A contributing 24 percent of the stanine losses in Phase 2 is most likely to be the nature of the participation of the school in the professional development. All teachers participated in Phase 1, and classrooms made stanine gains on average (i.e. every classroom average accelerated their achievement). But School A withdrew most of its teachers in Phase 2, in that only one teacher consistently attended the professional development, and no school leader did. Table 19 shows attendance by staff and the school leader for nine sessions in Phase 2. A low participation rate was associated with 7 out of 13 classrooms in School A which made stanine losses, i.e. they did not maintain the expected progress, compared with national norms. However, when more teachers in that school participated in Phase 3, only one of its classrooms made stanine losses, and the rest made accelerations in stanines.

One indication of the degree of participation in Phase 3 was the number of inquiry project presentations at the teacher conference (see Table 20). School A had a history of incomplete participation, with analyses of participation in another intervention prior to this one showing that they had only completed one of the two task requirements of the programme (Lai et al., 2003). Any future interventions with this school may need to stress the importance of fully participating in the programme.

Table 19 Ratings of Participation of Staff and School Leader in Ten Professional Development Sessions (Phase 2) by School

School	Teachers ¹	Leader ²
B	3	2
D	3	3
E	3	3
C	3	1
G	3	3
F	3	3
A	1	0

¹ 1= fewer than 5 sessions; 2= 5-7 sessions; 3= 8-10 sessions
² 0=did not attend; 1=fewer than 5 sessions; 2=5-7 sessions; 3= 8-10 sessions

Table 20 Participation of School in Presentation of Inquiry Projects (Phase 3) by School

School	Presentation¹
School B	3
School D	3
School E	3
School C	3
School G	3
School F	0
School A	1

¹ 0=no staff presentation; 1=one presentation (representing less than 50% classes); 2=more than 1 but not all classes represented; 3=all staff at all levels contributing.

Nonetheless, the school that had a high percentage of stanine losses in Phases 2 and 3 (School G) had high rates of participation in both phases (see Tables 19 and 20). Therefore, participation was unlikely to be the reason for the high percentage of losses in this school. Three hypotheses can be advanced to explain the results. The first relates to ceiling or floor effects. The other two relate to the school’s capacity to analyse and use achievement data (one key capacity needed to raise achievement in this intervention) and the teachers’ implementation of the professional development in their classrooms.

The first hypothesis was that this school had significantly lower or higher achievement than other schools, suggesting, respectively, either greater difficulty in accelerating achievement, due to large percentages of students being below expectations, or being closer to a ceiling. However, an analysis of the baseline information at Time 1 shows that the school that scored the lowest at this baseline (2.80) made significant accelerations in achievement in Phases 2 and 3. It is therefore unlikely that low baseline scores alone could be the reason for the patterns in achievement.

The second hypothesis is unlikely, because a previous evaluation of the capacity of school leaders in that school to analyse, interpret and use achievement data showed that the leaders had a good understanding of how to do so (Lai et al., 2003). In fact, that school showed higher ratings in their capacity to analyse, interpret and use achievement data than some other schools in the intervention. As the leaders had not changed since the beginning of the intervention, the leaders had the capacity to analyse, interpret and use data to inform classroom practices. Moreover, the leaders had also demonstrated that they had taught and checked the understanding of teachers as to how to analyse, interpret and use data in that school (Lai et al., 2004).

The final hypothesis relates to classroom practices. There was little difference in the ratings of School G teachers’ classroom practices, compared with other schools. In the second year, five Literacy leaders rated their teachers for aspects of their classroom programme (see Table 21). The ratings were from 0–3 on each dimension (see Appendix A). One leader did not rate teachers, because at the end of the year that lead teacher left the school; another did not close because of limited involvement in the professional development by that school. Each of the five Literacy leaders who rated their staff overall judged that they well had established classroom routines with high engagement by students (range 2.3 – 2.7); students were rated as immersed in a rich literacy environment (range 2.0 – 2.5), where teachers generally carried out the focus of the professional development (range 2.0 – 2.5).

Table 21 Ratings¹ by Literacy Leaders of Features of Reading Instruction

Teachers (N)					School
	D (6)		C (5)		G (11)		B (10)		E (12)
Routines and high engagement		2.7		2.2		2.5		2.5		2.3
Focus on PD		2.5		2.2		2.2		2.3		2.0
Richness of literacy Environment		2.5		2.0		2.2		2.0		2.1
Frequency of Instruction		2.8		3.0		2.0		2.8		3.0
Assessment		2.5		2.0		1.9		2.0		1.7
Contribution to PLC		2.3		2.0		1.9		2.0		1.7

Note: Two school literacy leaders did not rate staff.
¹ Ratings in a 0-3 scale (see Appendix A).

However, as these ratings were completed by one school leader, without any form of reliability check, we cannot discount the possibility that differences between the school leaders’ in how they rated their classroom practices may have been a factor influencing different levels of gain. Detailed descriptions for the ratings and the rating system were discussed with the literacy leaders in a group session, so this possibility is less likely.

Within these overall ratings, there were individual teachers who were rated quite low. The Literacy leaders all generally rated their teachers as less than fully contributing to the professional learning community of their school (range 1.7 – 2.3), but the average rating of 2.0 meant they judged that staff generally talked and shared ideas on a range of settings, and had discussions on what they did in their classrooms. One school Literacy leader (School D) rated their teachers consistently higher than the others. The only area in which School G appeared to be consistently different from the other schools was the frequency of dedicated reading instruction sessions per week. In school G, such sessions occurred only three times a week. In each of the other schools, sessions occurred in general four or more times a week (rating 3.0). As this is a description of the curriculum delivery, this aspect is less likely to contain from any systematic bias on the ratings. It is one possible reason for the lower achievement in School G in the second year.

	T:	… It’s our learning outcomes or what we’re learning to do. …We’re going to extend, and another word for extend is improve our vocabulary, okay? This is mainly a vocabulary lesson. And we’re going to write our own poems with the synonyms we find.
	[discussion of words “vocabulary” and “synonyms”]
	[Children read poem]
	T:	What were you thinking about as you were reading that poem?
	[further discussion focussed on response to poems and synonyms]
	T:	I agree with you that this is not a happy or a positive or a light hearted poem. This is a hard question. What is the deeper meaning of this poem? What do you think Pauline Cartwright’s trying to tell us in this poem? [looks at child with hand raised]
	C:	About war and how people die.
	T:	Could be. Think a little, but what um Willie said earlier on.
	C:	[another child] the newspaper’s horrible to read.
	T:	The newspaper’s horrible to read. Because look at the title of the poem. Read the title of the poem to me. [they read title]
	T:	My teacher said to read the newspaper so I did. Have I told you that before? To read the newspaper?
	C:	[all say yes]
	T:	So you did. So what did they find out? Horrible, sad, bad, war. Do you think all newspaper articles are about like that?
	C:	[all say no]
	T:	I’ve got a happy one here that we will read a little bit later, okay? Just to prove that not all newspaper articles are bad. Right but let’s get on with our um discussion of meanings.
	[children make lists of words known and not known and compare common unknown words – words include “depression”, “fascism”, “recession”, “greediness”, “neediness”]
	T:	Okay so tick “inflation”. What else do you have at the bottom there? “Mourning” Hey talking about that, is there another morning?
	C:	[one child says] Yes miss.
	T:	How’s that spelt? [looks at another child]
	C:	m-oh m-o-r-n-i-n-g.
	T:	What is that? What does it mean? What does that morning mean that you have just spelt?
	C:	The morning right now.
	T:	Yeah. In the …?
	C:	(children say ) Morning
	T:	Morning. What’s short, the abbreviation for morning?
	C:	Morn.
	T:	When you see it written next to a time?
	C:	[two children say] A.M.
	T:	A.M. well done. A.M. morning. Now that “mourning” that we’ve got written on our papers … what did they add up here? They added something.
	C:	U
	T:	They added “u” and therefore the meaning changes ….
	[discussion of words continues with use of dictionaries and thesaurus]
	T:	I just wanted you to be aware that the words they’ve written down there are all nouns okay? They changed the root words into nouns to write the poem. You don’t have to keep all your words nouns when you write your poem. Okay? It’s just so that you can understand the poem. So we’ve learnt what the deeper meaning is of the poem, we’ve seen that our poem rhymes, and that …

The group completed the session by jointly writing a poem: Racism is bad. Mourning is sad. Fears give you tears. War has gore. Open the doors, no more horrors and sores.

Case Study 2: Expected gain classroom
Teacher 2, who was at a different school from Teacher 1, was also a mainstream teacher in a Year 7/8 composite class, with 27 students. On average, students had made expected progress, remaining at the same stanine level at the end of the year. Their gains of component tests, shown in Table 26, were largest in decoding and in the fifth curriculum related task of identifying emotive words in a piece of persuasive writing.

Teaching in this classroom also had the attributes associated with the cluster focus in the first and second years. The teacher also had a well organised sequence of activities, using the general programme components of guided reading with ability groups and shared reading with those groups. She established clear statements of goals at the beginning of sessions. The focus on checking for evidence in the text was present both at the beginning and at the end of the year. Similarly, she had a consistent focus on vocabulary, as well as making life to text connections, trying to incorporate background event knowledge. The following sequence showing these attributes comes from a guided reading session at the beginning of the year with seven children reading an informational text about mangrove trees.

	T:	Now what we’re gonna do here, is we’re first of all gonna have a look at the first page and find out two things about these trees while we’re actually reading through it. The first thing is why they only grow in the northern part of New Zealand and why they are really different trees from most other trees. “C” can you start reading please.
	C:	Did you know that nga manwa [child stumbles on word]
	[further reading and correction of pronunciation] …
	C:	Nga Manawa will only grow in the northern part of the island – the North Island. In the southern land it is too cold for them to grow.
	T:	Okay now, why do they only grow in the northern part of the North Island of New Zealand.
	[teacher selects child with hand up]
	C:	[child does not answer]
	T:	We’ve been talking about it and it’s actually written down there on the page. Remember how I said I was going to ask you?
	C:	Yes.
	T:	Yes just to make sure you were concentrating [selects another child]
	C:	Because it’s warm.
	T:	Yeah because it’s warmer. That’s the important thing. Because it says down the bottom, further south than this it is too cold. So it’s warmer. Now what is the other thing about these mangrove trees that makes them really special tree that’s been mentioned on the first page?[teacher selects a child]
	C:	Because they can survive in um salt water.
	T:	Yeah that’s right. So that’s two interesting facts about it. Now have a look on that page and see if there’s any words that you don’t know what they mean. ‘M’?
	C:	‘Swampy’.
	T:	Okay does anyone know what ‘swampy’ is? What do you think?
	C:	Um like where ducks live.
	T:	Okay so where ducks live is not a bad idea. So what kind of places do ducks live in?
	C:	Swamps.
	T:	Yeah but we’re trying to find out what that word means eh? ‘M’?
	C:	Can I take a guess?
	T:	Yeah take a guess.
	C:	Is it like muddy and sort of watery?
	T:	Yeah that’s a good thing because if you look at this (refers to illustration) that’s got a bit much water in it at the moment, but it’s sort of along the right lines. Just look on the next page. This picture here. Also, some of us went to Western Springs (a local lake). Have any of you guys been to Western Springs? At Western Springs you get kind of muddy slushy areas like that. That’s swampy. But the difference between Western Springs and where the mangroves live is that the mangroves live in …? [child puts up hand]
	C:	[another child answers] Salty water.
	T:	Yeah salty water.
	[Other words were discussed, such as “anchor” and “poison”, over multiple turns. For example, the word ‘Coastal’ took 13 turns]
	C:	Can I ask a question?
	T:	Yeah.
	C:	Do they call mangroves around the world ‘mangroves’ or do they call it another name for it?
	T:	In countries where people speak English they call them mangroves. I don’t know what, I don’t know the word for mangrove in any other language at all, except Mäori which is Nga Manawa, but I don’t know in any other languages at all. You might like to ask Mrs. ‘H’ (a Samoan teacher in another class) afterwards if she knows what the word is in Samoan. She might.

Further examples of these attributes can be seen at the end of the year, in a guided reading session with four children reading an informational text about comets (“Shooting Through”; Connected 3 – 2003).

	T:	Oh the Orion nebula. Okay, Orion is a constellation. It’s a group of stars that are all together. Okay was there anything else anybody didn’t know what it was?
	C:	This one
	T:	Okay so let’s look at the word “occasionally”.
	[discussion of word for 22 turns]
	C:	Evaporate
	T:	Okay evaporate. Anyone know what “evaporate” means? Okay where abouts is it, oh that’s in that first sentence of that paragraph isn’t it?
	C:	[one child] yes.
	T:	Okay let’s have a listen to the sentence and you have a read of it while I read it to you and we will see if we can work out what it means. As the comet nears the sun some of it evaporates and becomes a strip of dust and vapour that looks like a huge tail. So as the comet nears the sun some of it ‘evaporates’.
	C:	[another child] Gets longer?
	T:	No, that’s a good guess though cos we learnt that it did get longer.
	C:	{Another child] Some of them starts to fall apart.
	T:	You are on the right track there.
	C:	[another child] It changes.
	T:	Yeah it does change. Now when you put your clothes out on the clothes line, they’re wet aren’t they?
	C:	[all say yes]
	T:	Later on in the day unless it’s been raining you go to bring them in, what are they like?
	C:	[one child] Wet. [another child] soaked.
	T:	If it hasn’t been raining … you put them out and they’re wet right? And it hasn’t been raining all day and you go back to bring them in.
	C:	[one child] It’s dry.
	T:	It’s dry. Okay so where has the water gone?
	C:	[one child] Oh the sun absorbed it.
	T:	Okay that’s evaporates okay? It turns into steam and it disappears off up into the clouds.
	[Further examples of extended question and answer sequences around the unknown words (one for “thermos” of 31 turns).]