Moving a school: Higher order thinking through SOLO and e-Learning

Introduction

Technological, social and economic change is encouraging increasing emphasis on the development of higher order thinking skills throughout the world and they are being incorporated into national curriculum goals in many countries, including New Zealand. Simultaneously the use of digital technologies is being promoted by many educators and authorities in this country and elsewhere as an approach that will enable students to develop these skills.

An increasingly popular tool for identifying higher order thinking is the SOLO (Structure of Observed Learning Outcomes) taxonomy (Biggs & Collis, 1982). This taxonomy describes the complexity of student responses to questions or tasks, and also can be applied to the questions or tasks themselves. SOLO levels distinguish between surface and deep thinking and one intention of using this approach is to enable teachers to guide students towards increasing their depth of thinking.

This TLRI project was set up to explore the school-wide introduction of the SOLO taxonomy to guide learning and teaching in a New Zealand secondary school. A particular focus of the project was the use of e-learning activities to support the planned learning outcomes which were derived from the use of SOLO.

As part of this exploratory project, the researchers studied a professional learning and development (PLD) programme which introduced the use of SOLO to the whole staff, and investigated the teaching and learning in the Year 10 classes of a small number of science and social studies teachers who participated in the study.

The educational context

Higher order thinking skills in the curriculum

National educational goals frequently call for an emphasis on generic cognitive skills, which are usually described in terms of critical, creative or reflective thinking, analysis and knowledge creation. These have become known as higher order thinking skills. In New Zealand, the first key competency required in the The New Zealand Curriculum (Ministry of Education, 2007) is “thinking”—“using creative, critical, and metacognitive processes to make sense of information, experiences, and ideas” (p. 12).

Alexander and colleagues (2011) have looked at the intellectual activity involved in higher order thinking, and propose the definition:

Higher order thinking is the mental engagement with ideas, objects, and situations in an analogical, elaborative, inductive, deductive, and otherwise transformational manner that is indicative of an orientation toward knowing as a complex, effortful, generative, evidence-seeking, and reflective enterprise. (p. 53)

Although this definition is general, the authors recognise that the distinctive qualities of higher order thinking depend on the nature of the domain and task. Schraw, McCrudden, Lehman, and Hoffman (2011) argue that there is a range of often overlapping higher order thinking skills, which they identify as (a) reasoning skills, (b) argumentation skills, (c) problem solving and critical thinking, and (d) metacognition.

Use of the SOLO taxonomy

Learning outcomes have often been described using the taxonomy of educational objectives (Bloom, Engelhart, Furst, Hill, & Krathwhohl, 1956)—often called “Bloom’s taxonomy”—which categorised cognitive processes as knowledge, comprehension, application, synthesis or evaluation. The latter two categories have been associated with objectives involving higher order thinking skills, although the phrase “higher order thinking” does not actually appear in the work. The taxonomy does not imply movement through the categories as a hierarchical process and treats them as discrete (Alexander et al., 2011).

A revised version of Bloom’s taxonomy was published in 2001 (Anderson et al., 2001) but a number of commentators still believe that this model is flawed (e.g., Hattie & Brown, 2004; Hattie & Purdie, 1998). These commentators consider the SOLO taxonomy an alternative that has advantages over Bloom’s taxonomy.

SOLO was originally designed to provide a means of identifying the cognitive level of student responses to questions, categorised on a five-level scale of stages based on the number of ideas presented and the relationships identified between them (see Figure 1).

The SOLO categories are ordered according to the depth of thinking demonstrated and recognises that students can operate at any SOLO level with different degrees of difficulty. SOLO also enables teachers to design questions targeted at different levels of complexity, though the answers that the students provide may demonstrate thinking at a different level from that targeted.

Figure 1. The structure of the SOLO taxonomy

	Level	Description	Simplified version*	Symbol
	Prestructural	No relevant aspects
Surface	Unistructural	One relevant aspect	One/Many Ideas
	Multistructural	Several relevant independent aspects
Deep	Relational	Integrated into a structure	Relate
	Extended abstract	Generalised to a new domain	Expand

Based on Biggs (1991)

*Simplified structure used in this project.

Studies have demonstrated that SOLO can also provide an effective tool for teachers to structure questions, construct tests and design activities (G. T. L. Brown, Irving, & Keegan, 2008; Hattie & Purdie, 1998). This can enable students to receive clear feedback on their performance in relation to the teaching intentions. Smith and Colby (2007) found that when teachers deliberately selected activities to elicit surface or deep learning, students responded correspondingly, as assessed using SOLO, while research by Hattie and Purdie (1994) has indicated that teachers found the SOLO taxonomy much easier to use than Bloom’s taxonomy.

The SOLO taxonomy forms the basis of the Assessment Tools for Teaching and Learning (asTTle) developed for the New Zealand Ministry of Education (G. T. L. Brown et al., 2008). Assessment for the achievement standards used in the New Zealand National Certificate of Educational Achievement (NCEA) is also similar to SOLO.

Despite widespread use of SOLO, there has been limited evidence of its use by teachers at the secondary level within New Zealand. It can, however, provide a relatively simple, comprehensible and flexible tool for teachers to design learning outcomes which make explicit the cognitive level at which students are operating. This enables both teachers and students themselves to assess the cognitive level at which the students are working and identifies higher order thinking skills within the students’ activities.

A SOLO-based approach to teaching

Martin (2011) used SOLO effectively with secondary science classes to make learning outcomes explicit for students through specific success criteria, which were presented to students in a learning log that they used to track their own progress. The teacher then planned activities which enabled students to achieve the learning intentions at the different levels.

Martin adopted the practice of identifying the learning intentions using a more manageable three levels, as shown in the fourth column in Figure 1, in which the unistructural and multistructural levels are combined and icons are used to aid students’ understanding. This is the approach used in the project school for the TLRI study reported here. Some examples of success criteria based on this simplified scheme are shown in Figure 2.

Figure 2. Examples of success criteria based on the SOLO taxonomy used in science

One/many ideas: I can identify and describe the different types of heat transfer. Relate: I can analyse the data collected, using graphs. Extend: I can justify the natural materials used in my building and the type of heat transfer they reduce.

e-Learning in the curriculum

The use of information and communications technology (ICT)—computers and similar devices, software and the internet—in schools as a medium for student learning can be described as “e-learning” and is widely encouraged, as in The New Zealand Curriculum (Ministry of Education, 2007).

There is a large body of evidence which shows that e-learning can have a positive effect on student achievement (e.g., Alton-Lee, 2003; BECTA, 2002; Crook, Harrison, Farrington-Flint, Tomás, & Underwood, 2010; Wright, 2010) though the conclusions are often equivocal (Eng, 2005). Hattie (2009) found the effect size for studies concerned with computer-aided instruction to be low but he identified some conditions which characterised more effective interventions. Wright (2010) noted that a trend in the literature on e-learning was to emphasise the importance of teachers’ active role and presence in classroom use of ICT, and Cox et al. (2003) also point to the crucial role of teacher planning, preparation and follow-up, particularly in linking resources, activities and objectives.

Wegerif (2002) points to three different roles of the computer in education, as tutor, mindtool and support for learning conversations, which echo the key characteristics summarised by Dyke, Conole, Ravenscroft and de Freitas (2007) as fostering (a) experience and activity, (b) thinking and reflection, and (c) conversation and interaction. Dyke et al. conclude that, “It is the aptitude for higher-level thinking that is the hallmark of human learning” and that, “e-learning needs to nurture this ability to think, reflect, deliberate and anticipate the possible consequences of our actions” (p. 89).

SOLO and e-learning

To achieve the perceived potential of e-learning in developing higher order thinking skills requires a focus on activities which encourage thinking, and teachers need to develop what Mishra and Koehler (2006) have described as “technological pedagogical content knowledge”—understanding of ways in which information technology can be applied to achieving specific pedagogical goals.

The use of SOLO-based learning intentions can facilitate a focus on appropriate e-learning activities. Martin’s (2011) approach was to use e-learning activities in the classroom linked directly to SOLO-based learning intentions, providing individual choice, shared meaning making, motivation and opportunities for teacher dialogue with students. The activities were available to students online through a “virtual classroom”, where students had access to their learning log with related resources and activities, both inside and outside the physical classroom; increased autonomy for the students increased motivation and provided the teacher with greater opportunity for individual attention.

Innovation in education

The introduction of curriculum innovations and new technologies into educational institutions has been widely studied, often related to computer technology. In recent years technology acceptance theories have been applied to e-learning in education (e.g., Teo, 2011). The basic technology acceptance model (Davis, 1989) proposed that intentions to use a technology were formed by the perceived usefulness and the perceived ease of use and may then lead to actual use. Further work has developed more complex models (e.g., Venkatesh and Davis, 1996; Taylor and Todd, 1995, cited in Smarkola, 2011, p. 11).

While these theories do not recognise all of the unique influences on teachers and students, we found that they provided a basis for studying responses to e-learning, and can also inform our understanding of introducing other innovative practices including, in this study, the use of SOLO.

Professional learning and development in secondary schools

The phrase “professional learning and development” (PLD) reflects a broad understanding of the needs of teachers to develop knowledge and understanding (Education Review Office, 2009). In a climate of rapid change in schools there has been an increasing focus on improving the effectiveness of PLD based on research evidence. A best evidence synthesis (Timperley, Wilson, Barrar, & Fung, 2007) has been a major influence on the development of PLD practice in New Zealand, concentrating on the emerging knowledge about how to promote teacher learning in ways that affect the learning of diverse students. Elsewhere, Timperley (2008) summarised important points of this synthesis, which are discussed later in this report. A key principle is that PLD should lead to improved student outcomes.

The 2009 ERO report on PLD in a number of secondary schools also emphasises the importance of effective management, planning and leadership in achieving successful student outcomes.

Baldwin (2008) draws attention to some specific features of secondary schools that challenge professional learning—their larger size and complexity, the specialist nature of most teachers, and organisation into subject-based departments. A number of authors see these features as creating cultural and organisational barriers to school-wide change. To overcome this, one strategy adopted by schools has been to establish professional learning groups which are organised across departments.

A professional learning group may be described as a particular kind of learning community. Timperley et al. (2007) found that professional learning communities were common in a large majority of studies and appeared to be a necessary, but not sufficient, condition for successful PLD. Successful communities provide a mechanism for teachers to work together to share and compare theories and practices, plan collaboratively and build revised practice and knowledge.

The specific school context

This TLRI project took place within a large suburban co-educational secondary school in a New Zealand city. In 2012 the school roll was 1,806, including 48 international students, with a diverse ethnic mix. Students come from a largely high socioeconomic community and the school decile rating is 10.

At the time of its 2010 Education Review Office (ERO) report, the school had been facing a number of problems, with a decreasing roll and a decline in student achievement in some areas of the curriculum. The report made positive comments about relationships within the school, the enthusiasm of the teachers and moves in the school towards dealing with the issues, including the directions being taken by the new principal, appointed at the start of that year. It urged an increased emphasis on critical reflective practice, including, “an evaluation of the effectiveness of professional learning and development models, and of teaching and learning initiatives” (Education Review Office, 2010, p. 7). It recommended a focus on improving Māori and Pasifika achievement, and building a learning culture across the whole school.

The TLRI project examined the PLD model which was developed in response to that situation, and the use of SOLO and e-learning to meet the identified needs.

The VGT programme

The model of teacher professional development, introduced in 2011 and called VGT (Very Good Teaching), aimed to (a) raise student achievement and (b) improve teacher effectiveness. The initial activity focused on gathering data from staff and students at the beginning of the year, and the programme was launched at the beginning of May.

The programme was based on cross-curricular professional learning groups of 16–20 members, led by a team of seven experienced teachers who met together to plan and develop their own knowledge and skills. The specialist classroom teacher was the co-ordinator and lead facilitator.

The focus in the first year (2011) was on relationships with students and between colleagues; in the second year on the SOLO taxonomy for clarifying learning goals; and in the third year on “improving with inquiry” (IWI)— teachers’ inquiry into their own practice, with a SOLO focus. Teachers were required to present the results of their inquiry to their subject department, and optionally to their professional learning group. In the fourth year, the focus is on learning partnerships through reciprocal mentoring.

Aims of this research

This exploratory research project was developed from a desire to study the effectiveness of a major PLD project introducing the use of SOLO across all curriculum areas and particularly the value of using SOLO to guide the use of e-learning in the curriculum with a goal of developing higher order thinking.

The research had two aims:

1. To study the effectiveness of a model of professional development designed to increase secondary teachers’ competence in using the SOLO teaching approach.
2. To study the combined use of the SOLO taxonomy and e-learning as a teaching strategy designed to develop higher order thinking in secondary school students.

The following broad questions arose from these aims.

1. In what ways does a professional development programme based on professional learning groups and professional inquiry enable secondary teachers to effectively implement the use of the SOLO taxonomy with e-learning in their teaching?
2. What is the effect on students’ performance on higher order thinking tasks of teachers using the SOLO taxonomy and e-learning in a secondary school, and what factors affect this?
3. What are the characteristics that describe different levels of implementation of complementary use of SOLO and e-learning in a secondary school?

Value of this research

This study should inform initiatives of a similar nature in other schools by providing a deeper understanding of a PLD approach for secondary schools, and the implications of introducing the SOLO taxonomy. We also intended that the study could provide some guidance about effective ways of using e-learning based on the use of SOLO.

The broad exploratory nature of the study should provide some insight into important issues for future research.

For the school within which the study took place, and the individual teachers involved, the project was intended to be a further opportunity for reflection, and for feedback which included an external perspective.

Research design and methods

The research design is based on the “design research” methodology (Collins, Joseph, & Bielaczyc, 2004), which arose largely out of work by Brown (1992) and has also been called design-based research (Sandoval & Bell, 2004). The methodology typically involves a four-stage process consisting of analysing practical problems, researching solutions and developing solutions, evaluating them in practice and, finally, documenting and reflecting on them (Margaryan, 2008). This project effectively begins at the third stage of evaluation and testing, where data is gathered through mixed methods to provide rich qualitative and quantitative information as a basis for reflection.

To make the project practicable, two subjects were chosen—science and social studies—and Year 10 selected as the level for study. In the first year, two teachers from each subject participated. One of the four had expertise in using SOLO and e-learning, and acted as a facilitator and mentor. Because of a late start in the first year, we decided to continue with the same subjects in the second year; two teachers, including the expert, remained the same, but three new participants were recruited giving a total of five.

Each teacher and their Year 10 class were observed during a unit of work at the start of each year, and again for a second unit in the second semester, during which the teachers were asked to incorporate the SOLO approach being studied. At the same time, the teachers were participating in the whole-school PLD programme focused on teaching with SOLO. Data were collected using a range of methods, listed below.

• A small number of lessons for each class were observed for each unit with video recording and a running record.
• Students in each class completed a written survey at the beginning and end of each year.
• Small groups of students and some individuals from the study classes were interviewed.
• Participating teachers were interviewed at least once each semester, drawing on the classroom observations.
• All teachers in the school were asked to complete a survey at the beginning and end of the project.
• Additional interviews were carried out with senior managers, including the principal, and selected middle managers and the heads of department for science and social studies.
• A small number of teachers from other subject areas, who were identified as particularly innovative in using SOLO, were interviewed.
• asTTle scores for the project classes, and anonymously for the whole cohort, were obtained, and scores on the Science: Thinking with Evidence test (NZCER, 2010) were obtained for the science classes.

Interviews were recorded, transcribed, and then analysed using the nVivo computer software to identify common themes. Classroom observation records were expanded to give written descriptions under standard headings, and key sections of video, identified in the records, were reviewed for inclusion.

Survey responses were coded into spreadsheets and then analysed using SPSS software to provide descriptive statistics for individual classes and the whole cohort. SPSS was used to analyse test scores and provide a comparison between classes and the whole cohort.

Because of the small sample sizes, few statistical inferences can be usefully drawn from the survey data, so the project draws heavily on descriptive statistics and qualitative data. However, the multiple sources of data provide checks on the validity of conclusions that are drawn from these sources.

Key findings

The professional development programme

This section reports on the school-wide professional development programme and considers Research Question 1.

Key finding 1: Design and implementation of the PLD programme

The VGT programme satisfies, to a high degree, the 10 key principles of effective professional learning and development listed by Timperley (2008) and shown in Table 1.

Table 1. Principles of effective professional learning and development (Timperley, 2008)

Focus on valued student outcomes Based on worthwhile knowledge and skills Integration of knowledge and skills Assessment for professional inquiry Multiple opportunities to learn and apply information Approach responsive to learning processes Opportunities to process new learning with others Knowledgeable expertise Active leadership Maintaining momentum

We examined the design and implementation of the VGT programme, and found that the programme displayed a high degree of compliance with all of these principles.

Key Finding 2: Teacher attitudes to the PLD programme

Teachers showed widespread, though not universal, support and enthusiasm for the VGT programme. They valued the support from senior school leaders, the work of the facilitators, the non-threatening and collegial nature of the programme and disconnection from appraisal. They saw this as providing valuable time for reflecting on teaching and sharing ideas with colleagues from other departments, as well as sharing their personal investigations with subject colleagues, and perceived it as benefitting students.

In our survey of all teachers at the end of the project (with 58 respondents), 64 percent responded that they agreed or strongly agreed that the programme had been beneficial for their teaching and 68 percent that it had helped them apply SOLO in their own teaching, though only 48 percent said that it supported their personal development needs.

The VGT organisers also carried out a survey of participants (with 32 responses) into aspects of the year’s programme. Asked about the value of the programme in meeting needs of teachers in “embedding, reviewing and revising the school’s core initiatives”, 91 percent chose “valuable” or “extremely valuable”. The inquiry project was particularly appreciated, with 88 percent rating it as “valuable or extremely valuable” in making a difference to their practice, and 94 percent responding the same way to sharing their reports in subject departments.

Senior school leaders reported impressions of improved teacher practice based on informal observations of teaching and other anecdotal evidence, whilst the most recent ERO report (2012) commended the school for the culture of learning among the teaching staff and noted the discussion and sharing of good teaching practice.

Key Finding 3: Teacher implementation of SOLO

A majority of teachers expressed confidence in their understanding of SOLO and reported using it in their teaching and planning, though a number appear to be still developing effective ways of communicating the intentions to students.

In our final survey of teachers, 69 percent believed that they had a “good” or “very good” understanding of the meaning of the SOLO levels, 62 percent considered that they had a “good” or “very good” understanding of how to design SOLO-based learning intentions in their subject area and 62 percent believed SOLO was “useful” or “very useful” in teaching in their subject area. A number of the written comments argued that SOLO was not applicable in all subjects and circumstances.

Of these teachers, 67 percent reported using SOLO in their planning, teaching and assessing “often” or “very often”, but 40 percent reported only “occasional” or “never” setting tasks with specified levels based on SOLO.

We observed lessons where planning might have been based on SOLO but students were not reminded of the learning goals or how classroom activities related to these, while in other classes teachers were observed to use SOLO concepts consistently in their discourse, emphasising learning goals, deep thinking and self-assessment.

In the end-of-year surveys, students in the project classes were asked about their use of the learning logs. Overall in the second year, of 115 responses from five classes, 27 percent of students reported that they had referred to the learning logs “often”, 30 percent “occasionally”, and 40 percent “never” or “1 or 2 times”. Usefulness for learning was also widely rated—49 percent believed that they were “very helpful” or “helpful” while 37 percent found them “a little helpful” and 15 percent “not helpful”. Less than 3 percent reported the success criteria as “never clear”, while 77 percent found them “usually clear” or “always clear”. Response patterns differed between classes, but it was difficult to identify clear trends, except in the case of one class where students rated all of these three aspects more highly than the other classes; their teacher was known to be skilled in, and enthusiastic about, the use of SOLO.

To gauge students’ practical grasp of the SOLO levels, we asked them to explain the three levels in their own words, and judged the level of understanding on a six-point scale. Overall, we judged that 46 percent demonstrated an adequate and useful understanding and there were again noticeable differences between classes. The students were also asked about ways they thought the learning logs helped them learn; the number of individual responses, and the complexity varied between classes—where we had observed frequent teacher reference to the learning outcomes, students appeared to have more ideas.

An analysis of student responses during small group interviews shows that there was an increase from the first year to the second in the percentages of mainstream students who had a useful understanding of SOLO, and those who perceived it to be useful. This supports our impression that teachers were becoming more effective in using SOLO as the PLD progressed.

The above findings rely on perceptions of both teachers and students. They indicate a positive climate for improved learning outcomes, but are seldom based on objective criteria. Even with the study classes we did not succeed in establishing clearly the criteria for establishing student success in achieving higher order learning goals. Informal judgements of experienced teachers can be reliable, but in a situation of developing new practices this judgement may still be developing.

Key finding 4: Student achievement

All measures available from the school showed overall gains in student achievement since the VGT programme has been in operation. Though a causal link cannot be demonstrated, the VGT programme is likely to be a factor in changing attitudes and practices.

Our limited data suggest that these gains vary widely between individual classes.

The most recent ERO report on the school (Education Review Office, 2012) noted that student achievement across the school had become a key priority, and reported improved NCEA grades in 2011, with those at all levels exceeding national averages; numeracy and literacy also showed improved progress at Years 9 and 10.

The VGT programme was, however, not emphasising the use of SOLO at that time. Although NCEA grades decreased slightly in 2012 they again increased appreciably in 2013, particularly for levels 1 and 2.

Since then teachers’ inquiry project presentations to the professional learning groups demonstrated that teachers were developing ways of measuring the effect of their teaching on students’ learning, and a number reported improved achievement in tests and school exams which they attributed to the use of SOLO.

Average scores from biannual asTTle and Science: Thinking with Evidence tests showed overall statistically significant annual gains for the individual students in Year 10 but we did not have access to long-term data for comparison. The increases could have been affected by the VGT programme, but could also have reflected student maturation, and other factors. We did find that for the three tests the degree of increase of average scores and the effect sizes varied considerably between the different study classes, and some showed no significant increase.

Students experience a number of different subjects and teachers throughout a year which can affect their learning in other settings. Students we observed were also using SOLO in other classes. In addition, two of the classes involved in the second year were high band, with students chosen for that subject on the basis of standardised tests at the end of the previous year and these two classes did demonstrate greater appreciation and use of SOLO, while also having experienced teachers who were comfortable and committed in using SOLO. Other teachers were in the initial stages of learning to apply the approach.

Thus, in the complex school environment it is extremely difficult to demonstrate a causal link between any PLD approach or teaching strategy and student achievement. In this case it is also possible that features of the VGT programme—other than the use of SOLO—could contribute to improved outcomes.

We intended to investigate appropriate subject-specific methods of assessing student levels of thinking but were unable to achieve this. However, we are aware of at least one department where teachers have since worked collaboratively developing SOLO-based assessments, and believe that this holds good prospects for improving information about teaching practice and assessing outcomes.

Classroom implementation of SOLO and e-learning

This section reports on the use of the SOLO taxonomy and e-learning. The findings draw on data from surveys and interviews with the wider teaching staff, as well as those in the immediate project group of teachers and students. Specifically, the section addresses Research Question 2 and Research Question 3.

Key finding 5: Teaching using SOLO

The degree to which teachers incorporated the use of SOLO into their classroom practice varied, but a substantial number of teachers reported benefits from using SOLO.

At the end of the project, 85 percent, or the 58 teachers responding to our questionnaire, agreed—selecting “agree” or “agree strongly”—that the use of SOLO could help students develop deep thinking; 64 percent that “the majority of students in Year 10 find the SOLO taxonomy helpful in their school work”; and 72 percent that SOLO was “readily applicable” to the subjects that they taught.

In teachers’ comments, frequently stated benefits for teacher practice were to clarify their own thinking about goals and the levels of achievement, for structuring units and lessons, providing for more student-centred learning, developing differentiated tasks to cater for the diversity of students, and designing assessment tasks.

These teachers reported that the use of SOLO helped students clarify expectations and criteria, appreciate the distinction between levels of achievement, engage in self and peer assessment and focus more on deep thinking. Teachers liked the simplicity of the SOLO model and the alignment with NCEA assessment levels, which they believe a benefit to students.

Key finding 6: Teaching using e-learning

A majority of teachers believed that it was important for students to use computer technology and that this could be beneficial for their learning, but most only occasionally built this into their lessons.

In the final survey of all teachers in the school, 66 percent agreed—selecting “agree” or “agree strongly”—that it was important for students to use computer technology in school to help their learning; 66 percent agreed that it was helpful for students learning content, though fewer (57 percent) thought it could help them develop reasoning skills. However, 47 percent reported that they “never” or only “occasionally” required students to use computers or mobile devices during the year. In a survey at the beginning of the second year, 5 percent of the 126 students recalled “never” having used computers and 67 percent “occasionally” having used them.

In general computers were used in limited ways: most teachers used computers for class teaching with PowerPoint presentations, video clips, still images and text, while the predominant use by students in class was researching topics on the internet. Other uses of e-learning were much less frequent.

These were the uses reported most frequently by teachers and students in surveys, and that we observed most frequently in the study classrooms. Some teachers, however, designed activities which made use of other software relevant to their subject, including student presentations, making and editing videos, graphics and design, simulations, dictionary software and educational games.

Other teachers integrated activities and resources online, through the school network, Microsoft OneNote or Edmodo (a free cloud-based virtual learning space). Students could access these in class or elsewhere (including at home) and uploaded materials included PowerPoint slides, reference material, research questions, links to video clips and online interactive sites, multiple-choice quizzes to test knowledge, surveys to gather student feedback, and discussion forums.

In interviews, teachers identified a number of pedagogical purposes for using e-learning. One was to enable students to work more independently, freeing up class time for the teacher to move around and focus on quality questioning and giving attention to individual students. Making material available online ahead of a class can also allow the teacher to move more rapidly to the more interesting higher order aspects of the topic, known as “flipping the classroom”. Some teachers believed it was important to help students develop independent research and learning skills by guided searching online, as an important educational outcome.

Teachers frequently referred to students using e-learning outside the classroom. From student surveys we found that over the two years almost all students had an internet-connected computer at home that they could use—in our final survey (with 115 responses), 97 percent had a computer to use at home, with 99 percent on the Internet. Most (96 percent) owned one or more mobile, internet-connected devices such as tablets and smart phones. In this survey all students, except one, used a computer at home for school work, with over a third (37 percent) doing so once or more times a week on average. However, in interviews, a number of students seemed reluctant to do classwork or communicate with their teachers online from home, and this appeared to vary depending on the interest engendered by their teachers.

Both teachers and students pointed out problems associated with e-learning, and these often coincided. Students reported information online was hard to find and often difficult to understand; some students complained that they did not know how to access sites from home or had technical problems, thus missing out on extra help, while others did not want to go online outside school time as more teachers were expecting them to do so more frequently; students were themselves aware of the distractions present when working online, particularly when working in groups. Both teachers and students found that unreliability in the school network and netbooks could be frustrating. Teachers were aware of the time and effort required to develop skills, keep abreast of changes and make choices between the many available applications.

Key finding 7: Factors affecting successful implementation

In this project we developed a model which was useful in considering the influences which affected teacher and student acceptance of the SOLO taxonomy and of e-learning. This model has been derived from theories of technology acceptance, modified for the unique environment of a school, in the light of relevant theories.

In interviews, we found that acceptance and use of these innovations was affected by the social influences of peers and superiors—teachers being a major influence on students—and by attitudes which derived from knowledge and perceptions about the demands of time, ease of use, availability of support and the usefulness of the innovation. Teachers were influenced by their perceptions of the value for student learning. With e-learning technical considerations became important—personal sense of efficacy in using computers, reliability and availability of technical support. Individuals’ personal innovativeness and, for teachers, personality and teaching style were also factors which appeared to affect willingness to make changes in practice. For many teachers, the support and knowledge provided by the VGT programme was an important influence.

Four adaptations of this model are presented graphically in Appendices 1 to 3, portraying factors influencing teachers and students in using SOLO and e-learning.

Key finding 8: SOLO with e-learning

During this exploratory study we were not able to obtain much evidence of SOLO enabling the effective use of e-learning and believe that this was an overambitious expectation. With a heavy focus on developing their skills with SOLO many teachers who were not already confident and skilful in using e-learning could not be expected to meet this additional challenge; teachers with competence in e-learning were able to demonstrate that e-Learning can support SOLO-based outcomes.

Two cases were studied in depth, illustrating different approaches which did combine the use of the SOLO taxonomy with e-learning and other classroom activity.

The first case was a science unit on heat energy. This was planned so that the final outcome of student work was the design of an energy-efficient building. The teacher created a sequence which, following an initial introduction to heat transfer, involved groups of students carrying out online research, designing, performing and reporting a practical experiment, designing a building and explaining the reasoning behind their decisions, and finally presenting this to the whole class. The students thus started with their existing skills and were guided through all SOLO levels, with appropriate scaffolding. The unit made use of the OneNote software: it provided a guide and template for the tasks and a portfolio of each group’s work as they were required to submit each sub task online when it was completed. This also provided the teacher with a convenient method of monitoring the groups’ progress. Other computer software was also used, such as PowerPoint and a 3D drawing program.

The second case was a social studies unit. The teacher had in the past experienced the use of computers in the classroom to be an inefficient process and so had been exploring the use of the Edmodo online “learning space” during the year so that students were able to use the system outside lessons. A number of features of the software were found to be helpful in developing students’ deeper thinking. These included making political cartoons available for regular homework analysis and posting newspaper articles as the basis for student exercises. These tasks were at times uploaded to Edmodo and sometimes students studied the resources in preparation for classroom discussion, which raised the level of thinking in the classroom. Students were also able to seek feedback and guidance from the teacher online, and receive timely responses through the same medium. Another strategy aimed at assisting student learning was the use of online self- or peer assessment before tasks were marked. The teacher also uploaded supplementary resources intended to aid students’ understanding, though on average in a class survey students indicated that they found this the least useful strategy to aid understanding, with 40 percent rating this “not useful”. By comparison, teacher explanation was rated the most useful, with 100 percent ranking it “useful” or “very useful”.

In feedback about the whole year’s work, 75 percent of the class (n = 29), however, agreed, or strongly agreed that the use of Edmodo provided “challenge and interest”. 75% also, agreed, or strongly agreed with the statement that the use of SOLO helped them understand their levels of thinking.

Major implications for practice

The model of PLD adopted in the project school has features which could be usefully adopted and adapted by other secondary schools, with good “buy-in” and positive outcomes for student learning. A more accurate evaluation of success, however, cannot be made for some time and will rely on teachers continuing to build on these initial steps in developing personal expertise and shared practices.

There are indications that the introduction of structured use of the SOLO taxonomy can have positive effects on teaching and learning across the curriculum in a New Zealand secondary school, and has a good fit with the curriculum and national assessment practices. It has the potential to support students’ development of higher order thinking when used by teachers who develop appropriate knowledge and skills.

The project has provided some examples of how SOLO can provide a means to focus the use of e-learning on achieving valuable cognitive outcomes for students, but there is a continuing need for professional learning and institutional support specifically focused on this. In secondary schools, subject-based collaboration offers an approach to develop effective models and resources and we believe that further research in this area would be valuable.

Limitations of the project

This study intended to illustrate the variety of responses to the use of SOLO or e-learning, rather than to catalogue them. It relies heavily on qualitative data and descriptive statistics derived from surveys, and opinions expressed in written surveys and recorded interviews. The accuracy of recording and interpretation of these data from different sources have been cross-checked, where possible, for “triangulation” to give confidence in conclusions. However, the major limitation derives from the non-random selection of respondents.

The surveys of the whole staff were not returned by all teachers; the major survey at the end of the project was answered by 58 teachers—about half of the staff—and we cannot know what bias is introduced by this self-selection. Student opinions from questionnaires and interviews were obtained from the science and social studies classes taking part in the project which were chosen because their teachers were invited and agreed to participate. These students and teachers may not represent all opinions of their peers, but we have no reason to expect a systematic bias. Other teachers were interviewed because of their particular roles in the school, or because they were known to have experiences which illustrated aspects of the study. For these reasons, our interviewees may have been more enthusiastic or experienced than many of their peers, in relation to the subject of the study.

Conclusion

The project was ambitious in its intentions, and the research team encountered a number of setbacks which resulted in modifications and changes of emphasis which the methodology allowed, but which reduced the effectiveness of our intended outcomes. However, a great deal of interesting and illuminative data was gathered, which has enabled us to gain a rich understanding of a complex environment and draw conclusions from those. Much credit, and gratitude, is due to the teachers who so willingly allowed others into their classrooms, and freely gave time to discuss their work in depth.

This report describes an intense and ongoing journey of a school which demonstrates many signs of successful outcomes. The story of this journey will be instructive for teachers in other schools who wish to follow a similar route.

The project raises questions that call for further research. This includes further research on the use of the SOLO taxonomy over a wide range of schools, all curriculum areas and student achievement levels; research into designing valid learning intentions and success criteria; quantitative approaches which can validate teacher judgements and compare teaching methods; and more work to assess the most useful ways of using e-learning. This project defined higher order thinking in terms of SOLO “deep” thinking, but other ways of considering the wide range of higher order thinking skills also need to be explored.

References

Alexander, P. A., Dinsmore, D. L., Fox, E., Grossnickle, E. M., Loughlin, S. M., Maggioni, L., . . . Winters, F. I. (2011). Higher order thinking and knowledge: Domain-general and domain-specific trends and future directions. In S. Schraw & D. H. Robinson (Eds.), Assessment of higher order thinking skills (pp. 47–88). Charlotte, NC: Information Age Publishing.

Alton-Lee, A. (2003). Quality teaching for diverse students in schooling: Best evidence synthesis. Wellington: Ministry of Education. Retrieved from http://www.educationcounts.govt.nz/__data/assets/pdf_file/0019/7705/bes-quality-teaching-diverse.pdf

Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., . . . Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Longman.

Baldwin, R. (2008). Professional learning groups in secondary schools: What makes them effective? Paper presented at the NZARE Conference, November 2008, Palmerston North.

BECTA. (2002). ImpaCT2: The impact of information and communication technologies on pupil learning and attainment. Coventry: British Educational Communications and Technology Agency.

Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcomes). New York, NY: Acadmic Press.

Biggs, J. B. (1991). Student learning in the context of school. In J. B. Biggs (Ed.), Teaching for learning (pp. 7–29). Hawthorn, VIC: Australian Council for Educational Research.

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwhohl, D. R. (Eds.). (1956). Taxonomy of educational objectives: Classification of educational goals— Handbook I cognitive domain. New York: David McKay.

Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141-178.

Brown, G. T. L., Irving, E., & Keegan, P. (2008). An introduction to educational assessment, measurment and evaluation (2nd ed.). Auckland: Pearson Education.

Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15-42.

Cox, M., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). ICT and pedagogy. London: Department of Education and Skills. Retrieved from http://www.bee-it.co.uk/Guidance%20Docs/Becta%20Files/Publications/46.%20Research%20report%20 ICT%20and%20pedagogy%20-%20a%20review%20of%20the%20research%20literature.pdf

Crook, C., Harrison, C., Farrington-Flint, L., Tomás, C., & Underwood, J. (2010). The impact of technology: Value-added classroom practice final report. Coventry: University of Nottingham & Nottingham Trent University. Retrieved from http://www.elearning.tki.org.nz/ content/download/624/4042/file/The%20impact%20of%20technology_Value%20added%20classroom%20practice.docx Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319-340.

Davis, F. D., & Venkatesh, V. (1996). A critical assessment of potential measurement biases in the technology acceptance model: Three experiments. Int. J. Human-Computer Studies, 45, 19-45.

Dyke, M., Conole, G., Ravenscroft, A., & de Freitas, S. (2007). Learning theory and its application to e-learning. In G. Conole & M. Oliver (Eds.), Contemporary perspectives in e-learning research: Themes, methods and impact on practice (pp. 82-97). Abingdon: Routledge.

Education Review Office. (2009). Managing professional learning and development in secondary schools. Wellington: Education Review Office. Retrieved from http://ero.govt.nz/content/download/52367/875785/version/10/file/pld-sec-may09.pdf

Education Review Office. (2010). Howick College Education Review Report. Wellington: Education Review Office. Retrieved from http:// www.ero.govt.nz/Early-Childhood-School-Reports/School-Reports/Howick-College-08-09-2010

Education Review Office. (2012). Howick College Education Review Report. Wellington: Education Review Office. Retrieved from http:// www.ero.govt.nz/Early-Childhood-School-Reports/School-Reports/Howick-College-15-10-2012

Eng, T. S. (2005). The impact of ICT on learning: A review of research. [J]. International Education Journal, 6(5), 635-650.

Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London: Routledge.

Hattie, J. A. C., & Brown, G. T. L. (2004). Cognitive processes in asTTle: The SOLO taxonomy. Auckland: The University of Auckland/Ministry of Education.

Hattie, J. A. C., & Purdie, N. (1994). Using the SOLO taxonomy to classify test items. Unpublished manuscript, University of Western Australia, Graduate School of Education, Perth, Aus.

Hattie, J. A. C., & Purdie, N. (1998). The SOLO model: Addressing fundamental measurement issues. In B. Dart & G. Boulton-Lewis (Eds.), Teaching and learning in higher education, (pp. 72-101). Melbourne: ACER Press.

Margaryan, A. (2008). Supporting instructors in innovation: A three-component approach. Journal of Workplace Learning, 20(6), 400-415. Martin, S. (2011). Using SOLO as a framework for teaching. A case study in maximising achievement in science. Invercargill: Essential Resources Educational Publishers.

Ministry of Education. (2007). The New Zealand Curriculum. Wellington: Learning Media.

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record., 108(6), 1017-1054.

New Zealand Council for Educational Research (2010). Science: thinking with evidence [Tests]. Wellington: NZCER.

Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: Introduction. Educational Psychologist, 39(4), 199-201.

Schraw, G., McCrudden, M., Lehman, S., & Hoffman, B. (2011). An overview of thinking skills. In G. Schraw & D. R. Robinson (Eds.), Assessment of higher order thinking skills (pp. 19-45). Charlotte, NC: Information Age Publishing.

Smarkola, C. (2011). A mixed-methodological trechnology adoptionb study: Cognitive belief-behavioral model assessments in predicting computer usage factors in the classroom. In T. Teo (Ed.), Technology acceptance in education: Research and issues (pp. 9–41). Rotterdam: Sense Publishers.

Smith, T. W., & Colby, S. A. (2007). Teaching for deep learning. The Clearing House, 80(5), 205-210.

Taylor, S., & Todd, P. A. (1995). Understanding information technology usage: A test of competing models. Information Systems Research, 6(2), 144-176.

Teo, T. (Ed.). (2011). Technology acceptance in educationL Research and issues. Rotterdam: Sense Publishers.

Timperley, H. (2008). Teacher professional learning and development. Brussels: International Academy of Education. Retrieved from http://www.ibe.unesco.org/fileadmin/user_upload/Publications/Educational_Practices/EdPractices_18.pdf

Timperley, H., Wilson, A., Barrar, H., & Fung, I. (2007). Teacher professional learning and development. Wellington: Ministry of Education. Venkatesh, V., & Davis, F. D. (1996). A model of the antecedents of perceived ease of use: Development and test. Decision Sciences, 27(3), 141-481.

Wegerif, R. (2002). Literature review in thinking skills, technology and learning. Slough: NESTA Future Lab.

Wright, N. (2010). e-Learning and implications for New Zealand schools: A literature review. Wellington: Ministry of Education. Retrieved from http://www.educationcounts.govt.nz/__data/assets/pdf_file/0006/77667/948_ELearnLitReview.pdf

Research Team

Anthony Hunt (principal investigator) is an Honorary Research Fellow in the School of Learning, Development and Professional Practice at the University of Auckland, where he was formerly a Senior Lecturer. His research interests are in e-learning, educational technology, teacher education and secondary education. He is a former secondary school teacher and educational media specialist.

Frank Walton (researcher) is a Professional Teaching Fellow, and formerly a Lecturer, in the School of Learning, Development and Professional Practice at the University of Auckland. His research has been into student attitudes to assessments, teacher education and physics teaching. He is a former science teacher, head of department, dean and deputy principal.

Stephen Martin (lead teacher–researcher) is a secondary school Head of Junior Science. His particular interest is in the use of the SOLO taxonomy and computer technology in science teaching and he has won a Prime Minister’s Prize for Science Teaching and the Microsoft Distinguished Teacher Award.

Dr Mavis Haigh (advisor on professional development and methodology) is an Associate Professor in Education, School of Learning, Development and Professional Practice at the University of Auckland. Her major research interests are in teacher education practice and science education.

Dr S. Earl Irving (advisor on assessment and evaluation) is a Senior Research Fellow in the Faculty of Education at the University of Auckland. His major research interest is in assessment

Appendix 1 Influences on teacher use of SOLO.

Adapted from Taylor & Todd, 1995; Davis & Venkatesh, 1996.

 

Appendix 2 Influences on student use of SOLO.

Adapted from Taylor & Todd, 1995; Davis & Venkatesh, 1996.

 

Appendix 3 Influences on teacher use of e-learning.

Adapted from Taylor & Todd, 1995; Davis & Venkatesh, 1996.