What works and what doesn't

This site is mainly about your own individual practice as a teacher, and as such it tries to take into account your particular circumstances, such as the students you teach (assumed largely to be over school-age), your subject, your setting (school, college, university, work-based or informal adult education).

It recognises that it is difficult and even unreasonable to generalise, but we ought to set alongside this the results of very generalised research in the form of meta-analyses. Meta-analysis is more commonly found in medicine and epidemiology that in education, and it has its limitations, but it can also make very strong points.

It is simply the technique of searching for all the existing research reports on a particular issue, and combining them to get an overall result. A moment's thought—particularly if you know anything about research methods—will tell you that this is fraught with problems. Has the issue been defined in exactly the same way by all researchers? If not, how do you adjust the results? If the research is on the interaction between two variables (say, the use of ICT with disaffected learners), what category do you put it in? Do you rate the validity and reliability of the findings, or just assume that if it has been published, it must be right? And what about all the unpublished research which did not make it because it questioned the conventional wisdom of the day? And so on.

However, its proponents argue that many of these problems cancel each other out when you take a large enough research base, and that others can be mitigated by the choice of the meta-assessment tool.

The most prominent meta-meta-analyst in education is probably John Hattie, whose work draws on upwards of half a million items of research.

Hattie's common denominator

In common with standard meta-analysis practice, Hattie's bottom line is the "effect size". An effect size of "1" indicates that a particular approach to teaching or technique advanced the learning of the students in the study by one standard deviation above the mean. OK, that's rather technical:

So an effect size of "1" is very good.

• Anything better is well worth considerable effort to achieve.
• Much less deserves less effort, and is marginal.

Given that we can't do everything, where should we concentrate our efforts?

Hattie's findings can be summarised in this chart:

[Based on Hattie, 2003]

OK, so the majority of innovations and methods "work", according to the meta-analysis (bearing in mind that unless substantial funding and contractual obligations to publish were involved, most researchers would not be inclined to publish negative findings).

But which work really well, and which have such a marginal effect that it is not worth the bother? That is the critical question. Here is the answer! (Follow links for comment; and please note that the comments are my spin on the topics, in relation to post-compulsory education, not Hattie's)

(edited from Hattie, 2003)

Feedback

(almost three times the average effect size)

It can well be argued that the older students get, the more cognitive feedback matters. As the quotation above indicates, there is a reinforcement component to feedback which helps to shape behaviour, but cognitive feedback also has clear information in it, which helps the student to correct errors and polish performance.

Given that Hattie's meta-analyses are very general, it might be expected that this influence is greatest in skill-based learning, not only in the psycho-motor domain, but also in other convergent areas such as language learning and mathematics. While still important, its influence on learning in the humanities and creative subjects may be less direct.   [Back to table]

Students' prior cognitive ability

(Two and a half times the average effect size)

This is of course largely beyond our control. According to Hattie what students bring to their learning accounts for 50% of the variation of achievement; but even so, 30% of the variation is still down to teaching variables. On this topic, see "Intelligence" and "Student baggage".  [Back to table]

Instructional quality

(Two and a half times the average effect size)

So 30% of what makes a difference is in the hands of teachers. Hattie emphasises that teachers make a difference, and also goes on to point out the differences between expert teachers and merely experienced ones, based on empirical research, for once:

He goes into much more detail than it is possible to pursue here, but it is well worth reading. For my own take on the nature of expertise, which is I think reasonably consistent with Hattie and Jaeger's work, see this paper. [Back to table]

Direct instruction

(Twice the average effect size)

This is an influence with an effect-size of less than one, but still double the average; but it is also more controversial. "Direct instruction" is what we often term "teacher-centred" rather than "student-centred" teaching; it is traditional teaching rather than discovery learning, for example. There is substantial evidence to support this finding in compulsory schooling, but perhaps rather less in the post-compulsory sector. This is an area in which considerations such as the nature of the assessments used to generate the initial data, and indeed their cultural settings, might make a considerable difference. It may indeed sit slightly uncomfortably with some of the observed features of the practice of expert teachers.

Going back to research in the '70s (old enough not to reference) the argument may be that indirect, student-centred, teaching is easy to do, but difficult to do well. We might also take into account that the meta-analysis does not discriminate between cultural settings, and in some of those (particularly in what Biggs calls "Confucian-heritage" countries) anything other than direct instruction is regarded as perverse. [Back to table]

Remediation/feedback

(One and a half times the average effect size)

I don't know how the reported effect-size relates to the overall figure for "feedback" or quite how "remediation" is construed, but I imagine it refers to specific advice on how to improve performance on an assessment. [Back to table]

Students' disposition to learn

(One and a half times the average effect size)

Also known as "motivation". But remember that the vast majority of students/pupils covered in the original research had little alternative but to attend school. "Post-compulsory" education is by definition not overtly compulsory, but that does not necessarily mean that all our students are enthusiastic about learning (in case you had not noticed). The different circumstances can result in different results.

Highly motivated students do better, if only because they are prepared to put time into learning, but whether the moderately and minimally motivated students do worse on a linear scale is unproven.  [Back to table]

Advance organisers

(Just below average effect size)

OK—I devoted a page to them (or at least part of one), and they are less effective than average. On the other hand, they involve very little effort, and every little helps.   [Back to table]

Computer-assisted instruction

(Only three-quarters of average effect size)

I read into this, and others of Hattie's papers, that much of the data was gathered starting in 1987. Pre-Windows(™ blah, blah) pre-internet; things are changing rapidly on this front, but I still have reservations, this site notwithstanding (lovely word! What does it mean?)   [Back to table]

Individualisation

(One-third of average effect size)

I told you so! The primary manifestation of individualisation is concern with "learning styles". The issue is being plugged all over the place, including by official bodies such as Ofsted and the Department for Education and Skills. As the Coffield et al (204) study shows, most of the much-hyped and purported assessment schemes have neither validity nor reliability, and the effort required to take them into account is clearly disproportionate to the pay-off.

On the other hand, one of Hattie's characteristics of expert teachers is that they have high respect for students" (Hattie, 2003:8). That, in my book, involves getting to know them, and getting a feel for how they learn, and responding to that, rather than to some mechanistic score on a dubious scale.  [Back to table]

Behavioural objectives

(One-third of average effect size)

If anything performs less well than individualisation, it is an insistence on behavioural objectives. Again, the data refers mainly to school learning, so the picture may be different in the post-compulsory sector, particularly if the focus is on work-related training. There is nothing wrong—and a lot right—with being clear about what you are planning to teach, of course; but a rigid insistence on merely behavioural objectives probably results in sterile, boring teaching, the effect of which outweighs any gain—particularly when you take into account the mental and epistemological contortions necessary to find behavioural correlates for an appreciation of "King Lear". (Shorn of the jargon; they are more hassle than they are worth.)

Reference

HATTIE J (1992) "What Works in Special Education" Presentation to the Special Education Conference, May 1992 [On-line; Acrobat file] NZ Available: http://www.arts.auckland.ac.nz/FileGet.cfm?ID=C302783E-1243-4B65-AC54-B7FD4A5B7EF7 Accessed 1 October 2005

For an other major meta-analysis, see Marzano, R. J., Pickering, D. J., & Pollock, J. E. (2001). Classroom instruction that works: Research-based strategies for increasing student achievement. Alexandria, VA: Association for Supervision and Curriculum Development, and the study underpinning it; Marzano, R. J. (1998). A theory-based meta-analysis of research on instruction. Aurora, CO: Mid-continent Research for Education and Learning

Hattie's and Marzano's work forms the basis of Geoff Petty's highly recommended book Evidence-based Teaching; a practical approach Cheltenham; Nelson Thornes, 2006

On meta-analysis in educational research

All about effect size

Hattie in a nutshell in his own words

Note that Hattie is working across all educational settings, all ages and all cultures.