Checklist for reporting a piece of empirical work on language

A checklist of some important things often included when reporting a piece of classical quantitative empirical research professionally in research reports and journal articles,... and assignments and dissertations. Much is similar for qualitative writeups too.

Using a professional structure and headings makes your work look so much more impressive!

o Information given here does NOT apply where your assignment or dissertation is purely sourced from reading about what other people say or have researched, or from your own experience, introspection etc. Follow the departmental ‘essay’ guidelines for that

o Information given here does NOT apply where your assignment is doing exercises of some sort (e.g. phonetic transcription or drawing syntactic trees)

o Information given here DOES apply wherever a dissertation or assignment has involved you gathering and analysing data from people, or even from texts, material previously taped by others, corpora etc. Especially where you have something to express as numbers at the end. I.e. wherever you have counted things in documents, interviewed people, given questionnaires or tests, etc. Anything called a project is likely to fall in this category. I.e. what you are writing is a report of (empirical) research, rather than an essay.

The list of things to report is of course a list of the things one should do as well!

Broadly any data gathering project has three parts, reflected in the three major parts of the write-up…. The checklist below elaborates on this nutshell:

o The Background. What you are researching, your research questions etc., why it is of interest and where it ‘comes from’ in other people’s theories or research studies

o The Method. How you gather the data to carry the project out

o The Results. What you found and what it means, in relation to what was said in Background

One does not necessarily write the report from start to finish in the logical order it has at the end, as presented here. Often people read some background sources on their research topic, write some research questions, work on the method of data gathering, then gather the data, and only then, while looking at the results, actually write the background review and the description of method.

Some of what I list may not be entirely comprehensible without attending LG575 or similar, or further reference to research methods books.

I cannot of course cover every eventuality, so watch for what you may need to mention in addition to what is below! Note also that everything listed here will not of course be relevant in every instance where someone does a piece of empirical work: generally the more reactive and experimental the methods used, the more there will be to say. Everything here would certainly not be appropriate in a non-classical research study (e.g. an ethnographic study or one relying for data purely on expert linguistic introspection).

Where, as often, several variables are measured from one lot of data, and/or more than one question/ hypothesis investigated, much of the following may all have to be gone through separately for each.

Finally, though sections below, reflect what seems a reasonable consensus on 'good practice' as far as labelling of sections goes and their contents, there is considerable variation on the ground in reports you will read both in how sections are labelled and what is mentioned in each (and indeed in how thorough the overall account is!). For APA conventions, see appendix in Christensen, or the Fifth Edition (2001) of the APA Publication Manual (It does not seem to be available online).

! marks common mistakes people make (other than of course missing out whole chunks, jumbled order, and general inaccuracy, vagueness, typos etc.) - mistakes in what is reported, and maybe also in what was done in the research.

******

TITLE

· containing all the relevant keywords to identify the topic

! Too long - can't put everything in the title

! Too vague and general, perhaps overclaiming the scope of the work

ABSTRACT

· a summary of the whole thing

! You give what is really an introduction, missing out what the results were

! You tell us what each section of the writeup is going to talk about (e.g. 'In the third section we will describe the method'), not briefly what you did (e.g. 'The method we used was...')

! You include wording that refers forward like ‘… as we shall see…’. Again it is not an introduction. It should read as referring back to the whole completed project

INTRODUCTION

· what the topic is, in brief

! You start telling us a lot of detail about the method and your results at this point

! Detailed research questions and hypotheses… premature to give them here

· reasons for doing the work, e.g.

importance as a research topic in itself, in the context of current knowledge in the relevant field. This entails saying a bit about what general areas of ELT, linguistics or whatever the study relates to

importance for local situation of researcher (esp. if teacher). This entails possibly a detailed description of what that situation or context is (e.g. if your study is on writing, then how that is taught throughout the educational system etc.)

! The research is presented as having interest only for the researcher's school/country. Classical research needs to be presented as having wider implications

! Long account of 'problems of teaching in my country'...none of which turn out to be the subject of your research

! Unfounded generalisations with no sources like ‘standards of English have become poor in recent years’

! Multiple sections with titles like Importance of the study, Significance of the study, which are really not differentiatedin content

· outline of what will come in the chapters/sections that follow

· maybe brief definitions of some key terms to be used later

REVIEW OF LITERATURE covering these things but not necessarily in this order

· review and critique of previous research in the same general area (shortcomings of methods or argumentation previously used, new areas to look at suggested by previous results). Their findings, esp. with respect to variables you are interested in. This should at every point be explicitly connected to your specific project.

! The background review reads like an MA survey essay on some area of investigation, cataloguing other people's studies, with no comparison of them with each other, or critique, and no use explicitly made of them to connect to your own work by showing what they suggested for it.

! Too broad… need to focus rapidly on just what bits of articles and books are relevant to your study

! You report previous work as ‘important’ when actually it has no relevance to your own research (though it may be highly regarded in the field generally).

! You retail other people’s criticisms of each other’s research but do not resolve opposing views, argue your own view, or draw implications for your research.

! Review feels like the literature got on top of you, rather than that you are on top of the literature, and is too long (more than a third of the writeup)

! You mention the results of your own later research in your review

! see also http://privatewww.essex.ac.uk/~scholp/litrevsarc.htm

· theoretical background(s) or 'models' from which the ideas come (both pure and applied linguistic, and maybe in psychology, sociology...), or which you hope to shed light on

! Ostrich: you stick with one model you have learnt about and don't cover the rival theories or look in other disciplines that have something to say.

· Discussion of definitions of key terms… esp, vague ones (e.g. in ELT ‘communicative’, ‘function’, ‘strategy’, ‘task’ etc….) where you disentangle different opinions of scholars

! You catalogue a lot of people’s definitions of X but fail to show where they agree/differ or which one you are adopting for your work and why.

· a review of methods used previously to gather relevant data, justifying yours (e.g. merits of interviews versus questionnaires etc.). Better here than in Method chapter/section if it is substantial.

RESEARCH QUESTIONS/HYPOTHESES

· the specific research questions/hypotheses this piece of research aims to deal with, mentioning the main variables in the study. Sometimes expressed as ‘aims’ or ‘purposes’.

! The hypotheses/questions seem to have no connection with the literature review, or the reader has to work hard to find exactly what prompted them in a mass of review.

! Hypotheses vaguely worded, e.g. what is being compared with what exactly? If you say one group will ‘do better in the cloze test’ then make sure you say ‘than who’ or ‘than in what other test’

! Hypotheses unclear as to which ones the researcher actually expects to be confirmed. E.g. if they are given as null hypotheses, is it the null hypothesis that the researcher actually expects to be confirmed or not?

METHOD

a) Cases/Subjects/Informants and Sampling

· explanatory variables (EVs) that consist of groupings of cases that are required for the comparisons that the study has to make may be mentioned here, though the measurements or manipulations involved to define them may be described elsewhere -

groupings by inherent features of subjects, such as age group, L1, sex, type of school attended, class in school

groupings related to conditions imposed by an intervention or experimental regime, such as a division between subjects taught reading by one method and others taught by another:

how subjects were assigned to such groups – intact classes? random assignment?

· for each group of cases involved –

how chosen:

random sample - what procedure for random selection used

quasi-random - how chosen (friends?...)

purposely selected individual(s) - on what basis

whether paid or otherwise rewarded

permissions obtained – informed consent (e.g. from parents, school)

whether subjects such as schoolchildren were free to decline to participate if they did not wish to (ethics)

how many

what sort of people (common features of the populations sampled), hence what wider populations the chosen subjects can be claimed to be representative of, e.g.:

native language, dialect etc., target language

age, social class, sex, school grade etc.

from what institution (school, university, hospital, etc.)

English learning/teaching history

subject variables that were controlled (i.e. deliberately eliminated from having any potential effect) and how each measured (distinguish subject variables used as EVs), e.g.:

intelligence

social class

reading age

etc.

any subjects rejected for any reason

! Insufficient information about what the subjects have in common and what population they could reasonably be regarded as representative of.

! Overclaiming the populations represented

! Just saying you used random samples, as a matter of routine, when obviously they weren’t

! Lack of clarity over what aspects of the cases you are controlling (eliminating), which you are comparing as variables in the study

b) Variables and their Observation/Measurement/Manipulation using Instruments

For each variable of any sort that is involved – dependent/response variable (DV), or one used to establish groups of subjects (EV), or one imposed on subjects to create conditions (independent variable, such as stimuli of three types to be responded to in an experiment, or a special teaching intervention), or one used just to screen subjects (control). How they were all manipulated or measured; instruments:

· Materials / Stimuli

standard published tests/questionnaires/observation schedules/ checklists etc. used, or own measures

how items, tests, questionnaires etc. were selected or constructed or adapted (random/deliberate, from what sources)

language or non-linguistic form of stimuli and/or responses

procedure for cloze gap choice, item selection, checklist point selection or the like

form of print/visual display; person whose voice was taped

response mode – written, spoken etc.; multiple choice, open choice…

variables controlled in choice of items, texts etc. (length, frequency, familiarity, readability etc. etc.)

distractors/fillers etc. introduced

incorporation of open/unstructured questions/items

realia - pictures, dolls etc.

topics / texts / task types etc.; interview questions

how many items used to measure a given variable or condition, and overall

how stimuli / materials etc. for different conditions were distinguished from each other; how they were ordered

equivalent, but different, versions of the same text, stimuli or whatever required for different groups

materials needed for teaching intervention, if any (texts, exercises, websites, practice materials, where from, how chosen, why created how they were…)

any piloting of materials and revision by item analysis before the current investigation .. how done and what was learnt from it and changed in the main study

! Insufficient justification for using the questionnaire, test or whatever that you did, rather than an alternative instrument

! Lack of detailed account of where individual items, texts etc. used came from, or how they were made, or why they were needed, given the research questions

! Pilot inadequately described, and what was learnt from it and improved in the main study not reported

! In teaching interventions, failure to be clear how exactly the ‘new’ materials being used differ from the usual ones, and if there is a control group, failure to be clear what materials they are using.

· Procedure

the measurer, or rater, judge, assessor, teacher ..etc.:

whether a proxy for the real measurer/researcher

sex, age, race, etc.

status (teacher/researcher etc.)

institution attached to

how introduced or known to subjects

the situation:

postal / take-away or done on the spot

place where measurements obtained (lab/home/school, description of room etc.)

time of day

activity subjects would otherwise have been doing (e.g. leisure, class...)

onlookers (peers, mother etc.)

any manipulation or separate attention to the situation to create different conditions – e.g. formal vs informal, in class versus out of class

the measurement itself, and the intervention, or imposition of different treatments, if there was any:

how much was explained to subjects about the real purpose of the research and what would be done with the data before or after; any risks/benefits for participants; assurance of confidentiality (ethics)

instructions given (quote verbatim), in what language

practice items or training in the instruments provided first

administration to cases singly or in groups together

measurer present or not

exact sequence of events - what equipment was turned on and off when, order of presentation of stimuli, different tasks etc. Maybe a table showing the timing and order in which procedures for various tests and treatments or teaching sessions were performed

exact task of subjects, learning schedule etc.

any separate procedures performed to create different conditions/treatments, e.g. different teaching provided to two groups, different instructions, etc.

disguise / distraction introduced or actual misdirection/lying about the task (ethical issues)

precautions taken against collusion

help/feedback provided during task (e.g. any correction after each item, second chances, encouragement etc.)

encouragement given to guess or not when in difficulty

time allowed per item/overall

opportunity for subjects to comment after on the task

any piloting of any aspects of procedure before the main data gathering

! Lack of detail: in principle the account should be explicit and detailed enough to enable someone else to repeat exactly what you or the investigator did by following it.

! Impression given that everything went perfectly when in practice we know it rarely does

! Account of how things were measured mixed up unclearly with account of an intervention

! In teaching interventions, failure to be clear how exactly the ‘new’ procedure being used differs from the usual one students had before, and in what respects it is identical; and if there is a control group, failure to be clear what procedure was followed there and where exactly it differed from the experimental one.

· Apparatus / Equipment

make and specification of anything like these used to present material, record responses etc:

tape / cassette recorder and player and microphone, earphones

language laboratory

video camera and player

earphones

slide projector

voice operated relay (to detect start of spoken response to a stimulus)

timer

computer and monitor (keys specially labelled?) and software used either to capture data or analyse it

· Scoring / Categorisation / Coding / Data Analysis

kind of scale on which each variable is recorded (interval numerical scores, categories of some sort etc.)

how the data was transcribed, translated etc., if relevant

what was counted as what when categorising etc. - coding of open responses to questionnaire items, observations, unstructured interview data, think aloud protocols etc.. Source of coding scheme - standard one used by others or made up/adapted in the light of data obtained

any data omitted and why (unanalysable, aberrant, subject uncooperative...)

how totals and % scores were derived

any procedure for combining scores from subtests or sets of items etc. to produce overall scores for each person

any standardisation of scores

how missing values were dealt with - non-responses, absences

reliability checks performed:

kind of check (test-retest, internal, etc.) and resulting coefficient of reliability

timing of test-retest check, on what cases, etc.

% of tests/protocols rescored by another measurer and who that measurer was, whether he/she knew purpose of measurement, his/her training etc.

validity checks - content, concurrent etc.

what was done to ensure anonymity of cases for reporting, and keep any information on real identities secure (ethics)

! Coding of data was done just by the researcher, not crosschecked by anyone else

! Linguistic aspects are not labelled with proper linguistic labels

! Things are counted as errors that are acceptable to an NS of English

! No account given of how the categories used to classify the data were arrived at

! Unclear how exactly the figures that appear in later tables were derived from the measures used

c) Design

· overall specification of how all the (groups) of cases and variables of all sorts interlock:

which are explanatory or independent variables, dependent variables, and controlled (eliminated) variables, with what values

which are experimental 'made' variables (including teacher interventions), which just 'measured' ex post facto

which variables are within-subject 'conditions' or ‘treatments’ (matched groups, repeated measures), which between-subjects 'groupings' (independent groups)

labels like:

single-subject or case study

cross-sectional or between-groups study

longitudinal study

correlational

factorial

multivariate

· where a lot of variables and conditions are involved, esp. in experiments, the overall plan and order in which measurements were made and stimuli presented, interventions made etc.; counterbalancing to take care of order effects, and so on

! Design is not mentioned, and it is unclear just how many variables there were, or which were potentially explaining which etc.

! Unclear which variables were controlled, in the sense of eliminated, as against which were the focus of research

! Calling every empirical study an 'experiment'.

RESULTS

· often usefully organised around each research hypothesis or question in turn

! The account follows the types of statistic used, e.g. all descriptive statistics like averages given first, then all the ANOVA test results, when in fact this means that information related to the same research question gets fragmented. The stats are the servant not the master.

! Account slavishly follows the instruments, e g. all questionnaire results first, then all interview results on the same issues, etc. This may make sense but if different instruments cover the same ground it can be more informative to combine the accounts. E.g. cover what all the instruments said about one thing in one place

· graphs and tables showing results for groups and conditions and distribution shapes

! Graphs and tables have unclear or abbreviated labels like 'NNSPC1' such as maybe you had to use for the computer, but which should not carry through to the write-up

! Graphs and tables not numbered and not referred to by number from the text (‘See below’ is no good)

· summary and dispersion statistics for groups and conditions:

means, percentages etc.

SDs, variances etc.

· descriptive measures and inferential tests for comparisons of results of different groups, conditions, variables etc.:

differences between means, percentages etc. of different groups or conditions

correlations, associations etc.

significance tests for differences and correlations

! Results are said to be 'significant' when they aren't so in the statistical sense, or a sig test has not been performed.

· qualitative account of things not reducible to figures, often in the form of summaries of what was said in interviews, or profiles of selected best and worst cases

! Insufficient examples cited from the data, where the data consisted of recordings, compositions etc. So the evidence is not supplied for the conclusions

· interpretation of all the above, as you go along, in the light of the original hypotheses / questions under investigation

! DIY: masses of figures/tables/graphs are given, but not much explanation of what they are telling us. Reader is bombed with detail.

! There are thirty pages of results without explanation, then thirty pages of the explanation starting over, by which time the reader has forgotten what the first results were.

! The interpretation of results does not connect back to the original hypotheses.

! Pseudo-explanations of results like 'The hypothesis was not met because there was not much difference in performance' (Why?!)

! Many new references are made to other research, which should have been in the original literature review

! No use is made of whatever was reviewed in the literature review. No comparisons with previous studies’ results.

· interesting things you noticed in the data that it had not been planned to look for - exploratory

DISCUSSION

· integrated account of the findings, pulling together what may have been covered in different places in the preceding coverage of Results

· wider implications of the results in relation to:

previous work already discussed above

possible future research work by oneself or someone else

theory – has the work added to or altered anything?

practical applications (e.g. to teaching, therapy)

! There is only a feeble gesture in two paragraphs to give 'practical implications for teaching' or the like. Vague statements that nobody could make any practical use of.

! Under the name of implications various perfectly sensible recommendations are made about teaching or whatever, but they are not derived from your study. They derive from the general literature on the subject. Implications should clearly come from YOUR findings.

! Overgeneralisation of findings to 'all learners' or the like, when only University English majors in one city in your country were studied

! Failure to show clearly what was the new information from this study and what was confirming what had been found elsewhere (lit review).

· what might better have been done differently, with hindsight

! Overconfidence in findings, despite shortcomings of the method

CONCLUSION

· brief underlining of main points again. Often combined with Discussion.

! Tedious repetition of what has already been said three times over.

! The summary is only of the factual results, not of your interpretation and discussion of what the results ‘mean’

REFERENCES/BIBLIOGRAPHY

· in the right format for the recipient (see guide in course booklet, targeted journal etc.)

! There are references in the text which are not in the bibliography, or vice versa.

! Names of authors are spelt differently/wrongly in different places.

! References not cited in the text in an accepted form or in different forms in different places

! Confusion of first and second name, such as reference to the works of Noam.

APPENDICES

· tables of all the individual scores of all the individual cases measured, properly labelled

· copies of any teaching materials, lesson plans etc. used

· one copy of every test, questionnaire, observation sheet, checklist etc. that was used, in English and in the language in which it was administered, if different

· samples of actual protocols (not all of them, unless they are very few). I.e. filled in questionnaires, compositions the subjects wrote, transcriptions of taped interviews and think aloud sessions....

! Far too much put in. One does not need every completed test script or questionnaire, or transcription of 20 hours of tape.

rev. 11