# Designing Research Instruments Research Methods in Education Session 2 Quantitative Approaches The (quasi) experimental approach Experiments are used to ascertain the presence, type and degree of a causal relationship between two variables. At its simplest, an experiment involves making

changes in the value of one variable - the independent variable - and observing the effect on another variable the dependent variable. Experiments usually involve The identification and control of variables Observation and measurement (Often) the use of controls to compare circumstances

So lets do an experiment! Cut out one spinner and see how it flies. Design a simple experiment to investigate changing one variable which you think will affect its flight. Collect some results to present to the group. Identify possible sources of

error in your experiment and suggest how it might be improved. The origins of experimental approaches in agriculture and botany borrowing scientific methodology in the context of growing crops, where all or at least most variables could be controlled; all can be held constant while variations are introduced one at a time to investigate the effect upon

subsequent growth. A statistical method of analysing the data and attributing significance can then be employed. Researchers believed that the same methodology and analysis could be applied to educational research. Issues which arise in the context of educational research include the extent to which pupils behave like plants, i.e. can they be thought to be identical?, and can all variables be controlled?

Experimental design Suppose you wish to investigate the effect of a teaching programme upon a group of pupils. The simplest design you could adopt is: Pre-test . (X) .. Post-test where pre- and post-tests are the same test and (X) is the treatment applied to the group e.g. the teaching method.

Results of the Simplest Design Assuming from the results that the distribution of data was reasonably symmetrical, and that they were reasonably separated, this might lead you to believe that a real difference exists between the pre-test results and the post-test results, which might have been caused by the teaching programme. However Was it the teaching method (X) or was it something

else like the ageing of the pupils that caused the change in test scores? Was there any Hawthorne effect? What is the effect of the pre-test upon the post-test? A statistical test (t-test) could be applied to demonstrate the significance of the difference between pre- and post-test results.

What does the frequency distribution look like?. A stronger design Experimental group: Pre-test (X) Post-test Control group: Pre-test . Post-test The pre-test for both groups should be administered at the same time. The post-test for both groups should be

administered at the same time. The control group does not experience the treatment (X). Some questions Are the groups identical? Ethically, is it right to experiment on pupils where possibly you may deprive some of the experience (X)? Still possible for their to be a Hawthorne effect with

the control group? Have you taken into account all the variables? o the time of day of the teaching o the age and ability of the pupils o the effectiveness of the teacher A more sophisticated experimental design: Experimental Groups: Teacher APre-test (X) Post-test

Teacher B Pre-test (X) Post-test Teacher C Pre-test (X) Post-test Control Group: Pre-test ... Post-test Significance (a dangerous word!) Significance in the statistical sense should always be quoted at a particular level.

Levels most commonly quoted are 5% (or 0.05) or 1% (or 0.01). To say that a relationship is significant at the 5% level means that the probability of that relationship happening by chance is less than or equal to 5 in 100: ie. the probability of it happening because of a particular reason is greater than 95 in 100.

Significance at the 5% level therefore indicates a stronger relationship than, e.g., significance at the 10% level. Significance levels are used with, e.g., the t-test; a value for t is calculated and then a table is referred to in order to determine the level at which that value of t is significant, given the size of population under consideration. Correlational research

Sometimes it is important to know how one set of figures compares with another set (e.g. maths results and English results for the same set of students). The degree of agreement between two variables can be measured by calculating the coefficient of correlation (r). The value of this quality always lies between -1 and +1. Two sets of figures can also be shown diagrammatically, by plotting them on the two axes of a graph. Its bananas! Suppose we wanted to look at the relationship between birth-rate

and banana production on an island. What do we make of the data? Birth rate x x x x x

x x x x x x

Banana production x Its bananas! What about if they looked like this? Birth rate x x x

x x x x x

x x Banana production x x Brexit and educational attainment

Brexit and age Correlation and Causality Correlation may be high, but what does this tell us about cause and effect? Does one cause the other? If so, which way round? Be careful about assuming causality in any relationship: high correlation does not necessarily mean that there is a causal

relationship though it may indicate a possible relationship that merits further investigation Surveys Typically surveys gather data with the intention of describing the nature of existing conditions; comparing existing conditions with a previously

documented situation; comparing existing conditions between different groups; determining the relationship between different events. Before starting a survey it is necessary to consider the exact purpose of the enquiry; the population to be investigated; the most appropriate sampling techniques (if any); the resources available to the researcher.

On-line resources Bristol Online Surveys (BOS) Other on-line survey possibilit ies Sampling Populations and samples Educational researchers can make inferences

about a large group of people by studying a smaller group (or subset). The large group is described as the population. The smaller group is described as the sample. Why sample? In many cases complete coverage is not possible. Complete coverage may not offer substantial advantage over a sample (extent of variation within

the population). Studies based on samples require less time and produce quick answers. but . Surveys may require more administration, planning and programming than saturation surveys (because of the need to design appropriate sampling procedures). Sample studies will always be open to criticism simply

because of having to work with a sample.. Principles of sampling A sample must be selected carefully so as to represent the overall population Sample units must be identifiable and defined Once selected, the sample should be used throughout the study

The sample must ensure a sufficient number of responses Some common types of sample Random sample: every member of the parent population has an equal chance of being selected. The method involves selecting at random from a list of the population. Systematic sample: members are selected according to a system rather than at random: for example, every tenth person on the list. The starting point for the selection is

chosen at random. Stratified random sample: the population is grouped (eg boys and girls, or different socioeconomic groups) before random selection is made. It is usual to select sample sizes from each group in proportion to the total size of the group. Probability sampling Based on the idea that the people/events you choose as the sample are chosen because you think it is probable (its likelihood) that they will be a representative cross section of the

population/event in the whole population under study. 30 Some more common types of sample Cluster sample: this uses pre-existing groups for the study. For example all pupils in a school or class may be selected for reasons of practicality. Convenience/opportunity sample: a group is chosen because they are nearby (or in a convenient location),

available, and willing to take part in the exercise. Purposive sample: the cases in the sample are handpicked on the basis of professional judgement about their typicality. Non- probability Based on the idea that each member of the population does not have an equal chance of being selected. Thus, there is pre-defined rationale for selection.

32 Sample size how many do I need?! What is the variation within the population? How much time/resources do I have to collect the data? What is my response rate likely to be? How much time/resources do I have to analyse the data? At least 30 for any statistical analysis Making sense of your data

Descriptive Statistics Counts Indices of centrality Measures of dispersion Indices of centrality

Mean average of all values Median mid-point in the range Mode most common value 9 21 22 23 23 25 25 26 27 28 28 28 31 32 33 45 Mean = 426/16 = 26.6 Median = 26.5 Mode = 28 Dispersion or spread Range

Semi-interquartile range Standard Deviation Normal distribution curves Inferential statistics These use descriptive statistics of samples to draw inferences about the corresponding population from which the samples are drawn. t-test

correlation coefficients factor analysis Chi-squared test Cronbach alpha Graham Gibbs on YouTube