Making Sense of Data & Evidence
Once the evidence has been collected, the next step is to make sense of what you have found in order to make changes and improvements for the future. This process usually involves the following steps:
Summarizing the data and evidence
The best approach to summarizing the evidence depends on the kind and volume of data you have collected. For quantitative data (e.g. data collected using multiple-choice type questions) providing a summary is usually fairly straightforward -- for example calculating average responses -- but interpreting this summary sometimes needs more effort. For qualitative data (e.g. students' answers to open-ended free response questions), the summarizing will be difficult, but because the data are richer in content and meaning, the interpretation is often easier.
Quantitative data
In FET, summarizing the quantitative data collected through questionnaire is usually rather straightforward. In fact, if you are using the HKUST survey tool, then the data will be summarized for you automatically. If you are using a paper questionnaire, then you will need to arrange to have the data inputted into, say, an Excel spreadsheet, or some statistical software package (e.g. SPSS) for processing. On the other hand, if your questionnaire is short and the number of student respondents is small, then it may be quicker to calculate mean responses by hand. If you need additional information about processing quantitative data, please click here .
Qualitative data
For qualitative data like students' answers to open-ended questions, what needs to be done usually involves reading through responses once or twice and then summarizing the main points the students raised. It is not uncommon to find students expressing a diverse range of views about a course. You will need to identify one or more themes that run through at least some of the responses. For example, you might determine that 30% of the students would like the class to be more active even though they differ in their preferences for particular class activities.
More sophisticated methods may be needed if the class is really large or you intend to acquire a more in-depth understanding of the students' views, say through focus group meetings. For large classes, it might be necessary to use a table or matrix format to help you summarize the data. Please click here for more details.
Interpreting the results
With the data summarized, it is necessary to make sense of what you have discovered, reflect on the feedback, and think about what follow-up actions might be taken in the short and longer terms.
With quantitative data, what we often get is a general picture of students' views on one or more aspects of our teaching. For example, from the responses of students in section A to the question below, we found that 70% were able to follow what was taught in class most of the times, but 30% failed to do so.
To make sense of the percentages, the instructor can consider using one of the following approaches:
- Comparison with other sections
- Comparison with the evaluation of the previous semester
- Comparison with a pre-defined criterion
If the same question was asked in section B taught by the same instructor, and in a previous semester by another instructor (assuming same course content), a comparison can be made. The example above shows that the students in section B were quite similar in their responses, but students in the previous semester were more positive. Comparison can also be made with a criterion defined by the instructor. It could be based on his or her past experience with students, or his or her beliefs in teaching. For example, if the instructor expects and would like 80% of the students to follow all of the information presented in class, he or she might need to change something about the class presentation methods.
While it is useful to know the percentages of students who could follow what the instructor taught in class, we don't know what it was they failed to understand unless we have a follow-up question which asks them to specify what they did not understand in class. Without such knowledge, it is difficult to identify appropriate follow-up actions. Hence in FET it is important that we include some open-ended questions to let students share with us their negative and positive experiences.
With qualitative data, the meanings of the student feedback are usually self-evident and easier to interpret. However the following points might be worth taking note of when reading students' comments:
- While we should not ignore the views of a few students (positive or negative), it is the majority's views that we should be focusing on.
- Undue emphasis should not be placed on the negative views of a few students, especially if the students failed to provide relevant information regarding their complaints.
- For the same reason, caution should be exercised about adopting an idea proposed by a small number of students, even it is pedagogically sound. For example, in a class of 60, if 10 students indicated their preference for project-based learning, the instructor should bear in mind that the majority of the students probably are unfamiliar with PBL and would need extra guidance if it is to be introduced
- Students' expectations of the learning outcomes of a course might be different from ours. Students sometimes enroll into a course with the expectation of learning something which is not part of the key intended learning outcomes of the instructor. For example, students could enroll into a course with the title "Career management" and expected to learn job searching skills such as job interview techniques and writing job application letters, whereas the instructor may be more concerned with helping students to improve their self-understanding and plan their careers.
Reflection on teaching
Having collected, processed and examined the students' feedback, and possibly performance data such as students' assignments, projects, and examination scripts, it is time for the instructor to pool the information and reflect on his or her teaching, possibly with the help of a colleague.
In this process, the purpose would be to find out what the data and evidence have to say about the teaching and learning processes and their effectiveness in bringing about the intended outcomes.
Brookfield (1995) encouraged teachers to adopt a critical stance in this reflection process. He pointed out that instructors should be alert to the many implicit assumptions they have about their students and their actions, which might not always be true. Brookfield gave many examples to illustrate how students might interpret teachers' behaviors in different ways - which may be very different from those of the teacher. Such understanding is crucial to a proper understanding of why students behaved the way they did. He also alerted us to how the way we learned as a student and past teaching experience affect the way we teach.
Tools are also available for instructors to do some self-examination of their teaching. Below are some examples.
- Instructor self-evaluation form (Weimer, Parrett and Kerns (1988)) - This form evaluates an instructor along four dimensions:
- Adequency of classroom procedures
- Enthusiasm for teaching and knowledge of subject matter
- Stimulation of cognitive and affective gains in students
- Relations with students
- Teaching orientation questionnaire (Kember and Gow, 1994) - This instrument allows an instructor to find out about his or her teaching orientation (transmissive or facilitative).
- Transmissive teaching - Teaching as a teacher-centred activity with the aim of transmitting knowledge to students
- Facilitative teaching - Teaching as a student-centered activity with the aim of facilitating students' learning and helping them to be independent learners.
Through such reflection, instructors can come to a better understanding of their teaching, its strengths and weaknesses, and more importantly what should be done to improve it. Teaching as a professional practice should be subject to a process of continuous improvement.
- 5944 reads
