Study on Generic Skills (Summer 2013) – Interpretation Guide for Individual Results

About the Pilot Study

The Pilot Study on Generic Skills aims to evaluate the effectiveness and reliability of various instruments designed for accessing students’ creativity, their effectiveness of teamwork skills and their intercultural competence. The study was conducted in July 2013 and near 300 students participated in our study. They were invited to complete one test and four surveys under proctor’s supervision. Below is a list of the instruments employed in this study:

1. Torrance® Tests of Creative Thinking
2. Intercultural Sensitivity Scale
3. Global Perspective Inventory
4. General Teamwork Skills
5. Comprehensive Assessment of Team Member Effectiveness

Individual results of these instruments were already emailed to participants. To help participants better interpret their results, this webpage provides some background information about these instruments and the meanings of different scales.

1) Torrance Tests of Creative Thinking (TTCT)

The Torrance Tests of Creative Thinking (TTCT) was developed by Dr. E. Paul Torrance and is currently owned by Scholastic Testing Service. It consists of two forms, the Figural Form and the Verbal Form and the latter was employed in this pilot. The instrument consists of 6 exercises and they were designed to evaluate a student’s creativity in three mental characteristics described as follows:

• Fluency: This reflects the student’s ability to produce a large number of ideas using words.
• Flexibility: This represents the student’s ability to produce a variety of ideas, to shift from one approach to another, or to use a variety of strategies.
• Originality: This refers to the student’s ability to produce new ideas that are uncommon or deviated from the normal pattern.

Based on the above three scores, an average score is calculated to provide an overall indicator of a student’s creativity.

In order to provide students’ a frame of reference, all individual results are compared with the scores of two normative samples: (i) US students with the same level of education (i.e. grade-based score), OR (ii) US students with the same age (i.e. age-based score). By using these two samples, two relative figures are calculated:

• Standard scores: This is a normalized score such that the mean of the benchmark group (age-based or grade-based group) is 100 and the standard deviation (SD) is 20.
• National percentile: This represents the relative ranking of a student’s score compared with a random sample of 100 students selected from the benchmark group.

Take your age-based scores as an example. If your age-based standard score is 116, that means that by comparing your results with the population of US students in the same age, your score is 0.8 SD higher than the mean. If your age-based national percentile of average is 80, that means on average your score is equal to or higher than 80 students in a randomly drawn sample of 100 out of all US students in the same age.

The following table provides a list of descriptions about the ranking of percentile according to the Scholastic Testing Service.

2) Intercultural Sensitivity Scale (ISS)

The Intercultural Sensitivity Scale (ISS) was developed by Chen and Starosta (1997) to measure students’ willingness to motivate themselves to understand, appreciate and accept differences among cultures. It is a self-report survey which includes 24 question items on a 5-point scale (i.e. from “Strongly disagree” to “Strongly agree”). Based on a student’s responses to these items, his or her score in each of the following five scales is calculated.

• Interaction attentiveness refers to one’s level of responsiveness to cultural differences in multi-cultural settings;
• Interaction confidence shows one’s confidence and relaxation in communicating with people from different cultures;
• Interaction engagement reflects one’s open-mindedness, appreciation and acceptance of cultural differences;
• Interaction enjoyment indicates the extent to which one finds his/her interaction with people from different cultures encouraging and enjoyable; and
• Respect for cultural differences means one’s level of recognition, empathy and receptiveness of cultural differences.

A higher rating in a particular scale (e.g. Interaction attentiveness) indicates a stronger disposition of his/her characteristics (e.g. being more responsive) in interacting with people from diverse cultures.

3) Global Perspective Inventory (GPI)

The Global Perspective Inventory was developed by Dr. Larry Braskamp and is currently hosted by Central College — Pella, Iowa. It is designed to reflect the extent to which students have a global and holistic view of their own learning and development. It measures how a student thinks, views himself or herself as a person with a cultural heritage, and relates to others from other cultures, backgrounds and values. It consists of 6 sub-scales in three dimensions - cognitive, intrapersonal and interpersonal.  

• Cognitive dimension consists of two subscales of Knowing and Knowledge. The former shows the degree of complexity of one’s view of the importance of cultural contexts when judging what is important to know and value. The latter reflects the degree of understanding and awareness of various cultures and their impact on our global society.
• Intrapersonal dimension includes the subscales of Identity and Affect. The identity subscale assesses the level of awareness of one’s unique identity, purpose, and meaningful philosophy of life, whereas the affect subscale indicates the level of respect for and acceptance of cultural perspectives different from one’s own and degree of emotional confidence when living in complex situations. This in turn reflects an “emotional intelligence” that is important in one’s processing encounters with other cultures.
• Interpersonal dimension involves two subscales: Social responsibility, which refers to the level of interdependence and social concern for others; and Social interactions, which means the degree of engagement with others who are different from oneself and degree of cultural sensitivity in living in pluralistic settings.

In each statement, participants are to respond on a 5-point scale from “Strongly disagree” to “Strongly agree”. The higher the student scored on a particular sub-scale (e.g. Intrapersonal Awareness), the more he or she exhibits his or her way of seeing his or her own learning and development in that dimension (e.g. one’s unique identity, purpose, and meaningful philosophy of life) from a global and holistic perspective.

4) General Teamwork Skills

The general teamwork skills were evaluated by a self-report survey developed by the research team of the Educational Testing Service (ETS) (Zhuang et al 2008). The survey consists of 30 statements, each asking students to rate on a 6-point scale from “Never” to “Always”. The survey comprises of the following three scales:

• Cooperation measures the students’ inclination to bring ideas together, seek solutions and provide feedback to team members.
• Advocating/Influence reflects the student’s tendency to direct others, provide appropriate suggestion and criticism, and persuade others.
• Negotiation reveals the student’s inclination to listen, adapt to change during conflicts and solve conflicts.

The higher the student scored on a particular scale (e.g. Negotiation), the more he or she is his or her tendency in working as part of a team (e.g. to listen, adapt to change during conflicts and solve conflicts).

5) Comprehensive Assessment of Team Member Effectiveness (CATME)

CATME was created for use with college students to measure five broad categories of team member contributions in 33 statements:

• Contributing to the team’s work, e.g. in taking up fair share of work and completing it in time and in good quality, being well-prepared for team meetings and helpful to teammates;
• Interacting with Teammates, e.g. in giving/receiving feedback and encouragement to/from teammates, communicating with teammates effectively and timely;
• Keeping the team on track, e.g. in helping to plan, organize and monitor the progress of the team’s work;
• Expecting quality, e.g. in terms of one’s beliefs in the team’s ability to achieve high standards and produce high-quality work;
• Having relevant knowledge, skills, and abilities, e.g. in possessing relevant expertise and skills in completing one’s own and others’ jobs in the team.

Each statement requires students to rate on 7-point scale from “Strongly disagree” to “Strongly agree”. The higher the student scored on a particular scale (e.g. Keeping the team on track), the more he or she tends to contribute to the team’s work in that aspect (e.g. offering help to plan, organize and monitor the progress of the team’s work).

References

1. Scholastic Testing Service Inc. Torrance Tests of Creative Thinking - Manual for Scoring and Interpreting Results (2008)
2. Chen G., Starosta W. J., A review of the Concept of Intercultural Sensitivity. (1997)
3. Larry A. Braskamp, David C. Braskamp, Kelly Carter Merrill and Mark Engberg, Global Perspective Institute. The Global Perspective Inventory (GPI) – Its Purpose, Construction, Potential Uses and Psychometric Characteristics (2012).
4. Zhuang, X., MacCann, C., Wang, L., Liu, L., Roberts, R. D., Educational Testing Service. Development and Validity Evidence Supporting a Teamwork and Collaboration Assessment for High School Students (2008).
5. Ohland, M. W., Loughry, M. L., Woehr, D. J., Bullard, L. G., Felder, R. M., Finelli, C. J., Layton, R. A., Pomeranz, H. R., Schmucker, D. G., The Comprehensive Assessment of Team Member Effectiveness: Development of a Behaviorally Anchored Rating Scale for Self- and Peer Evaluation.  

Enquiry

For enquiries, please contact us at ctir@ust.hk.