Next Steps in Gap Year Assessment

We’re proud of the impact of our gap year programs on the lives of our students. Why does this matter? Our theory of change states that if TBB programs help students develop a sense of purpose and direction rooted in justice and equity, they will better leverage opportunities higher ed and their careers offer to create social impact. Additionally, if we can help students develop higher order empathy and their capacities as powerful learners, they will create meaningful social impact throughout their lives. This impact assessment demonstrates that we have successfully developed these capacities — the same ones that made great social impact leaders so effective — in high potential students.

But, some of the results of the assessment did not reflect such clear student growth and development. Some of the outcomes suggest little positive change or even regression in students’ thinking. Examining these areas is a critical next step for our work. Our findings will inform changes in our program model and improvements in our assessment tool.

Assessing Impact

Assessing impact in education is difficult. One of the biggest challenges is determining how to identify and measure evidence of the outcomes we are seeking. The previous posts in this series identified 11 outcomes that demonstrated significant growth. We are confident of these outcomes, in part, because the measurements align with observed student behaviors.

There are 8 outcomes in this impact assessment that suggest either little or negative growth. In each case, we were surprised by this outcome because, based on our direct experience with the students, we expected to see substantial growth. We are now working to understand whether these measurements reflect weaknesses in TBB’s programs or in the assessment.

Below is a list and graph of 5 of the outcomes we are investigating more deeply (the remaining 3 will be addressed in the next section).

• Demonstrates a working knowledge of social, political, and economic processes as they relate to critical global issues and societal injustices.
• Can reassess, reorganize, and restructure assumptions and knowledge based on new information.
• Incorporates and appreciates the value of diverse types of knowledge and expertise in learning and knowledge construction.
• Demonstrates affinity and preference for inquiry-based discussion and learning.
• Expresses comfort with ambiguity and complexity in academic and social settings.

As the graph shows, the measurements of these outcomes showed growth ranging from 17% to -47%. These results have raised the following questions regarding the efficacy of TBB programs:

• The data shows high levels of engagement but little growth with these outcomes. How do we improve upon growth? If the scores are starting at relatively high levels – like with “Ambiguity & Complexity” – is it safe to assume that growth is possible or even necessary?
• Does the program and curriculum provide enough academic focus? Does it mean students don’t understand the concepts or that they haven’t learned to articulate them clearly?
• Should there be more academic accountability? Does the lack of grades and formal assessment reduce student engagement in the learning opportunities that would increase growth in these areas?
• Is the program and curriculum too ideologically narrow? If students are not incorporating or appreciating diverse types of knowledge or expressing comfort with ambiguity, are they not being challenged enough with assumptions and perspectives that run counter to their own?
• Are students developing enough metacognitive self-awareness? Can the program better develop the skills of being a self-aware learner?

While we are actively engaged in evaluating our programs, we are also evaluating the assessment tool utilizing following questions:

• Does growth for certain outcomes require a longer time horizon than that afforded by this study? Would an appreciation for inquiry-based learning and diverse types of knowledge become more prevalent after rejoining a traditional schooling?
• Does the unprompted nature of the blogs mean certain outcomes are less likely to be represented? Would students blogging for family and friends articulate evidence of these outcomes that accurately reflects the scope of their growth?
• How do the emotional high’s and low’s of the program impact the outcomes? How do homesickness, fatigue, and group dynamics affect blogs?

Inter-Cultural Competency

The 3 remaining outcomes pertain to Inter-Cultural Competency (IC). There has been extensive research in IC and how to assess it in recent years. We measured the following outcomes, resulting in the graph below:

• Can recognize one’s own cultural perspective in relation to other cultural perspectives, assigning equal standing and value to both.
• Expresses appreciation, curiosity, and respect for cultural difference, as opposed to fear, indifference, or “mere tolerance.”
• Communicates effectively with others from varied backgrounds and cultures.

These measures suggest that students may be struggling with culture shock throughout the program and not being supported enough to move past it. We’re also considering how to parse emotional expressions between

However, we’re also considering whether these outcomes accurately represent IC. Presenting the literature review that resulted in the adoption of these three outcomes requires more space than this post allows. But, trying to measure these outcomes raised important questions:

• Are these measures too focused on cultural difference? Much of the IC literature highlights the importance of overcoming cultural differences in forming meaningful cross-cultural relationships. But, aren’t relationships formed by recognizing the common humanity of members of other cultures?
• How can a humanized view of others be assessed? While explicit statements appreciating cultural difference are relatively easy to identify, is it safe to assume students will explicitly express their appreciation for human sameness?

Ultimately, we believe these are important questions for the international education sector to consider when defining IC.

Conclusions

Thinking Beyond Borders is proud to share the results of our first impact assessment. As educators, we are focused on understanding how best to support the learning growth of our students, particularly as it pertains to advancing our mission. The outcomes shared over the course of this blog series serve to inform the continued development of our programs. We hope they also contribute to the discourse around international education and the college transition.