I just hit the e-grants “submit” button for the final report for Constitutional Connections, the TAH grant that brought me to ESD 112 almost 4 years ago. While 49 pages of evalu-speak might be a little much for anyone, I thought I’d share the final section of the report. The report is written by Dr. Rick Dills from Portland State University’s Center for Student Success, our third party evaluator. Beyond his work as evaluator, Rick was a true partner in running the project: he held my hand as I made the switch from classroom teacher to TAH Project Director.
Final Evaluation Question #1: Utilizing your evaluation results, draw conclusions about the success of the project and its impact. Describe any unanticipated outcomes or benefits from your project and any barriers that you may have encountered.
Three-year trends in the evaluation data for the Constitutional Connections TAH Project indicate that participating teachers have reported significant changes in their historical knowledge, classroom practices, and students’ learning; moreover, trained raters have observed related improvements when evaluating teacher and student work samples. These positive trends are most notable in areas receiving the greatest focus within the Project’s professional development activities: 1) the understanding of Constitutional history; 2) the reading and instructional use of primary source documents; and, 3) the development of historical thinking skills required by Washington’s Classroom Based Assessments (CBA’s), the state’s method for evaluating achievement in history.
From 2006 to 2008, the percentage of elementary, middle, and high school teachers reporting a high level of instructional emphasis on the Constitution increased from 18% to 74% (a difference of 56%); the percentage reporting a high level of student learning in this area increased from 16% to 78% (a difference of 62%). Raters similarly noted that the percentage of teacher work samples showing evidence of emphasis on the Constitution increased from 39% to 78% (a difference of 39%). When raters evaluated student achievement in Year 3 of the Project, the mean rating for “understanding key ideals and principles… in the US Constitution” was at grade level or better for all three instructional levels. Teacher comments and testimonials, from surveys, interviews, and focus groups, underscore this data-based conclusion: a major success of the Project has been to help teachers and students make “constitutional connections,” to infuse the Constitution into US History instruction and to help teachers and students develop an active understanding of Constitutional principles and issues.
To what might this success and instructional impact be attributed? First, the Project focused narrowly on one arena of US history (the Constitution) – albeit a theme that is woven throughout the history of the nation – and sustained that focus over four years of engaging, high-quality professional development. The Academies, through which teachers gained most of their content knowledge, not only sustained a multi-year focus on the Constitution, but brought both a range of top historians (provided by the Organization of American Historians’ Distinguished Lectureship Program) as well as repeated exposure to a few selected historians. The area of Academy focus also mattered to participating teachers; their initial survey responses indicated that they believed teaching the Constitution was very important, but were not confident that they knew enough to do so, and therefore had not sufficiently emphasized it within their history instruction. The four years of grant-funded Academies and Institutes seem to have closed the gap between teachers’ values and their practices, empowering them to successfully implement Constitutional history within their classrooms.
The patterns in the data related to the instructional use of primary source documents are similar, as are the explanations for Project success. From 2006 to 2008, the percentage of teachers reporting a high level of instructional emphasis on the reading and interpretation of primary source documents increased from 39% to 70% (a difference of 31%); the percentage reporting a high level of student learning in this area increased from 43% to 74% (again a difference of 31%). Raters similarly noted that the percentage of teacher work samples showing evidence of emphasis on primary sources increased by 25%, to 88% in 2008. When raters evaluated student achievement in Year 3 of the Project, the mean rating for “interprets primary sources or case studies” was at grade level or better for all high school students. These successes in changing teacher practices, and apparent impact on student learning, can again be attributed to a consistent, focused emphasis on using primary sources, as modeled by the historians in Academy presentations and as discussed regularly in history professional learning team (HPLT) meetings. As with the related content emphasis on the Constitution, teachers believed that using primary sources with students was important, but had previously not been fully confident in doing so. Through Project activities, they developed their own skills for reading and interpreting documents, leaned how to teach those skills to students, and discovered that students were surprisingly capable of reading documents as historians do, given appropriate instructional scaffolding.
Teachers were additionally motivated to address the use of primary sources by the expectation in most Washington Classroom Based Assessments (CBA’s) that students interpret primary documents in completing a complex CBA task. Not surprisingly then, the data related to the implementation of CBA assessments (Project Objective 4), and the initial successes in this area, parallel the patterns noted in the two previous areas of significant Project impact. Though Washington has expected CBA’s to be fully implemented by 2009, most teachers who entered the Constitutional Connections Project in 2006 had minimal familiarity with what CBA’s required and had done little work to implement them in their classrooms. From 2006 to 2008, however, the percentage of teachers reporting a high level of alignment with CBA expectations increased from 28% to 80% (a difference of 62%). Over that same period, work sample reviewers noted an increase of 45% (from 51% to 96%) in the number of teacher assessment tasks that showed alignment with CBA expectations. That the Project supported teachers in understanding and responding to the challenging demands of Washington’s classroom based assessment system, bringing essentially all teachers to a point where they could meet those demands, is a significant Project success. That it did so while still emphasizing the content of “traditional American history” and connecting teachers’ CBA work to the Constitution is an even greater success. Finally, that teachers came to see CBA’s as “authentic intellectual work” closely related to the real work of historians (rather than as just a state requirement to be complied with) is especially important.
As the preliminary nature of the Year 3 student achievement data attests, it is still too soon to conclude that the Project directly affected student achievement during its three years. Students did show significant growth in many historical knowledge and skill areas from pre- to post-assessment, and almost two-thirds were judged to meet grade level expectations by the end of the year (when only a sixth had demonstrated grade-level skills at the start of the year). These data suggest a possible correlation between increased student learning and the changes noted above in teachers’ practices and perceptions of student achievement, but do not offer empirical proof of effect. This dilemma is one that has posed a challenge (and in some cases a barrier) to achieving the Project’s goals and carrying out an empirically-based evaluation. Unlike many states, Washington does not administer a multiple-choice test to measure students’ achievement in history. Instead, the state has initiated a system based on classroom based assessments and teacher judgment. This system was still being designed during the period of this grant, and in fact the Constitutional Connections teachers were at the vanguard of implementation efforts in the state. There have been no reference points to which student achievement on CBA’s can be compared, no state-level data to show how successful other Washington teachers and students have been in meeting the sophisticated expectations of the CBA tasks. Evaluators, Project leaders, and teachers have been forced to develop their own methods for measuring student achievement within this context.
Having had to do so has also brought some unanticipated outcomes and benefits. First, the Project was able to identify a set of knowledge- and skill-based criteria through which student learning and achievement could be viewed. Using these criteria, the Project developed criterion-referenced tools for reviewing and evaluating student opportunity and performance, which teachers have endorsed as very valuable in the classroom as well. Because measurement of student achievement had to occur through examining work samples, teachers gained tremendously through the experience of both assembling and reviewing collegially (in their HPLT’s) examples of their students’ work. This effect was particularly strong for lead teachers in the Project, who not only facilitated work sample reviews in their HPLT’s but also participated as raters in the semi-annual work sample reviews organized by the Project Evaluator. These teachers in particular were able to sharpen their understanding of what CBA’s expected and what successful student performance looked like, an unexpected benefit to them as lead teachers and a very positive aspect of the Project’s success.
Finally, the Project’s successes (as well as barriers to its success) should be discussed in relation to how it accomplished its goals as well as to what it accomplished in changing teachers’ practices and affecting student learning. While the Academies provided the primary vehicle for delivering content to teachers, the History Professional Learning Teams (HPLT’s), which met eight times each year, formed the Project’s backbone. It was through team discussions that teachers processed and implemented what they were learning about the Constitution, and that they came to understand what their students would need to do to meet CBA expectations. From the start, these teams were both powerful and problematic. While they potentially provided professional networking that had been missing from many teachers’ practices, they also required teachers to participate in professional development outside the school day, on top of work place demands, and, in many cases, at a great distance from their schools (some teams involved teachers from large geographic areas). Because there were so many teachers, from so many different districts, geographic areas, grade levels, and backgrounds, it was not easy to direct or coordinate common professional development experiences. Once teams were turned over to lead teachers selected from their midst, they necessarily developed their own areas of focus, norms, and processes, making quantifying, capturing, and reporting the results of their collegial work challenging. And as a result, the Project had to “measure” student achievement in a diverse range of student outcome areas (drawn from the Washington Grade Level Expectations that teachers elected to target) rather than a single, more focused arena. Nevertheless, survey data and teacher testimonials suggest that the impact of three-year involvement with HPLT’s was in many cases profound. And, as with the content focus on the Constitution, primary source documents, and classroom based assessment, Project leaders learned some valuable lessons that will hopefully translate to other contexts, among them:
- Participating in collegial teams refreshes and rejuvenates teachers; sustained collegial learning professionalizes their development in a meaningful and powerful way.
- Teams can be effective across grade levels and teaching assignments, but are strengthened by having similar assignments and/or instructional emphases.
- Teams can be effective across districts, but geographical distance is challenging.
- Teams can (and should) be encouraged to find their own paths and focus on what matters most to themselves and their students, but allowing them to do so makes evaluation problematic.
- Teachers’ practices are significantly affected by engaging in collegial, criterion-based review of student work.
- Given sustained collegial support, teachers at all levels (as well as their students) are very capable of understanding sophisticated historical content such as Constitutional history and its interpretation through case studies and primary document review.
Overall, the Evaluators conclude that Constitutional Connections was an unusually powerful and effective professional development effort. While it is not possible to assert that student achievement was conclusively improved within its three-year span, there is every indication that the Project will have a positive long-term impact on students’ learning of American History. Clearly, teachers, their knowledge of history, and their instructional practices were changed in lasting ways. Year 4 informants, in the follow-up interviews conducted by the Evaluators, universally indicated that Constitutional Connections was the strongest professional development opportunity in which they have participated during their careers. They also said that the 2008-10 ESD 112 TAH grant, Causes of Conflict, promises to be even better. In one case, a teacher new to the profession said that she had been involved in the Constitutional Connections Project and now Causes of Conflict, and that she had just assumed all teachers were having high quality professional development opportunities like she was. Unfortunately, she found that she was the rare beginning teacher who was truly being inducted into the history teaching profession with very sophisticated support. Her comment was that the Project’s combination of sophisticated historical content and the instructional improvement teachers gained from the focus on classroom based assessment and Lesson Study is very unique in professional development for educators. She said, “How do we promote more of this kind of professional development for all educators instead of the often more trivial stuff we are exposed to?” She and others in the respondent group are concerned that they may never have such a rich opportunity again.
Finally, all respondents indicated, often without being asked, that project director Matt Karlsen’s leadership and experience in managing these projects was outstanding. His direction and ability to attract high quality speakers was cited several times by the grant participants. Virtually all the focus group participants indicated that Matt was always available to respond to their questions and that he utilized group comments to better respond to participant needs. A couple of participants indicated that ESD 112’s experience in facilitating three Teaching American History grants has made them experts in the process and should be acknowledged. The Evaluators would like to echo this acknowledgement of excellence in professional leadership, and commend the Constitutional Connections Project for its lasting impact on teachers, students, and history education in Southwest Washington.