External Evaluation for Grant Programs

Rockman et al frequently serves as an independent, external evaluator for grant-funded, multi-year projects supported by state and federal agencies, private industry, and foundations. The substantial investment in these projects means they often require rigorous evaluations to assess evidence of impact, as well as analyses that examine the critical links between implementation and impact. Our evaluations also frequently include studies to document and inform implementation throughout the life of the project.

We have a strong reputation for conducting evaluations of formal and informal science education programs funded by the National Science Foundation, National Institutes of Health, and NASA. We have also served as the external evaluator for numerous K-12 school reform initiatives funded by the US Department of Education.

Selected examples of our work in evaluating educational initiatives for state- and federally-funded grant programs are below. We welcome the opportunity to discuss your project with you and to provide you with more information about our tailored services. Please Contact Us to schedule a free 30-minute Q&A session with a consultant.

Selected Projects Include

  • Coastal Marine Biolabs – Neurolab M3

    NeuroLab M3 is a comprehensive research education program that carefully guides high school students through a series of in-depth, interdisciplinary, and discovery-based explorations of scientific models, model organisms, and model systems in developmental neuroscience. The 5-year project includes immersive residential research institutes in comparative functional genomics for rising juniors and seniors. The evaluation will examine outcomes for participating students using longitudinal surveys and content knowledge tests.

  • Radiolab

    Rockman et al conducted a comprehensive evaluation of Radiolab initiatives funded as part of its “What does curiosity sound like?” project. The project included evaluation of live events, online content, Radiolab’s social media and mobile phone app, and citizen science activities. The evaluation also sought to differentiate outcomes on STEM professionals and lay-audiences, and the extent to which Radiolab programming can facilitate better communication between these two audiences.

  • Bay Area Video Coalition Bridges to STEM Careers

    REA conducted a formative and summative evaluation of Bay Area Video Coalition’s Bridges to STEM Careers, a three-year Advanced Technological Education (ATE) Program Improvement Project designed to enhance the relevance of technician education in Computer Science and Multimedia. The project targeted two-year college students, their families and their college and workplace mentors. The ultimate goal of the Bridges project was to keep traditionally underrepresented and low-income students in the STEM pipeline by demystifying the process of preparing for a STEM career through internships and informational videos from past internship participants.

  • Astronomical Society of the Pacific Galileo Educator Network (GEN)

    With funding from NASA, the Astronomical Society of the Pacific (ASP) developed the Galileo Educator Network (GEN), which prepared teacher educators nationwide to design and deliver professional development workshops on astronomy content and science practices aligned with Next Generation Science Standards (NGSS). REA collected surveys and observations to describe the ways in which GEN engaged K-12 teachers and promoted the effective use of NASA-developed resources. REA presented its evaluation findings at two ASP annual meetings.

  • American Museum of Natural History Space Shows

    Rockman et al has conducted evaluative studies on four dome-style planetarium shows created by the American Museum for Natural History. Rockman’s evaluations of “The Search for Life,” “Cosmic Collisions,” “Journey to the Stars” and “Dark Universe” have examined audience responses to the shows and a range of outcomes including attitudes toward science, interest in science, and content knowledge. We’ve conducted research at several planetaria around the United States to better understand the impact of presentation context, and with a variety of audiences in order to understand the impacts that programming has on audience members of all ages and prior knowledge levels.

  • East Bay Center for the Performing Arts Learning Without Borders

    The East Bay Center for the Performing Arts (EBCPA) Learning without Borders (LWOB) professional development program served elementary school teachers in the West Contra Costa Unified School District. Through professional development workshops, coaching, and in-class visits from artists, the program aimed to increase the capacity and confidence of the teachers to integrate arts with other core subject areas. REA collected survey, focus group, and pre- and post-test data from the participating teachers, and this data led to important insights about the teachers’ expectations for the program, feedback about the program, perceptions of the program’s impacts, and recommendations for the program moving forward.

  • Convergence Academies Project

    The Center for Community Arts Partnerships at Columbia College Chicago, in partnership with Chicago Public Schools, received a development grant from the Investing in Innovation Fund (i3) of the U.S. Department of Education for their Convergence Academies project (http://convergenceacademies.org/ourorigins.html). To support 21st Century learning, CCAP developed a whole school reform model, inspired by principles and values of connected learning, that involved inquiry-based instruction and integration of digital media arts and technology throughout the school. The model was implemented in Chicago in both a high school and a feeder elementary school (K-8). As the external evaluator for the project, Rockman et al conducted studies of both implementation and impact, including an interrupted time-series design to investigate gains on state tests in math and reading.

  • KQED QUEST

    QUEST, initiated by KQED in San Francisco, is a unique cross-platform science journalism collaborative designed to tap into the region’s interest in science, environment, and nature, while providing entertaining and informative content about current local issues in science-environment-nature, and engaging users in conversations about actions for these issues. Over a six-year period, REA evaluated pilot materials, including radio, TV and web content, and educational outreach workshops, and documented internal operations and partnerships with external organizations. The project served to identify the internal collaboration processes and resources needed for full production of a weekly half hour magazine show on KQED-TV, a weekly insert in Morning Edition, and Web site and education materials for use by general audiences. The formative and summative evaluation of QUEST’s six-station collaborative over a three-year period included focus groups and interviews with diverse stakeholders, and revealed that QUEST successfully served a number of interests across diverse groups, addressing environmental issues with high quality content and current science information.

  • Jackson State University: Mississippi Academy for Science Teaching (MAST)

    Rockman et al has served as the external evaluator for four professional development programs provided by Jackson State University for science teachers in elementary, middle and high school. REA has employed a mixed methods approach, including pre- and post surveys, qualitative interviews, and observations to measure the quality of implementation and impacts on teachers and students. Our most recent evaluation, a long-term impact study of MAST alums between 2004-2014, showed that MAST can achieve meaningful and sustained impacts on teaching and learning.

  • New Visions Accessing Algebra through Inquiry (a2i)

    The Accessing Algebra Through Inquiry (a2i) project is an i3 validation grant conducted by New Visions for Public Schools, in partnership with the Silicon Valley Mathematics Initiative and the New York City Department of Education. The project provides formative assessment lessons, ongoing coaching and training, and opportunities to examine student work in school-based inquiry team meetings. As the external evaluator, REA will conduct both implementation and impact studies to assess how students in a2i project classrooms perform on annual standardized achievement tests compared to matched groups of students whose teachers are not participating.

    Report: a2i narrative
  • Exploratorium Carbon Networks

    REA is the external evaluator for ‘Carbon Networks,’ a federally-funded project which is a collaboration between the Exploratorium, the Pacific Science Center, and the Waikiki Aquarium. This project aims to assist museum educators in building capacity for developing educational programs and activities for public audiences and formal educators on the topics of climate change and ocean acidification. REA is conducting formative evaluations of educator workshops and summative evaluation of program impacts at each of the participating institutions, using online surveys, interviews and observations.

  • Leading with Learning

    Rockman et al is the external evaluator for the U.S. Department of Education Investing in Innovation Fund (i3) Development award to WestEd for the Leading with Learning (LWL) project, (formerly known as the English Learner Professional Learning (ELPL) Project). The three-year evaluation includes both a quasi-experimental impact study with matched comparison schools, and an implementation study in six California elementary schools in two districts. For the impact study, REA is using hierarchical linear modeling to compare student and teacher outcomes. Expected outcomes for students—as measured by state tests—are improved reading comprehension, speaking and listening competencies, oral and written production of English, and content understanding. Participating teachers are expected to develop greater knowledge and skills in new practices for accelerating EL students’ understanding and use of academic language. To study implementation, REA will use surveys, interviews, focus groups, observations, and logs. The LWL model aims to develop teachers’ and instructional coaches’ knowledge and skills for implementing new instructional approaches that are sensitive to the language demands necessary for ELs to excel in academically and linguistically challenging content.

  • Museum of Science and Industry “Our Place in Space”

    Rockman et al is the external evaluator for “Our Place in Space,” a professional development program for formal and informal science educators designed and implemented by the Museum of Science and Industry in Chicago, with funding from NASA. Through observations, surveys, and interviews, REA is studying how the program helps prepare educators to cover Next Generation Science Standards and implement a series of high-quality STEM instructional activities. REA’s evaluation seeks to demonstrate the value of the program as well as suggest potential areas for improvements in subsequent programming.

  • New Visions Urban Teacher Residency Program

    The New Visions for Public Schools-Hunter College Urban Teacher Residency (UTR) project is a partnership between New Visions, the Hunter College Schools of Education and Arts and Sciences, and the New York City Department of Education. The 5-year project is designed to increase the number of well-prepared teachers entering the city’s classrooms, provide ongoing support during the critical early years of teaching, and ultimately raise teacher retention rates and student achievement. To assess the impact of the UTR model, REA conducted surveys and focus groups with each cohort of UTR residents at key points during their 14-month program, and during their first years as NYC public school teachers. REA also gathered data from Hunter faculty and host school mentors and principals focusing on residents’ preparation, readiness, confidence, and integration into school culture. Finally, REA reviewed achievement and other institutional data to assess the program’s impact on student performance, including state licensing and certification records, graduation rates, and NYC teacher retention rates.

  • PBS Kids Transmedia Games Usability Testing

    Rockman et al has run a series of usability tests for media products developed by PBS Kids and its partners as part of the Ready To Learn initiative, including several rounds of testing in Boston, Washington DC, and Bloomington IN.

  • Peg + Cat: Adventures in Learning

    As external evaluators for an NSF-funded AISL grant awarded to The Fred Rogers Company, Rockman et al tested early versions of Peg + Cat interactive media and science camp activities with pre-schoolers, their families, and their teachers. REA also evaluated youth experiences and outcomes in an affiliated summer camp, including observations, focus groups, and interviews with participating educators and children. Peg + Cat is a PBS math-based television program for preschoolers, which is accompanied by interactive media including digital storybooks, online games and apps.

  • Project Jericho

    In collaboration with International Society for Technology in Education (ISTE), REA evaluated a pilot effort of the South Carolina State Department of Education to test the feasibility of virtual schooling for the state, examining its online learning course management system (Blackboard), registration system, quality assurance measures, and the need for and response to the program around the state. REA conducted site visits in pilot sites and interviews with virtual school coordinators, and consulted on surveys of students, teachers, and counselors involved in the pilot.