External Evaluation for Grant Programs

Selected Projects Include

• 

  Photo courtesy of NWABR via CC licensing.

  Coastal Marine Biolabs

  Coastal Marine Biolabs’ NeuroLab 2.0 project has developed a multidisciplinary, storyline-based high school curriculum unit about a complex movement disorder, congenital mirror movement disorder (CMM). Over the ten lesson sequence, students generate questions, analyze and interpret data, and construct and revise explanatory models. The project seeks to expand access to investigative research experiences for students by training teachers on the new curriculum. REA is conducting a small-scale classroom study to evaluate the efficacy of the NeuroLab 2.0 unit on students’ science knowledge, beliefs, and attitudes. Evaluators are also studying teachers’ implementation of the materials.
• 

  Photograph courtesy of Jared Kelly via creative commons licensing.

  RadioLab

  Radiolab’s “What Curiosity Sounds Like: Discovering, Challenging, and Sharing Scientific Ideas” project, funded by the NSF Advancing Informal Science Learning (AISL) program, explored new ways to actively engage both lay and professional audiences and foster meaningful communication between scientists and the general public. As the external evaluator for the project, REA examined participants’ experiences and explored project outcomes through a series of surveys and interviews.

  Report: Radiolab. What Curiosity Sounds Like
• 

  Photo: BAVC News

  Bay Area Video Coalition Bridges to STEM Careers

  REA conducted a formative and summative evaluation of Bay Area Video Coalition’s Bridges to STEM Careers, a three-year Advanced Technological Education (ATE) Program Improvement Project designed to enhance the relevance of technician education in Computer Science and Multimedia. The project targeted two-year college students, their families and their college and workplace mentors. The ultimate goal of the Bridges project was to keep traditionally underrepresented and low-income students in the STEM pipeline by demystifying the process of preparing for a STEM career through internships and informational videos from past internship participants.
• 

  Photo courtesy of Ryan Somma via CC licensing.

  Astronomical Society of the Pacific Galileo Educator Network (GEN)

  With funding from NASA, the Astronomical Society of the Pacific (ASP) developed the Galileo Educator Network (GEN), which prepared teacher educators nationwide to design and deliver professional development workshops on astronomy content and science practices aligned with Next Generation Science Standards (NGSS). REA collected surveys and observations to describe the ways in which GEN engaged K-12 teachers and promoted the effective use of NASA-developed resources. REA presented its evaluation findings at two ASP annual meetings.
• 

  Photo courtesy of Daniel Torres Bonatto via CC licensing.

  American Museum of Natural History Space Shows

  Rockman et al has conducted evaluative studies on four dome-style planetarium shows created by the American Museum for Natural History. Rockman’s evaluations of “The Search for Life,” “Cosmic Collisions,” “Journey to the Stars” and “Dark Universe” have examined audience responses to the shows and a range of outcomes including attitudes toward science, interest in science, and content knowledge. We’ve conducted research at several planetaria around the United States to better understand the impact of presentation context, and with a variety of audiences in order to understand the impacts that programming has on audience members of all ages and prior knowledge levels.
• 

  Photo courtesy of Dean Hochman via CC licensing.

  East Bay Center for the Performing Arts Learning Without Borders

  The East Bay Center for the Performing Arts (EBCPA) Learning without Borders (LWOB) professional development program served elementary school teachers in the West Contra Costa Unified School District. Through professional development workshops, coaching, and in-class visits from artists, the program aimed to increase the capacity and confidence of the teachers to integrate arts with other core subject areas. REA collected survey, focus group, and pre- and post-test data from the participating teachers, and this data led to important insights about the teachers’ expectations for the program, feedback about the program, perceptions of the program’s impacts, and recommendations for the program moving forward.
• 

  Photo courtesy of Chris Betcher via CC licensing.

  Convergence Academies Project

  The Center for Community Arts Partnerships at Columbia College Chicago, in partnership with Chicago Public Schools, received a development grant from the Investing in Innovation Fund (i3) of the U.S. Department of Education for their Convergence Academies project (http://convergenceacademies.org/ourorigins.html). To support 21st Century learning, CCAP developed a whole school reform model, inspired by principles and values of connected learning, that involved inquiry-based instruction and integration of digital media arts and technology throughout the school. The model was implemented in Chicago in both a high school and a feeder elementary school (K-8). As the external evaluator for the project, Rockman et al conducted studies of both implementation and impact, including an interrupted time-series design to investigate gains on state tests in math and reading.

  Report: Evaluation of the Convergence Academies Initiative Final Report
• 

  Photo courtesy of KQED QUEST via CC licensing.

  KQED

  REA has a long history of partnering with KQED, the PBS affiliate in Northern California, on diverse research and evaluation projects involving teacher professional development, organizational development, environmental education, public science communication, and science journalism. Specific projects include evaluation of KQED Teach (an online professional development tool for teachers), a two-year organizational development study, and two NSF-funded projects applying social science research methods to investigate best practices for engaging millennial audiences in science content, and improving science media production and journalism practices.

  Report: QUEST Summative Evaluation Report
• 

  Photo: Dr. Peter Chang.

  Jackson State University: Mississippi Academy for Science Teaching (MAST)

  Rockman et al has served as the external evaluator for four professional development programs provided by Jackson State University for science teachers in elementary, middle and high school. REA has employed a mixed methods approach, including pre- and post surveys, qualitative interviews, and observations to measure the quality of implementation and impacts on teachers and students. Our most recent evaluation, a long-term impact study of MAST alums between 2004-2017, showed that MAST can achieve meaningful and sustained impacts on teaching and learning.
• 

  Photo courtesy of Tim Farrell, New Visions.

  New Visions Accessing Algebra through Inquiry (a2i)

  Accessing Algebra Through Inquiry (a2i) was a high school mathematics project conducted by New Visions for Public Schools (New Visions), in partnership with the Silicon Valley Mathematics Initiative (SVMI) and the New York City Department of Education. The project was supported with a 2013 i3 validation grant (#U411B110116) from the U.S. Department of Education. Rockman et al (REA) conducted the independent evaluation of a2i. The central focus of a2i was on the use of high-quality formative assessment processes to help teachers implement high standards and create effective math learning environments. The major categories of support for teachers included: (a) inquiry teams focused on data-driven instructional practices, (b) adaptable formative assessment lessons and strategies, and (c) instructional coaches. A quasi-experimental study involving 40 schools (across 2 cohorts) compared the effectiveness of a2i on students’ achievement in algebra and geometry compared to business-as-usual instruction in matched comparison schools. While the main findings did reveal statistically positive effects of a2i instruction on state standardized test performances, additional analyses by REA found evidence of improved achievement during the second year of the study after professional development was restructured and day-to-day pedagogical practices were introduced. REA evaluation of classroom changes revealed that students engaged in more group work, talked more about math, and solved problems collaboratively.

  Report: Accessing Algebra Through Inquiry Final Report
• 

  Photo courtesy of NOAA Photo Library via CC licensing.

  Exploratorium Carbon Networks

  REA is the external evaluator for ‘Carbon Networks,’ a federally-funded project which is a collaboration between the Exploratorium, the Pacific Science Center, and the Waikiki Aquarium. This project aims to assist museum educators in building capacity for developing educational programs and activities for public audiences and formal educators on the topics of climate change and ocean acidification. REA is conducting formative evaluations of educator workshops and summative evaluation of program impacts at each of the participating institutions, using online surveys, interviews and observations.
• 

  Photo courtesy of Enokson via CC licensing.

  Leading with Learning

  To improve teaching for English learners (ELs), WestEd partnered with two, large school districts in California on a multiyear project called Leading with Learning: Systematically Transforming Teaching for English Learners. The Leading with Learning (LwL) project was awarded a three-year development grant from the U.S. Department of Education’s Investing in Innovation (i3) fund. Rockman et al conducted the independent evaluation of LwL to study its implementation and to test its effectiveness on elementary student outcomes using a quasi-experimental design. The LwL model involved an intensive, two-year program of professional learning for K–6 teachers, instructional coaches, principals, district co-facilitators, and district leaders. Teacher practice changes were aimed at improving students’ understanding of complex texts, engagement with complex tasks, and understanding of academic English; which was expected to lead to improved academic achievement. Results showed that while the state assessment outcomes did not provide evidence of effectiveness, other data sources revealed important improvements in outcomes for students, teachers, and coaches.

  Report: Leading with Learning Final Report
• 

  Photo: Jennifer Borland

  Museum of Science and Industry “Our Place in Space”

  Rockman et al is the external evaluator for “Our Place in Space,” a professional development program for formal and informal science educators designed and implemented by the Museum of Science and Industry in Chicago, with funding from NASA. Through observations, surveys, and interviews, REA is studying how the program helps prepare educators to cover Next Generation Science Standards and implement a series of high-quality STEM instructional activities. REA’s evaluation seeks to demonstrate the value of the program as well as suggest potential areas for improvements in subsequent programming.
• 

  Photo courtesy of Tim Farrell, New Visions.

  New Visions Hunter College: Urban Teacher Residency Program

  A 10-year partnership between New Visions for Public Schools, Hunter College, and the New York City Department of Education, the Urban Teacher residency (UTR) program was funded through multiple U.S. Department of Education grants, along with the Mathematics and Science Teacher Education Residency (MASTER), funded by the National Science Foundation. The programs focused on the placement of new secondary teachers in high-needs classrooms in New York City where resident teachers were trained alongside a mentor teacher during a year-long residency. Rockman et al, served as the external evaluator to assess the impact and implementation of training on student performance, teacher practice and school culture.

  Report: 2018 Summative Report for New Visions for Public Schools – Hunter College Urban Teacher Residency Project
• 

  Photo courtesy of Lucélia Ribeiro via CC licensing.

  PBS Ready to Learn

  For nearly two decades, REA has conducted formative evaluations on behalf of PBS Kids and their partners as part of the US Department of Education “Ready to Learn” grant program. Through the years, our evaluation efforts have included youth and family playtesting for digital games at various stages during the design process, formative testing of stories for broadcast programming, and extended play studies that examine the impacts of longer-term product-usage, including gameplay and video programming.
• 

  Photo courtesy of Julia Li.

  Peg + Cat: Adventures in Learning

  Peg + Cat is a math-based PBS television program for preschoolers, supplemented by online games, digital storybooks, and apps. We conducted formative and summative evaluation of Peg + Cat interactive media and themed science camp activities, developed by Fred Rogers Productions (formerly The Fred Rogers Company) and Carnegie Science Center as part of an NSF-funded AISL project. We examined pre-schoolers’, families’, and educators’ experiences using project resources, as well as pre-schoolers’ social-emotional and learning outcomes via observations, focus groups, interviews, and a quasi-experimental design that incorporated pre-post surveys and content assessments.
• 

  Photo: Wikimedia Commons

  Project Jericho

  In collaboration with International Society for Technology in Education (ISTE), REA evaluated a pilot effort of the South Carolina State Department of Education to test the feasibility of virtual schooling for the state, examining its online learning course management system (Blackboard), registration system, quality assurance measures, and the need for and response to the program around the state. REA conducted site visits in pilot sites and interviews with virtual school coordinators, and consulted on surveys of students, teachers, and counselors involved in the pilot.