Independent Evaluation

Selected Projects Include

• 

  Photo courtesy of spacedust2019 via CC licensing.

  California Academy of Sciences

  Over a 4 year period, Rockman et al has conducted more than a dozen evaluation and research projects with the California Academy of Sciences in San Francisco using onsite and online surveys, visitor intercepts, interviews, focus groups and observations with members and visitors. Evaluation projects include formative and summative evaluation of exhibits, planetarium shows, museum websites, and K-12 school field trip programs, as well as summative evaluation of teacher professional development programs, curriculum kits and evening programs for adults. REA has also delivered research studies on capacity building, institutional partnerships, and organizational development.
• 

  Photo courtesy of Lucélia Ribeiro via CC licensing.

  PBS Ready to Learn

  For nearly two decades, REA has conducted formative evaluations on behalf of PBS Kids and their partners as part of the US Department of Education “Ready to Learn” grant program. Through the years, our evaluation efforts have included youth and family playtesting for digital games at various stages during the design process, formative testing of stories for broadcast programming, and extended play studies that examine the impacts of longer-term product-usage, including gameplay and video programming.
• 

  Photo courtesy of Tim Farrell, New Visions.

  New Visions Accessing Algebra through Inquiry (a2i)

  Accessing Algebra Through Inquiry (a2i) was a high school mathematics project conducted by New Visions for Public Schools (New Visions), in partnership with the Silicon Valley Mathematics Initiative (SVMI) and the New York City Department of Education. The project was supported with a 2013 i3 validation grant (#U411B110116) from the U.S. Department of Education. Rockman et al (REA) conducted the independent evaluation of a2i. The central focus of a2i was on the use of high-quality formative assessment processes to help teachers implement high standards and create effective math learning environments. The major categories of support for teachers included: (a) inquiry teams focused on data-driven instructional practices, (b) adaptable formative assessment lessons and strategies, and (c) instructional coaches. A quasi-experimental study involving 40 schools (across 2 cohorts) compared the effectiveness of a2i on students’ achievement in algebra and geometry compared to business-as-usual instruction in matched comparison schools. While the main findings did reveal statistically positive effects of a2i instruction on state standardized test performances, additional analyses by REA found evidence of improved achievement during the second year of the study after professional development was restructured and day-to-day pedagogical practices were introduced. REA evaluation of classroom changes revealed that students engaged in more group work, talked more about math, and solved problems collaboratively.

  Report: Accessing Algebra Through Inquiry Final Report
• 

  Photograph courtesy of Jared Kelly via creative commons licensing.

  RadioLab

  Radiolab’s “What Curiosity Sounds Like: Discovering, Challenging, and Sharing Scientific Ideas” project, funded by the NSF Advancing Informal Science Learning (AISL) program, explored new ways to actively engage both lay and professional audiences and foster meaningful communication between scientists and the general public. As the external evaluator for the project, REA examined participants’ experiences and explored project outcomes through a series of surveys and interviews.

  Report: Radiolab. What Curiosity Sounds Like
• 

  Photo courtesy of ENERGY.GOV via CC licensing.

  Systems Approach to Green Energy (SAGE) Program

  With a grant award from the NSF IGERT program, the University of California, Berkeley’s (UCB) Systems Approach to Green Energy (SAGE) program aims to provide interdisciplinary education and training for UCB graduate students, in order to develop a workforce capable of achieving greener energy technologies. As the external evaluator of the program, Rockman et al is conducting a five-year, mixed-methods study to track progress towards and attainment of SAGE program goals. The study involves multiple questionnaires and focus groups with students and faculty, and focuses on three areas: (1) program infrastructure, (2) faculty collaboration and productivity, and (3) student expectations and experiences.
• 

  Photo courtesy of joaquin uy via CC licensing.

  YR Media

  Since 2006, REA has served as the independent evaluator on YR Media’s education initiatives to engage under-represented youth in informal STEM learning through interest-driven inquiry that combines journalism, design, data, and code. Building on YR Media and MIT App Inventor’s longstanding collaboration to democratize youth media production, the Innovative Approaches to Informal Education in Artificial Intelligence project, funded by NSF-AISL, aims to improve public understanding of artificial intelligence (AI) among young creators and their audiences through youth-led journalism about AI, digital media powered by AI, and open educational resources about how AI works and its role in society. REA conducted the project’s independent evaluation, which included project team interviews, participant observations, focus groups with youth producers, online youth surveys, and audience testing to capture outcomes and lessons learned about effectively engaging youth in learning about AI.

  Report: Youth Radio Innovation Lab Final Evaluation Report
• 

  Photo courtesy of Enokson via CC licensing.

  Leading with Learning

  To improve teaching for English learners (ELs), WestEd partnered with two, large school districts in California on a multiyear project called Leading with Learning: Systematically Transforming Teaching for English Learners. The Leading with Learning (LwL) project was awarded a three-year development grant from the U.S. Department of Education’s Investing in Innovation (i3) fund. Rockman et al conducted the independent evaluation of LwL to study its implementation and to test its effectiveness on elementary student outcomes using a quasi-experimental design. The LwL model involved an intensive, two-year program of professional learning for K–6 teachers, instructional coaches, principals, district co-facilitators, and district leaders. Teacher practice changes were aimed at improving students’ understanding of complex texts, engagement with complex tasks, and understanding of academic English; which was expected to lead to improved academic achievement. Results showed that while the state assessment outcomes did not provide evidence of effectiveness, other data sources revealed important improvements in outcomes for students, teachers, and coaches.

  Report: Leading with Learning Final Report
• 

  Photo courtesy of Maryland GovPics via CC licensing.

  Oakland Unified School District Salad Program

  REA conducted an evaluation of the impact of a salad bar implementation at 5 schools within the Oakland Unified School District (OUSD). Project evaluation included pre-implementation student surveys and interviews, lunchroom observations, and post-implementation interviews with key project stakeholders (students, teachers, and administrators). The evaluation focused on three key areas: 1) Implementation (training, staffing, promotion, availability, methods for sustainability, curricular connections), 2) Participation (methods of adoption/education, level of participation, parent/teacher/food service worker roles) and 3) Attitudes (changes in diet/nutrition-related knowledge, behavior, impact of cultural differences, food likes/dislikes, expectations, influences on food selection, etc.).
• 

  Photo courtesy of Dean Hochman via CC licensing.

  East Bay Center for the Performing Arts Learning Without Borders

  The East Bay Center for the Performing Arts (EBCPA) Learning without Borders (LWOB) professional development program served elementary school teachers in the West Contra Costa Unified School District. Through professional development workshops, coaching, and in-class visits from artists, the program aimed to increase the capacity and confidence of the teachers to integrate arts with other core subject areas. REA collected survey, focus group, and pre- and post-test data from the participating teachers, and this data led to important insights about the teachers’ expectations for the program, feedback about the program, perceptions of the program’s impacts, and recommendations for the program moving forward.
• 

  Photo courtesy of NASA's Earth Observatory via CC licensing.

  American Museum of Natural History “Our Earth’s Future”

  Rockman et al collaborated with the American Museum of Natural History to evaluate onsite and online course offerings as part of the “Our Earth’s Future” series of educational programming about climate change. Evaluation efforts included observations, surveys, and focus groups with face-to-face as well as online course participants, and in addition to examining the unique outcomes for each course offering, compared findings across programs of different lenghts and formats.
• 

  Photo courtesy of Ryan Somma via CC licensing.

  Astronomical Society of the Pacific Galileo Educator Network (GEN)

  With funding from NASA, the Astronomical Society of the Pacific (ASP) developed the Galileo Educator Network (GEN), which prepared teacher educators nationwide to design and deliver professional development workshops on astronomy content and science practices aligned with Next Generation Science Standards (NGSS). REA collected surveys and observations to describe the ways in which GEN engaged K-12 teachers and promoted the effective use of NASA-developed resources. REA presented its evaluation findings at two ASP annual meetings.
• 

  Photo courtesy of Wars via CC licensing.

  Water: Exploring Science in the Studio

  In collaboration with California College of the Art (CCA), Rockman et al conducted an evaluation of “Water: Exploring Science in the Studio,” including course development and delivery processes, and instructional and assessment elements. The broad objective of this project was to increase the ability of CCA students to interact with scientists within the context of a series of four design courses. REA developed and administrated online surveys to CCA art faculty and visiting science faculty who collaborated on the development and delivery of these courses.
• 

  Photo courtesy of MusikAnimal via CC licensing.

  NYSCI Design Lab

  REA provided formative and summative evaluation for The New York Hall of Science’s (NYSCI) Design Lab, a multi-faceted project which consists of a unique, museum-based innovation laboratory, a series of permanent onsite exhibits and teacher professional development program that supports design-based approaches to science, technology, engineering, and math (STEM) teaching and learning. Over a three year period, REA staff conducted formative evaluation of onsite Design Lab exhibits, summative evaluation of Teacher Summer Institutes. and documented the impact of ongoing professional development through observations of participating teachers and their students when back in the classroom. REA also completed a retrospective study to assess long term impact with teacher graduates from each of program’s four cohorts. Evaluation methods included online surveys, interviews and observations.
• 

  Photo: BAVC News

  Bay Area Video Coalition Bridges to STEM Careers

  REA conducted a formative and summative evaluation of Bay Area Video Coalition’s Bridges to STEM Careers, a three-year Advanced Technological Education (ATE) Program Improvement Project designed to enhance the relevance of technician education in Computer Science and Multimedia. The project targeted two-year college students, their families and their college and workplace mentors. The ultimate goal of the Bridges project was to keep traditionally underrepresented and low-income students in the STEM pipeline by demystifying the process of preparing for a STEM career through internships and informational videos from past internship participants.