Observation Protocols

Selected Projects Include

• de Young Museum’s Get Smart With Art Program

  REA conducted an evaluation of the de Young Museum’s Get Smart With Art program for grades 1-3, and 4-8. Get Smart with Art @ the de Young is an interdisciplinary curriculum package that uses art objects as primary documents, sparking investigations into the diverse cultures represented by the Museum’s collections. The evaluation included a logic model workshop, two museum gallery and museum classroom observations, and a review of curricular materials. A primary purpose of this evaluation was to provide de Young Education staff with methods and criteria for conducting a more thorough and comprehensive evaluation of the entire program at a later date. The final report also included proposed evaluation methods and instruments (for classroom, gallery and curriculum materials) and aligned research questions.
• Tech Know Build

  The TechKnowBuild project was a partnership between the Crawfordsville Community Schools and Indianapolis Public Schools that provided “one to one” laptop computers and wireless Internet access to 2,500 middle school students and approximately 100 teachers, combining ubiquitous computing and problem-based learning. As the external evaluator, REA assessed impacts on teachers and students through suveys and secondary data analyses. During the final year, student achievement was measured by standardized test scores and writing prompts. Almost all teachers incorporated internet research and exploration, one of the most popular and regular use of laptops, and generally found problem-based learning to be an effective way to meet their academic content standards. REA found that problem-based learning activities increased students’ engagement in school, and gave students a sense of ownership for the topics they researched and a deeper awareness of community issues and their roles as citizens.

  Report: One to One Laptop Programs
• 

  Photo courtesy of Tim Farrell, New Visions.

  New Visions Accessing Algebra through Inquiry (a2i)

  Accessing Algebra Through Inquiry (a2i) was a high school mathematics project conducted by New Visions for Public Schools (New Visions), in partnership with the Silicon Valley Mathematics Initiative (SVMI) and the New York City Department of Education. The project was supported with a 2013 i3 validation grant (#U411B110116) from the U.S. Department of Education. Rockman et al (REA) conducted the independent evaluation of a2i. The central focus of a2i was on the use of high-quality formative assessment processes to help teachers implement high standards and create effective math learning environments. The major categories of support for teachers included: (a) inquiry teams focused on data-driven instructional practices, (b) adaptable formative assessment lessons and strategies, and (c) instructional coaches. A quasi-experimental study involving 40 schools (across 2 cohorts) compared the effectiveness of a2i on students’ achievement in algebra and geometry compared to business-as-usual instruction in matched comparison schools. While the main findings did reveal statistically positive effects of a2i instruction on state standardized test performances, additional analyses by REA found evidence of improved achievement during the second year of the study after professional development was restructured and day-to-day pedagogical practices were introduced. REA evaluation of classroom changes revealed that students engaged in more group work, talked more about math, and solved problems collaboratively.

  Report: Accessing Algebra Through Inquiry Final Report
• 

  Appalachian Support for Specialized Education Training (ASSET)

  The Appalachian Support for Specialized Education Training (ASSET) project was funded through the Department of Education’s Supporting Effective Educator Development (SEED) grant program and sought to develop and provide high-quality professional development micro-credentials for Appalachian teachers. The REA research team designed and implemented a randomized control trial with more than 600 participants. In addition to the RCT, we gathered formative evaluation data on each micro-credential from a teacher advisory board and early micro-credential completers and studied iterative versions of simulations designed to assess participants’ performance within one of the mathematics micro-credentials.
• 

  Photo courtesy of CERDEC Math and Science Summer Camp via CC licensing.

  Odd Squad Summer Camp

  Rockman et al worked with PBS KIDS to evaluate a pilot of the Odd Squad Camp program, including observations at two summer camps, youth and parent surveys, and interviews with participants, program facilitators, and participants’ parents.
• 

  Dinosaur Train Evaluation

  Rockman et al conducted a series of evaluative studies for Dinosaur Train, an animated series geared towards a pre-school aged audience, produced by the Jim Henson Company for PBS Kids. The evaluation included an online survey, focus group sessions with parents and children, data collection from families at the Pittsburgh Children’s Museum, and in-home observations and interviews with families children viewing Dinosaur Train episodes in Bloomington, IN.
• 

  Photo courtesy of AfroTech.

  Black Girls Code

  Black Girls Code (BGC) is an out-of-school program designed to provide girls of color opportunities to learn in-demand skills in technology and computer programming during a time when they are considering career paths. The goal of this program is twofold: To introduce programming and technology to a new generation of coders and to sustain this interest well into high school and college. Since 2012, REA has examined the impact of BGC’s Culturally-Relevant STEM programming that embraces the intersectionality of its students (culture, race/ethnicity/gender), sense of belonging for girls of color, short and long-term program impact, and career inspirations and pathways. We have utilized case study design, focus groups, interviews, and surveys to measure BGC’s impact.

  Report: Black Girls Code Final Report
• Star Schools

  Rockman et al conducted a three-year study of the impact of the online, Supplementary Education Services (SES) provided by Educate Online, Inc. (EO), funded by the Star Schools Program of the U.S. Department of Education. The evaluation activities under the grant measured the effectiveness of Catapult Online, a real-time, one-on-one tutoring service designed to address the needs of underserved rural, middle and high school students. The formative and summative evaluations examined online student assessment instruments, content, delivery mechanism, and the enhanced parental communication via mobile technology. REA also conducted a five-year evaluation of critical issues associated with the delivery of SES via online technologies in rural and urban areas. Among these issues were strategies for motivating and engaging middle and high schools students in remedial and supplemental programs, serving the needs of students with limited English proficiency, the impact of evolving technology platforms on the delivery of instructional programs, and emerging online assessments.

  Report: starschoolsreport
• 

  Photo courtesy of Amanda Golden via CC licensing.

  Elmo Loves ABCs and Elmo Loves 123s

  Rockman et al worked with the Sesame Workshop to develop and administer a basic literacy and mathematics assessment for preschool-aged students in order to test outcomes of using either Elmo Loves ABC’s or Elmo Loves 123’s for a month-long period as part of the normal school-day experience. To facilitate this project, the Rockman team managed a set of 10 iPad devices (with unique copies of apps for each of 90 students in the study) and a team of 8 researchers.
• 

  Photo courtesy of Juan Cristóbal Cobo via CC licensing.

  PBS Kids: Raising Readers

  REA conducted a study of PBS Kids Island with twenty-one English-speaking children ages 3-8 to learn about site usage patterns over an extended period of time, whether children found the content to be age-appropriate, fun, and accessible, and the overall appeal of the site to children. REA conducted usability testing for the PBS Kids Raising Readers website, including versions in Spanish and English. Working with PBS contacts, Hispanic community organizations, and public libraries in three US sites, we conducted more than 69 user testing sessions with parents, teachers and caregivers of children ages 2-8, focusing on children from low-income families and including bilingual teachers and caregivers of Spanish-speaking children.
• 

  Photo: Jennifer Borland

  Museum of Science and Industry “Our Place in Space”

  Rockman et al is the external evaluator for “Our Place in Space,” a professional development program for formal and informal science educators designed and implemented by the Museum of Science and Industry in Chicago, with funding from NASA. Through observations, surveys, and interviews, REA is studying how the program helps prepare educators to cover Next Generation Science Standards and implement a series of high-quality STEM instructional activities. REA’s evaluation seeks to demonstrate the value of the program as well as suggest potential areas for improvements in subsequent programming.
• Kamehameha Maui Laptop Project

  The Kamehameha Schools (KS) Maui Campus implemented a laptop program at the beginning of the 2003 – 2004 academic year. As part of their “one to one” program, KS provided laptop computers to all high school faculty and students for use at both home and school. Rockman et al collaborated with the Policy Analysis and System Evaluation (PASE) research staff to design a model and instruments for evaluation of the laptop program. Rockman et al was contracted to conduct administrator interviews with key program staff as well as classroom observations and teacher interviews.

  Report: Kamehameha Schools Laptop Report
• 

  Photo courtesy of US ED via CC licensing.

  ReadWorks

  ReadWorks worked to support reading comprehension instruction by providing research-based, grade-level aligned instructional materials and tools to over 1 million K–12 teachers and 13 million students. REA partnered with ReadWorks to design and conduct a series of multi-year evaluation studies investigating both the implementation and impact of ReadWorks’ curriculum and teacher training materials. The evaluation research included an investigation of web analytics data, a mixed-methods retrospective study focused on teacher usage of the website and curriculum, a multi-year study of ReadWorks-supported teacher practice networks, and a prospective efficacy study of their Article-A-Day curriculum in high-needs, urban classrooms. Teachers reported valuable gains from using ReadWorks to develop students’ content knowledge, vocabulary, reading comprehension skills, and reading motivation.
• 

  Photo courtesy of Enokson via CC licensing.

  Leading with Learning

  To improve teaching for English learners (ELs), WestEd partnered with two, large school districts in California on a multiyear project called Leading with Learning: Systematically Transforming Teaching for English Learners. The Leading with Learning (LwL) project was awarded a three-year development grant from the U.S. Department of Education’s Investing in Innovation (i3) fund. Rockman et al conducted the independent evaluation of LwL to study its implementation and to test its effectiveness on elementary student outcomes using a quasi-experimental design. The LwL model involved an intensive, two-year program of professional learning for K–6 teachers, instructional coaches, principals, district co-facilitators, and district leaders. Teacher practice changes were aimed at improving students’ understanding of complex texts, engagement with complex tasks, and understanding of academic English; which was expected to lead to improved academic achievement. Results showed that while the state assessment outcomes did not provide evidence of effectiveness, other data sources revealed important improvements in outcomes for students, teachers, and coaches.

  Report: Leading with Learning Final Report
• 

  Photo courtesy of NOAA Photo Library via CC licensing.

  Exploratorium Carbon Networks

  REA is the external evaluator for ‘Carbon Networks,’ a federally-funded project which is a collaboration between the Exploratorium, the Pacific Science Center, and the Waikiki Aquarium. This project aims to assist museum educators in building capacity for developing educational programs and activities for public audiences and formal educators on the topics of climate change and ocean acidification. REA is conducting formative evaluations of educator workshops and summative evaluation of program impacts at each of the participating institutions, using online surveys, interviews and observations.
• 

  Photo courtesy of Hudson Alpha.

  It’s Complex! Engaging Student Discussions around Complex Genetics and Individualized Medicine

  Rockman et al is studying the impact of Touching Triton, an online space-themed interactive created by the HudsonAlpha Institute for Biotechnology. The game is designed to increase high school students’ knowledge about and interest in genetic and environmental factors that might influence the likelihood of developing a complex disease. To inform the design, REA conducted student focus groups, teacher interviews, and classroom observations. REA has also provided consulting on quasi-experimental studies to assess summative impacts. Touching Triton was selected as a finalist in the 2015 Serious Games Showcase and Challenge in the Business Category.
• DragonflyTV’s Informal Science Education Project

  Rockman et al developed an impact study for the educational television program: DragonflyTV, a science program for and about 9-12 year old children produced by Twin Cities Public Television. A controlled classroom viewing protocol was established for schools in Chicago and San Francisco. Students were surveyed following each of three episodes selected for the study. Pre and post- interviews with students introduced a hypothetical scenario in which they were asked to design an experimental study. An additional in-class viewing study was also implemented with educators in a variety of locations across the United States.
• 

  Photo courtesy of spacedust2019 via CC licensing.

  Bay Area Discovery Museum

  REA has conducted numerous research and evaluation projects with the Bay Area Discovery Museum (BADM), one of the country’s leading children’s museums dedicated to the emotional and cognitive growth of young children. REA has led evaluations of the Try-it-Truck, a mobile engineering lab serving K-5 students, libraries and community centers, and workshops for Head Start parents on the importance of executive function in young children. We also developed of an observation protocol for BADM education staff to assess impact of onsite museum programs.
• 

  Photo courtesy of Chris Betcher via CC licensing.

  Convergence Academies Project

  The Center for Community Arts Partnerships at Columbia College Chicago, in partnership with Chicago Public Schools, received a development grant from the Investing in Innovation Fund (i3) of the U.S. Department of Education for their Convergence Academies project (http://convergenceacademies.org/ourorigins.html). To support 21st Century learning, CCAP developed a whole school reform model, inspired by principles and values of connected learning, that involved inquiry-based instruction and integration of digital media arts and technology throughout the school. The model was implemented in Chicago in both a high school and a feeder elementary school (K-8). As the external evaluator for the project, Rockman et al conducted studies of both implementation and impact, including an interrupted time-series design to investigate gains on state tests in math and reading.

  Report: Evaluation of the Convergence Academies Initiative Final Report
• 

  Photo courtesy of Julia Li.

  Peg + Cat: Adventures in Learning

  Peg + Cat is a math-based PBS television program for preschoolers, supplemented by online games, digital storybooks, and apps. We conducted formative and summative evaluation of Peg + Cat interactive media and themed science camp activities, developed by Fred Rogers Productions (formerly The Fred Rogers Company) and Carnegie Science Center as part of an NSF-funded AISL project. We examined pre-schoolers’, families’, and educators’ experiences using project resources, as well as pre-schoolers’ social-emotional and learning outcomes via observations, focus groups, interviews, and a quasi-experimental design that incorporated pre-post surveys and content assessments.
• Game-Enhanced Interactive Physical Science (GIPS)

  Through funding from the NSF’s SBIR program, Filament Games developed a suite of online educational games and curricular activities for middle school classrooms that target core scientific concepts, such as engineering, plant genetics, and planetary characteristics. Rockman et al evaluated the “promise of efficacy” from each game via pre-post surveys, classroom observations, and embedded assessments, comparing content learning and engagement between students who played the game and engaged with the curricular activities, only played the game, or received traditional instruction. Differences in outcomes based on students’ interest in and aptitude for reading were also explored to ensure that the games were inclusive of a range of student abilities.