Evaluation and assessment is an integral part of The Materials World Modules Program. MWM takes pride in customizing our products to suit users in terms of diversity, culture, language, and grade levels, by continuously reviewing and enhancing our contents, services, and communication. This unique service is achieved through qualitative and quantitative evaluation and assessment methodologies. The following examples illustrate how evaluation and assessment have been carried out in various parts of the world.
U.S. National Field Test Assessment Protocol
A randomized nation-wide field test was conducted between 2002 and 2006 to evaluate the effectiveness of the MWM modules as supplementary curricular materials. Prior to the actual evaluation activities, a team of highly experienced secondary science educators and MWM content developers wrote and validated student assessment items for an item bank that would accompany the modules being evaluated. The "Validity Team" addressed content validity, construct validity, face validity as well as issues related to gender and cultural bias, syntactical style, hierarchy of questioning asking, etc. Each module consists of a set of 60 to 80 assessment items, which included multiple choice, short answer, and long answer questions.
Field-test teachers used the bank of validated assessment items to compose unique classroom pre and post tests that matched their goals. We also limited the study to one science classroom per school. Eight MWM modules were field-tested in 118 randomly selected schools located in 42 states across six (6) geographical regions of the country. Each of the randomly selected 118 field test sites, in effect, became a separate research entity. The field test sample was found to be statistically representative of the nation as a whole in terms of U.S. geographical distribution, NCES population code, student gender, teacher gender, teacher level of academic preparation, and teacher years of experience. A total of 2297 students participated in the study that spanned 40 titles of science classes, (physical science, AP chemistry, biotechnology, physics, intro to engineering, etc.) From the total sample of students, we received complete data packets from 2026 of them, amounting to a usable classroom return rate of 88.2%. The study used a quasi-experimental pre-post design and then aggregated results using meta analytical techniques.
This study was reported in “Materials World Modules - 2002: A Nationally Representative Evaluation of Classroom Gains” by Barbara J. Pellegrini, in Journal of Materials Education, 32(5–6) 185, 2010. The study showed similar gains regardless of student gender or socio-economic status, years of teacher experience or classroom settin
Wherever MWM is introduced, students respond with enthusiasm, creativity and high achievement gains. Field test teachers from 118 schools and across 42 U.S. States used the Materials World Modules as a two-week design supplement to the typical canon of science curricula. On average, classrooms gained 2.66 standard deviations, or an average of 31.8% over their pretest means (see accompanying figure). Girls gained significantly more than boys both in terms of content acquisition and design achievement. However, boys gained more in terms of science esteem. Teachers and students reported improved acquisition of science processes and design skills, and both teachers and students reported being moderately satisfied with the module experience.
The evaluation examined the results of classroom field tests according to their geographical location across six regions of the country (NE, SE, NC, SC, NW, SW). The results showed that there was no significant difference in effect size (p <.061); normalized gain ( p <.549); percent value added (p <.279), and design score (p <.123). Classrooms across all regions of the country could do just as well.
The evaluation also examined the results of classroom field tests in eight locales of school settings that were described by the National Center for Educational Statistics (NCES) according to population density codes (e.g. large city, mid sized city, suburban, small town, rural, etc.) The results of the evaluation showed that there was no significant difference among the eight types of NCES locales in terms of effect size (p < .983); normalized gain (p <.895); percent value added (p <.462), and design score (p <.905.) Classrooms in all types of locale settings appeared to do just as well.
In summary, the U.S. national field-test data suggested that the modules successfully provided students with a sense of excitement about science and engineering design. The data also supported a significant rise in classroom science esteem levels as well as successful implementation of modules as supplemental science curricula with notable gains, regardless of classroom setting, length of teacher experience, or student socio-economic background.
MWM program was adopted in Chihuahua, Mexico since 2005. Statistical results from assessment studies during the past several years are provided in Journal of Materials Education Vol. 32 (5-6): 245 - 254 (2010) by Fuentes et al. Some of the pertinent results are listed below:
The MWM project has grown steadily since its inception in 2005.
The number of new teachers who have been trained in the MWM methodology per year, from 2005 to 2010:
A survey of students who have completed the Modules have shown the following:
- 68% of the students agreed that the MWM program has improved their progress in STEM education.
- 74% of the students agreed that the MWM program has motivated them to reassess the scientific and technology knowledge.
- 62% of the students agreed that the MWM has inspired them to consider a career in science or technology.
In the fall of 2011, our research team conducted another pilot experiment using MWM modules in 13 independent secondary schools in Qatar over a two-month period between October and November of 2011. A total of eight modules were used with roughly 300 students at seven boys' schools and six girls' schools. Each module was implemented at the local school site and taught by a MWM staff member from Northwestern University, assisted by a Qatar University staff member (also a Qatar high school science teacher) as well as the classroom teacher and technicians. Typically, students at each school spent an entire week on one module predetermined for the school. The total time students were engaged in the module activities averaged about 10-12 hours, roughly 2.5-3 hours each day spread out over a four-day period. During the first two days, students worked in teams of 3-5 students and engaged in a series of 4-5 hands-on, exploratory experiments that focused on material properties and concepts central to the module theme. Students then spent the third and fourth days engaged in two design experiences. The first was a "structured" design challenge, which provided students a clear target and a goal to tie together their previous learning experiences. Throughout this "inquiry through design" process, students were expected to ask questions to inform their design decisions, weigh the inevitable trade-offs, debate about a particular design approach's values and limitations, connect these design decisions to scientific principles, make and test prototype designs, and generate data to validate their decisions. The second design challenge was an "open-ended" product design, relevant to the module theme, in which students were challenged to innovate and create an entirely new product or an improved product design based on the concepts they had learned. On the fifth and final day, students were given the opportunity to present the results of their design projects and share their module experiences with the staff at Qatar University's Materials Technology Unit.
Based on three complete data sets (out of 13 classes) gathered to date, the pre-post assessments showed that students have made positive gains in the key concepts taught in all three modules: Nanotechnology, Polymers, and Composites. Though the sample size was small, there is a clear trend that students were progressing towards higher post-test scores. The combined average effect size from pre-test to post-test was 1.12. This encouraging result led the MWM team to continue its implementation efforts in the spring of 2012.
Results regarding students' attitudes, specifically self-efficacy were also studied. These results showed an improvement in the students' attitudes towards research and invention. The percentage of the students who strongly agreed that they were capable of invention increased from 12% to 24%, and those who agreed went from 40% to 60%. The percentage of students who strongly agreed with the statement regarding the importance of collaborative work increased from 40% to 48% while the students who strongly agreed that their skills had increased after participating in scientific experiments increased from 48% to 72%.
In conclusion, even from these preliminary data sets, we have seen a positive shift in students' gain of new scientific knowledge. The design projects have also solidified this knowledge as they demonstrated their ability to complete the design challenge, as well as showing their ingenuity in creating new or improved products.
In July, 2012, a two-week MWM workshop was held in China. Three selected MWM modules were used by 30 teachers and 120 students from 6 Xian Metropolitan high schools. The pre-post assessment of the teachers' workshop yielded an effect size of 2.7 for the combined score of the three modules, while a combined effect size of 1.3 was obtained by the students. The lower student gain was attributed to linguistic proficiency, which can be resolved by translation.
Based on these initial positive results, MWM is working jointly with the local Educational Ministry of Shaanxi province to expand the teacher training program so that students in the province will start using the MWM program.