Scientific-Based Research Study conducted by WestEd: Evaluation of the First In Math® Online Mathematics Program

Scientific-Based Research Study conducted by WestEd: Evaluation of the First In Math® Online Mathematics Program

Education industry giant WestEd recently completed a scientific-based research study on the First In Math (FIM) Program in the National City School District in San Diego County. The evaluation involved 2,412 students in grades 3 through 6 and focused primarily on the impact of the FIM program on student achievement. The evaluation studied if the amount of time students spent on FIM affected their mathematics skills, and if participation in FIM had a positive effect on students' attitudes.

WestEd's evaluation of the First In Math® (FIM) Online Mathematics Program in the National School District in San Diego County focused on three research questions:
1) Does participation in FIM have a positive effect on students' attitudes;
2) Do teachers use students' performance on FIM to make instructional decisions; and
3) Does the amount of time students spend on FIM (dosage) affect their mathematics skills?

WestEd drew upon multiple data sources to answer the research questions. Evaluators interviewed teachers regarding their use of the FIM program and surveyed students on their attitudes toward FIM and mathematics in general. Evaluators also analyzed quantitative data from secondary sources, including FIM usage data and student achievement data as measured by the California Standardized Test and Reporting Program (STAR).

WestEd used analysis of covariance modeling techniques to examine characteristics that were associated with program use, Skill Set® attainment, and student achievement. This enabled WestEd to statistically control for student demographics and previous year achievement, as well as school related factors.

The evaluation involved a relatively large sample group of 2,412 students in grades 3 through 6 during the 2003-2004 school year. The evaluation group is comparable to the demographic composition of the National School District in terms of student ethnicity. Nearly 8 of the 10 students in this evaluation group (78.6%) are identified by the STAR data as Hispanic.

Download an 8 1/2'' x 11'' printable version of the full, 40-page WestEd report


FIM and Effects on Student Achievement
• FIM had small but significant effects on California Standards Test (CST) student achievement scores across each grade level. There was a differential effect of the program by grade level, with 3rd grade students benefiting the most from FIM exposure.

• FIM had small but significant effects on California Achievement Test (CAT/6) student scores across all grade levels. FIM exposure had a larger impact on student achievement than student demographic variables, except for students' CAT/6 pre-test score (in all grades) and the effect of attending a particular school (in some grades).

• Controlling for students' previous achievement record, students at each grade level who complete higher FIM Skill Set® levels also had higher CST and CAT/6 achievement scores.

• Comparing the mathematics NCE (Normal Curve Equivalents) gain scores of the FIM cohort with the average gain scores of California as a whole, the study found that students in 3rd, 5th, and 6th grades made positive achievement gains in their CAT/6 scores above and beyond the progress typically experienced from year to year.

• Examination of the impact of FIM on particular CST question clusters revealed differential effects of FIM on CST questions clusters in each grade. Further, the program has small but significant effect on particular CST content areas. For example, FIM had the largest effects in Clusters 1-3 for 3rd graders. FIM seems particularly beneficial in facilitating 3rd graders understanding of basic number sense, operations, and algebraic relationships. FIM seems to be particularly effective for 5th grade students in reinforcing concepts that are introduced at Skill Set® levels 5-8, including estimation, percents, factors, algebra, and functions. For 6th grade students, FIM has the largest effect on statistics, data analysis, and probability. The California content standards for 6th graders in this area for this grade level include students' understanding of how to use fractions and percentages to compare data. Placing this finding in the context of both underlying concept of FIM and its' focus on fractions and decimals, provides one explanation of this large effect.

Student/Teacher Attitudes about FIM 
• Nearly three-quarters of the students surveyed (72 percent) agreed with the statement "math lessons are fun." Teachers overwhelmingly agreed that students enjoyed the program and sought out time to use the FIM website.

• Teachers reported that FIM increased student's interest in mathematics, especially those students who were previously not engaged in the classroom. Student survey data also revealed that students found mathematics interesting and relevant beyond the classroom.

• Teachers reported that the program seemed well-suited for their underachieving or disadvantaged students. Several teachers speculated that because students used FIM alone (as opposed to playing the 24® game with peers) that students were more likely to explore the games using the computer.

• Teachers reported that FIM encouraged students to try problems on their own, even those problems not yet covered in class. Seventy seven percent of students believed that it was better to "solve a math problem myself" than to ask the teacher for the answer, and 67 percent said they liked trying to solve hard math questions on FIM without their teachers help.

• Teachers reported that students were excited about the game format and motivated to accumulate the award stickers and help their class "team" achieve higher rankings. Because students are grouped in a "team" with their fellow classmates, they helped one another learn by discussing and sharing strategies for solving the problems.

• Nearly all teachers noted that they used the program to motivate students. Students were allowed to use the program after using their "regular" classwork, which prompted students to finish their assigned work on time. No teacher interviewed used FIM for grading purposes. They felt tying FIM to grades would negatively impact students' motivation and enthusiasm for the program.

Teachers' Comments
"This is the first time I've had so many kids do so well and have such a good buy-in. Usually I have maybe four or five kids that really like something like that. The rest are like, 'oh, do I have to.' But if I say you can do First In Math game, it's like, 'yay!' You know, it's everybody now. We have to almost take numbers or do first-come-first served. I have seven computers at lunchtime, and whoever gets in there first, the first seven people get on. They love it."

"I have a class that has had a lot of turnover this year. And some of my new kids are from Mexico and have difficulty with English. And they've taken right up with (FIM). They want to come. That's all they want to do. They want to sit there and do that. One of the students never went to second grade anywhere, and they put her in third grade here, and she's improving. She keeps plodding away and doing it. I have noticed that some kids that have done really well on First in Math were the low-achievers, which to me is a big thing, because they were struggling with math before."

"After the student is clicking the mouse, it's clicking in their brain. And they're catching on. The highest student in my class is a low achiever, historically. But she just took off, and she just clicked and clicked, and it took a long time for each Skill Set, but she's finished every one of them. And she is proud of the fact that she's at the top of the class. But she taught herself. That wasn't me teaching her. And that's what math is, looking for the patterns and doing that work over and over again, figuring out your own algorithms. That's going to stick with her longer than what I say."

Download an 8 1/2'' x 11'' printable version of the full, 40-page WestEd report

WestEd is known for the quality of their assessment work has earned the designation by the U.S. Department of Education as the nation's leading Regional Education Laboratory (REL) in assessment. (