Education industry giant WestEd recently completed a scientific-based research study on the First In Math (FIM) Program in the National City School District in San Diego County. The evaluation involved 2,412 students in grades 3 through 6 and focused primarily on the impact of the FIM program on student achievement. The evaluation studied if the amount of time students spent on FIM affected their mathematics skills, and if participation in FIM had a positive effect on students' attitudes.
WestEd's evaluation of the First In
Math® (FIM) Online Mathematics Program in the National School
District in San Diego County focused on three research
questions:
1) Does participation in FIM have a positive effect on students'
attitudes;
2) Do teachers use students' performance on FIM to make
instructional decisions; and
3) Does the amount of time students spend on FIM (dosage) affect
their mathematics skills?
WestEd drew upon multiple data sources to
answer the research questions. Evaluators interviewed teachers
regarding their use of the FIM program and surveyed students on
their attitudes toward FIM and mathematics in general. Evaluators
also analyzed quantitative data from secondary sources, including
FIM usage data and student achievement data as measured by the
California Standardized Test and Reporting Program (STAR).
WestEd used analysis of covariance
modeling techniques to examine characteristics that were associated
with program use, Skill Set® attainment, and student achievement.
This enabled WestEd to statistically control for student
demographics and previous year achievement, as well as school
related factors.
The evaluation involved a relatively large
sample group of 2,412 students in grades 3 through 6 during the
2003-2004 school year. The evaluation group is comparable to the
demographic composition of the National School District in terms of
student ethnicity. Nearly 8 of the 10 students in this evaluation
group (78.6%) are identified by the STAR data as Hispanic.
Download an 8
1/2'' x 11'' printable version of the full, 40-page WestEd
report
FINDINGS
FIM and Effects on Student
Achievement
• FIM had small but significant effects on California Standards
Test (CST) student achievement scores across each grade level.
There was a differential effect of the program by grade level, with
3rd grade students benefiting the most from FIM exposure.
• FIM had small but significant effects on
California Achievement Test (CAT/6) student scores across all grade
levels. FIM exposure had a larger impact on student achievement
than student demographic variables, except for students' CAT/6
pre-test score (in all grades) and the effect of attending a
particular school (in some grades).
• Controlling for students' previous
achievement record, students at each grade level who complete
higher FIM Skill Set® levels also had higher CST and CAT/6
achievement scores.
• Comparing the mathematics NCE (Normal
Curve Equivalents) gain scores of the FIM cohort with the average
gain scores of California as a whole, the study found that students
in 3rd, 5th, and 6th grades made positive achievement gains in
their CAT/6 scores above and beyond the progress typically
experienced from year to year.
• Examination of the impact of FIM on
particular CST question clusters revealed differential effects of
FIM on CST questions clusters in each grade. Further, the program
has small but significant effect on particular CST content areas.
For example, FIM had the largest effects in Clusters 1-3 for 3rd
graders. FIM seems particularly beneficial in facilitating 3rd
graders understanding of basic number sense, operations, and
algebraic relationships. FIM seems to be particularly effective for
5th grade students in reinforcing concepts that are introduced at
Skill Set® levels 5-8, including estimation, percents, factors,
algebra, and functions. For 6th grade students, FIM has the largest
effect on statistics, data analysis, and probability. The
California content standards for 6th graders in this area for this
grade level include students' understanding of how to use fractions
and percentages to compare data. Placing this finding in the
context of both underlying concept of FIM and its' focus on
fractions and decimals, provides one explanation of this large
effect.
Student/Teacher Attitudes about
FIM
• Nearly three-quarters of the students surveyed (72 percent)
agreed with the statement "math lessons are fun." Teachers
overwhelmingly agreed that students enjoyed the program and sought
out time to use the FIM website.
• Teachers reported that FIM increased
student's interest in mathematics, especially those students who
were previously not engaged in the classroom. Student survey data
also revealed that students found mathematics interesting and
relevant beyond the classroom.
• Teachers reported that the program
seemed well-suited for their underachieving or disadvantaged
students. Several teachers speculated that because students used
FIM alone (as opposed to playing the 24® game with peers) that
students were more likely to explore the games using the
computer.
• Teachers reported that FIM encouraged
students to try problems on their own, even those problems not yet
covered in class. Seventy seven percent of students believed that
it was better to "solve a math problem myself" than to ask the
teacher for the answer, and 67 percent said they liked trying to
solve hard math questions on FIM without their teachers help.
• Teachers reported that students were
excited about the game format and motivated to accumulate the award
stickers and help their class "team" achieve higher rankings.
Because students are grouped in a "team" with their fellow
classmates, they helped one another learn by discussing and sharing
strategies for solving the problems.
• Nearly all teachers noted that they used
the program to motivate students. Students were allowed to use the
program after using their "regular" classwork, which prompted
students to finish their assigned work on time. No teacher
interviewed used FIM for grading purposes. They felt tying FIM to
grades would negatively impact students' motivation and enthusiasm
for the program.
Teachers' Comments
"This is the first time I've had so many kids do so well and have
such a good buy-in. Usually I have maybe four or five kids that
really like something like that. The rest are like, 'oh, do I have
to.' But if I say you can do First In Math game, it's like, 'yay!'
You know, it's everybody now. We have to almost take numbers or do
first-come-first served. I have seven computers at lunchtime, and
whoever gets in there first, the first seven people get on. They
love it."
"I have a class that has had a lot of
turnover this year. And some of my new kids are from Mexico and
have difficulty with English. And they've taken right up with
(FIM). They want to come. That's all they want to do. They want to
sit there and do that. One of the students never went to second
grade anywhere, and they put her in third grade here, and she's
improving. She keeps plodding away and doing it. I have noticed
that some kids that have done really well on First in Math were the
low-achievers, which to me is a big thing, because they were
struggling with math before."
"After the student is clicking the mouse,
it's clicking in their brain. And they're catching on. The highest
student in my class is a low achiever, historically. But she just
took off, and she just clicked and clicked, and it took a long time
for each Skill Set, but she's finished every one of them. And she
is proud of the fact that she's at the top of the class. But she
taught herself. That wasn't me teaching her. And that's what math
is, looking for the patterns and doing that work over and over
again, figuring out your own algorithms. That's going to stick with
her longer than what I say."
Download an 8
1/2'' x 11'' printable version of the full, 40-page WestEd
report
WestEd is known for the quality of their
assessment work has earned the designation by the U.S. Department
of Education as the nation's leading Regional Education Laboratory
(REL) in assessment. (www.wested.org)