What does it mean to field test standards?

A reader over at my other blog asked some questions about how the standards were developed and about  this idea of field testing standards that has been floating around recently. On the question of field testing, he says

She [Diane Ravitch] also talks about how the standards were never field-tested. I may be wrong, but it seems to me there’s nothing to test about the standards themselves. It’s simply a decision about what we want students to know and be able to do.

This is eminently sensible. Standards are an agreement; you can’t field test an agreement. You can field test curriculum and assessment based on the agreement, and at some point you may want to revise the agreement as a result of what you learn. But field testing the standards themselves would have meant not having common standards; and so would have told you nothing about the advantages or disadvantages of having common standards. The only way to field test the standards would be to have some parallel United States, just like ours, but where different common standards were adopted. And follow it for 10 years.

In fact, we did do the closest thing we could do to field testing: we looked carefully at standards of high achieving countries and states. You could think of this as a sort of observational study, which is what you do when a field test isn’t feasible.

The other question my reader had was about the role of teachers in developing the standards, and the opportunity for public comment. I’ll answer that one tomorrow.

About Bill McCallum

I was born in Australia and came to the United States to pursue a Ph. D. in mathematics at Harvard University, met my wife, and never went back. I am a professor at the University of Arizona, working in number theory and mathematics education.

4 thoughts on “What does it mean to field test standards?

  1. I agree with the post but think there’s a point people are trying to make which you’re not quite addressing. Why didn’t we for example take a set of N(=5?) new carefully written standards and field test each of those at some smaller level before choosing which one to use as the basis of a Common Core?

    I know one answer: we were doing that already to large degree with N=50, with results which were disconcerting to anyone who looked closely enough (e.g. 40% remediation rates in college). And another answer: if we tried to run such a trial who could tell if success or lack thereof was due to the quality of the standards vs. the policy framework of the districts/states in the trial vs. the resources vs.… And another: for such trials, you wouldn’t attract nearly the amount of talented people as we are seeing now to produce strong curricula, assessments, instructional techniques and professional learning which are needed, because such people would be dispersed and wouldn’t be so interested in jumping in if their work is likely to be tossed aside. So maybe you could have a sense for which of the standards may be better, but you’d still be far from certain about the amount of impact.

    Finally, as a parent and a mathematician, I think the sooner we can all start working together to put out quality resources to support learning, the better.

    1. Thanks Dev, I think you’ve answered your question very well. But I agree that the idea of field testing standards is not quite as infeasible as I suggested in my post; i.e., you don’t literally need a parallel universe. However, I don’t know of any country or state that has done what you suggest. They’ve done what we did: build on previous experience, the wisdom of practice, and the knowledge of teachers and scholars.

  2. The question about field testing is really a question about whether there are good grounds to believe that the Common Core is a good idea and is likely to raise mathematics achievement. I think there are. Research informed the development of the standards, including international and domestic studies of mathematics performance. The working group included researchers in mathematics education who drew on their expertise. In 2012, a peer-reviewed study found that states whose previous standards better matched the Common Core tended to have better NAEP scores. Has any previous set of state standards ever passed such a test, or even been subjected to it?

  3. You might consider field testing (whatever that means for standards) to be a form of research. Questions about the relationship of standards and research are not new.

    What is the relationship between what is known from research in mathematics education and what is expressed in the . . . standards? Can we say, for example, that research supports the Standards? These questions have become increasingly important as debates about reform reach fever pitch. They are fair questions, even though they do not have simple answers. The answers are not simple because (a) standards, in any field, are rarely based solely on research, so the connection between research and standards is never straightforward; and (b) research in mathematics education does not shine equally brightly on all aspects of the . . . Standards, so we cannot provide blanket statements.

    The paragraph above was written by James Hiebert, who directed the TIMSS 1999 Video Study of Teaching. It was written in 1999 and refers to the NCTM standards (PSSM) but it and the article in which it appears are relevant for the CCSS.

    A persistent type of claim is that a given policy (e.g., the implementation of specific standards) caused a certain type of event to occur (e.g., better test scores). For example, states with better scores on NAEP or TIMSS (like Massachusetts) might be considered to have better standards which have been field tested. But, did Massachusetts standards affect Massachusetts test scores? Is there some other reason? Perhaps the explanation is really the tests for teacher licensure which were initiated in 1998: http://www.doe.mass.edu/mtel/about.html. Or maybe we don’t know enough about how either were implemented to say that they are even possible explanations. Research can help to answer those questions, for example, by looking at relationships of teachers’ scores on licensure tests and their students’ scores on NAEP or TIMSS—or other factors, such as school organization or professional development. But, as far as I know, that research has not been done.

    A different perspective on field testing might come from research on learning progressions (aka learning trajectories). This might be thought of as field testing sequences of mathematical goals. The two paragraphs below come from the draft introduction for the Progressions.

    “A learning trajectory has three parts: a specific mathematical goal, a developmental path along which children develop to reach that goal, and a set of instructional activities that help children move along that path. (Clements & Sarama, 2009, Learning and Teaching Early Math: The Learning Trajectories Approach, Routledge, p. viii.)

    Well-documented learning progressions for all of K–12 mathematics do not exist. However, the Progressions for Counting and Cardinality, Operations and Algebraic Thinking, Number and Operations in Base Ten, Geometry, and Geometric Measurement are informed by such learning progressions and are thus able to outline central instructional sequences and activities which have informed the Standards.”

    For more discussion of research and the CCSS, see the draft introduction for the Progressions which is part of the front matter document at: http://ime.math.arizona.edu/progressions/

    Hiebert’s article is posted here: http://mel.zbedu.net/互动课程在理科学习中的应用/理论与方法/英文文献/Relationships%20Between%20Research%20and%20the%20NCTM%20Standards.pdf

    Hiebert’s article is also here (JSTOR access needed): http://www.jstor.org/discover/10.2307/749627?uid=3739560&uid=2&uid=4&uid=3739256&sid=21102673449671

Comments are closed.