You can hardly take a
step in Edland without tripping over a portfolio.
Little
kids in fourth grade (not that there are many BIG kids in fourth grade) are
busily selecting, cutting, pasting, magic markering, stapling, and binding
“artifacts“ and “evidences” of their “authorship” of “literacy materials” for
“authentic assessment” of portfolios. And when they bring these foul creations
home—covered with glitter and half-dried Elmer’s Glue dripping off the sides--their
parents Oo and Ah and assume their kids have learned something.
It takes a cynical and heart-hearted parent to look at his kid’s portfolio and
say, “How’s this different from toting home all your junk in a sack?”
Well,
the portfolio biz is no longer limited to kids. Having made the little ones
illiterate (with whole language) and unable to make change (with fuzziest
math), the ed establishment in some districts now requires graduating high
schoolers to present their portfolios to a board of portfolioticians for
evaluation.
Not
satisfied with turning students in all grade levels into ninconpoops, some
states have taken that all-important next step--turning EVERYone
into a nincompoop. So, they're requiring new teachers to make a huge portfolio
(I mean ten pounds of flapdoodle) that must be passed by a team of high-level
practitioners with Ph.D.s in Portfoliology before they can be licensed.
It’s just one more example of the slow domestication of
Americans by big bureaucracies in whose interests it is to turn everything into
something for them to evaluate according to “rubrics” (possibly the most stupid
word) that represent THEIR daffy “progressive philosophies” (therapy by any
other name) and which serve as yet another hurdle, obstacle, contingency, and
demand in a field that most closely resembles a con game run by
eduhacks whose motto is "It's all to further
your personal development--to make you more reflective, to help you
share your feelings and private thoughts."
We
say Ha! How do YOU spell c o
n t r o l ?
Let
me tell you something, neighbor.
You
may recall that Professor Plum once asked members of a state department of
public instruction if they had any evidence that portfolio assessment of new
teachers predicted teacher effectiveness better than observations by their
principal and mentors. [Or, you may not.] The immediate and uniform
chastisement from colleagues in response to
So,
he (Professor Plum) decided to write a critique of portfolio assessment of
teachers. And here, as they say, it is. If you like it, feel free to use it.
Portfolio Assessment of
Teachers: Another Expensive, Worthless, and Altogether Stupid Idea
Brought to You By the Education Establishment
Portfolio
assessment is offered as a form of assessment that may largely replace or
supplement more objective forms of assessment, as when a principal repeatedly
observes a teacher and her students’ achievement. However, as with almost
every educational reform over the past 100 years, as discussed by Diane Ravitch http://www.dianeravitch.com/
in Left
Back (and others of her books), it is CERTAIN that portfolio
assessment of new teachers will be useless, ridiculously expensive, and
destructive. [In other words, another cleverly disguised eduscam paid for
by the public.]
I.
The Argument
Portfolios
appeal to many persons--perhaps the same folks who make scrapbooks of "My
visit to
A
critical examination of wide-scale (e.g., state level) portfolio assessment of
new teachers must address the following propositions.
1.
The wide-scale use of portfolios is enormously expensive and
organizationally unwieldy.
2.
Portfolio assessment has neither sufficient face validity, criterion
validity, nor pragmatic validity.
3.
Portfolios have not been shown to be more valid, more feasible, and less
costly than existing school-based assessment methods.
4.
Wide-scale portfolio assessment disempowers individuals and groups at the
local level and increasingly empowers dominant minorities who control the
assessment process. Note that wide-scale portfolio assessment was created
by the very groups who benefit most (social position, power) from its
wide-scale use.
5.
Wide-scale portfolio assessment is an unwarranted invasion of individual
privacy and a clear and present danger to individual liberties and to the civil
liberties of minority groups.
II. The Wide-Scale Use of Portfolios is Expensive and
Unwieldy
The
wide-scale use of portfolio assessment of new teachers means that every year
state Departments of Public Instruction will have to process tens
of thousands of portfolios created by ed
students and by teachers seeking initial licensure. These portfolios will
have to be stored, mailed from one assessor to another, and mailed back to
assessees. Assessors will have to be selected and trained and be
subjected to frequent reliability checks. The assessment of any portfolio
must require at least several hours. These facts will require the
development of a large and costly assessment apparatus, without which there
will surely be chaos. Indeed, considering the enormous number of
portfolios every year, it is doubtful that the job can be done at all.
III. Portfolio Assessment Is Neither Validated
Nor Validatable
The
portfolio assessment process rests on: (1) the validity of scoring rubrics; (2)
the validity of scorings; and (3) the validity or usefulness of portfolio
assessment compared to other forms of assessment. In all three ways,
wide-scale portfolio assessment either has weak validity or is unvalidated.
Questionable Validity of
Scoring Rubrics
There are at least three sorts of validity with respect to scoring rubrics:
face validity, criterion validity and pragmatic validity.
1.
Face
validity is the subjective judgment (or intersubjective judgments) of rubric
creators that an item (measure) in a rubric actually measures what it purports
to measure. Face validity requires that words in items are
sufficiently clear that their meaning (referent) is obvious. For example,
hitting another person probably has face validity as an indicator or measure of
the concept "aggression." However, many items or measures in portfolio
scoring rubrics dont have face validity. In view of the many
"approaches," "theories," and "perspectives" in
education, it's doubtful that terms such as "learning,"
"classroom management," and "developmentally appropriate,"
to name a few, have common meaning to assessors and to the individuals (e.g.,
teachers seeking licensure) who are creating portfolios guided by these
terms. Moreover, there are no reported studies of the face validity
(e.g., inter-assessor judgments) of the concepts used in portfolio rubrics.
2.
Criterion
validity is the extent to which a new (candidate) instrument, measure, or item
is correlated with a widely used, already-validated and generally standard
measure. For example, a standard measure of upper-body strength is
the number of repetitions of bench presses a person can do with certain amounts
of weight. The criterion validity of a new candidate measure (e.g., the
density of fiber bundles in chest muscles) would be indicated by the degree of
correlation between the number of bench presses persons do and the density of
muscle fiber bundles. If the correlation is low, we judge the candidate
measure to have little validity.
However,
there are no substantial studies of the criterion validity of items in
portfolio rubrics. Moreover, criterion validation is simply impossible for most
items. This is because the items are actually hypothetical constructs,
psychological dispositions, or what some philosophers call "mental
predicates." In other words, they can't be measured--perhaps because
they DON'T EXIST. Therefore, one cannot determine their correlation with
standard measures. Examples include dispositions towards
**Considering the whole child. "We're sorry, Ms.
Parsley. But your portfolio suggests that you consider only three-eights
of a child."
**Life-long learning. "We're sorry, Ms. Sage. But you
haven't adequately addressed learning in the elderly."
**Continual reflectiveness. "So, sorry, Ms. Rosemary, but you
don't spend much time discussing how you think about what you think about why
you think."
**Openness to new ideas. " We're sorry, Ms. Thyme, but you seem
closed minded towards ideas that strike you as amazingly stupid."
3.
Pragmatic
validity is perhaps the most important test of portfolio assessment.
After all, the whole point is to determine how well a new teacher (for example)
teaches. Yet, there are no significant studies (indeed, there do not appear to
be any studies) of the extent to which scores on any part of the portfolio
assessment, or scores on the portfolio overall, predict or are correlated with
the achievement of a teachers' students.
Naturally, portfolio scores ought to predict many other things that good
teachers and principals do. But there is no validation of portfolios as
predictors of these, either. In other words, there
is little if any validation of the extent to which portfolio assessment serves
its stated function (Hambeton et al.,
1995; Koretz at al., 1994; Pacific Research Institute,
1999).
V.
Portfolios Are of Little Use
In
addition to questionable validity, there's the question of the usefulness of
portfolios. No substantial studies have been done on whether portfolio
assessment provides more information, more useful information, or more
pragmatically valid information than:
1.
The current method of (for example) assessing and improving initial teacher
performance by year-long interaction, supervision, and coaching by school
principal and mentors; or
2.
An enriched method of this sort of assessment in which principals and mentors
are trained to supervise, coach and provide formative and summative evaluation.
In
other words, there's little reason to replace current forms of assessment with
wide-scale and expensive portfolio assessment--except of course to
provide well-paying jobs for eduhacks and their edubuddies whose intellects top
out at making lists of meaningless "standards" for evaluating
teachers.
VI. Portfolio Assessment Is Disempowering
The case for portfolio assessment often appeals to self-empowerment. Portfolios
are supposed to enable persons to express themselves in a way that is
comfortable and without the intimidating scrutiny by supervisors. This is
nice. Its
also bunk. At the mentor-mentee (Is there a dumber term? I
don't think so) level, portfolio creation and evaluation may have these
beneficial effects. However, at the wide-scale level, where sociological
and political processes come into play, portfolio assessment reduces the power
of individuals, mentors, and school principals, and shifts power to persons
and agencies who run and profit from the assessment
apparatus. The cops call this an extortion racket.
First,
the items to be assessed and the scoring rubrics were not created by persons
being assessed or their representatives. This is an obvious example of
alienation, disempowerment and "loss of voice." How can portfolios
reveal and encourage self-reflection and self-development when the scoring
rubric and portfolio instructions tell persons being assessed what to write
about and how they will be assessed?
Second,
school principals may hire teachers and write letters of recommendation, and
mentors may observe and meet with assessees, but the major decision-making
power is shifted to the agencies who have created and
manage the assessment apparatus. This is an example of disempowerment and loss
of voice--in the very persons who work most closely with assessees and who,
logically, ought to have the most compelling voice.
VII. Wide-scale Portfolio
Assessment is an Invasion of Privacy and a Threat to Individual Liberties and
to the Civil Liberties of Minority Groups
This
assertion may strike the reader as hyperbole. However, if the reader will
bear with me, I believe he or she will find the argument tenable. I have
spent at least 20 years studying despotism, totalitarian societies, witch
hunts, religious and political persecution, spouse and child abuse, and the
abuse of "inmates" in nursing homes, mental hospitals and other
"total institutions." Informative works include Reiff (Triumph
of the therapeutic), Henry (Pathways to madness), Solzhenitsyn (The Gulag
archipelago), Huxley (The devils of Loudon), Frank (Persuasion and healing),
Gubrium (Living and dying at Murray Manor), Glasser (Prisoners of benevolence),
Emberely (Values education and technology), Kelman (Crimes of obedience),
Foucault (Discipline and punish), Sobsey (Violence and abuse in the lives of
people with disabilities), Hughes ("Good people and dirty work"),
Weber (Theory of social and economic organization), Bourdieu (Reproduction and
Outline of a theory of practice), Mosca (The ruling class), Talmon (Origins of
totalitarian democracy), Arendt (The orgins of totalitarianism) and others.
Three
empirical generalizations stand out.
First,
all forms of control (superordinate-subordinate relations, or what Max Weber
called "forms of domination") involve an apparatus; i.e.,
1.
A set of ideas that legitimize the relations and methods of domination.
2.
A division of labor (often a bureaucracy) among persons running the apparatus;
e.g., persons who find "deviant" persons and groups; who
"test" suspects; who transport convicted or identified
"deviants"; and who carry out "treatments."
Second,
forms of domination are not necessarily overtly harsh or violent.
Orwell's 1984 and Huxley's Brave new world are examples of the difference
between tyranny using force and tyranny operating under the guise of
benevolence--sometimes called "friendly fascism." Examples are
found in the sort of psychotherapy that convinces vulnerable children that they
have been subjected to satanic abuse by their parents and teachers.
Alexis de Tocqueville (Democracy in America)
saw the same thing happening in the
It would seem
that if despotism were to be established among the democratic nations of
our days, it might assume a different character; it
would be more extensive and more mild; it would degrade men
without tormenting them. (p. 335)
Above this race
of men stands an immense and tutelary power, which takes upon itself alone to
secure their gratifications and to watch over their fate. The power is
absolute, minute, regular, provident, and mild. It would be like the
authority of a parent if, like that authority, its object was to prepare men
for manhood; but it seeks, on the contrary, to keep them in perpetual
childhood: it is well content that the people should rejoice, provided they
think of nothing but rejoicing. For their happiness such a government
willingly labors, but it chooses to be the sole agent and the only arbiter of
that happiness; it provides for their security, foresees and supplies their
necessities, facilitates their pleasures, manages their principal concerns,
directs their industry, regulates the descent of property, and subdivides their
inheritance: what remains but to spare them all the care of thinking and all
the trouble of living. Thus it every day renders the exercise of the free
agency of man less useful and less frequent; it circumscribes the will within a
narrower range and gradually robs a man of all the uses of himself.
Such a power
does not destroy, but it prevents existence; it does not tyrannize, but it
compresses, enervates, extinguishes, and stupefies a people, till each nation
is reduced to nothing better than a flock of timid and industrious animals, of
which the government is the shepherd. (pp. 336-337) [Alexis de
Tocqueville. From Democracy in
Third, forms of domination involve an
obligation on the part of persons being tested or assessed to speak--to reveal
their thoughts and feelings. What
is lost is not only the right to speak (e.g., to challenge the assessment) but
the right to remain silent.
Silence--either
the refusal to speak or reluctance to speak--are understood by those running
the apparatus as signs of something to hide. Both silence and incorrect
thinking (religious, political, psychiatric, domestic, and now pedagogical) are
punished by ostracism, imprisonment, loss of jobs, and sometimes physical
violence. Note that the details or the contents may vary from one
situation to another (e.g., witch hunt, competency hearing), but the structure
of domination, the ways it is legitimized, and its effects are exactly the
same--one group increasingly controls another group.
Wide-scale
portfolio assessment that is conceived, planned, and controlled by a small
minority of officials, is an example of the loss of the right to remain silent;
loss of the right simply to do one's job well and not have to reveal how one
thinks or what one feels. It is an example of the use of
therapeutic and humanistic terms (reflection, authenticity, self-development)
to make palatable--even desirable--what is in fact coercion to speak or be
punished; i.e., not be licensed.
In
addition to loss of the right to privacy, wide-scale portfolio assessment is a
clear and present threat to minority groups. It is well known, for
example, that members of Asian cultures are embarrassed by the requirement to
reveal personal information. And it is well known that different cultures
(e.g., African American) have different ways of revealing themselves and
different histories that leave them better or worse prepared to satisfy the
"reflection" rubrics created by individuals who do not share their
culture. Just as so-called IQ tests and other standardized
(dominant-culture-biased) tests leave cultural minorities at a distinct
test-taking disadvantage, there is every reason to believe that this will be
true of wide-scale portfolio assessment.
Moreover,
what evidence is there that the assessors are knowledgeable about and open to
what are currently minority pedagogies? Is it not fair to ask
whether teachers who strongly advocate focused or direct instruction on
phonics and math will receive lower scores than teachers who express a more
constructivist or allegedly "child-centered" philosophy? And is
it not fair to suggest that--when the pendulum swings once again (as it is
already doing regarding reading)--that whole language teachers will be
reluctant to reveal what may become a "deviant" orientation.
Even if the assessment process were "orientation-fair," assessees'
knowledge of what is "in" and where they stand is likely to have a
chilling effect on their experience of freedom to express themselves.
VIII. Summary
At
the mentor-mentee levels, portfolio creation and review may be helpful.
They may foster reflection, guide improved practice, and facilitate communication.
At the macro or wide-spread level, however, portfolio assessment transfers
substantial power from individuals and local mentors to a small minority of
functionaries (of dubious intelligence and motive) in whose interests it is to
increase the scope of portfolio assessment. Moreover, as devices
requiring individuals to speak, and whose scoring rubrics are clear
reflections of narrow pedagogical-cultural (progressive) orientations,
wide-spread portfolio assessment is an invasion of the right to privacy and a
threat to minority cultures and pedagogical orientations. In addition,
there is no solid evidence that wide-scale portfolio assessment is valid, has
an advantage over local, more empowering, and clearly less expensive forms of
school-based assessment and teacher development.
References
Hambleton,
R.K., Jaeger, R.M., Koretz, D., Linn, R.L., Millman, J., and Phillips, S.E.,
"Review of the Measurement Quality of the Kentucky Instructional Results
Information System, 1991-1994," Office of Educational Accountability,
Kentucky General Assembly, 20 June 1995: 4.
Koretz, D., Stecher, B., Klein, S., and
McCaffrey, D, "The
Portfolio Assessment Program," Educational Measurement: Issues and
Practice, Fall 1994: 12-13.
Pacific
Research Institute (1999). Developing and implementing academic
standards. On-line at http://www.pacificresearch.org/pub/sab/educat/ac_standards/main.html