Anonymous FTP: Can We Risk It?Dare We Publish Pre-prints?
Gene V Glass
|
|
Abstract: A redefinition of scholarly communications will significantly impact publishers, scholars, and libraries. Our concept of a scholarly journal, as established in the traditional world of paper media, is under serious challenge due to recent advances in telecommunications technology. The idea of creating "pre-print" digital archives, while in use in the "hard sciences" (physics, chemistry, mathematics), has yet to catch-on in the "soft sciences" (education and psychology, to name just two). Creation of an electronic system that provides a venue for scholars to share their early work and benefit from immediate review by peers world-wide has now been a technical possibility for some years and has recently gone online for educational research in the United Kingdom. Why is its existence largely unknown? Why is the prospect of such archives playing a central role in scholarly communications so dismal in the soft sciences? |
My friend Larry Rudner tells me
not to expect an Anonymous FTP site for
educational research to succeed any time soon. I am strongly inclined to
trust his judgment on
this matter since I know no one with more experience in these matters than
he. And how ever much I yearn for the day when all scholars will
communicate quickly and widely without the intercession of private
commercial interests, I nonetheless respect my colleagues'
prediction about the possibility that we will not see this day any time
soon. In this article, I am searching for the reasons why this is so.
At the beginning of the search for reasons is the discrepancy between
how the hard and soft sciences communicate. (I shall rely heavily on the
distinction between hard and soft sciences without attempting a careful
analysis of what these designations mean. If pressed, I can do no better than
cite a couple of authors whose ideas here have influenced me, viz.,
Paul Meehl (1978) in his famous "Two Knights" paper, and David Labaree (1998)
in his paper "Educational researchers: Living with a lesser form of
knowledge." Going back further, the reader can find foundation for the arguments
presented here in Gergen (1973), Andreski (1972), and Nozick (1974).
I will have to return to some of these authors below to make
several points.)
Question: xxx.lanl.gov: Where Does Hard Science Go to Find Research?
Ans.:
It goes to Los Alamos, New Mexico.
Specifically, it goes to the arXiv.org e-Print archives better known
as xxx.lanl.gov.
xxx.lanl.gov, or arXiv.org as it now wishes to be called, was created in
August of 1991 as a fully
automated electronic archive and distribution server for
research papers. The disciplines covered include physics and related disciplines,
mathematics, nonlinear sciences, computational
linguistics, and neuroscience.
See Appendix A for a detailed list of sub-disciplines archived at xxx.lanl.gov.
Users retrieve articles from the archive either through web browsers or by sending email commands to the system. Authors submit their papers to the archive either using the web interface, anonymous ftp, or e-mail. Authors can update their submissions, though prior versions remain available.
The access statistics for xxx.lanl.gov are truly staggering. Over 2,500 new articles were archived on the system in the month of March, 2000. The number of "unique hosts" (roughly, persons) connecting to the system to download articles was over 150,000 for the same month. The following graphs (Figures 1 and 2) show the growth in use of the system over the past few years.
Figure 1. Numbers of Articles Submitted to xxx.lanl.gov Since 1991
Figure 2. Numbers of "Users" of xxx.lanl.gov July 1997 - March 2000
- Red-Number of connections in each week
- Blue-Number of hosts connecting that week (divide by 10 for
correct number)
- Green-Number of new hosts that week (divide by 10)
What Hard Scientists say about xxx.lanl.gov
On the "Larry King Weekend" telecast on CNN on December 25, 1999, Stephen Hawking, the famous cosmologist, was asked,"[KING] What, Professor Hawking, do you consider the most important discovery of this millennium?"and Hawking replied,
"[HAWKING] I think the invention of printing was a breakthrough for the human race. It meant that information and discoveries could be disseminated widely and not just on a one-to-one basis by word of mouth or handwritten manuscript. It led to an ever increasing rate of scientific and technological development. This has now made printing almost obsolete and replaced it by the Internet."
"[KING, in his inimitable fashion] Do you surf the net? What do you think of this Internet thing?"
"[HAWKING] I use the Internet each day to get physics papers and to get the news from the BBC or CNN."
Brand (1999) conducted a series of interviews with physicists, mathematicians, chemists, psychologists and educational researchers in which he questioned these conspicuously productive scholars on their habits of communicating their research to their respective communities, and in particular, their use of computer mediated telecommunications to do so. As one might expect, the hard scientists reported that the internet and, specifically xxx.lanl.gov, was the primary means by which they keep current with research in their field. A physicist at a Research I university explained his use of the online preprint archive: "Suddenly, there's a new way I can instantly broadcast to everybody. Why shouldn't I be able to do it? ... that's the incentive to research, which is to maximize ease, dissemination, and maximize the availability of their research in a timely fashion."
Numerous physicists, asked if they foresaw electronic journals replacing paper, answered with a resounding "Yes!" An experienced physicist from a large Research I university described his view of the timetable for the transition as follows: "It's already happening! An important set of journals for us [the physics reviews] . . . and those are now available online so you can download the articles." Asked the same question, a young chemist from a smaller comprehensive university, responded: "Yes, almost all of our journals now have gone to online versions...."
Question: Where Does Soft Science Go to Find Research?
Ans: It waits for the letter carrier.
Few realize that an Anonymous FTP site for education exists:
It's called Education-line and it resides at
Education-line was started on
January 2, 1997, when it was announced at
the North of England Education Conference in Sheffield, England. Its guidelines
for submissions signaled its interest in unreviewed "pre-prints," much like
xxx.lanl.gov for the hard sciences.
Guidelines for submission to the Education-lineEducation-line will consider electronic texts in the following categories of literature:
|
Figure 3. Acquisitions of Education-line in its First Year (1997)
In its first year, Education-line acquired and archived 600 articles. But by January 14, 2000, it had recorded only its 1,315th submission. This is a monthly rate of 35 submissions compared to the submission rate at xxx.lanl.gov of 2,500 per month. Access logs for March 2000, show visitors to Education-line coming from from about 7,500 separate internet addresses. Education-line is a superb facility, technically, but it is not unfair to say that relatively few educational researchers use it. Its current monthly activity compares unfavorably with the weekly activity of a single scholarly peer reviewed journal of the traditional type that is confined to one narrow sub-discipline of education:
Does this difference in usage between xxx.lanl.gov and Education-line have to do with the fact that there are so many more physicists in the world than there are educational researchers? Well, the differences in the two populations are not as striking as one might assume. For the American Physics Society, membership currently consists of roughly 42,000 people in the following proportions: 50% academia (including students, postdocs, faculty, research staff), 25% industry, and the remainder national laboratories and other venues. The current membership of AERA is approximately 25,000. And in fact, AERA membership is skewed more toward academics than is the membership of the American Physics Society; for AERA, the breakdown of membership is 79% academia, 9% school districts, 2% federal agencies, 1% industry, and 8% R&D and testing organizations. In fact, the numbers of persons with academic affiliations are 21,000 for the American Physics Society and 19,000 for AERA. So, educational research does not want for researchers, but it does have very different habits when it comes to communicating about research.
Why the Difference Between How the Hard Sciences and the Soft Sciences Communicate?
In Brand's (1999) study, referred to earlier, the hard scientists reported that the internet and, specifically xxx.lanl.gov, was the primary means by which they keep current; soft scientists (psychologists and educationists) reported continuing to rely on traditional paper journals, and in fact, expressed some distrust of scholarly communications over the internet. Brand's interview data illustrated not only the status of each discipline in the formation of their pre-print archives, but also their feelings toward such a development. According to a scientific librarian who was interviewed, preprints have been available for quite a while in physics:"Physics had a strong tradition of preprints in paper far before it did electronically. And so it was kind of a natural thing that worked there. I think it will work it other disciplines. Other disciplines are just much more slowly coming to it. But probably something of that type will be much more heavily in play in a variety of disciplines in the next few years." (Brand, 1999, p. 137)A chemist who revealed in the interview that he preferred to receive materials electronically noted the following:
"I would say chemistry lags behind many other fields like physics and mathematics where they really are; everything comes up as preprints first. Chemistry is a little worse about that for some reason. . . . There's hardly any, nothing compared to physics or mathematics as far as the preprint community. It's a shame. I mean you write, all you have to do is spend much time with math or physics and you've really spoiled, if you like online things, you start to love their society because half my work is in physics and I can get everything online without ever worrying about it." (Brand, 1999, p. 137)Educators, on the other hand, have a history of a form of eprint/preprint known as ERIC (Educational Research Information Center). The majority of the educational researchers interviewed did not view ERIC very highly. A prominent educationist spoke thusly about ERIC:
"Another factor in education, in particular I think is that the electronic media may be a bit tainted by the experience we've had with ERIC. ERIC doesn't really have a very good reputation. It's a tremendous resource and we use it, but the first thing anybody is told about ERIC is, you know, anybody who wants to can send a bunch of term papers in there, and they'll stick them in ERIC. There's no quality control at all. Things end up in ERIC and are sent as ERIC documents, only if the author has been unable to place them in a more prestigious format. So I can submit something to ERIC and I'll try to publish it, but if I don't manage to publish it, then it will end up being an ERIC citation someplace. So ERIC has a reputation, I'm afraid, as being more like the leavings. And I think that probably may have very well created some prejudice against the electronic media. I don't know that. Again, it's speculation. I'm not even sure how you go about tracking it down." (Brand, 1999, p. 137)One might be tempted to describe this researcher's attitude toward a mere archive as supercilious (a raised, doubtful eyebrow conveying contempt and scorn), but it must be taken seriously. Undoubtedly, this person spoke for many and gave voice to a culture that has and continues to shape educational researchers modes of communication. Indeed, Brand observed throughout his study that the soft sciences were much more concerned with the "quality" of its communications media than the hard sciences. More about this below when attention focuses more closely on peer review.
The Greater Technical Expertise of the Hard Scientists?
That physicists are more adept at operating PCs than educationists was once probably very true, today only somewhat true and increasingly carries little weight in accounting for the very different communication habits of the two groups. Perhaps they—physicists— are somewhat more familiar with Adobe Acrobat or file transfer protocols, but since mere toddlers have had the ability to browse and click for three or four years now, it is difficult to believe that accessing information on the internet is beyond the skills of anyone working in a university today. Indeed, if anyone accused educationists of being inferior to hard scientists in their ability to use a library card catalogue, the educationists would be justifiably outraged.The Absence of "Paradigms" in the Soft Sciences?
The term "paradigm" has become a modern cliché, applied to nearly anything the speaker wants to aggrandize, and the slightest change of no particular significance is christened a "paradigm shift." This usage surely stems from Thomas Kuhn's 1962 The Structure of Scientific Revolutions in which he advanced his then controversial views on how science progresses. In the late 1960s, Dick Dershimer, who was AERA Executive Officer at the time, went to Princeton to interview Kuhn and glean what wisdom he might vouchsafe to educational researchers. I read the transcript of that interview about 30 years ago, and one thing has stuck in my memory. Kuhn said that his ideas had no relevance for educational research whatsoever since educational research had no paradigm—by which he meant, a prevailing, agreed upon set of concepts, methods and questions. In deed, if you read the 1961 book carefully, you will see that Kuhn thought that biology and the social sciences were also in what he termed a "pre-paradigm" stage.What then do the soft sciences have if they don't have paradigms? They have fads (Meehl, 1978) and tribal leaders (Becher, 1989) [The case of Edward Tolman.] . The sociology and the politics of the science become as important or more important than the science itself.
The culture and history of the disciplines in the hard sciences, as asserted by the participants in Brand's study, facilitated the use of non-reviewed electronic archives of research. Members of the soft disciplines spoke of a history of unreliable data archives such as ERIC, no apparent urgent need for discovery and dissemination, and argued unconvincingly that their colleagues possessed lower technical ability and familiarity in working on computers.
Labaree (1998) argued that education lives with a lesser form of knowledge than physics due to the kinds of knowledge produced. Soft applied knowledge production is "rural" and "divergent," to use Becher's terms; the researchers can not build towers on the foundation laid by others since education is always altering the foundation. On the other hand, hard, pure knowledge production is "urban" and "convergent," allowing the field to build upon solid foundations of agreed upon methods, constructs and established empirical findings (Kuhn's "paradigm," as it were). Labaree's assertions were visible in the fast growth of electronic journals in the hard disciplines and its eprint cumulative nature. Soft disciplines, according to Labaree, place senior educators in less control over the intellectual knowledge due to the ease with which even novices can challenge favored theories and published studies. On a couple of occasions in Brand's study, senior researchers in the soft sciences reported not being willing to relinquish their data to other scientists unless they were deemed qualified to review their work. On the other hand, the majority of the hard scientists mentioned the time and effort that each of them contribute to making the data or the information available to all other colleagues and researchers for immediate replication of the experiment. Where even the highest are subject to critique by the lowest, a premium is placed on tribal leaders and gate- keeping and orthodoxy of thought—hence, the premium placed by soft scientists on controlling the messages emanating from the various media.
The Role of Peer Review in Scholarly Communications
The most palpable difference between the hard sciences and the soft sciences appears to be with respect to their reverence for "peer review." Brand (1999, p. 101) found that the hard sciences were less enamored of peer review than soft sciences:"Hard scientists favored peer review over nothing at all, but if they had to choose between timely access and peer review, many would have chosen timely access. This was not the case in the soft disciplines.""One senior educational scholar, Educationist A stated, I think quality control and perception of quality control are really serious issues [in considering the speed of development of electronic communications]." (Brand, p. 101)
Another scholar and editor, Educationist B, stated: "As soon as the promotion and tenure all the way up the line believes in it and believes it has just as tough a review [as paper journals], then I think it will be accepted. I think as yet, I don't think electronic journals have as tough a review. I need to be convinced, and people like me sit on those boards." (Brand, 1999, p. 105)
It is generally believed that "peer review" provides for
- protection against error
- improved version resulting from feedback
- efficiency of readers' time since "bad" articles are suppressed.
- a higher level of scholarship because of the threat of rejection.
One would think that hard science has more to fear from "error" than soft science does. If this were so, then hard scientists would be more dedicated to peer review of articles than soft scientists. But as we see, the opposite is true. In actuality neither has strong reasons to fear "error," since the hard sciences are not fundamentally threatened by error or fudging or fraud (it stands only to lose a little bit of time—Franklin, 1990, 1997—recall the case of "cold fusion" and how quickly it was examined and dispensed with), and the soft sciences can scarcely discriminate between error and truth amid its rampant ideological and methodological wars.
The irony discovered through comparing and contrasting the disciplines is that those scholars and researchers most at risk of being misled by "errors" in communications were least inclined to support the institution of peer review to protect themselves from error. Those scholars who had least to fear from such "error" were the most intent on controlling access to publication outlets. Physicists said, "Don't slow things down by tying them up in the peer review process; give me the data and I'll decide whether it is right or not." Psychologist, educationists and other practitioners of the soft disciplines emphasized the risk to their disciplines of allowing "errors" to enter the literature. One becomes suspicious that this reliance on the peer review process in the soft disciplines (e.g., rejection rates in the soft sciences often approach 80%-90% while remaining extremely low—5% to10%—in the hard sciences) has more to do with controlling whose message and what message is disseminated than it has to do with protecting the discipline from error.
Stevan Harnad of Southampton University in the UK and of Princeton University has been a leader in exploring new modes of scholarly communication—"scholarly skywriting, as he calls it. Harnad has been a visionary in the area of scholarly communications, and what little progress has been made in psychology in internet communications owes much to his efforts. Throughout the past decade, Harnad maintained a consistent position in favor of peer review of electronic publications. "The scholarly communicative potential of electronic networks is revolutionary. There is only one sector in which the Net will have to be traditional, and that is in the validation of scholarly ideas and findings by peer review. Refereeing can be implemented much more rapidly, equitably and efficiently on the Net, but it cannot be dispensed with, as many naive enthusiasts (who equate it with 'censorship') seem to think." (Harnad, 1993)
By far the most productive discussion of these issues occurred in September 1998 in an online forum conducted by the American Scientist, the archives of which are available here: listserver.sigmaxi.org/scripts/wa.exe?A1=ind98&L=september98-forum&F=l. Harnad moderated the discussion and contributed most to it. But his ideas were already well known, and it was the contributions of Joseph Ransdell, a professor of philosophy at Texas Tech University and an early experimenter with internet communications in philosophy, that impressed me most. I have quoted Ransdell below at considerable length—indeed, the length of the quotations will strike many a modern scholar as odd, that person's expectations having been developed in paper media where space is precious—because his ideas are so close to my own and so well put. Indeed, the exchange between Ransdell and Harnad is so compelling and illuminating that it stands itself as testimony to Ransdell's point that the unfiltered exchange of scholars in open discussion is one of the highest forms of intellectual expression, surpassing the unrebutted soliloquies that fill our journals. Consequently, I have appended (C) portions of the forum to this article in case the website disappears or the reader is seeing these words on paper at the moment.
We join the discussion at the point where Ransdell is tempering the enthusiasm of a cyberphile:
|
> Granted that the way the physics community uses xxx is not the only > possible model, but it is clear to me that any community would benefit > by promoting unencumbered, free circulation of authors work through > centralized, globally mirrored archives. Ransdell: I agree with this completely, but I suggest that once we move out of the hard sciences we are going to find that the number of academics who disagree with you and me on this will turn out to be much greater and more influential than one might suspect. There will be many who will find it unacceptable that ANY work in the field in question should be made unrestrictedly available. Disciplinary authoritarianism in fields outside of the hard sciences is more the rule than the exception, and any realistic attempt at following the lead of the sciences in taking the sort of free access that xxx exemplifies as paradigmatic has to take that into account. Stevan's reassurances and concrete demonstration that the hierarchical structures presently associated with peer review can be ported to the net and perhaps even made more rigorous there will not pacify these people: they will want ONLY refereed material available and do what they can to insure it, as they are doing now by letting it be known that it is risky to have a network presence of that sort. These people will, moreover, be disproportionately influential both among faculty and administration—and this for obvious reasons: the present system of restricted access tends by and large to favor those in the most powerful positions in the professorial hierarchy by protecting them and their work from criticisms other than from those who are similarly positioned in the hierarchy, whom they have long since learned how to accommodate or effectively ignore. Old dogs of a certain academic breed (some of them seemingly young) not only will not be learning new tricks but are going to be—as they already are—discrediting new tricks as thoroughly as possible precisely because they do not want to have to deal with young dogs that know these tricks. They can see no place for themselves in a networked professional environment—their lives are already planned out in accordance with other assumptions—and some of them at least are justifiably worried that they may have to answer to criticism posed by their professional inferiors, since they regard themselves as officially certified as superiors by their institutional rank. Their professional lives are built around the kind of protection from the barking dogs of criticism this hierarchical system provides. Perhaps nobody is like that in the hard sciences. ;-) There is reason to suppose that this kind of authoritarianism is not so prevalent in the sciences, at least, because there is a traceable connection with evidence and substantial results that is not there in these other fields, where people can and sometimes do rely entirely upon personal judgment and institutional privilege for intellectual control. This is much more frequent than one might suspect, and not merely an occasional personal aberration. But it is not primarily a question of how many such people there are but of who they are and where located in the system: power and position are at stake in publication practices, and people do not normally cooperate in changes that seem to threaten their power. I don't know that the Ginsparg movement will affect the people in the sciences much at all in this respect. It seems to me that he simply took institutionally based science one very important step forward by clarifying its pre-existing publication practices by universalizing it in the archive, and he did it beautifully—as best I can make out—by keeping his eye on exactly what had to be done at every step. But the implications of that clarification for the rest of academia are much more radical. Others may disagree with me on the difficulties ahead, and perhaps Stevan in particular will. But if he does I think it might be because of his experience from the rather special position he stands in, about midway between the hard sciences and the humanities, with some substantial basis going in both directions. For the result of the mediated contact with the sciences which he has promoted so well has been to make the "softer" side of the several disciplines he is mediating much more like the sciences than they would otherwise be: in short, his own success in elevating the quality of thinking in some of those fields may be misleading him. But I am confident myself that there are going to be major and highly influential areas of academe where the present institutionally reinforced authoritarianism of the professorial system is going to stop the expansion of the Ginsparg model in its tracks if the reactionary tendencies and maneuvers are not understood and outflanked in some way. Second, I suggest that the progress of the implementation of the Ginsparg model or any model basically compatible with it, be it centralized or distributed, will begin to develop hitches even in the sciences in the areas where the fields begin to "soften" through connections with the human sciences, social and psychological. This can even be predicted, I believe, if one is willing to get solid information on the extent to which people on the leading edge in a given field already rely extensively on pre-prints. I wonder if this might not even be a good rule-of-thumb method of measuring the "hardness" of a science: to what extent does if rely upon pre-prints? The "soft" areas in academe hardly use preprints at all and people in them sometimes even think of use of preprints as some sort of cheating! |
Later, in an exchange with Harnad, Ransdell pushed his analysis of the function of peer review further:
|
Ransdell:
I ... move now to the people—and they
most assuredly exist—who do not want anything of anybody's that is
unfiltered by referees to be made available because they regard
refereeing as the elimination of trash, and they do not believe that
people in academia should be polluting the international networks with
trash. They certainly don't want junior faculty, or untenured profs or
grad students from their own departments to be polluting the nets with
what may well be trash since nobody has filtered it. They are
encouraged in this view by people who stress constantly the importance
of refereeing and peer review as something whose chief function it is to
identify trash and eliminate trash (and, by implication, intended or
unintended, to eliminate trashy people from the academic scene). Not
only is this systematically denigrative of the value of the net as
intellectually open, but it is highly questionable as a proper
description of the purpose of refereeing to begin with.
I find it difficult to believe that refereeing was established historically to trap and dispose of trash, and that has not in fact been the usual function of much of it even in recent years, when the constant harping on the dangers of the barbarians at the gates has moved academics more and more into thinking of themselves as thought police and garbage experts. Refereeing has often served the function simply of organizing the secondary literature within the appropriate journals, each of which has some more or less well-defined niche within a larger informal librarial scheme. Yes, some filtering is happening there, of course, but the imposition on this of the scheme of hierarchical grading of the value of a contribution by authorities in intellectual grading who have been appointed to the task by . . . whom? is an addition to the organizational function of refereeing which is by no means necessarily implicit in it, and it is questionable in its own right in the sense that the time has surely come to question it in order to find out precisely what IS going on in all of this grading and sorting and ranking. I take that to be one of the most important questions that the success of the Ginsparg archives has forced upon us. If refereeing is not needed at the point where the research reports flow back into the inquiry process and the process itself enriched and developed by those results insofar as they are actually accepted by inquirers by being used by them in inquiry—not graded by "authorities" but accepted (or not accepted) by inquirers—then when and why, precisely, IS refereeing required for inquiry? This is a real question now, and we need a good answer for it, but I haven't seen any such answer thus far, perhaps because nobody wants to ask it for fear of being thought guilty of "having no standards" or being a proponent of unrestricted trash distribution. |
Harnad deserves equal space, but since this article is pushing against the limits of normal attention spans already, I shall append (D) his most concise statement of his view rather than quote it in the body.
Excerpts from the Garvey Study of AERA that Seem Relevant But Don't Fit Nicely Anywhere YetGarvey (1979) suggested the following in his research of peer review and scholarly communication: "That eclectic ("soft") nature of the social scientist's subject matter probably contributes to this situation. For example, social science authors and editors disagree more often that physical science authors and editors do on the appropriateness of the required revisions; the editorial process in the social sciences focuses more on the mechanics of the work, such as statistical procedures and methodology, than on the controversiality of research findings; and whereas "core" journals in physical sciences receive few manuscripts previously rejected elsewhere, social science authors repeatedly recycle manuscripts rejected by "core" journals and resubmit them to other "core" journals. (p. 297) |
I have edited three journals on paper, going back to 1968 with the Review of Educational Research. Today I edit two electronic journals (http://epaa.asu.edu, http://cie.ed.asu.edu). None of the paper journals I edited had peer review even remotely approaching the quality of what my editorial board gives me on my electronic education policy journal. An author who published with me two years ago received 14 peer reviews in two weeks; he said it was one of the most rigorous and helpful experiences of his publishing career. But this is less an argument for peer review than it is a suggestion that the 14 reviews should be as public as the amended and finally published article. But this experience speaks only to a tiny marginal improvement in scholarly communications afforded by the new technologies. It does not go to the heart of the matter of justifying our current practices of peer review. Every editor I know can tell stories of how the reviews have sometimes surpassed the thing reviewed, only to be filed out of sight of the reading public. I see my function as Editor at EPAA as more a selector for relevance of content than a guard at the gates of the temple of truth. Even the packaging of scholarly electronic communications in forms resembling traditional journals (with volume and issue numbers and names like "The Journal of Online Whatever") concedes too much to the power of the political and social forces that rule the academic life, in my opinion. And yet, I have in my own efforts reluctantly acceded to these forces.
The British Medical Journal is currently conducting an experiment with pre-print posting and online public review of research articles. See how it is working at www.bmj.com/cgi/shtml/misc/peer/index.shtml .
I believe that as a culture, educational research and the other minor disciplines are ambivalent on the role of peer review. Four inconsistencies that are instructive to examine reveal this ambivalence.
|
What This All Has To Do With Meta-analysis
My interests in scholarly communications and meta-analysis (the quantitative integration of empirical research) meet unexpectedly at this point. It has long been a matter of concern to meta-analysts that the findings of empirical research might be filtered and distorted somehow by the review process of journals and other media. Where the matter has been explicitly addressed (Smith, 1980), surprisingly small differences have been observed between the findings that are made public in different media (books, peer-reviewed journals, dissertations, convention papers, unpublished reports). My own experience with having other people select what I read is this: if I have only a passing interest in a topic and can spare little time to educate myself on it, then heavily "filtered" articles in the most "select" outlets are useful; but if I am intensely interested in a topic because it is central to my own research, then I want as few people as possible placing themselves between me and the other researcher with whom I am communicating—peer-review detracts from these communications, rather than adding to it. (I hasten to add that although I acknowledge some utility to "filters" where I have less interest in the topic, it is easy to conceive of many types of filters that are more useful and less dangerous than the traditional peer-review that sifts and winnows.) Like the physicist, the more important the topic is to me, the more I wish to make up my own mind about what I read. (I am curious whether other researchers feel the same way. If you have a view on this to share, please write me: glass@asu.edu.A Final Clarification
So that my message is not misread, I add this final point in clarification. I do not suppose that it is in the best interests of the soft disciplines (educational research, for example) to seek to imitate the communications practices of the hard sciences. The differences between them are great, and most efforts of the former to imitate the latter in more fundamental respects have been unproductive. The soft sciences resemble the humanities much more than the hard sciences (Andreski, 1972; Gergen, 1973), and they will have to invent their own patterns of communicating in the digital future. My fond hope is that they will do so thoughtfully and democratically, rather than merely transfer the paper culture to the electronic medium.Notes
This article has two titles because it served as the basis for two presentations at the Annual Meeting of the American Educational Research Association in New Orleans, in April 2000. There is far more in this article than can be presented in the time allotted for presentations at the Annual Meeting. Each of the presentations (see below) will sample from various portions of the article.An article with two titles raises a great many questions. Why isn't the article broken into two, one for each presentation? "Why did this guy get accepted for two presentations on the 'same' topic and my paper got rejected?" someone might ask. How does one cite an article with two titles? Does this article count double for tenure and merit pay? Was this article(s) peer reviewed? All of these questions are legitimate, and the article hopes to raise them for discussion both explicitly and by example.
Session #18.69: Communicating Knowledge in the 21st Century;
4:05-6:05, April 25, 2000
Marriott, Mardi Gras B, 3rd Fl T.
Session #43.66:
Scholarly Electronic Communication and the Challenges of
Creating Digital Archives;
8:15a.m.-10:15a.m., April 28, 2000
Marriott Hotel, La Galerie 5, 2nd Floor.
References
Becher, Tony. (1989). Academic tribes and territories: intellectual enquiry and the cultures of disciplines. London: Milton Keynes; Bristol, PA: Society for Research into Higher Education & Open University Press.
Brand, W.J. (1999). Papyrophiles, philistines and electroncentrics: The slow growth of electronic scholarly journals. Doctoral dissertation. Tempe, AZ: Arizona State University.
Dow, Ronald F., (1997). Gatekeeper Attitudes Toward Supplanting Paper Journals with Electronic Alternatives. Doctoral dissertation. University Park: The Pennsylvania State University.
Franklin, A. (1990). Experiment, Right or Wrongs. Cambridge University Press.
Franklin, A. (1997). Can That Be Right? Essays on Experiment, Evidence, and Science. Dordrecht: Kluwer Academic Publishers.
Garvey, W. D. (1979). Communications: The essence of science facilitating information exchange among librarians, scientists, engineers, and students. New York: Pergamon Press.
Gergen, K. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26, 309-320.
Harnad, S. (1990). Scholarly skywriting and the prepublication continuum of scientific inquiry. Psychological Science, 1(6), 342-344. [Available online at http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad90.skywriting.html]
Harnad, S. (1993). Implementing peer review on the net: Scientific quality control in scholarly electronic journals. Laboratoire Cognition et Mouvement URA CNRS 1166 I.B.H.O.P. Universite d'Aix Marseille II 13388 Marseille cedex 13, France.
Kuhn, T. S. (1962). The Structure of Scientific Revolutions. Chicago: University of Chicago Press.
Labaree, D. F. (1998). Educational researchers: Living with a lesser form of knowledge. Educational Researcher, 27(8), 4-12.
Meehl. P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-34.
Nozick, R. (1974). Anarchy, state and utopia. New York: Basic Books.
Schauder, D. (1994). Electronic publishing of professional articles: Attitudes of academics and implications for the scholarly communication industry. Journal of the American Society for Information Science, 45(2), 73-100.