A reality of the path to publishing in top academic journals

And You Thought It Was Bad:

The Editorial Process of Journals

John Saunders^[1]

Richard Speed^[2]

Abstract

Academics are under increasing pressure to publish yet the literature on the editorial process for academic journals reveals many defects that block innovative work while failing to identify flawed or fraudulent research. There is a significant gap between the reviewing processes “as it purports to be,” a “gatekeeping” process to ensure the quality and significance of the material published reported in journals from many disciplines. An alternative interpretation of the publishing process, the “invisible college” places much more emphasis on the pre-submission processes and the social networks of research in generating quality material for publishing. The conclusions suggest authors view academic publishing as a human process supported by networks of kindred spirits rather than an abstract exercise pitched at inanimate journals.

Introduction

The pressure to publish in quality journals or perish has been strong in the US for many years. Decades ago, the pressure was lower in the UK although a few British marketing academics close to the American scene achieved international acclaim. Peter Doyle, Andrew Ehrenberg, Ken Simmonds, and Robin Wensley were typical of these in having London Business School links. They achieved international excellence despite university’s receiving no income to support their effort. Outside their small circle, few people in Britain were aware of their endeavours. Even fewer shared their academic contacts, attended the international marketing conferences that they did, or felt their urge learn the international game.

The decoupling of income continued through the early Higher Education Funding Council’s (HEFC) Research Assessment Exercise (RAE), when a top 5 rating earned recognition but no money. The Netherlands has a similar unfunded system but with published reports that would make the litigiously sensitive British blanch (VSNU 2002). In Britain, funding changed with successive RAEs until the 2001 round. It has left many marketing groups with no research income while supporting research active marketers in 5* rated schools over £20,000 per year (HEFCE 2003). Now Britain’s paper chase is on, and the paper is money as well as publications.

This puts pressure on academics to publish, review articles, create academic journals and edit them. Yet Brits at top international conferences continue to be rare and British publications in the top marketing journals are still as rare as in the days of Doyle, Ehrenberg, Simmonds, and Wensley reigned (Bassant et al. 2003). The message is now writ large: fund research in internationally rated departments lavishly; give little or no funds to those outside that golden circle. As one Vice-chancellor recently proclaimed, “The ivory tower now has a moat and portcullis.”  

With such skewed funding, breaking into the fortified tower of research privilege is more important and more difficult than ever before. A department’s international standing is multi-faceted and refers to a whole research group, not individuals. The often-heard claim “I am a  5* researchers” is both arrogant and ludicrous. Research communities are 5*, not individuals. A top researcher, with a string of international publications, is a 4, 3a, 3b or less grade researcher if other people in the research group are struggling for national standing. Additionally, international recognition extends beyond publications to include in the scale and success of its research degrees programmes, the scale and source of its research contracts income and signs of international research leadership as well as the esteem of a few key researchers.

However, the ability to publish in highly cited international academic journals is a necessary, if not sufficient, grounds for a group claim international standing. Publishing in top international marketing journals is not easy. Frustration of marketing academics is evident. The top journals are American, they are never read by practitioners, biased in the data sets and methodologies they accept, time consuming to write for and reject everything. Yet, a few Europeans have broken through the barrier. Moreover, marketers are not alone in finding top publishing so hard. This review of academic publishing across many disciplines reveals a surprising uniformity in the challenges faced. It also holds clues on how to succeed.

The aim of this study is to help authors submitting to academic journals by sharing with them the experience of those publishing in many academic fields. It has four parts. The first part looks at the reviewing process “as it purports to be.”  The second reviews the evidence from many disciplines on the performance of the reviewing and editorial process “as it is.”  The third part examines an alternative interpretation of the evidence, and some proposals for restructuring journals. Finally, conclusions are drawn.

A Rationale for the Reviewing Process

Academic journals are the principal outlet for the communication of results in marketing, as they are in every other academic discipline. Publication in academic journals is a mechanism by which the discipline signals the acceptance of new results as part of the body of accepted knowledge (Campanario 1998a). For authors, publication of their work in an academic journal serves as a signal of professional performance to current and future employers, current and potential colleagues and to research funding bodies. The reliability and validity of these two signals, about the value of the results and about the associated standing of the author, is ensured by the pre-publication process employed by the journals. The cornerstone of academic publication is the peer review process. The role of the editor and the reviewers is to minimise both type I error, (ensure that results that are not worthy of publication are not accepted) and type II error (ensure that results that are worthy of publication are not rejected) (LaBand and Piette 1994a)

They also seek to increase the quality of the contribution through advice to and demands on the author, with the objective of ensuring that, should the authors win space in the journal, the value the readership gains from that space is maximised (Ellison 2002b).

The Reviewing Process “as it purports to be”

The processing of articles for publication is similar for all refereed journals. Authors submit manuscripts to a journal of their choosing and, on receipt, the editor screens them to see if they are within the scope of the journal in terms of quality or topic. The aim of this screening is to give authors quick responses to papers and to guard the time of the reviewers. Manuscripts that do not pass the screen become “desk rejects.”

Manuscripts that pass screening then enter the formal review process. The editor handles choice of reviewers, and how many are selected, whether they are selected from within a set pool (the editorial board) or without, and how much information they have about the authors depends on journal policy.

The reviewing stage of the publication process can be long, slow, and painful for authors, reviewers, and editors. There is evidence that it has been becoming longer and slower over time (see Ellison 2002a and b).  To address these problems many incoming editors revamp the review process, often setting both quantitative and qualitative targets to be met at their journal (recent examples in marketing include Shugan 2002, Kamakura 2001, Varadarajan 2000, Stewart 1999). For an unlucky majority, the review process is quick and painful. Acceptance rates vary dramatically, but top journals reject most papers. A minority are congratulated and asked to revise their paper for resubmission to the journal. Today it is rare that papers are accepted at the first round or need only minor revisions (see Ellison 2002a).

When editors do invite an author to resubmit their work in line with the reviewers’ suggestions, the authors often face the challenge that it is rare that reviewers agree completely on what needs doing to the paper. Smart authors know this and are as expert at revising papers as they are at writing them. They are also quick and very thorough. As the rounds continue, the editor’s involvement increases in balancing the views of the authors and reviewers. Finally, the paper is accepted, copy edited and sent to the publishers.

This model of the publication process makes journal editors, review boards, and reviewers “gatekeepers of science” (Crane 1967, Glogoff 1988). On behalf of the discipline, they selflessly sift papers sent to their journal, reject some papers while advising other authors on how to make their submission more suitable for publication. The process where peers anonymously and selflessly evaluate and comment on each other's work is exemplary and is the foundation of knowledge development and dissemination. Sitting behind this model of the publication process are some significant assumptions. In the submission stage for the top journals to publish the high-impact material, it must be assumed that all the potentially high-impact material is being sent to those journals, and that all authors have equal opportunity to submit. It is assumed that the first editorial screen is unbiased and accurately reflects what the reviewers would have advised. It is assumed that in reviewer choice the allocation of reviewers is unbiased, and that the reviewers selected for a paper are authorities on the topic under investigation, the method used and is conscientious and quick in reviewing.

The review process is carried out by volunteers, normally without compensation (Engers and Gans 1998). Rewards are primarily intrinsic (contribution to the discipline), or symbolic (acknowledgement in the journal). The Gatekeeping model assumes such rewards are sufficient to encourage reviewers to provide prompt, reliable, accurateand unbiased reviews.

In the Gatekeeping Model, researchers send their work, and by extension, themselves, to the judgement of the wisest of their peers. It is a submittant led process, with the author choosing the journal to which to submit and handling the development of the manuscript for submission. The editor and reviewers are passive, sitting Sphinx-like as each intrepid author approaches for judgement.

The Reviewing Process “as it is.”

The accuracy of the “Gatekeeping Model” of the review process rests on some significant assumptions about the motivations and behaviours of the academics who act in these roles. A basic knowledge of attribution theory (Weiner 1986) suggests that every author whose wisdom has ever been rejected should seek an external locus of causality, scoff at the validity of these assumptions and descend into cynicism about the review process. Fortunately, the publication process has been subject to thirty years of academic research that allow us to form some opinions of the degree to which the cynicism is justified.

Submission

Authors do not submit equally. A survey of 28 education journals (MacNealy, Speck and Clements 1994) found that 89% of the editors had invited “inside track submissions”. Opinions split on the ethics of this insider dealing. Rodman and Mancini (1977) condemn the practice while Roediger (1987) and Armstrong (1996) argue that such an approach encourages innovation.

Editorial Inspection

The primary inspection of submissions to a journal is as important as it is time consuming. Few would agree with Tullock (1980) that editors should read all submissions and then decide whether to go ahead with it or not. However, many “clearly inappropriate” articles become “desk rejects.”  Although this action is harsh, it is necessary to protect reviewers (Absher 1990). At this stage, editors should use a light hand since even they are also human beings with prejudices.

Choice of Reviewers

The editor’s choice of reviewers has a huge influence on a submission’s chance of success and the quality of the feedback provided (Weller 1990). Some describe the assigning reviewers to papers as the editor's major task (Michels 1995). Sherrell, Hair, and Griffin (1989) found that out of a list of 15 editorial practices selection of reviewers that have a strong bias (pro or con) about a manuscript was rated as one of the two least ethical acts.

There is evidence that the selection process is not based on an objective assessment of who has the most ability, but rather editorial access is the major determinant of reviewer selection. Hammermesh (1994) studied the appointment of reviewers for seven economic journals and found that a disconcertingly high proportion of reviewers were drawn from the editor’s own department. In medicine, Stossel (1985) found that a disproportion number of refusals to review came from the most eminent reviewers. Evidence has also been found of reviewers passing on the reviewing task to more junior colleagues (Glogoff 1988), and of editors deliberately selecting more junior staff, even doctoral students, as reviewers since they are less overburdened (Romanelli 1985).

One benign explanation for editors assigning reviewing based on grounds other than expertise is that in modern reviewing, a single reviewer is rare. Editors therefore engage in a form of reviewer portfolio management. Each reviewer has idiosyncrasies, and a good editor knows their editorial board, their rejection rates, and their academic prejudices (Siegelman 1991). This enables editors to “balance the portfolio” and adjust for particularly harsh or benign reviewers.

Reviewer Contact

The mode of reviewing varies across disciplines. In the physical sciences, one reviewer often suffices (Hargens and Herting 1990) while multiple reviewing predominates elsewhere. Recently blind reviewing, where the reviewer is unknown to the author, has become the dominant form (Dalton 1995). Although this “secrecy of the Mafia” (Wright 1970) is controversial in the natural sciences (McCutchen 1991, Robinove 1990, Godlee 2002), double blind reviewing is increasingly used (McNutt et al. 1990). In this, the identities of authors and reviewers are kept from each other.

There are arguments for, and against, blind reviewing (Armstrong 1996, Fabianto 1994, Godlee 2002). Blind reviewing gives authors equal treatment and helps the scholarly disadvantaged, such as women, ethnic minorities, or authors from universities that lack a research reputation (McGiffert 1988). Some of these concerns seem justified. From a quasi-experiment, Peters and Ceci (1982) concluded that reviewers were biased against less prestigious institutions. Rosenthal’s (1982) experience of once rejected papers being accepted when he moved to Harvard also shows evidence of institutional bias. Lloyd (1990) also confirmed gender bias when papers by female authors were accepted more often by female reviewers (62%) than by male reviewers (21%).

Evidence does not support Armstrong’s (1996) view that naming referees forces them to produce better reviews. Besides improving fairness, authors judged blind reviews to be of better quality (McNutt et al. 1990) and to be more knowledgeable (Fletcher and Fletcher 1997) than open ones. Citation rates fuel the debate on bias. If reviewers were biased, papers from lesser institutions would be cited most because they would have to be strong to survive reviewing. Perlman (1982) found the opposite to be true.

After finding that citation rates are a good guide to future citation rates, Abrams (1991) argued that revealing authors names would help find good papers with high future citation rates. This suggestion is problematic. Authors may cite papers from leading institutions because they give clear weight to their argument or to increases the chance of their citing reviewers. Based on the evidence, McNutt et al. (1990) concluded that there was minor difference between blind and non-blind reviews although blind reviews do have the edge on quality.

A final argument for open reviewing is honesty. Blind reviewing is really as "opaque reviewing". Experts know one another’s work so can often guess authorship. Making the process open removes the pretence of blindness.

Reviews

Many support Eysenck and Eysenck’s (1992) criticism that refereeing is: unreliable, since referees often disagree (Cicchetti 1991, Petty, Fleming and Fabrigar 1999); not valid, since referees ratings do not relate to eventual citation rates (Gordon 1977); biased; inefficient, since it delays publication (Horrobin 1990, Ellison 2002a), inhibiting the publication of new ideas (Horrobin 1990) and damaging, painful and depressing for authors. Nobel Laureate George Akerloff’s “Market for Lemons” paper was rejected at three top journals (American Economic Review, Journal of Political Economy and Review of Economic Studies) before being published in a fourth (Quarterly Review of Economics), and Akerloff thinks the rocky reception of this paper did affect his subsequent work, making him reluctant to follow up what was a seminal contribution to information economics (Gans and Shepherd 1994). Marketing Science editor Steven Shugan points out Akerloff was lucky not to be in marketing, where he would have had fewer A-journals as options (Shugan 2002). The final item drives many people’s feelings about reviews. Referees are acculturated to the negative (Eysenck and Eysenck 1992), ignorant (Horrobin 1982), arrogant and obnoxious (Bradley 1981a, b). People also review the best papers less able than the author, so are prone to be misunderstood (Armstrong 1996).

Besides providing an outlet for weak and unpleasant academics, refereeing has a moral dilemma (Lafollette 1992) when authors and referees are competing for the same recognition or money (Forscher 1965, Ellison 2002b). Being experts in their field, referees gain little from research that demolishes the world that they helped to create. This could explain why reviewers tend to reject research that is innovative (Kuhn 1962) or controversial (Armstrong and Hubbard 1991).

Reviewers are not explicit about their reasons for rejection. Mahoney (1977) found controversial findings rejected on methodological grounds while papers with the same method, but supporting the established thinking, were accepted. This prejudice is conscious. Sixty-one per cent of reviewers questioned by Chubin and Hackett (1990) admitted that they were “reluctant to support unorthodox or high-risk research”.

The resistance faced by novel ideas is part of scientific tradition. Examples span Galileo’s persecution during the Inquisition (Carey 1995) to Pope John Paul II warning Stephen Hawking not to inquire into the big bang’s creation (White and Gibbin 1992). There are many stories of researchers, such as Newton and Darwin, struggling to have their ideas accepted. We know of them because they fought and won. How many more ideas died in the process?

Studies of refereeing show that modern ideas are resisted by strong and weak forces. The strong forces occur when findings are not what people want to know. Armed security guards protected Rushton and Ankney’s (1996) presentation of their findings relating brain size, race, and gender to intelligence. Ironically, their findings were not even innovative since similar politically incorrect findings have existed since 1880. The weak force acts against negative (Mahoney 1977) or inconclusive (Stirling 1959) findings. Weak forces accumulate. If influences are inconclusive and negative results suppressed, there will be mounting evidence of the small positive results that are desired. This can occur early in analyses. When statistical results do not follow expectations, the offending items are often removedbecause they lack “face validity.”

Editor’s decision

The editor’s decision is informed by reviews and results in a paper being rejected, accepted outright or, more often, accepted subject to revisions being successfully completed. Rejection rates vary. Zucherman and Merton (1971) found rejection rates ranging from below 30% in linguistics, geology, physics, and the biological sciences to over 80% in the political sciences, philosophy, history, languages, and literature. Gordon (1978) found rejection rates in the social sciences (75-80%) to be much higher than in the sciences (10-67%). Within disciplines, the variations are even wider. The rejection rates for library and information sciences journals (Budd 1988) ranged between 0 and 92% while in the humanities they ranged between 25 and 98% (Budd 1981).

High rejection rates are often associated with high quality (Dalton 1995). If this is true, the figures reflect a difference in the quality of disciplines as well as journals. The pattern of rejection has implications for the cynics who liken the acceptance of an article for publication to a roll of the dice (Glenn 1982, Lindsey 1988). If they are correct, the odds a gambler (author) faces depend upon the game (discipline or career) and the table (journal) they play.

Even if the odds vary, the optimist’s expectation is that if an article meets a journal’s standard, referees will recognise and agree on its qualities (Cicchetti 1991). A “trip-up model” of publishing follows the optimist’s expectations. If a paper is good there is little variability in the referee’s perception of the paper being good. On the other hand, if a paper is poor, at least one of the reviewers will recognise this and trip it up by destroying the unanimity of positive reviews.

Some evidence supports the trip-up model. Editor’s decisions are highly correlated with those of reviewers (Mahoney et al. 1978, Marsh and Ball 1989) and a single recommendation to reject often results in a paper's rejection (Bakanic et al. 1990). Cicchetti (1997) reports that in the Journal of Applied Psychology, two reviewers recommending acceptance was associated with 88% acceptance, but if one of the two urged rejection this figure fell to 14%.

Editors often reject papers that eventually become citation classics (Campanario 1993, 1995, 1996) or that lead to Nobel prizes (Gans and Shepherd 1994, Yalow 1982). Azbel (1993) reports that Albert Einstein became so outraged by reviewing at the Physical Review he ceased to publish in the journal as a protest. Outstanding contributions are likely to be innovative or controversial, so upset at least one referee with investments in the status quo.

Responding to referees

Most authors accept the rejection of their papers without question. Others are so discouraged by the demands for a major revision that they give up or submit the paper to another journal. Hirst et al (1999) noted that author’s response to rejection varies with country and the experience of authors. Experienced authors are most likely to question the rejection and rarely fail to publish once they have a chance to resubmit. With successful academic publishing comes the realizatdion that journals are social, political (Rip 1985) and economic enterprises. In 1996 The Lancet recognised this explicitly and appointed an ombudsman, “to record and where necessary to investigate episodes of alleged editorial maladministration” (Horton 1996 p6).

Simon et al. (1986) examined the results of challenges to rejection at the American Sociological Review, reporting that the editor changed their decisions on 13 % of occasions. This is data from a single editor, and as Dalton (1995), editors’ receptiveness to their decision being challenged is not uniform. Some methods of challenging editors are more likely to succeed than others (Hirst et al. 1999). Aggressive attacks on a referee’s judgement are far less likely to succeed than challenges that explain how, thanks to the referees' comments, the paper can be improved substantially.

Once allowed to resubmit a paper, an author becomes part of the editorial process (Schneider 1990, Weick 1985) where publication is the result of structured negotiations. Only 23% of the authors surveyed by Bradley (1981b) thought that the changes recommended by referees were not based on “whim, bias or personal preference.”  In contrast, 31% of the referees demanded significant changes. Forty per cent of the authors also thought that some of the referees had not read the paper properly. Consequently, 8% of the authors swallowed their pride – along with a pinch of professional ethics - and resubmitted their paper after making changes that they knew to be wrong.

The publication

Editors accept bad papers as consistently as they reject good ones. Armstrong (1996) cites much evidence of the failure of reviewing to detect errors or fraud. In an analysis of the statistical procedures in 38 papers in medical journals, Murray (1988) found only 6 worthy of publication as they were, 11 needed minor revisions, 7 needed major revisions and 4 should have been rejected outright. The reliability of citations is equally depressing. Evans et al. (1990) found error in 48% of the citations in three medical journals while Eichorn and Yanjaur (1987) found 31% of references in public health journals had errors.

Given its failings, can reviewing expose or reveal the incompetence or fraud of authors? Studies of John Darsee’s retracted work highlight this question. Stewart and Feder (1987) studied 18 of Darsee’s publications that appeared shortly before he admitted to scientific fraud. Sixteen of the papers had major or minor errors, with an average of 12 errors per publication.

Two opportunities for the fraudulent academic are plagiarism and the repeated publication of the same work. To test the ability of reviewers to find these forms of malpractice Epstein (1990) submitted two modified but methodologically flawed versions of a well-cited paper to 110 journals. Only two journals said that the paper had been published elsewhere, and few reviewers mentioned the methodological problems.

Replicate studies cast further doubt on the ability of reviewers to recognise good science. An analysis of 266 replications found that only a quarter supported the original findings. The rest conflicted with the original findings in a minor (1/4) or major (1/2) way. Whether these results reflect incompetence, fraud or other causes, post publication refereeing shows that many criticisms of the reviewing of academic journals are well founded. Not only is the process biased against innovative and controversial studies, but it also does not detect fraudulent or flawed research.

Validity of the Gatekeeper Model

The hopes and experience of the Gatekeeper Model of the editorial process are far apart. More concerning are the implications of this research for the body of knowledge disseminated. The overwhelming evidence is that the peer review process is not a neutral process through which the wisest in the discipline select the best work to join the discipline’s knowledge base. Of what value are streams of publications with methodological flaws and errors that support the establishment? If the evidence on process is representative, the findings strike at the heart of science by undermining the credibility of the journals that support it. Is science being strangled by the processes designed to serve it?

Of course, this research begs a further question:  are the empirical results on journals reliable? Unless the critical articles are islands of purity in a sea of errors, fraud, and incompetence, they are as prone to error as the articles they criticize. The research on journal reviewing could itself be biased. Researchers into journal reviewing are self-selected groups who have chosen to study their discipline’s reviewing. The people motivated to that choice are those most hurt by the process. Once a critical perspective is set up, it becomes the orthodoxy. Articles that are positive about reviewing become uninteresting and a challenge to the critical establishment.

An alternative model of the review process exists that draws on the same evidential base but has received less attention from researchers. In the next section we briefly review this, before outlining some of the more radical proposals put forward by concerned authors.

Alternative Perspectives

Naiveté, arrogance, and a simple application of attribution theory could explain some researcher’s disappointment with the reviewing process. However, flawed though the process may be, many have mastered it. Researchers have examined to what extent those who have mastered the art of publishing differ from those who have not. This research dates back to the sixties (see Campanario 1998b for a review) and can be termed the “Invisible College” Model (Crane 1972).

The premise of the Invisible College Model is that publication does not happen by accident. Journals should not be modelled as portals guarded by editors and reviewers, and the review process should not be modelled as a neutral selection process. Journals should be modelled as a central node in a social network that consists of like-minded editors, reviewers and authors, and the review process just the final stage in a set of interactions within the network. The network is forged when players interact professionally, through their institutions and their academic training, through meeting at conferences, seminars, and workshops. Early researchers showed that journals have disproportionate acceptance rates for papers where the author has an institutional connection to a journal (Pfeffer, Leong and Strehl 1977, Willis and McNamee 1990, Peters, and Ceci 1982). Such findings were taken to show the existence of an Invisible College, and of bias, where insiders receive favourable treatment, and rejection rates are built on the aspirations of outsiders. Those with an institutional relationship with the editor are seen to be using this relationship to receive preferential treatment at the journal.

One alternative explanation is that this is bias, but it is bias to quality. The insider articles are accepted because they are higher quality. Laband and Piette (1994b) investigated this using data drawn from 28 top economics journals and found higher citation rates for papers by authors with a connection to the editor. Smith and LaBand (1995) found equivalent results for 15 accounting journals. The alternative explanation is that the pre-existing connection to the editor does not affect the review process, or the editorial decision, but it affects acceptance through the submission process. Laband and Piette (1994b) argue that editors will search for research right for their journal at the presubmission stage through conference attendance, and use of personal and professional relationships. Hence, this is research that is not randomly submitted but has been effectively pre-screened by the editor for its suitability. Under this scenario, those with an institutional relationship with the editor find the editor using this relationship to solicit good papers for their journal.

The Invisible College model is much more in tune with modern notions of knowledge generation through communities of common interest, held together by social capital, be they research teams in organisations, inter-firm networks in industry. In this model of the publication process, the pre-submission process is vital. Through this process, research is pre-tested, knocked into shape or abandoned. This exchange amongst researchers with shared interests also allows them to gauge who is doing what and the latest methodological twists and trends. Within these communities, researchers find collaborators who have complementary skills and who can share the burden of research. They also establish relationships with colleagues, who offer advice on papers, many of whom could be reviewers. Submitting an article to a journal in this context is no longer like the role of a dice from an invisible hand, rather it is the final stage in a collaborative process of product development. If these processes work properly, the interactions within the Invisible College at the pre-submission phase minimise type I and II error, and the peer review process can become much more a process of quality improvement.

It is arrogant to assume one’s research and insights are so wonderful that building academic relationships and pre-testing research can be cut out. Humility and recognition of other people’s work is a feature of most successful researchers. As Sir Isaac Newton wrote to his friend Robert Hooke in 1675: “If I have seen further, it is by standing on the shoulders of giants.”  Even in this, Newton was building upon earlier thoughts of Bernard of Chartres (John of Salisbury, 1159):

We are like dwarfs on the shoulders of giants, so that we can see more than they, and things at a great distance, not by virtue of any sharpness of sight on our part, or on any physical distinction, but because we are carried high and raised up by their giant size.

Radical Solutions

Of course, there are weaknesses to the Invisible College model, the most obvious of which is that not everyone has the same access to the networks that enable the necessary pre-submission interactions. Two academics that have long-standing research interests in the academic publications process have proposed radical redesigns of the dissemination process to address the problems of access. Developed independently, they have some key features in common.

The first proposal is from Armstrong (2003), who has long argued that the key issue for marketing is the importance of the knowledge we generate. The key question for Armstrong is how to discover and communicate important findings. Having recognised the failure of the current dissemination process to encourage this, he makes recommendations to journals and the profession. The most radical is to publish something about every paper submitted. The minimum published would be what was discovered, how it was discovered and why it is important. These gobbets could be published with reviews or recommendations from the editorial board, and more details provided online. Armstrong also recommends significant changes to the peer review process, all aimed at manuscript improvement rather than screening, and the dedication of space in journals for both controversial findings and for replications.

Campanario (1997), like Armstrong, is a leader in studying the peer review process. His suggestion is to reverse this process and make it outlet led. Each discipline or speciality has a single “metajournal,” which might be like internet USENETs. The metajournal is effectively a clearinghouse for academic research findings. Any author could submit an abstract or a full manuscript to the metajournal, with no reviewing or screening. The role of the journal editorial boards would be to routinely scan the metajournal to find manuscripts that fit well with the positioning of their journal. Editors would then contact the author about publishing the articles in their journal. If more than one offer is made, the author would choose the journal in which to publish.

The metajournal places much more responsibility for content onto the journals. They must positively compete for the best work from all authors. Campanario envisages fresh players in the publication process, “journal scouts,” who would seek out content for specific journals. Journal scouts would be experts in their fields, could convince editors of the importance of papers and, like a literary agent, would look to secure publication for the chosen work. With experts scouring the available papers for hidden gems, the odds on an important paper being tripped up by a single outlying reviewer would be minimised. Journals would be free to continue to use peer review for screening but run the risk that an author may opt for a lower prestige, but faster publishing journal.

A common feature of these two proposals is to abandon gatekeeping, at least at the initial stage. All material gets out somewhere, whether as a tightly focused gobbet in one case, or in the metajournal in the other. As the costs of publication fall with online opportunities, the need for editors to guard their pages is diminished. Once it is no longer about guarding access to a scarce resource of pages the role of the journals and their editorial team's change. They become a virtual college, seeking out new contributors and working with them to develop better and more important material.

Conclusions

The literature on academic publishing produces hardly a study that does not reveal failings or condemns the process. Can the failure be ignored? Maybe now is the time to adopt one of the more radical solutions. More mundanely what is needed is for some organisation to endorse good journals. However, such an organisation is likely to be as establishment oriented and resistant to innovation as the chaos of journals that currently exists.

However, things are not so bleak. Within all disciplines there are a few journals that are well known, highly cited, professionally run, and essential to understanding the latest developments. Alongside them are masses of journals that are little read and of questionable value. Many are published by organisations with a greater interest in mammon than academic excellence. It would be valuable to find out if the documented failures of academic journals applied particularly to the elite, to the also-rans, or to all.

We have highlighted an alternative interpretation of the publishing process – that it is a social, human, networked process, involving a whole series of other interactions beyond those in the traditional peer review process. Access to this network is the key step for success. Academics are encouraged to seek out these networks and take part, journals and their publishing organisations are encouraged to explore mechanisms, like those discussed as radical solutions, which enable access to networks to be maximised.

Any process that has bright people pushing at the limits of knowledge will be challenging and fraught with errors. That is true of any innovative process. The frustrations, agonies, unfairness, and ecstasies of innovation will never be removed. Academic publishing is like any other innovative process in having two parts. The first is a mechanistic, bounded, professional process that chugs ahead but requires ever-changing skills that can be learned by bright people who want to succeed. The second is the process of innovation where people succeed by recognising and adapting to the chaotic organisations that produce change. Order can be imposed. Under fascism, the trains run on time, but innovative people use the trains to escape the order.  

References

Abrams, P.A. (1991), “The Predictive Ability of Peer Review of Grand Proposals: The Case of Ecology and the US National Science Foundation” Social Studies in Science, 21, pp. 111-32

Absher, John. R. (1990), “Peer Review and the Philosophy of Science.” Journal of the American Medical Association, 264, pp. 3143

Armstrong, J. Scott, and R. Hubbard (1991) “Does the Need for Agreement among Reviewers Inhibit the Publication of Controversial Findings?” Behavioural and Brain Sciences, 14, pp. 136-7

Armstrong, J. Scott. (1996), “Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation”, Science and Engineering Ethics, 3, pp. 1-22

Armstrong, J. Scott. (2003), “Discovery and Communication of Important Marketing Findings:  Evidence and Proposals” Journal of Business Research Forthcoming

Azbel, Mark, (1993) “Could Columbus have Passed Peer Review?” Physics Today 46 6 pp. 423-8

Bakanic, Von, Clark McPhail, and Rita J. Simon (1990) “If at First You Don’t Succeed: Review Procedures for Revised and Resubmitted Manuscripts” American Sociologist, 21, pp. 373-91

Bassant, John, Sue Birley, Cary Cooper, Sandra Dawson, John Gennard, Andrew Grey, Peter Jones, John McGee, Colin Mayer, Michael Pidd, John Saunders, Andrew Stark, and M. Gardiner (2003), “Talking Business: The State of the Field in UK Management Research” British Journal of Management 13 1, pp. in press 

Bradley, James V. (1981a), “Overconfidence in Ignorant Experts” Bulletin of the Psychonomic Society, 17, pp. 82-4

Bradley, James V. (1981b), “Pernicious Publication Practices” Bulletin of the Psychonomic Society, 18, pp. 31-4

Budd, John. (1981), “Humanities Journals in 1979” Scholarly Publishing, pp. 12171-81

Budd, John. (1988), “Publication in the Library and Information Sciences: The State of the Literature” Library Journal, 113, pp125-31

Campanario, Juan Miguel, (1995), “Commentary:  On Influential Books and Journal Articles Initially Rejected Because of Negative Referees’ Evaluations” Science Communications 16 pp.304-25

Campanario, Juan Miguel, (1996), “Have Referees Rejected some of the Most-Cited Articles of All Times?”  Journal of the American Society for Information Science 47 pp. 302-10

Campanario, Juan Miguel, (1997) “The journal scout: An alternative to peer review” The Scientist, 11 10, pp. 9

Campanario, Juan Miguel, (1998a) “Peer Review for Journals as it Stands Today – Part 1” Science Communications 193 pp. 181-211

Campanario, Juan Miguel, (1998b) “Peer Review for Journals as it Stands Today – Part 2” Science Communications 194 pp. 277-306

Campanario, Juan Miguel. (1993), “Consolation for the Scientist: Sometimes It Is Hard to Publish Papers That Are Later Highly-Cited” Social Studies of Science, pp. 23342-62 

Carey, John. (1995), The Faber Book of Science, London, Faber, and Faber

Chubin, Daryl E, and Edward J. Hackett. (1990), Peerless Science: Peer Review and U.S. Science Policy, Albany, NJ, State University of New York Press

Cicchetti, Dominic V. (1991), “The reliability of Peer Review for Manuscripts and Grant Submissions: A Cross-Disciplinary Investigation” Behavioural and Brain Sciences 14, pp.119-35.

Cicchetti, Dominic V. (1997) “Referees, Editors and Publication Practices: Improving the Reliability and Usefulness of the Peer Review System” Science and Engineering Ethics 3 pp. 51-62

Crane, Dian. (1967), “The Gatekeepers of Science: Some Factors Affecting the Selection of Articles for Scientific Journals”, American Sociologist November 2, pp. 195-201.

Crane, Dian. (1972), Invisible Colleges:  Diffusion of Knowledge in Scientific Communities Chicago:  University of Chicago Press

Dalton, Margaret Stieg. (1995), “Refereeing of Scholarly Works for Primary Publishing” In Martha E. Williams (ed.) Annual Review of Information Science and Technology, 30, Medford, NJ, Information Today Press, pp. 213-50

Eichorn, P., and A. Yankauer (1987), “Do Authors Check Their References? A Survey of Accuracy of References in Three Public Health Journals” American Journal of Public Health, 77, pp. 1011-2

Ellison, Glenn (2002a), “The slowdown of the economics publishing process”, The Journal of Political Economy, 110 5; pp. 1035-82

Ellison, Glenn (2002b), “Evolving standards for academic publishing: A q-r theory”, The Journal of Political Economy, 110 5; pp. 994-1035

Engers, Maxim and Joshua S. Gans, (1998), “Why Referees are not Paid (Enough)” American Economic Review 88 5 1341-50

Epstein, W.M. (1990) “Confirmational Response Bias among Social Work Journals.” Science Technology and Human Values, 15, pp. 9-38

Evans, James T., Howard I. Nadjari, and Sherry A. Burchell (1990) “Quotational and Reference Accuracy in Surgical Journals: A Continuing Peer Review Problem” Journal of the American Medical Association, 263, pp. 1353-4

Eysenck, H.J., and S.B.J Eysenck (1992) “Peer Review: Advice to Referees and Contributors.” Personality and Individual Differences, 13, pp. 393-9

Fabianto, A. (1994), “Anonymity of Reviewers” Cardiovascular Research 28 1134-9

Fletcher, Susanne W., and Robert H. Fletcher (1997), “Evidence for the Effectiveness of Peer Review”, Science and Engineering Ethics, 3, pp. 35-50

Forscher, Bernard K. (1965) “Rules for Referees”, Science, 50 October 15, pp. 319-21

Gans, Joshua S., and George B. Shepherd (1994) “How are the Mighty Fallen:  Rejected Classic Articles by Leading Economists” Journal of Economic Perspectives 8 1 pp.165-80

Glenn, Norvald, (1982), “The Journal Article Review Process as a Game of Chance” Behavioural and Brain Sciences, 5, pp. 211-2

Glogoff, Stuart (1988), “Reviewing the Gatekeepers: A Survey of Referees of Library Journals” Journal of the American Society for Information Science, 39/6 pp. 400-8

Godlee, Fiona, (2002) “Making Reviewers Visible:  Openness, Accountability and Credit” Journal of the American Medical Association 287 21 pp. 2762-5

Gordon, Michael D. (1977) “Evaluating the Evaluators” New Scientist, 73 February 10, pp. 342-3

Gordon, Michael D. (1978) “Disciplinary Differences, Editorial Practices and the Patterning of Rejection Rates for UK Research Journals” Journal of Research Communications Studies, 1, pp. 139-59.

Hamermesh Daniel S. (1994), “Facts and Myths about Refereeing” Journal of Economic Perspectives 8 1 153-64

Hargens, Lowell L. and Jerald R. Herting (1990) “A New Approach to Referee’s Assessment of Manuscripts”, Social Science Research, 19, pp. 1-16

HEFCE (2003), Re currant Grant for 2003-4: March 2003/10 Bristol: Higher Education Funding Council for England

Hirst, Andy, Chris Stagg, and John Saunders (1999) “From IJRM Editor to EMAC President: Some Lessons, One Painful.”  EMAC Chronicle, January, pp. 1-2

Horrobin, David F. (1982), “Peer Review: A Philosophically Faulty Concept Which Is Proving Disastrous for Science” Behavioural and Brain Sciences, 5, pp. 217-8

Horrobin, David F. (1990), “The Philosophical Basis of Peer Review and the Suppression of Innovation” Journal of the American Medical Association, 263, pp.1438-41

Horton, R. (1996) “The Lancet’s Ombudsman” The Lancet 348:6

Kamakura, Wagner A; (2001) “From the editor” Journal of Marketing Research, 38, 1; pp. 1-2

Kerr, S., J. Tolliverand D. Petree (1977), “Manuscript Characteristics which Influence Acceptance for Management and Social Science Journals”, Academy of Management Journal, 20, pp. 132-41

Kuhn, T.S. (1962), The Structure of Scientific Revolutions, Chicago, IL, University of Chicago Press

Laband, David N., and Michael J. Piette, (1994a) “Does the "blindness" of peer review influence manuscript selection efficiency?” Southern Economic Journal 60 4 pp. 896-905

Laband, David N., and Michael J. Piette, (1994b) “Favoritism versus Search for Good papers:  Empirical Evidence Regarding the Behavior of Journal Editors” Journal of Political Economy 102 1 pp. 196-204

Lafollette, Marcel C. (1992), Stealing into Print: Fraud, Plagiarism, and Misconduct in Scientific Publishing, Berkeley, CA, University of California Press

Lindsey, Duncan. (1988), “Assessing Precision in the Manuscript Review Process: A Little Better Than a Dice Roll” Scientometrics, pp. 1475-82

Lloyd, Margaret E. (1990), “Gender Factors in Reviewer Recommendations for Manuscript Publication” Journal of Applied Behaviour Analysis, 23, pp. 539-43

MacNealy, M. S., B. W. Speek and N. Clements (1994), “Publishing in Technical Communication Journals from the Successful Author’s Point of View” Technical Communications, 41, pp. 240-59

Mahoney, Michael J., Alane Kazdin, and Martin Kenigsgerg (1977), “Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System”, Cognitive Therapy and Research, 1, pp. 161-75

Mahoney, Michael J., Alane Kazdin, and Martin Kenigsgerg (1978), “Getting Published.” Cognitive Therapy and Research, 2, pp. 69-70.

Marsh, H.W. and S. Ball (1989), “The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals: Inter-judgement Reliability” Journal of Experimental Education, 57, pp. 151-69

McCutchen, Charles W. (1991), “Peer Review: Treacherous Servant, Disastrous Master”, Technology Review, 94, pp. 29-40

McGiffert, Michael. (1988), “Is Justice Blind? An Inquiry into Peer Review” Scholarly Publishing, 20, pp. 43-8.

Michels, R. (1995), “Peer Review” International Journal of Psycho-Analysis 76 pp.217-21

Murray, G.D. (1988) “The Task of a Statistical Referee”, British Journal of Surgery, 75, pp. 664-7

NcNutt, Robert A., Arthur T. Evans, Robert H. Fletcher, and Suzanne W. Fletcher (1990), “The Effects of Blinding on the Quality of Peer Review” Journal of the American Medical Association, 263, pp. 1371-6

Perlman, David. (1982), “Reviewer ‘Bias’: Do Peters and Ceci Protest Too Much?” Behavioural and Brain Sciences, 5, pp. 231-232

Peters, Douglas P., Stephen J. Ceci (1982), “Peer-Review Practices of Psychological Journals: The Fate of Published Articles Submitted Again” Behavioural and Brain Sciences, 5, pp. 187-95

Petty, Richard E., Monique A. Fleming and Leandre R. Fabrigar, (1999) “The Review Process at PSPB: Correlates of Inter-reviewer Agreement and Manuscript Acceptance” Personality and Social Psychology Bulletin 25 2 pp. 188-203

Pfeffer, J., A. Leong, and K. Strehl, (1977), “Paradigm Development and Particularism:  Journal Publication in Three Scientific Disciplines”, Social Forces 55 pp.938-51

Rip, Arie. (1985), “Peer Review Is Alive and Well in the United States.” Science, Technology and Human Values, 10, pp. 83-6.

Robinove, Charles J. (1990), “Refereeing Policy: Against Anonymity”, Physics Today, 43, pp. 11

Rodman, H. and J.A. Mancini, (1977), “Errors, Manuscripts and Equal Treatment” Research in Higher Education 7 pp. 369-74

Roediger, Henry L. (1987), “The Role of Journal Editors in the Scientific Process” In D. N. Jackson and J. P. Rushton (eds.) Scientific Excellence, London, Sage

Romanelli 1985

Rosenthal, Robert. (1982), “Reliability and Bias in Peer-Review Practices” Behavioural and Brain Sciences, 5, pp. 235-6

Rushton, J. P., and C.D. Ankney, (1996), “Brain Size and Cognitive Ability: Correlations with Age, Sex, Social Class, and Race”, Psychonomic Bulletin and Review, 3, pp. 21-36

Schneider, Joseph W. (1990), “The Case of the ‘Unfair’ Review: Ethical Issues from an Editor’s File” American Sociologist, 21, pp. 88-95

Sekaran, Uma. (1992), Research Methods for Business: A Skills-Building Approach. 2nd Ed., John Wiley & Son Inc.

Sherrell, D.L., J.F. Hair and M. Griffin (1989) “Marketing Academicians Perceptions of Ethical Research and Publishing Behavior”, Journal of the Academy of Marketing Science 17 315-24

Shugan, Steven M., (2002), “Editorial:  The Mission of Marketing Science”, Marketing Science 21 1 pp, 1-13

Siegelman, Stanley. (1991), “Assassins and Zealots: Variations in Peer Review” Radiology, 178, pp. 637-42

Simon, Rita J., Von Bakanic, and Clark McPhail (1986), “Who Complains to the Editor and What Happens”, Sociological Inquiry, 56, pp. 259-71

Smith K. and David N. LaBand (1995), “The Role of Editors’ Professional Connections in Determining which Papers get Published:  Evidence from Accounting Research Journals” Accounting Perspectives 1 21-30

Stewart, David W., (1999) “Beginning again: Change and renewal in intellectual communities” Journal of Marketing 634; pp. 2-4

Stewart, W. W., and N. Feder (1987), “The Integrity of the Scientific Literature.” Nature, 325, pp. 207-14

Stirling, T.D. (1959), “Publications Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance – or Vice Versa” Journal of the American Statistical Association, 54, pp. 30-4

Stossel, T.P., (1985), “Reviewer Status and Reviewer Quality:  Experience of the Journal of Clinical Investigation” NewEngland Journal of Medicine 312 658-59

Tullock, Gordon. (1980), “What’s Wrong with Editing” Speculations in Science and Technology, 3, pp. 610-6

Varadarajan, P Rajan (2000), “From the editor-elect” Journal of the Academy of Marketing Science 28, 3; pp. 327-9

VSNU (2002), Assessment of Research Quality: Business Administration and Management Utrecht: Vereniging van Universiteiten

Weick, Karle E. (1985), “Editing Innovation into Administrative Science Quarterly” In L. L. Cummings and Peter J. Frost (eds.) Publishing in the Organisational Sciences, Homewood, IL, Richard D. Irwin, pp. 366-76

Weiner, Bernard (1986), An Attributional Theory of Motivation and Emotion New York: Springer-Verlag

Weller, Ann C. (1990) “Editorial Peer Review in U.S. Medical Journals”, Journal of the American Medical Association, 263, pp. 1344-7  

White, Michael, and John Gibbin (1992), Stephen Hawking: A Life in Science, London, Plume

Willis, C.L. and S.J. McNamee, (1990), “Social Networks of Science and Patterns of Publication in Leading Sociology Journals 1960 to 1985” Knowledge:  Creation, Diffusion, Utilization 11 pp. 363-81

Wright, R. Douglas (1970), “Truth and its Keepers”, New Scientist, 45 February 12, pp. 402-4

Yalow, Rosalyn S. (1982), “Competency Testing for Reviewers and Editors” Behavioural and Brain Sciences, 5 pp. 244-5

Zucherman, Harriet and Robert K. Merton (1971), “Patterns of Evaluation in Science: Institutionalisation, Structure and Function of the Referee System” Minerva, 1, pp. 66-100

^[1] Address for correspondence: Aston Business School, England; email jsxtsr2@gmail.com

^[2] Melbourne Business School, Australia.