Assessing quality of peer review:
An uphill task
Dr. Jamshed Akhtar*
Peer Review is the process of subjecting a work or idea of an investigator to the scrutiny of others who are “Experts” in the field. Peer review is needed because it is assumed that it improves the quality of research work. But there is a great question mark at the end of this statement.
CPSP and JCPSP
Before going into the subject further I would like to explain as to how we work at Journal of College of Physicians & Surgeons Pakistan (JCPSP). We feel proud that we are among few journals indexed in Medline and is also enrolled with many others indexing services.
CPSP is the postgraduate diploma awarding body of Pakistan. More than 80% of the specialists in various medical fields are Fellows and Members of CPSP. CPSP recognizes various departments of different medical institutes for imparting training to postgraduate trainees. For imparting training consultants and faculty members have to undergo few mandatory workshops. CPSP offers fellowship in nearly 50 specialties including dentistry. We therefore have a long list of faculty members from all over Pakistan who act as a pool for reviewing articles submitted for publication to JCPSP. This opportunity lies only with us.
The circulation of JCPSP is more than any other medical journal of the Pakistan. It is distributed free of cost to all fellows of CPSP more than 7000 in number. It is also sent on complimentary basis to all libraries of medical colleges. In addition many foreign medical institutes also receive its copies.
Attributes of a Reviewer
To become reviewer at JCPSP an individual must be a member of teaching faculty of an educational institute. The person must have attended a workshop on Research Methodology & Biostatistics at CPSP. He should preferably be a supervisor of FCPS trainee and must have some publications to his credit.
Process of review
It involves initial assessment (format, newness, novelty, category, language)
Review by statistician and Assessment by bibliographer.
Following this the manuscript being processed meets the following fates:
Rejection (nearly 50%), send for redressing or proceed for review.
Process of review remains confidential. We pay remuneration to reviewers & we do not charge anything from authors. We encourage on line submission to facilitate the process. Following review the article can be:
* Accepted as such.
* Accepted with minor changes.
* Accepted if major changes incorporated.
* Accepted if category of article is changed.
The purpose of having last category is to encourage writing habits but as most of the authors contribute articles as these are mandatory for their promotion, they insist on original article category only. We discourage this attitude as we believe that a beautifully conducted randomized control trial and a simple retrospective record of patients can not be at par and distinction must be made otherwise RCTs will never be found in Pakistani journals. We offer different categories like clinical practice articles, current practices, short articles, debate, issues etc. So in a way JCPSP has a very author friendly approach.
Response of Reviewers
The response of reviewers is very disappointing. It can be briefly described as:
* Mostly casual.
* Not bothered to give timely response.
* Reminders are needed often time.
* Even when responses are facilitated by attached prforma no regard is shown to it.
* Many fail to give a consolidated view which matters most, to us.
* Language at times is rude and attitude appears hostile.
* Conflict of interest is not revealed.
* The review lacks critical analysis.
* Most are not aware of statistical tests and thus fail to analyze this part of manuscript.
* Most have no access to references quoted by authors so can not give any comment upon truthfulness of issues compared with various studies.
* Most do not even know what guidelines are to be followed during the process of review as adopted by specific journal (as they vary from journal to journal). We can classify bias reviews as Discouraging, Dishonest and Inadequate.
This was the response of reviewer when article was sent to them after initial in house assessment We recently conducted an experiment where articles were sent to reviewers without initially being assessed by assistant editor (who assessed the article independently – not subject specialist). To our surprise reviewers failed to identify even gross inadequacies. So we put a question: Is peer review process at all needed considering its failure on most of the occasions even when reviewers were carefully selected from a large pool?
This concern has been raised on many forums. So one really asks what do readers get out of review? It is natural to assume a filtered article which is accurate and brings clarity to practically done scientific work. But still many articles get published in high esteemed peer review journals that later turned out to be fraudulent.
So it is right to ask “Should Every Article Submitted Get Published?” It is a controversial statement. While BioMed Central an electronic journal operates as “Bias to Publish” accepting anything that meets minimal scientific and ethical standards while rejection rate at Lancet and BMJ is well over 90%.
Another issue that surfaced is that an open review improves the status quo. Open review was introduced in 1996 by Journal of Interactive Media of Education. Online journal Philicia adopted a new approach. Articles submitted are published immediately and review takes place later, while Naboj invites readers to write review.
We are therefore still unclear what peer review actually does and can its omission bring any change. Assessing the quality of peer review thus becomes a tricky job. It may be clarified here that Editor is under no formal obligation to accept the opinion of the referee.
Our (Pakistani Editors) position in this regard at present is not very clear. We are still in process of evolution as medical journalism is Pakistan in not recognized as a distinct entity. Editors of various journals can hold such meetings to define the milestones that have been achieved and what still to be done in this regard.
* Both authors and reviewers need to show maturity in their approaches.
* More dedication is needed on their part to improve the job that they are carrying out.
* More workshops are needed to update the knowledge of authors and reviewers.
* Former Associate Editor
Journal of College of Physicians & Surgeons Pakistan
Editor, Journal of Surgery Pakistan
Department of Pediatric Surgery
National Institute Of Child Health Karachi.