Nature Blog Network

Wednesday, August 8, 2007

How to Retard Scientific Progress

I found some a great quote and analogy from an essay in the most current Current Biology by Peter Lawrence titled The mismeasurement of science. This essay takes a look at how science is measured and examines the use of impact factors and other metrics used to measure scientific progress for individuals, departments and institutions.

The quote is actually from Leo Szilard, the famous Manhattan project physicist. When asked by a wealthy entrepreneur who believes that science has progressed too quickly, what should be done to retard this progress, Szilard replied:

"You could set up a foundation with an annual endowment of thirty million dollars. Research workers in need of funds could apply for grants, if they could make a convincing case. Have ten committees, each composed of twelve scientists, appointed to pass on these applications. Take the most active scientists out of the laboratory and make them members of these committees. …First of all, the best scientists would be removed from their laboratories and kept busy on committees passing on applications for funds. Secondly the scientific workers in need of funds would concentrate on problems which were considered promising and were pretty certain to lead to publishable results. …By going after the obvious, pretty soon science would dry out. Science would become something like a parlor game. …There would be fashions. Those who followed the fashions would get grants. Those who wouldn't would not."

The analogy is Lawrence's own and relates to song writers being assessed in the same way as scientists, an analogy I can understand having came to science from the music industry.
"It is fun to imagine song writers being assessed in the way that scientists are today. Bureaucrats employed by DAFTA (Ditty, Aria, Fugue and Toccata Assessment) would count the number of songs produced and rank them by which radio stations they were played on during the first two weeks after release. The song writers would soon find that producing junky Christmas tunes and cosying up to DJs from top radio stations advanced their careers more than composing proper music. It is not so funny that, in the real world of science, dodgy evaluation criteria such as impact factors and citations are dominating minds, distorting behaviour and determining careers."

A Scientific "Audit Society"

Lawrence suggests that our scientific "audit society" has put meeting the demands of the holy impact factor above understanding nature and disease. He predicts that citation-fishing and citation-bartering will increase. Citation-fishing is putting your name on authors lists when you basically did nothing, like provide a reagent. I am not entirely sure what citation-bartering means, but I suspect it has to do with journals "encouraging" submitters to cite more articles out that journal to facilitate acceptance of the paper.

One problem I have thought about using standardized metrics to evaluate scientific progress is the aggregation of high citation potential research in only a handful o the "top" journals. Instead of a more even spread of the literature in journals appropriate for certain paper, the trendy research of the day is published in high profile journal, while the more topical journals are left in the dirt. This prevents lower "impact" journals from escaping a certain impact factor range, making them less appealing to new researchers whose papers fit more in the journals interests and could have been potential read by more people in their field. Getting your paper read by the right people is the real impact in my opinion. I recently submitted a species description to a journal in which several descriptions of species in the in family have been published, in hopes it would reach the broadest audience. My next paper I hope to be open access though.

"Impact" and Taxonomy

Standardize metrics, namely the impact factor, have a tremendously negative impact on taxonomy and taxonomists. This has in part to do with the behavior of most scientists with regards to the field. Taxonomic works are virtually never cited in references of papers. In ecology this is the most pervasive because ecologists often use detailed keys and species descriptions in their work to confirm the identifications. The species is the fundamental unit of biology, especially in ecology. Yet you never see statements in the materials and methods section of such papers that "polychaetes were identified to genus using Fauchald 1977". Meigen 1830, which described Drosophila melanogaster, should be the most cited publication that ever existed, based on the amount research published using that species.

Ecologists often ID specimens on hearsay. By this, I mean someone told them what this species was so therefore that is what it will be called. This is how I started out. Senior grad students had done most of the identifying work for me, I just needed remember the species list or compare what I find with what was on the shelfs. The problem with this was either the preliminary IDs were sometimes wrong or or closer examination resulted in one species being 2 or more. The former has resulted in the shrimp description now in review and the latter resulted in an several anemone descriptions still in progress. I can't underscore the importance of checking the facts for yourself!

The profile of taxonomy will greatly benefit if people include citations to the works they used, whether individual species descriptions, large monographs, revisions or identification keys. Taxonomic works are consistently the only publications that get more used and more systematically ignored by the "big science" community, including medicinhttp://www.blogger.com/img/gl.link.gife, molecular biology & biochemistry, and evolutionary biology in addition to ecology. This attitude towards taxonomy, and the managerial approach to modern science practices, has devaluated the stature of the taxonomist to providing a service, essentially for free, for the greater community without due recognition. This is exploitation. While jobs become scarce, funding even scarcer, demand increases because of the top heavy age structure of taxonomists. This is especially true in the U.S. where most biology majors will never see a systematics class in their undergraduate handbook.

I would also like to highlight a blog, called DC's Goodscience run by David Colquhoun at University College London, that I recently discovered that
"...exposes the idiocy of ‘metrics’ for the assessment of research quality by means of actual real-life examples. The aim of the blog is to provide a forum for discussion of the effects of the cult of managerialism on universities in general, and on science in particular."

Update: Christopher Taylor at the Catalogue of Organisms gives a perspective on the importance of taxonomy using an example from Harvestman (Opiliones, an arachnid).

8 comments:

  1. The only thing I can't work out is if the misspelling of 'scientific' in the title is accident or irony :-).

    ReplyDelete
  2. LOL thanks, consider it both. its accidentally ironic... I fixed it though

    ReplyDelete
  3. Excellent, I missed this one when you originally posted it... the quote is actually from one of Szilard's sci-fi short stories (multi-talented man he was...) called The Voice of the Dolphins: And Other Stories. The particular story is titled The Mark Gable Foundation and I highly recommend it.

    Gave my copy of the book to the head of the department after she lamented what she perceived to be a trend towards awards of high value research grants to whatever was currently "in vogue".

    ReplyDelete
  4. I have a question about where the funding for science should come from. Obviously it seems that funding should come from the private sector, but what worries me is the large number of cases where scientists are pressured from donors to do certain research and reach certain results. This happens in the drug industry all the time. It's nice to think that philanthropists would just donate money to universities and allow them to do whatever research they desire, but can we really count on them getting enough money? (Sorry to pull up such an old post, by the way, I'm a newbie libertarian and I'm trying to figure this all out.)

    ReplyDelete
  5. Kelly,not a problem (old post) glad to open the discussion again. Take everything I say with a grain of salt, I'm not a researcher, but a student who works for/with researching professors.

    No, we can't count on them getting enough money from philanthropists, etc. But...

    My personal opinion is that we should have a good mix of funding sources. Federal and State Governments, philanthropy organizations and private sources including the commercial sector. Yes there is pressure to come up with results, but that is there no matter where the funding comes from. Even philanthropic fellowships seem to have a certain pressure. Different funding sources bring different issues that scientists must consider for ethical issues, but each is valuable.

    My main concern is what I perceive as a sharp decline in funding for basic research, especially as government funding is cut.

    ReplyDelete
  6. Kelly, blogs are meant to be archived discussions that we can keep coming back to at anytime! So its no worry. The case is never closed on science progress and funding.

    It is my view, much like my coblogger Eric, that science funding come from as many variety of sources as possible. The more the merrier. But you bring up a good point about funding from private companies. It has been shown in the past that certain companies, in particular drug development companies as you mention, pressure scientists to find the "right" results. I don't know how this pressure really means anything though. The peer review system, as imperfect as it is, is set up to catch dishonesties and misleading/wrong interpretations. It does a fair job for the most part. Even after a publication, eventually a result may be refuted by future research from another lab. My point is that given time wrong interpretations and dishonest publications will be discovered one way or another. I would love to know what this rate is for Biology, but I do remember seeing something about this in the last 2 years.

    It is also known, as we observed in the Bush Jr. administration, that government can suppress findings or put pressure on government funded researchers as well. NGOs and conservation groups might only report results that favor their agendas as well. You could take the cynics view that everything should be taken with a grain of salt, but that isn't very productive.

    As a researcher, I feel the government should fund a majority of basic research because most, if not all, basic research benefits the general public. Government can also incentivize researchers for promoting educational outreach, hiring unrepresented minorities/disabled people, purchasing american made tools and products, etc etc. But corporations have a role in providing funding for research that leads to commercial applications. IBM, Johnson & Johnson, John Deere are all examples or large corporations that have heavily invested in researchers both within the company and in academia. Non-profits also have an important role in funding research often with little tangible benefit to the general public and little commercial application. For example, protecting watersheds and coastlines is extremely important and have functional benefits to society at large (water filtration and erosion prevention).

    On your last point about getting enough money. That is why a constant source of funding from government for basic research is vital. The scientific community has a pool from the compete for funds that is relatively stable. Corporate funding, and especially non-profits funding can be highly volatile at times. So some years will be better than other but at least there is a baseline.

    ReplyDelete
  7. Cool, thank you for the feedback! Hopefully Obama will be friendlier to science than Bush was, and people start getting the funding they need.

    ReplyDelete

Note: Only a member of this blog may post a comment.