Assessment of impacts and outputs should take account of how each discipline works, conference told
Australia’s research assessment system should be reformed to recognise a wider range of research outputs, a national conference on open-access publishing has been told.
The Open Access Australasia conference session on 26 October heard that a one-size-fits-all assessment system, weighted heavily towards citations, no longer reflected actual practices.
Western Sydney University associate professor Robert Mailhammer said that for the humanities, some measures of “impact” were inadequate.
There needs to be a “close relationship between disciplinary practices and forms of assessment”, he said, adding that research metrics often did not fit with the sort of “non-traditional” outputs allowed in the humanities.
“Good assessment is going to reflect the discipline’s values…Impact is not the same as quality.”
Non-traditional outputs, such as creative works, are most commonly used in research in the creative arts and writing, and in studies around the built environment, according to Australian Research Council data.
Kimberlee Weatherall, a professor of law at the University of Sydney, said her research into the way impact was assessed in law showed it did not fit the way the discipline worked. “When we looked into it, we did find that the law discipline’s engagement with metrics was a disaster,” she said, with citations failing to reflect the standing of work, and major biases towards work done by men.
She said she would like to see metrics that asked what researchers had done with their discovery and how it had contributed to the work of others. Not all good researchers simply write books, she said, with some choosing to work more directly with the community. “There are different ways of being a successful academic and a good academic and doing good work.”
‘The targets become the reason’
Emma Burrows, a researcher at the Florey Institute of Neuroscience and Mental Health who has investigated health research funding in Australia, said that in creating new assessment methods, it was important to train assessors and guard against overly onerous processes. “There is a natural instinct when you are time-poor to want to rank quantitatively or numerically,” she said.
PhD student and session moderator Dion Detterer said that the current system “really saps the energy that researchers have. What they should be doing is research. There’s an inordinate amount of time spent doing these administrative tasks…The targets become the reason and that can really skew the research.”
Weatherall said the aim should be to set up systems that embedded the right metrics. Poor measurements could take a long time to move past, she said, citing an example of “when the Australian Research Council decided to just create a ranked list of law journals in 2010”.
“It’s been very, very difficult to get rid of since. Ten years down the track when it’s been abandoned by the ARC and anyone with any sense, you still see references to it.”
Open-access deals
The calls for reform came as open-access publishing in Australia received a major boost, with three deals announced within a month.
Oxford University Press is to give the staff and student members of the Council of Australian University Librarians access to 366 peer-reviewed publications. Caul members will be able to publish in the journals on an open-access basis. That deal came on 25 October, five days after the announcement of a similar deal with Springer Nature. Cambridge University Press and Caul will also begin an open-access publishing agreement in 2022 in a deal announced on 14 October, which will see publication costs waived.