How does research evaluation impact educational research? Exploring intended and unintended consequences of research assessment in the United Kingdom, 1986–2014
research evaluation; research assessment; Research Excellence Framework; UK; educational research; higher education
Abstract :
[en] Research evaluation systems in many countries aim to improve the quality of higher education. Among the first such systems, the UK's Research Assessment Exercise (RAE) from 1986 is now the Research Excellence Framework (REF). Highly-institutionalized, it holds research(ers) accountable. While studies describe the effects at different levels, this longitudinal analysis examines the gradual institutionalization and (un)intended consequences from 1986 to 2014. First, we analyze historically RAE/REF's rational, formalization, standardization, and transparency, framing it as a strong research evaluation system. Second, we locate the multidisciplinary field of education, analyzing submission behavior (staff, outputs, funding) of Departments of Education over time. We find: decreases in submitted staff; the research article as preferred publication format; the rise of quantitative analysis; and high and stable concentration of funding among few Departments. Policy instruments invoke varied responses, wit such reactivity shown by the increasing selectivity of submitted staff as a form of reverse engineering and the research article as the preferred output as a self-fulfilling prophecy. The funding concentration manifests an intended consequence, facilitating greater disparities between Departments of Education. These findings emphasize how research assessment impacts the structural organization and cognitive development of educational research in the UK.
Research center :
- Education, Culture, Cognition & Society (ECCS) > Institute of Education & Society (InES)
Disciplines :
Education & instruction Sociology & social sciences
Author, co-author :
MARQUES, Marcelo ; University of Luxembourg > Faculty of Language and Literature, Humanities, Arts and Education (FLSHASE) > Education, Culture, Cognition and Society (ECCS)
POWELL, Justin J W ; University of Luxembourg > Faculty of Language and Literature, Humanities, Arts and Education (FLSHASE) > Education, Culture, Cognition and Society (ECCS)
ZAPP, Mike ; University of Luxembourg > Faculty of Language and Literature, Humanities, Arts and Education (FLSHASE) > Education, Culture, Cognition and Society (ECCS)
Biesta, Gert; Brunel University London
External co-authors :
yes
Language :
English
Title :
How does research evaluation impact educational research? Exploring intended and unintended consequences of research assessment in the United Kingdom, 1986–2014
Aagaard K, Bloch C, Schneider J, (2015) Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator. Research Evaluation 24(2): 106–117.
Alldred P, Miller T, (2007) Measuring what’s valued or valuing what’s measured? Knowledge production and the Research Assessment Exercise. In: Gillies V., Lucey H., (eds) Power, Knowledge and the Academy. Basingstoke, UK: Palgrave, pp.147–67.
Barker K, (2007) The UK Research Assessment Exercise: The evolution of a national research evaluation system. Research Evaluation 16(1): 3–12.
Bassey M, (2002) RAE 2001: Analysis of the education results. Research Intelligence 78(February): 8–14.
Bassey M, Constable H, (1997) Higher education research in education, 1992–1996: Fields of enquiry reported the HEFCs RAE. Research Intelligence 61(July): 6–8.
Besley T, Peters M, (2009) Neoliberalism, performance and the assessment of educational research quality: Comparing United Kingdom, Australia and New Zealand. In: Besley T, (ed.) Assessing the Quality of Educational Research in Higher Education. International perspectives. Rotterdam, The Netherlands: Sense, pp.27–48.
Biesta G, (2004) Education, accountability and the ethical demand: Can the democratic potential of accountability be regained? Educational Theory 54(3): 233–250.
Biesta G, (2007) Why ‘what works’ won’t work: Evidence-based practice and the democratic deficit in educational research. Educational Theory 57(1): 1–22.
Biesta G, (2010) Good Education in an Age of Measurement: Ethics, politics, democracy. Boulder, CO: Paradigm Publishers.
Biesta G, (2011) Disciplines and theory in the academic study of education: A comparative analysis of the Anglo-American and continental construction of the field. Pedagogy, Culture and Society 19(2): 175–192.
Biesta G, (2016) Improving education through research? From effectiveness, causality and technology, to purpose, complexity and culture. Policy Futures in Education 14(2): 194–210.
Bleiklie I, Enders J, Lepori B, Musselin C, (2011) New public management, network governance and the university as a changing professional organization. In: Christensen T, Laegried P., (eds) The Ashgate Research Companion to New Public Management. Farnham, UK: Ashgate, pp.161–176.
British Educational Research Association (1997) Results of the Research Assessment Exercise. Research Intelligence 59(February): 4–7.
Demeritt D, (2000) The new social contract for science: Accountability, relevance and value in US and UK science and research policy. Antipode 32(3): 308–329.
Department for Education and Skills (2003) The Future of Higher Education. London: HMSO.
Elton L, (2000) The UK Research Assessment Exercise: Unintended Consequences. Higher Education Quarterly 54(3): 274–283.
Engels TCE, Ossenblok TLB, Spruyt EHJ, (2012) Changing publication patterns in the social sciences and humanities, 2000–2009. Scientometrics 93(2): 373–390.
Espeland W, (2016) Reverse engineering and emotional attachments as mechanisms mediating the effects of quantification. Historical Social Research 41(2): 280–304.
Espeland W, Sauder M, (2007) Rankings and reactivity: How public measures recreate social worlds. American Journal of Sociology 113(1): 1–40.
Espeland W, Sauder M, (2016) Engines of Anxiety: Academic rankings, reputation, and accountability. New York: Russell Sage Foundation.
Espeland W, Stevens M, (1998) Commensuration as a social process. Annual Review of Sociology 24: 313–343.
European Commission (2010) Assessing Europe’s University-Based Research. Brussels, Belgium: European Commission.
Fisher M, Marsh P, (2003) Social work research and the 2001 Research Assessment Exercise. Social Work Education 22(1): 71–80.
Furlong J, (2013a) Research assessment and the shaping of educational research in the UK. The Australian Educational Researcher 40(4): 513–520.
Furlong J, (2013b) Education. An anatomy of the discipline. Abingdon, UK: Routledge.
Gay B, (1981) Accountability in education: A review of the literature and research in the United Kingdom. Westminster Studies in Education 4(1): 29–43.
Geuna A, Martin B, (2003) University research evaluation and funding. Minerva 41(4): 277–304.
Gilroy P, McNamara O, (2009) A critical history of research assessment in the United Kingdom and its post-1992 impact on education. Journal of Education for Teaching 35(4): 321–335.
Gläser J, (2007) The social order of research evaluation systems. In: Whitley R, Gläser J, (eds.) The Changing Governance of the Sciences. Dordrecht, The Netherlands: Springer, pp.245–264.
Gorur R, (2014) Towards a sociology of measurement in education policy. European Educational Research Journal 13(1): 58–72.
Hamann J, (2016) The visible hand of research performance assessment. Higher Education 72(6): 761–779.
Harley S, (2002) The impact of research selectivity on academic work and identity in UK universities. Studies in Higher Education 27(2): 187–203.
Harley S, Lee FS, (1997) Research selectivity, managerialism, and the academic labour process. Human Relations 50(11): 1427–1460.
Hicks D, (2012) Performance-based university research funding systems. Research Policy 41(2): 251–261.
Hillage J, Pearson R, Anderson A, Tamkin P, (1998) Excellence in Research on Schools. London: Department for Education and Employment.
Holligan C, Wilson M, Humes W, (2011) Research cultures in English and Scottish university education departments. British Educational Research Journal 37(4): 713–734.
Jimenez-Contreras E, Anegon F, Lopez-Cozar E, (2003) The evolution of research activity in Spain: The impact of the National Commission for the Evaluation of Research Activity (CNEAI). Research Policy 32(1): 123–142.
Jones PK, Sizer J, (1990) The Universities Council’s 1989 Research Selectivity Exercise. Beiträge zur Hochschulforschung, 4: 309–348.
Karlsson S, (2017) Evaluation as a travelling idea: Assessing the consequences of Research Assessment Exercises. Research Evaluation 26(2): 55–65.
Kerr D, Lines A, MacDonald A, Andrews L, (1998) Mapping education research in England: An analysis of the 1996 RAE. Bristol, UK: Higher Education Funding Council for England.
Knights D, Richards W, (2003) Sex discrimination in UK academia. Gender, Work and Organization 10(2): 213–238.
Langfeldt L, (2001) The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 31(6): 820–841.
Lawn M, Furlong J, (2007) The social organization of educational research in England. European Educational Research Journal 6(1): 55–70.
Lawn M, Furlong J, (2009) The disciplines of education in the UK. Oxford Review of Education 35(5): 541–552.
Leathwood C, Read B, (2012) Final report: Assessing the impact of developments in research policy for research on higher education. London: Society for Research on Higher Education.
Lucas L, (2006) The Research Game in Academic Life. Maidenhead, UK: Open University Press.
Martin BR, (2011) The Research Excellence Framework and the ‘impact agenda’: Are we creating a Frankenstein monster? Research Evaluation 20(3): 247–254.
Martin BR, Whitley R, (2010) The UK Research Assessment Exercise: a case of regulatory capture? In: Whitley R, Gläser J, Engwall L, (eds) Reconfiguring Knowledge Production. Oxford: Oxford University Press, pp.51–80.
Merton RK, (1968) The Matthew effect in science. Science 159(3810): 56–63.
McNay I, (1997) The impact of the 1992 RAE on institutional and individual behaviour in English higher education. Bristol, UK: Higher Education Funding Council for England.
McNay I, (2003) Assessing the assessment: An analysis of the UK Research Assessment Exercise, 2001, and its outcomes, with special reference to research in education. Science and Public Policy 30(1): 47–54.
Mills D, Jepson A, Coxon T, Easterby-Smith MPV, Hawkins P, Spencer J, (2006) Demographic Review of the UK Social Sciences. UK: Economic and Social Research Council.
Moed HF, (2008) UK Research Assessment Exercises. Scientometrics 74(1): 153–161.
Morgan KJ, (2004) The Research Assessment Exercise in English universities, 2001. Higher Education 48(4): 461–482.
Murphy T, Sage D, (2014) Perceptions of the UK’s Research Excellence Framework 2014. Journal of Higher Education Policy and Management 36(6): 603–615.
Oancea A, (2004) The distribution of education research expertise: Findings from the analysis of RAE 2001 submissions (I, II). Research Intelligence 87(May)/88(August): 3–8 / 3–9.
Oancea A, (2005) Criticisms of educational research: Key topics and level of analysis. British Educational Research Journal 31(2): 157–183.
Oancea A, (2008) Performative accountability and the UK Research Assessment Exercise. ACCESS: Critical Perspectives on Communication, Cultural & Policy Studies 27(1/2):153–173.
Oancea A, (2010) The BERA/UCET Review of the Impacts of RAE 2008 on Education Research in UK Higher Education Institutions. Macclesfield, UK: Universities’ Council for the Education of Teachers / British Educational Research Association.
Oancea A, (2014) Research assessment as governance technology in the United Kingdom. Zeitschrift für Erziehungswissenschaft 17(6): 83–110.
Oancea A, Mills D, (2014) Educational Research. Final report of the British Educational Research Association Observatory 2014. London: British Educational Research Association.
OECD (2010) Performance-based Funding for Public Research in Tertiary Education Institutions. Paris: OECD Publishing.
Pajić D, (2015) Globalization of the social sciences in Eastern Europe: Genuine breakthrough or a slippery slope of the research evaluation practice? Scientometrics 102(3): 2131–2150.
Power M, (1997) The audit society: Rituals of verification. Oxford: Oxford University Press.
Powell JJW, Baker DP, Fernandez F, (eds) (2017) The Century of Science: The global triumph of the research university. International Perspectives on Education & Society, vol.33. Bingley, UK: Emerald Publishing.
Powell JJW, Dusdal J, (2017) The European center of science productivity: Research universities and institutes in France, Germany, and the United Kingdom. In: Powell JJW, Baker DP, Fernandez F, (eds). The Century of Science: The global triumph of the research university. International Perspectives on Education & Society, vol.33. Bingley, UK: Emerald Publishing, pp. 55–84.
Sayer D, (2014) Rank Hypocrisies. London: SAGE.
Sousa SB, Brennan J, (2014) The UK Research Excellence Framework and the transformation of research production. In: Musselin C, Teixeira P, (eds) Reforming Higher Education: Public policy design and implementation. Dordrecht, The Netherlands: Springer, pp.65–80.
RAE (1992) Research Assessment Exercise 1992: The outcome. Circular 26/92. Available at: http://www.rae.ac.uk/1992/c26_92.html#annexa
RAE (1994) Research Assessment Exercise: Units of assessment. RAE96 3/94. Available at: http://www.rae.ac.uk/1996/c3_94.html
RAE (2001a) Research Assessment Exercise: The outcome. RAE 4/01. Available at: http://www.rae.ac.uk/2001/Pubs/4_01/section1.asp
RAE (2001b) Research Assessment Exercise. Overview reports from the panels. Education. Available at: http://www.rae.ac.uk/2001/overview/docs/UoA68.pdf
RAE (2005) Research Assessment Exercise: Guidance to panels. RAE 01/2005. Available at: http://www.rae.ac.uk/pubs/2005/01/rae0105.pdf
RAE (2009a) Research Assessment Exercise: Manager’s report. Available at: http://www.rae.ac.uk/pubs/2009/manager/manager.pdf
RAE (2009b) Research Assessment Exercise. Overview report. Sub-Panel 45 education. Available at: http://www.rae.ac.uk/pubs/2009/ov/
REF (2011) Assessment framework and guidance for submissions. REF 02.2011. Available at: http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf
REF (2015) Research Excellence Framework 2014: Overview report by Main Panel C and sub-panels 16–26. Available at: http://www.ref.ac.uk/media/ref/content/expanel/member/Main%20Panel%20C%20overview%20report.pdf
REF (2016) Building on Success and Learning from Experience: An independent review of the Research Excellence Framework. London: Department for Business, Energy and Industrial Strategy.
Roberts G, (2003) Review of research assessment. Available at: http://www.ra-review.ac.uk/reports/roberts/roberts_summary.pdf
Roberts P, (2006) Performativity, measurement and research. In: Ozga J, Seddon T, Popkewitz T, (eds) Education Research and Policy. London: Routledge, pp.185–199.
Talib A, (2001) The continuing behavioural modification of academics since the 1992 Research Assessment Exercise. Higher Education Review 33(3): 30–46.
Tooley J, Darby D, (1998) Educational Research: A critique. A survey of published educational research. London: Office for Standards in Education.
University Funding Council (1989) Report on the 1989 RAE. London: University Funding Council.
Viseu S, (2016) Play the game or get played? Researchers’ strategies around R&D policies. In Trimmer K., (ed.). Political Pressures on Educational and Social Research – International Perspectives. Abingdon, UK: Routledge, pp.55–56.
Watermeyer R, (2014) Issues in the articulation of ‘impact’: The responses of UK academics to ‘impact’ as a new measure of research assessment. Studies in Higher Education 39(2): 359–377.
Whitley R, (2007) Changing governance of the public sciences. In: Whitley R, Gläser J, (eds) The Changing Governance of the Sciences. Dordrecht, The Netherlands: Springer, pp.3–26.
Whitley R, Gläser J, (2007) The Changing Governance of the Sciences. Dordrecht, The Netherlands: Springer.
Whitty G, Donoghue M, Christie D, Kirk G, Menter I, McNamara M, Moss G, Oancea A, Rogers C, Thomson P, (2012) Prospects for Education Research in Education Departments in Higher Education Institutions in the UK. London: Universities’ Council for the Education of Teachers / British Educational Research Association.
Zapp M, (in press) Beyond large-scale achievement testing – The psychological turn in international organizations’ work on educational assessment. In: Wiseman A., (ed.) Cross-nationally Comparative, Evidence-based Educational Policymaking and Reform. Bingley, UK: Emerald Publishing.
Zapp M, Dahmen C, (2017) The diffusion of educational ideas. An event history analysis of lifelong learning. Comparative Education Review 61(1): 492-518.
