Measuring culture

• The gathering of evidence about the impact of the arts has assumed centre-stage in cultural policy. The rise of statistics has paralleled an extension of government control over the arts, and the tendency to value culture for its ‘impact’ rather than its intrinsic value.

The first attempt to collect and analyse statistics about the arts in the UK was made 20 years ago by the Policy Studies Institute (1). Facts about the Arts sought to quantify what was spent on the arts; how many people they employed; what they earned; the number of events provided; and who went to see what.

This exercise inevitably attracted the scorn of those who believed that the concept of the arts is, by definition, ‘elusive and indefinable and that any attempt to measure it cannot begin to represent its essential quality’ (2).

In one sense, at least, not much has changed. Attempts to collect and analyse data on the cultural sector are still derided by some as being ‘no way to measure the true value of the arts’ (3).

But, in another sense, everything has changed. Driven by the notion that the arts constitute ‘a civilizing influence’ (4) and that they contribute to the quality of life as a matter of course, the subsidised cultural sector has been pressed into delivering on a number of government objectives – combating social exclusion, in particular. This is a far cry from the previous perception of culture as ‘a self-contained realm which exists in opposition to the material and the economic’ (5).

The aspirations ostensibly held by the Department for Culture, Media and Sport (DCMS) for data collection are far more sophisticated than the Policy Studies Institute’s initial pragmatism allowed for. In its attempt to make cultural provision accessible to ‘the many, not just the few’ and to cast cultural institutions in the role of ‘centres for social change’ (6), the department hopes to monitor the value for money and efficiency of those organisations to which it grants funds; measure the extent to which its ambitions are achieved; and, evaluate the precise impact that they are having. Consequently, data collection has become central – theoretically pivotal, even – to the DCMS’s operations.

This has resulted in the politicisation of the gathering of data and the state’s blurring of the relationship between advocacy and research. Despite manifest improvements, data collection in the UK cultural sector is currently a long way off being a disinterested, objective or even well-regarded pursuit and – whatever the rhetoric – the implementation of evidence-based policy is a long way off.

Over the past 20 years, data collection has moved to the forefront of cultural policymaking. There is now a mass of primary data claiming to measure the impact of cultural policy.

So, what has been driving this rise and rise of statistics – and how valuable has the pursuit of data collection actually been? This article looks at the experience of museums, galleries and the arts in England.

• The Department of National Heritage

The current government’s preoccupation with data collection was foreshadowed by successive Conservative governments’ demands for accountability and strategic management within the public sector. The Department for Culture’s predecessor, the Department of National Heritage (DNH), sought to broaden the cultural sector’s funding base, which drove the sector to justify its subsidies in economic terms and identify itself as a wealth creator.

The DNH was also concerned to increase ‘access’; and – in the months immediately preceding the 1997 election – to encourage participation and community development. These concerns, and the initiatives related to them, both depended upon and encouraged the development of data collection.

The DNH was established after the Conservatives’ 1992 election victory. Unlike the ministries that preceded it, the DNH was accorded status as a Department of State with representation at Cabinet. The implications of this marked the growth of political interest in the cultural sector, and were most keenly and emotively felt in the department’s relationship with the arts.

While the Secretary of State for the National Heritage was still publicly committed to the so-called ‘arm’s length principle’ in the early 1990s, by 1996 the proceedings of a parliamentary committee revealed that the principle had actually come to be regarded as ‘stunt[ing] the growth of the Department’ (7), and that as a Cabinet Minister, the Secretary of State expected to be able to have a view about how money was spent and what decisions should be taken about the future direction of culture in the country (8).

• Accountability and strategic management

Following the Conservatives’ Financial Management Initiative in 1982, attempts to introduce greater accountability and strategic management into the arts and the rest of the cultural sector in the form of the so-called three ‘Es’ (economy, efficiency and effectiveness) were inevitable. Much of this was mediated by the DNH.

The DNH’s operations were largely determined by its need to ‘act as a catalyst for effective action’ (9). Unlike most other government departments, as much as 95 percent of the monies voted to it were spent through its range of diverse, executive non-departmental public bodies on which it depended for the achievement of its aims and objectives. These included the then Arts Council of Great Britain, the national museums and galleries, and so on.

Consequently, when the department came to describe itself as ‘in part-policy orientated and strategic and in part supervisory’ (10), it followed that a fundamental principle of its relationship with its sponsored bodies would be their ‘accountability and value for money’ (11).

Given that the DNH had itself been encouraged to make effective use of the corporate planning process, similar demands were passed on to its sponsored bodies. The department oversaw a number of initiatives intended to improve the organisation and management of its sponsored organisations’ services to the public, including the introduction of corporate plans, support for the development of staff training, and the devising of a range of performance indicators which touched on measures of access and use, income generation and financial management.

These were intended to provide objective benchmarks against which organisations’ progress in achieving the aims and objectives identified in their corporate plans and elsewhere could be measured; to identify areas of relative strength and weakness; to inform decision-making on the allocation of resources; and to motivate staff. In 1996, the department introduced funding agreements as a way of clarifying what expectations could be made of sponsored bodies and to ‘stand as a concise and public statement of what [Parliament] and the taxpayer should expect for the grant in aid’ (12).

The DNH regarded it as vital ‘that policies are formulated in the light of the best available and most up to date information about the sectors in which they are to operate’. It consequently commissioned reviews of existing data sources as a way of clearing the ground for its own initiatives and to review its need for statistical information. At about the same time, its first-tier bodies – the then Arts Council and Museums & Galleries Commission – set up collections of time series data, intended to collect statistics from their respective domains and inform their strategic development.

• The economic agenda

Another imperative which encouraged the development of data collection stemmed from the DNH’s desire for the sector to break its largely exclusive dependence on public subsidies, by generating its own income and attracting and levering in private sector funds. Pressures to comply with an economic agenda – albeit a different one – also came from the Left.

In the brief period between 1985 and its abolition in 1986, the Greater London Council proposed funding ‘interventions’ in the ‘cultural industries’ in order to increase people’s choice of cultural provision and create new jobs and training opportunities, particularly among those previously excluded. This was a political and pragmatic response to the then massive rise in unemployment and ‘free time’ (13).

The debates and policies that ensued ultimately combined to encourage the sector to justify its value in economic terms.

So, during the second half of the 1980s, as the ‘new financial reality’ began to bite, the Arts Council actively sought to increase its grant in aid on the basis of making an economic case for increased public ‘investment’ in the arts. It introduced the notion of the arts as an ‘industry’, and listed the returns that the government could expect for ‘a small increase in public funding’, including the low cost of creating new jobs, savings on unemployment and recycled tax (14).

The Arts Council also proposed that arts projects could help ‘bring our cities back to life’ by acting as a magnet, attracting people, tourism, businesses and jobs to an area, and as a catalyst for regeneration; by enhancing the visual quality of the built environment; providing a focal point for community pride and identity; and, helping to build self-confidence in individuals (15). The Cabinet Office itself endorsed the role of the arts in urban regeneration, on the basis of their potential to deal with the problems of unemployment and alienation in the inner cities, as well as contributing to the creation of a classless and tolerant society.

Within the space of the mid-1980s alone, arts advocacy had shifted from ‘special pleading’ for extra funding – predicated on a loose equation based on the relationship between the size of the sector, the amount of investment sought, and the potential dividends – to arguments about the arts’ role in ‘an era of industrial restructuring’ (16).

The notion that the arts and the cultural industries were grounded in economic reality was further boosted by John Myserscough’s research into the Economic Importance of the Arts in Britain, published in 1988. Based on a set of surveys, this presented the arts and the cultural industries as a major contributor to the economy – as stimulating tourism, contributing to urban renewal, generating substantial exports and as responsible for mass employment. These findings were regarded as extraordinarily persuasive by a number of local authorities, many of which subsequently invested in the arts and cultural industries as a way of encouraging employment and inward investment in cities suffering the effects of post-industrialisation.

• Access, community development and participation

From the early 1990s, the impetus to increase public ‘access’ to the arts and the promotion of community development and participation became closely associated with data collection.

As the decade progressed, ‘access’ became a highly emotive and strategic issue, subject to both top-down and bottom-up pressures. The findings of the Target Group Index, which the Arts Council commissioned from 1986, doubtless played some part. This revealed a high degree of consistency from year to year of the percentages of the UK population who said they attended arts activities.

Acknowledging that ‘the available statistics suggest that some areas are enjoyed by a relatively narrow section of the population’, DNH established a strategic Access Initiative intended to ensure ‘the opportunity for as many people as possible to benefit from the arts, heritage and sport’, not least because ‘the investment of taxpayers’ money brings with it a responsibility to ensure that those who pay have the opportunity to benefit’ (17).

But, in addition to finding out about the numbers benefiting, there was also concern about what informed people’s attitudes to cultural provision. This promoted the funding system’s collection of audience and non-audience data, which interrogated, for example, which publics were being served; whether and how they personally valued cultural provision; and whether it improved their quality of life. Academic publications of the period, in particular, considered issues pertaining to the opening up of access to museums; the quality of visitors’ experiences in museums; and the sociology of museums’ publics.

One way of forwarding access initiatives was to remove various barriers which were assumed to inhibit access to the arts. This prompted the introduction of ‘cultural pluralism’ and ‘diversity’ into the mainstream. By the early 1990s a number of constituencies were vying with each other for the Arts Council’s recognition of their needs (18). Despite the fact that the majority of the DNH and the Arts Council grant funding was dedicated to portfolios of historically supported organisations, the appropriation of alternative activities within the funding mainstream signalled the theoretical cessation of the privileging of one cultural activity over another.

One such alternative activity was community and participatory arts. In the months before the 1997 election, the DNH assumed responsibility for the promotion and development of volunteering throughout the country. It established several initiatives intended to encourage participation as a way of fostering ‘pride, satisfaction, and self-confidence’ and community development to help people to improve their quality of life (19).

But these precursors were taken to an entirely new level with the election of New Labour in 1997. New Labour attempted to bring culture closer into line with government policy, and required that the cultural sector justify its funding by demonstrating the social impact it was making, particularly in the area of social inclusion.

• The Department for Culture, Media and Sport

From its establishment following New Labour’s election victory, one of DCMS’s most striking characteristics – and one that distinguished it from its predecessor departments – has been its explicit concern to contribute to the government’s manifesto commitments. From the start, the department intended playing ‘a full part in “joined-up government”‘ as well as ‘arguing for recognition of the part the arts, sport, tourism etc can play in delivering government policies beyond this department’s direct interests’ (20).

The department’s first annual report described its aims as being ‘guided by four central themes: the promotion of access, for the many not just the few; the pursuit of excellence and innovation; the nurturing of educational opportunity and the fostering of the creative industries, which provide a growing proportion of Britain’s employment and wealth… aims which resonate across every other department of Government’ (21).

• Social inclusion

Within months of the election, the Cabinet Office had also established a Social Exclusion Unit to report to the prime minister on ways in which government departments could work together to reduce the incidence of social exclusion (22) to create a more equitable and inclusive society. The unit recommended the formation of a number of Policy Action Teams (PATs) to look at different areas of public policy and propose ways in which social exclusion might be tackled head on (23).

In response to this, DCMS reported back on how ‘art and sport can not only make a valuable contribution to delivering key outcomes of lower long-term unemployment, less crime, better health and better qualifications, but can also help to develop the individual pride, community spirit and capacity for responsibility that enable communities to run regeneration programmes themselves’ (24).

DCMS subsequently established its own social inclusion action plan, designed to ensure that social inclusion objectives were incorporated across the whole of its remit, including museums, galleries, archives and libraries – every one of which ‘can recognise its social responsibilities and should be considering and reaching informed decisions about how it can best meet the needs of its communities’ (25).

The cultural sector’s support of community projects has been further enhanced through a series of reforms which shifted the distribution of national lottery funding away from ‘grand plans towards smaller “micro” projects for local communities’ (26).

• Accountability

New Labour’s promises to deliver on its priorities, improve public services generally, and to use evidence in the formulation of policy are predicated on the collection and analysis of data. On coming to power, DCMS found that because expenditure decisions were largely taken outside the department, this had ‘resulted in little strategic direction from the centre and only limited knowledge of how public money has delivered government objectives’ (27).

DCMS’s reforming ‘approach to investment in culture’, as outlined in 1998, set out how it saw the relationship between the department and its sponsored bodies as functioning in the future insofar as it:

• defined a new ‘strategic role’ for the department, in the delivery of cultural policy and funding;




• set out the terms of a ‘new relationship’ between the department and the bodies that it funds to ensure the delivery of the appropriate outputs and benefits to the public to be articulated through the introduction of three-year funding agreements, ‘placing clear responsibilities on those bodies to deliver against demanding targets’;




• streamlined the way policies and programmes are delivered – not least, through the establishment of new strategic sectoral bodies; and




• sought to raise standards of efficiency and financial management across all its sectors reinforced by the establishment of a new watchdog, the Quality, Efficiency and Standards Team (QUEST), intended to work alongside the National Audit Office, the Audit Commission and sponsored bodies’ own auditors ‘to monitor and improve standards of efficiency and promote quality across our sectors’ (28).

This framework, which was ultimately driven by the advent of ‘resource accounting’ across government, meant ‘that DCMS ties its expenditure to its objectives’ and that it needs ‘to be assured that public money is being used appropriately to meet public objectives’. Moreover, ‘investing for reform’, as the rhetoric has it, means that the ‘more money invested, [the] more results are required’.

For DCMS, this implied closure on any possibility of ‘grants for grants’ sake’: ‘This is not something for nothing. We want to see measurable outcomes for the investment which is being made. From now on there will be real partnership with obligations and responsibilities.’ (29)

By way of example, the £290million boost for DCMS sectors from 1999-2002, announced in 1998, was predicated on the expectation of: ‘outcomes which reflect our four central themes…. They will be linked to the delivery of increased outputs, improved access and efficiency and increased private sector support, reflected in the funding agreements from 1999.’ (30)

• Public Service Agreements

DCMS’s obligation to account for its progress and demonstrate that it is ‘delivering improvements in public services’ is articulated in its Public Service Agreement. Public Service Agreements, which were introduced through the 1998 Comprehensive Spending Review, essentially motivate departments to deliver on the government’s agenda.

They are tied to biennial Spending Reviews, and outline each department’s aims, objectives and its specific and measurable targets for the following three years, thus enabling government to monitor whether they are indeed delivering ‘better public services’ and how effectively its resources are being used (31). They determine the nature of departments’ core activities, and inform their annual reporting.

DCMS’s targets are ultimately delivered by its non-departmental public bodies, as specified in their funding agreements, which were reformulated to set out what the department views as ‘explicit and challenging statement of the outputs and levels of performance expected of sponsored bodies over the funding period’ (32).

So, while there might be some question as to how challenging they might be, funding agreements constitute the tool through which DCMS gives direction; sets targets and chases progress; and ‘where appropriate will take direct action to make sure that its objectives are achieved’ (33).

While DCMS primarily depends on its sponsored bodies, it is also concerned with developing the efficacy of its policies and improving the delivery of public services via local authorities and regional bodies. It regards local authorities’ cultural services as ideally placed to ‘make a considerable contribution to the wider aims of both central and local government, whether tackling social exclusion; encouraging healthier lifestyles; providing opportunities for voluntary and community activity; sparking urban and rural regeneration; or stimulating a commitment to lifelong learning, as well as enriching people’s lives’ (34).

Local authorities are encouraged to produce Local Cultural Strategies, which take on board the aims and objectives of DCMS, other government departments and lottery developments. These documents are also expected to set out how their implementation will be reviewed – what performance indicators and means of quantitative and qualitative assessment will be used, and what targets set. Individual local authorities are also entering into Local Public Service Agreements, which set out the authority’s commitment to deliver specific improvements and the government’s commitment to reward these improvements.

• ‘The new orthodoxy’

Both the arts and the museums sector have recently been accused of apathy in the face of the ‘new orthodoxy’ – described in the introduction to the publication ‘Museums for “The People?”‘ as ‘the monotonous whine’ of ‘vacuous slogans of relevance, accessibility and inclusiveness’ (35). This is probably most clearly articulated in the litany of claims attributed to the arts by DCMS: ‘the arts are good for your health’; ‘the arts have a key role in making our society a better place to live’; ‘the arts…can help tackle crime’ (36).

But the essay by Josie Appleton in ‘Museums for “The People?”‘ perceives ‘people-centred museums’ as being at the core of this new orthodoxy. She write that museums’ missions around ‘empowerment, inclusiveness, diversity and customer satisfaction’ combine the ideologies of the economic right and the cultural left, and suggests that they threaten the very ‘existence of the museum as such…the collection, preservation and study of objects deemed to be of artistic, historic or scientific interest’ (37).

Questions have been asked about the appropriateness of the government’s recruitment of cultural organisations to combat social exclusion, and it is intimated that social policy issues could be delivered more cost-effectively by other institutions. Some have argued that the cultural sector is making itself susceptible to having to accommodate shifting perceptions of benefits which might be subject to revised political judgements.

Nevertheless, responses by the cultural sector to such accusations imply that parts of it, at least, are willing to participate in, if not actively inform, this new orthodoxy. A high degree of convergence, for instance, seems to have existed between the government and the Regional Museums Task Force, whose case for additional funding for major regional museums was made on the basis of developing the capacity to deliver on the government’s agenda (38).

Numerous other initiatives suggest that cultural sector organisations are not only amenable to the government’s concerns, but that they actively believe in, or are opportunistically prepared to advocate, their domain’s potential to deliver.

• Claiming to deliver

Ex-culture secretary Chris Smith’s assertion that the arts and sport deliver key outcomes in terms of combating social exclusion (39) echoed a number of similar claims, and inspired a cascade of similar assertions by arts bodies. The literature is replete with publications asserting the vital contribution of the arts. But what kind of impact is it that these activities claim to be delivering?

The basis of the model by which claims have been made for the sector was established in the mid-1990s in relation to small-scale, arts-based, neighbourhood community and participatory projects. These projects were often temporary, event- or process-driven, and had effectively been neglected by the emphasis on economically inspired, prestige, cultural regeneration. The promotion of social impact was initiated by the consultancy Comedia, which in 1993 began lobbying for support to prove the effectiveness of ‘investing in socially orientated arts initiatives’. In The Social Impact of the Arts, Charles Landry et al argued that:

‘All kinds of arts projects can have a social impact be that a neighbourhood project to increase commitment and involvement with a local community; a project with disabled people to increase skills and life choices; and arts in hospital project to increase the quality of that caring environment, and initiative with prisoners to prepare them to re-enter the outside world or a community play to create common ground between people of different ages.’ (40)

In its series of publications around social impact, which most notably included François Matarasso’s report, Use or Ornament? The social impact of participation in the arts (1997), Comedia claimed arts-based projects’ credentials as a tool for social renewal – in that they enhanced social cohesion; improved local image; reduced offending behaviours; promoted interest in the local environment; developed self-confidence; built public/private sector partnerships; explored identities and visions of the future; enhanced organisational capacity; and, supported independence.

Other claims for public projects are couched in broadly similar terms. But despite – or perhaps because of – what is now the prevailing orthodoxy, it has become increasingly pressing for a distinction to be drawn between advocacy and evidence, potential and actual fact. While there is a large body of research, documented case studies and anecdotes, which are used to illustrate the impact of the arts, the sector is under increasing pressure to provide ‘robust’ evidence.

• Constructing the evidence-base

In fact, despite the existence of funding agreements for its sponsored bodies, DCMS is failing to produce much in the way of evidence of social impact from the cultural sector (41). The bulk of responsibility to deliver on the government’s agenda is manifest in the burgeoning research capacity of the department’s first-tier organisations and number of research reports commissioned.

These bodies are attempting to involve the rest of the sector through the dissemination of project evaluations (which have come to be regarded as providing evidence of social and learning impact as well as raising awareness of what can be achieved and how) and toolkits, guidelines, frameworks and standards, designed to encourage and help organisations themselves to capture evidence of the impact of their projects.

This very concentration of effort dedicated to identifying and extracting evidence of impact across the whole of the sector is such that it has been described as ‘the huge search to find the “holy grail”- the answer to the question, why fund the arts or more precisely show us why funding the arts will cure the sick, raise the dead and eradicate world poverty’. (42)

Commentators have been remarking on the lack of a serious and robust evidence-base for the arts and museums for nearly a decade (43). A number of recent reports published by DCMS, the Arts Council of England, Resource (the Council for Museums, Archives and Libraries) and on behalf of local authorities make the same point.

So, given its very evident frustrations with the lack of evidence, what has the cultural sector been doing to improve the situation and move beyond the rhetoric? How is it bridging the gap between advocates’ enthusiasm to show what difference the sector is making and the cassandras who suggest that what limited evidence exists indicates that ‘museums have had really very little impact on their communities’ (44).

The Arts Council and Resource have been approaching the problem along broadly similar lines. Both organisations’ current research agendas list their commitments to ground clearing exercises – scanning what data already exists and critically assessing it in the light of its potential value to the establishment of an evidence-base for informing strategic initiatives.

However, having established what already exists in terms of statistics and impact evaluations, Resource’s experience, in particular, suggests that mining extant data for evidence of social impact is relatively unproductive, if critically revealing. One recent statistical report, for example, noted that the existing body of quantitative data is not only ‘patchy, inconsistent and incomplete, but that it fails to generate any useful information on the outcomes of the services or on the impact they make’ (45). By definition, its findings comply with a growing consensus of opinion that ‘the scientific model of research is unlikely to provide the kind of assessment required’.

Given the widespread acceptance of the value of qualitative data in understanding and assessing impact, another piece of research, commissioned by Resource, pulled together the available evidence from a varied range of evaluations based on ‘soft’ (qualitative) indicators related to social impact, learning impact, economic impact and access (46).

This found that among the most frequent approaches used were the aggregation of case studies, project evaluations, cameos, ‘pen portraits of particular projects or initiatives, brief accounts of engagement with specific groups in the community, and anecdotal quotes from staff, project workers and…project participants’ (47). These characterise a number of advocacy cum policy documents published by DCMS and the Arts Council. But few pass muster in relation to current critiques of the evidence, which seem to have been prompted as much by scepticism about the apparent bias in most of the literature, as by the need to base policy development on an impartial and robust evidence-base.

• Criticisms of evidence of impact assessment

Several critical reviews of the available evidence on the impact of cultural projects have been published within the past year or so. They include meta-analyses which cover a mass of individual documents; they tend to have been driven by the government’s priorities around education, employment, crime and health; and they mostly analyse the robustness of the existing evidence and the methodologies employed.

These reviews reveal a consensus of opinion which questions the validity of much of the evidence presented, particularly with respect to:

• the non-substantiation of claims;




• the prevailing culture of cultural institutions, and their lack of evaluative experience;




• the limited jurisdiction of projects and their potential to influence outcomes; and




• the lack of robustness of the methodologies used, the quality of evidence gathered and a failure of reporting of methods, which ultimately undermines the validity of what evidence exists.

In terms of the non-substantiation of claims and the prevailing culture, for example, one review suggested that: ‘Rather than illustrating actual impact, these studies tend to provide evidence of engagement with the local community and of the organisations’ belief in their potential to make a social impact.’ (48).

They identify various sectoral characteristics that are impeding institutions’ ability to move forward with the collection of relevant data, including senior managers’ fear of the implications of visitor-centeredness – not least dumbing down – and arts practitioners’ fear that ‘the complexity of an experience which includes relationship, enjoyment, learning, exploration, expression will be destroyed, diluted or reduced’ (49).

Project organisers are regarded as having little or no incentive to collect data, and many lack the resources to do so. They may have little understanding of evaluation processes, and what has been done may have been carried out in a haphazard manner. Even when data are collected, it may not be analysed. Institutions may prioritise promoting their activities to key stakeholders and casting projects in a good light.

Consequently, outputs (such as participants’ continued involvement) often suffice for outcomes, and users are effectively bypassed in the process. Relatively few projects are even able to identify the socio-demographic characteristics of the participants.

One review observed that projects are liable to raise expectations beyond what they can deliver – not least in order to increase their chances of securing funding. In practice, some can only offer a limited sphere of influence. Projects may simply be ‘self-fulfilling’ and satisfy participants’ needs by definition; others can only have limited success in ‘opening doors’ to ‘people typically seen to be disenfranchised’.

Moreover, the remit of individual project evaluations is fairly limited, and the manifestations by which the success of projects might be judged – increased self-esteem, employment, social inclusion – tend to occur outside their jurisdiction (50).

A key issue affecting the quality of current evaluations is the lack of, and need for, appropriate methodologies to examine impact. Criticisms focus on the lack of systematically gathered hard evidence which might, for instance, enable projects’ effectiveness against their proposed outcomes to be assessed; causality to be attributed; or long-term, as opposed to immediate, impacts to be examined.

So while there may be evidence of actual impact for particular individuals at specific points in time – something reinforced by an emphasis on ‘critical incidents’ – the methodologies used cannot reveal long-term or sustained impacts (how, for instance, immediate engagement with the learning environment creates change or impact in the individual, group or community). It follows that little consideration is paid to securing baseline data against which change might be measured.

Other shortcomings identified include the reference groups that evaluations are based on. In general, these tend to refer to staff and project workers rather than project participants – which undermines the theoretical validity of user-based evaluation. Another frequently cited shortcoming is the identification and use of appropriate and meaningful indicators of impact.

It is also said that ‘interpretation problems plague studies’ (51) and that the over-determined use of ‘project evaluations by policy-makers may even be distorting the true picture of an impact or deflecting attention away from the more significant areas of impact’ (52). Commentators have called for a culture of more ‘balanced’ research, in which the reporting of specific examples of people who have achieved something valuable through projects rises above the status of ‘pure happenstance’ by a more compelling case being made by the inclusion of counter examples (53).

The lack of reporting of methodologies themselves introduces another layer of doubt. Credibility as to the validity of evidence and its analysis is undermined by the absence of such details as the number of participants surveyed; how data were collected; what techniques or procedures were used; and the presentation of detailed results.

This lack of robust evidence frustrates the possibility of meeting the political aspirations of building up a comprehensive picture of the social impact of the sector, measuring change, and indicating its relative value for money. There are real problems in aggregating existing data. As DCMS itself has observed, neither case studies, nor research based on small samples can be generalised from, and given that ‘reliable’ studies tend to vary in design and assessment, ‘the evidence they provide is often inconsistent and even contradictory’ (54).

Whatever the benefits of the arts, the lack of identifiable outcomes means that ‘we do not have enough information to judge whether such gains are enough or are efficiently and effectively gained’.

Many of these same criticisms have been levelled at Matarasso’s Use or Ornament? (55). This not only claimed to be the first report specifically to identify and assess evidence of the social impact of participation in the arts, but proposed much of what subsequently became accepted within mainstream qualitative research – surveying participants rather then institutions, addressing outcomes rather than outputs.

Like The Economic Importance of the Arts in Britain, Use or Ornament? exerted considerable political influence and effectively consolidated a near-consensus in Britain among cultural policymakers. But, its critics assert that the report is ‘flawed in its design, execution and conceptual basis’, that the data collected cannot support the conclusions reached, and that the methodology and the analysis are suspect.

One particular example cited is Matarasso’s default to intuition in the face of not being able to attribute causality:

‘[I]t cannot be denied that there is a cumulative power in the hundreds of voices we have heard over the past 18 months, in vastly different circumstances, explaining again and again how important they feel participation in arts projects has been for them. How many swallows does it take to make a summer?’ (56)

Given the extent of Matarasso’s influence, these criticisms throw the whole project of evidencing the social impact of the cultural sector into doubt. They also implicate the methodological competence of those policymakers who have promoted approaches to evaluating social impact based on his findings – not least DCMS, which aided and abetted the successful dissemination of Matarasso’s work by its citation in the culture secretary’s speeches (57) and the author’s involvement in initiatives set up by the department.

• Data collection and cultural policy

Many of DCMS’s concerns – both from the perspective of its objectives and in terms of its accountability – were inherited from the DNH. Before the 1997 election, DNH had already established initiatives around: access, including plans for culture online; museum education; the nurturing of young talent; participatory projects and community development, as a way of contributing to neighbourhood renewal. It had begun to shift the focus of national lottery funding away from capital prestige projects to local community projects. It introduced funding agreements and attempted to monitor the impact of its funding.

The development of all these has massively accelerated under DCMS via a controlling bureaucratic system intended to ensure the delivery of government objectives and increased funding, including sums specifically earmarked for the delivery of specific objectives.

So, to all intents and purposes, cultural policy is driving an immanent process in which the gathering of evidence about its impact has to all intents and purposes assumed centre stage. And, as a result of initiatives going back to the 1980s we have by now accumulated a mass of primary data.

But how valuable has the pursuit of that data for the cultural sector actually been?

Whatever the discussions about the quality of evidence gathered, questions also need to be asked about the use, or the lack of use, to which it is put. At the end of the day, this has to do with tensions implicit in government – in the relationship between investigation and political decision-making; between politicians’ receptivity to new research and determined ideology; between rational, evidence-based policy and intuitive politics; between the timescale required to research impact and politicians’ immediate requirements for information (58).

Until the cultural bureaucracy’s analysis of data is guaranteed, and until the evidence gathered can be seen to be being used constructively, it could be argued that collecting data has been a relatively spurious exercise.

Sara Selwood is Quintin Hogg Research Fellow at the School of Communication & Creative Industries, University of Westminster and editor of Cultural Trends.

This is an edited version of a paper given at the symposium Statistics in the Wake of Challenges Posed by Cultural Diversity in a Globalisation Context in Montreal, at the UNESCO Institute for Statistics and l’Observatoire de la culture et des communications de l’Institut de la Statistique du Quebec.

(1) Nissel M (1983) Facts about the Arts. A summary of the available statistics. London; PSI

(2) Nissel M (1983) Facts about the Arts. A summary of the available statistics. London; PSI, p1

(3) Marr A, ‘Painting by Numbers: statistics may rule our lives, but they are no way to measure the true value of the arts’, Observer, 29 July 2001

(4) Smith C (1999) ‘Government and the Arts’. Lecture at the RSA, London – 22 July 1999, in Wallinger M and Warnock M (eds) Art for All. Their policies and our culture. London: Peer, p14

(5) Ward M and Pitt P (1985) ‘Introduction to’ GLC (1985). The State of the Art or the Art of the State? Strategies for the cultural industries in London. London: Industry and Employment Branch, Department for Recreation and the Arts, Greater London Council, p6

(6) DCMS (2000a). Centres for Social Change: Museums, Galleries and Archives for All. Policy guidance on social inclusion for DCMS funded and local authority museums, galleries and archived in England. London: Department for Culture, Media and Sport

(7) House of Commons, National Heritage Committee (1996) Third Report. Session 1995-96. The Structure and Remit of the Department of National Heritage. Report together with the Proceedings of the Committee, Minutes of Evidence and Appendices. London: HMSO, p16

(8) House of Commons, National Heritage Committee (1996) Third Report. Session 1995-96. The Structure and Remit of the Department of National Heritage. Report together with the Proceedings of the Committee, Minutes of Evidence and Appendices. London: HMSO

(9) DNH (1994) Annual Report 1994. London: HMSO, p8

(10) DNH (1994) Annual Report 1994. London: HMSO, p12

(11) DNH (1994) Annual Report 1994. London: HMSO, p11

(12) House of Commons, National Heritage Committee (1996) Third Report. Session 1995-96. The Structure and Remit of the Department of National Heritage. Report together with the Proceedings of the Committee, Minutes of Evidence and Appendices. London: HMSO House of Commons, p2

(13) GLC (1985) The State of the Art or the Art of the State? Strategies for the cultural industries in London. London: Industry and Employment Branch, Department for Recreation and the Arts, Greater London Council

(14) ACGB (1985) A Great British Success Story. London: Arts Council of Great Britain; and ACGB (1986) Partnership: Making Arts Money Work Harder. London: Arts Council of Great Britain

(15) ACGB (1989) An Urban Renaissance. London: Arts Council of Great Britain

(16) Myerscough J (1988) The Economic Importance of the Arts in Britain. London: PSI , p2

(17) DNH (1994) Annual Report 1994. London: HMSO, p4-5

(18) ACGB (1993) A Creative Future. The way forward for the arts, crafts and media in England. London: HMSO

(19) Bottomly, V (1996) Building Communities: Community Development, Participation and Partnership. London: Department of National Heritage

(20) DCMS (1998) A New Cultural Framework. London: Department for Culture, Media and Sport

(21) DCMS (1998) DCMS Annual Report 1998 The Government’s Expenditure Plans 1998-1999. London: The Stationery Office

(22) ‘A shorthand term for what can happen when people or areas suffer from a combination of linked problems such as unemployment, poor skills, low incomes, poor housing, high crime environments, bad health and family breakdown’, quoted in Cabinet Office, (2000) The Social Exclusion Unit leaflet

(23) Cabinet Office (1998). Bringing Britain Together: a national strategy for neighbourhood renewal

(24) Policy Action Team 10 (1999) Arts & Sport. A Report to the Social Exclusion Unit. London: Department for Culture, Media and Sport, p2

(25)DCMS (2001) Libraries, Museums Galleries and Archives for All: Co-operating Across the Sectors to Tackle Social Exclusion. London: Department for Culture, Media and Sport

(26) Woolf M, ‘Jowell signals the end for big money lottery projects’, Independent, 30 July 2002

(27) DCMS (1998) The DCMS Comprehensive Spending review: A New Approach to Investment in Culture (mimeo)

(28) DCMS (2000) Departmental Investment Strategy. Spending Review 2000: New Public Spending Plans 2001-2004. Prudent for a Purpose: Building Opportunity and Security for All; and DCMS (1998a) A New Cultural Framework. London: Department for Culture, Media and Sport

(29) DCMS (1998) Chris Smith details biggest ever increase in cultural funding. A new contract: Increased Investment: Greater Efficiency: Structural Reform. DCMS press release 167/98

(30) DCMS (1998) A New Cultural Framework. London: Department for Culture, Media and Sport

(31) HM Treasury (2000) Public Spending Review Public Services White Paper

(32) House of Commons, Select Committee on Culture, Media and Sport (1999). Sixth Report 1998-99 The Department for Culture, Media and Sport and its Quangos, para 11 (accessed 16 August 2002)

(33) DCMS (1998) A New Cultural Framework. London: Department for Culture, Media and Sport

(34) DCMS (1999) Local Cultural Strategies. Draft Guidance for Local Authorities in England. London: Department for Culture, Media and Sport, p3

(35) Ryan M (2001) ‘Introduction’ in Institute of Ideas (2001) Museums for ‘The People’? Conversations in Print. London: Institute of Ideas, p8

(36) DCMS (2002) ‘Arts and sport can help tackle crime’ says Tessa Jowell. DCMS press release 45/2002; DCMS (2001) The arts have key role in making our society a better place to live says arts minister, Tessa Blackstone. DCMS press release 273/01; DCMS (2000) The arts are good for your health, says Alan Howarth. DCMS press release 87/2000

(37) Appleton J (2001)‘Museums for ‘the People’?’, Institute of Ideas (2001) Museums for ‘The People’? Conversations in Print. London: Institute of Ideas: 14-26

(38) Regional Museums Task Force (2001) Renaissance in the Regions: a new vision for England’s museums. London: Resource

(39) Policy Action Team 10 (1999) Arts & Sport. A Report to the Social Exclusion Unit. London: Department for Culture, Media and Sport, p2

(40) Landry C, Bianchini F, Maguire M Warpole K with Greenhalgh L and Six P (1993) The Social Impact of the Arts, A Discussion Document (mimeo), executive summary

(41) Selwood S (2002) ‘What difference do museums make? Producing evidence on the impact of museums’, Critical Quarterly, vol 44, no 4: 65-81

(42) Hamilton C (2002) Arts Research Digest Seminar. Presentation. Unpublished

(43) Landry C, Bianchini F, Maguire M Warpole K with Greenhalgh L and Six P (1993) The Social Impact of the Arts, A Discussion Document (mimeo), executive summary

(44) Newman A and McLean F, cited by Wavell C, Baxter G, Johnson, I and Williams D for Robert Gordon University (2002) Impact Evaluation of Museums, Archives and Libraries: Available Evidence Project. Draft report (mimeo)

(45) Cultural Heritage Consortium (2002) ‘Impact evaluation of museums, archives and libraries: quantitative time series data identification exercise’

(46) Wavell C, Baxter G, Johnson, I and Williams D for Robert Gordon University (2002) Impact Evaluation of Museums, Archives and Libraries: Available Evidence Project. Draft report (mimeo)

(47) Wavell C, Baxter G, Johnson, I and Williams D for Robert Gordon University (2002) Impact Evaluation of Museums, Archives and Libraries: Available Evidence Project. Draft report (mimeo), p12

(48) Wavell C, Baxter G, Johnson, I and Williams D for Robert Gordon University (2002) Impact Evaluation of Museums, Archives and Libraries: Available Evidence Project. Draft report (mimeo), p31

(49) Moriarty G (1997) Taliruni’s Travellers: An arts worker’s view of evaluation. The social impact of the arts: working paper 7. Stroud: Comedia

(50) Leeds Metropolitan University, Centre for Leisure & Sport Research (2002) Count Me In. The Dimensions of Social Inclusion through Culture & Sport. Leeds: Leeds Metropolitan University, p82-83

(51) DCMS, Economics Branch, Analytical Services Division (2002) Evidence on Cultural /Creative/Sporting Effects (mimeo), p3

(52) Wavell C, Baxter G, Johnson, I and Williams D for Robert Gordon University (2002) Impact Evaluation of Museums, Archives and Libraries: Available Evidence Project. Draft report (mimeo)

(53) Leeds Metropolitan University, Centre for Leisure & Sport Research (2002) Count Me In. The Dimensions of Social Inclusion through Culture & Sport. Leeds: Leeds Metropolitan University

(54) DCMS, Economics Branch, Analytical Services Division (2002) Evidence on Cultural /Creative/Sporting Effects (mimeo)

(55) Merli P (2002)‘Evaluating the social impact of participation in arts activities. A critical review of François Matarasso’s Use or Ornament?’, International Journal of Cultural Policy, vol 8 (1): 107-118; Belfiore E (2002) ‘Art as a means of alleviating social exclusion: does it really work? A critique of instrumental cultural policies and social impact studies in the UK’, International Journal of Cultural Policy, vol 8 (1): 91-106

(56) Matarasso, cited by Wavell C, Baxter G, Johnson, I and Williams D for Robert Gordon University (2002) Impact Evaluation of Museums, Archives and Libraries: Available Evidence Project. Draft report (mimeo)

(57) Merli P (2002) ‘Evaluating the social impact of participation in arts activities. A critical review of François Matarasso’s Use or Ornament?’, International Journal of Cultural Policy, vol 8 (1): 107-118

(58) Walker D (presenter) (26 July 2001) ‘Analysis: Unavailable Evidence?’, transcript of a recorded documentary