Contenu du sommaire : Évaluation des politiques publiques sanitaires et sociales

Elements for a sociology of public policy evaluation in France. At first sight, as a professional activity, evaluation of public policies and programmes (EPP) has met more difficulties in France than in the United States and in many other European countries. It is, in fact, possible to show that its integration among administrative and political practices has come relatively late when one compares the French case especially to English-speaking and Scandinavian countries. However, since the report for the Commissariat général du Plan (State Planning Commission) in 1986, numerous stakeholders have defined their activity as “evaluation of public policies”. Multiple initiatives from the central government were recorded, while the EPP resolutely developed in local territorial governments. After the National Evaluation Council eventually ground to a halt from 2002 on, numerous developments did, however, occur, such as the devising of “ministerial reform strategies”, and the launching of a “general revision of public policies”, but also the modification of the constitution in the summer of 2008, etc. New stakeholders appeared in the EPP professional field and it is difficult today to anticipate how their systems of action will be reorganised: in order to do so, one has to also consider activities which are in a way ‘contiguous' to evaluation, regardless of their undeniable specificities, whether they be complementary or in competition with. Activities such as ‘performance audit', for instance, should be seen as processing a polymorphic commodity that can be considered as “evaluative information”, which serves the management of public actions in more or less transparent conditions. It is possible that, with the new developments, evaluation will tend to be mainstreamed and become more trivial in the future.

For evaluative action. Urged on by the ambitions of a new public management, evaluation seems to have a rich future ahead of it, although this prosperity is not without equivocation. Despite this surge in esteem, evaluation still remains feared and even contested, and is often misunderstood. Reflecting upon the singular destiny of evaluation in France, the author observes that the processes' reach is weakened by the combined effects of institutional factors and the methodological derivatives that play a role in distancing evaluation from action. The influence of evaluation is thus limited, especially since it has more and more competition from other procedures. Thus suggesting making evaluation mover active, the author proposes a change of perspective and commits to a pragmatic idea of evaluation. New procedures are thus proposed to increase the credibility and the reach of evaluations, make them a tool for collective intelligence and definitively promote “evaluative action”.

Control and evaluation, beyond semantic arguments, relationship and difference factors. This article aims to compare ex post policy evaluation with formal systems of management control. The comparison is difficult due to the plurality of conceptions that exist equally for both of the objects to compare. It is also made arduous by the gap that can exist between ideal typical and practical models. The clearest rapprochement-differentiation is based on the object pair (controlled or evaluated) technology used (periodic information systems or punctual studies). À more delicate comparison concerns both the functions of each of the two instruments and their centring, as well as the major characteristics of the two procedures. Such a comparison leads to the conclusion that there is complementarity between the two tools, particularly in terms of contributing to acquisition. However, it seems to be too optimistic to believe that policy evaluation would be capable of compensating the ensemble of flaws currently denounced in formal management control systems.

Measuring the performance of public management using economic analysis. The need to modernise the state by developing management through performance is at the heart of a budget reform that began with the Organic Law of 2001 (LOLF). Said law is generally a process aiming to make explicit the objectives of public action and to evaluate the results thereof. As there can be no performances without appropriate indicators, the definition thereof is an important factor in its implementation since using imperfect indicators in salary incentive mechanisms or allocation of budgetary means can sometimes cause undesirable effects that could delegitimise the reform process currently underway. Performance management mechanisms require fully grasping the problems concerning information and the incentives to which public management is confronted, as well as understanding of what needs to be done in order to reduce said problems. This article provides economic reference points to complete the legislative, sociological and managerial approaches that are usually prevalent when reflecting upon the different levels of evaluating public actions. In this respect, economic analysis clarifies the informational obligations to consider when an explicit social interest objective has been set.

Econometric evaluation methods. Our survey covers the recent developments of the micrœconometric literature on evaluation methods. In this field, the canonical model is Rubin's model. This model is the relevant framework for defining and for examining the identifiability conditions of the parameters of interest in any evaluation study. We insist on the definition of these parameters, which include the average effect of the treatment on the treated and on the non-treated individuals. For each set of assumptions (selectivity on observable or unobservable characteristics, conditional independence between outcomes and treatment indicators, etc.), we present the most adapted estimation method. We put a special emphasis on matching estimators in the situation where the selectivity depends only on observables, and on differences-in-differences methods and on regression-discontinuity techniques when the selectivity depends both on observable and unobservable characteristics.

Use and misuse of statistical arguments: steering public policies by performance. This article aims to clarify several phenomena that emerge from the growing use of indicators in the management and reform of public policies. It concentrates on the conditions of validating statistical arguments, that is, using statistical figures to support political agendas. Taking the European Employment Strategy as an example, the article highlights the possible risks and derives weighing on the definition and implementation of public policies, particularly in the social realm; among which: the rational acquisition of optimizing quantitative results, the difficult conciliation between performance and social justice, the danger of bad indicators dispelling good ones, normativity hidden in batteries of indicators, the use of statistical conventions as political instruments. Finally, it questions the hypothesis of a decline in democratic procedures through which a society takes cognizance of itself and the collective problems facing it.

Quality and efficiency programmes, the instruments serving social security policy evaluation and the mobilisation of their stakeholders. Quality and efficiency programmes, which have been an annex to the social security's funding laws since 2007, aspire to extend the “objectives / indicators / results” procedure implemented in 2001 in the process of the elaboration and examination of funding law projects to the domain of social security policies. While they are certainly fructuous for the analysis and allocation of budgetary resources on a relatively aggregate level, this procedure is more controversial as regards its power to incite public managers to be efficient. In the case of social security policies, it does effectively complete the current performance assessment measures that were put in place in 1996 by the objective and management conventions linking the government and the national organisations responsible for implementing them. As a result, quality and efficiency programmes concentrate on the appreciation of the final impact of policies on the populations' income and living conditions. These programmes thus enable us to make a table of the ensemble of success reached by social security policies, as well as to measure the progress yet to be accomplished. In this respect, they are used to identify the measures for which a more in depth evaluation procedure, based on methodology to assess their exact impact, is needed.

Social aspects in health technology assessment. After having favoured the development of evaluating public policies, sociology is called upon to play a role within scientific evaluation procedures. In the realm of health technology assessment, particularly, sociological knowledge and know-how are solicited by specialized national agencies so as to assess objects that are not only technical in nature, but also social and political. This article presents the manner in which sociology has been involved in the evaluations carried out by the French National Authority for Health. It presents the results of a group of work on “Social aspects in health technology assessment”, the objective of which was to formalize an increased recourse to sociology. These results propose a “grid of standardized sociological questions” susceptible of being used by non-specialized experts within the agency, in view of possible further explorations.

Evaluating organisational change using the intervention research approach. Taking the impact of T2A (activity-based funding) as an example. This article examines a conference given at the seminar organised by the DREES (French Directorate for Research, Studies, Evaluation and Statistics) in 2009 on evaluation methods. In the specific case of organisational change, the author defends choosing a qualitative methodology based on the principles of intervention research. This methodology is essentially based upon the concept of instrumental interaction, for which the researcher helps the players to conceptualise the changes to which they are committed and to reveal the force field of which these changes are a component. The example then developed is derived from research aiming to evaluate the organisational effects of a new funding method for healthcare facilities, activity-based funding (T2A). It has been forwarded that the T2A has effectively induced diverse organisational evolutions, but very few at the heart of the healthcare process. This fact can be explained through a comparison of hospital activities with a hypercomplex production system.

From Public Policies to programmes: was evaluation saved by the LOLF (Organic Law relative to the Financial Laws) ? By switching state policies to a rationale of programmes and management by performance, the Organic Law relative to Financial Laws of 1 August 2001 set the stage for an evaluative revival, ending the way in which evaluation had been misused by public policy in the 1990s. The implementation of the Organic Law is too recent to precisely evaluate its effects on this subject. An examination of the Politique de la ville (policy for disadvantaged neighbourhoods) and its evaluation does, however, lead us to question the validity of the neo-managerial debates so in fashion today, which present the passage to a programme whose rationale was a decisive advance for evaluation and, beyond that, for public performance as well. Not only because it is not enough to create the conditions of a programme's evaluability in order for it to be evaluated, but also because the institutional transformations to this end prove to be very unfavourable for a pluralistic argument.

Defining control zones for Active Solidarity Income experimentation. In 2007 and 2008, experiments were carried out on Active Solidarity Income (RSA) in 34 French départements. Each of these départements defined one or more test zones or experimental zones, where the new benefits were implemented. In total, 51 test zones were chosen by the départements. So as to evaluate the effects of the RSA, we built and implemented a methodology enabling the definition of control zones in each experimenting département. These control territories were selected in such a way as to provide the most satisfying counterfactual possible for the evaluation of the test zones, taking into account the inherent limits of local statistics on return to employment behaviour. The first step, for a given département, was to establish a list of possible control zones whose socio-demographic characteristics were comparable to those of the test zone, notably as regards size (population, number of RMI, or Minimum Income Allowance, beneficiaries), the density of the population and the administrative situation (canton, CLI, UTAS, CAS, etc.). The second step was to classify all of these possible zones, from the closest to the test zones as regards the possibility to entry into a job for the RMI beneficiaries as estimated using the CNAF's (National Family Allowance Fund) administrative data. To do this, every semester there is a reconstruction of the rate of entry into a job for the Minimum Income beneficiaries on the communal level from June 2002 to June 2007 whilst neutralising the effects of constituting the beneficiaries according to characteristics observable in the administrative files. In the third step, the control zones were proposed to the experimenting départements, which were then able to make counterproposals backed up with arguments in case of disagreement. In practice, the choice of this iterative procedure enabled a good compromise between the principle of national unity in the selection process of control zones and putting local informational systems and knowledge of the field to the best possible use.

What assessments can be derived from the RSA experiments ? Thirty-three départements implemented the RSA on a trial basis, for the most part between November 2007 and May 2008. Based on the information made available over this time, it is now possible to examine the outcome of this experiment and also to assess the experimentation process itself. The experiments had a positive effect in terms of governance in that it reinforced cooperation among the different actors involved. It also improved knowledge of stakeholders' behaviour and the difficulties facing RSA beneficiaries when seeking employment. The experiments seem to have paved the way for large-scale implementation of the RSA. However, it did not provide robust data on the extent to which the RSA helped beneficiaries find jobs. Furthermore, it was not possible to identify the factors determining results observed, i. e., was it the reinforced social and/or professional support ? Was it the financial incentive or the new types of benefits specifically aimed at going back to work ? Finally, the experiments did not allow the poverty rate to be assessed. All of this can be partially explained by the need to experiment over a longer period, thereby enabling the effects to be evaluated.

The use of statistics: from aiding decision making to the evaluation of public policies. Evaluation, performance and management by results have driven the construction of quantitative data for following the implementation of public actions (measures, programmes, public policies). As decreed by the LOLF (Organic Law relative to the Financial Laws), all public action must hereafter specify its objectives and supply indicators intended to enable their monitoring and evaluation. Moreover, the demonstration of proving the efficiency or the pertinence of an action is all the stronger when it is supported by quantitative arguments. That statistical data is called upon to furnish arguments regarding public action is old news, what is newer is the extension of its use to domains thus far out of public statistics' field, to all branches of managerial activity. The diffusion of managerial concerns accords great importance to figures, which questions those of statistics. This article reviews the fundamentals of statistics and the links it has to evaluating public actions in order to consider them in the new politico-administrative context (decentralisation and the revival of evaluation by constitutional laws).

The General Inspectorate of Finance in an era of performance evaluation. With the implementation of the LOLF, a service such as the General Inspectorate of Finance (IGF) dedicates most of its work to performance evaluation for public policies, public organisations and public procedures. Its traditional methods of intervention were no longer useful to it, but the IGF did, nevertheless, evolve significantly in order to adapt to this profession in expansion. This work does not comply with the academic standards of evaluation, but the limits are due less to the procedural modalities or the positioning of an inspectorate than from the ambiguity of the objectives ascribed to the evaluative procedures, to its intrinsic difficulties and to questioning the general uses made of this procedure. Academic works on pubic policy evaluations often oppose the approach of scientific evaluation and that of ministerial and interministerial inspectorates: they are more interested in verifying the application of the procedures and policies decided upon and would be less-equipped to evaluate their performances. This judgment, which holds some truths, nevertheless, deserves to be qualified further in light of the recent experience with a particular inspection by the General Inspectorate of Finance, who, today, dedicates the majority of its workforce to the evaluation of different forms of public performance. As a consequence, the IGF had to change its methods and its organisation. Although its past work is incontestably limited in relation to ideal evaluation standards, these limits are widely shared and question the very evaluative process and the uses to which it can be put.

This article focuses on the “mixed practices” of law and evaluation in the implementation of public actions that have been in place in France since the 1990s. For the past twenty years, public policy laws have been undergoing an identity change that is no stranger to developments in public administration, whereas evaluation practices have entered into an institutionalisation phase. Before examining eventual relationships between law and evaluation, the very conditions of the possibility of such an undertaking will be evoked. Once this preliminary work is accomplished, we shall address the transpositions operating between these two procedures: that of the Jurist and that of the Evaluator. This article depicts that which the law shows of evaluation (its status has evolved in legal texts), and that which evaluation shows of the law, which becomes, in turn, an assessable production. In the end, we observe a redistribution of roles in public action: more and more, jurists must become evaluators and evaluators must borrow a part of jurists' identity.