MERCI – Monitoring European Research Council’s Implementation of Excellence

Objectives
MERCI evaluates the European Research Council's (ERC) "Starting Grants" program by focusing on the effects on the individual career development of young researchers. Additionally, the study considers institutional effects (i.e. how the ERC grant affects research organizations around Europe) and structural effects with regard to the consolidation of the European Research Area (ERA). We aim to draw a broad and valid picture concerning the question of whether the ERC target group is reached, whether the Starting Grant applicants' needs are being addressed effectively and whether the funding scheme brings about the intended effects in terms of conducting "pioneering research", carrying out research independently or increasing mobility. MERCI aims to provide information for program improvement by assessing both intended and unintended effects of the Starting Grant funding scheme. In order to help the new ERC scheme to perform well, the focus of MERCI is on the implementation and operation assessment and, thus, on the program's performance and effectiveness. The overall objective of MERCI is to generate strategically relevant information which will aid the ERC in adjusting its program operation, in performing necessary functions adequately and thereby enhancing the quality of service delivery.

Research design
The value added of the MERCI evaluation design does not only arise from combined application of qualitative (semi-structured interviews), quantitative methods (online survey in a panel design) and bibliometric analysis but also from the functional interlacing of intermediate results from the qualitative interviews and the online survey, instrument development and interpretation of data (triangulation approach).

Semi-structured Interviews
The interview studies primarily focus on the Starting Grants application and selection process, as well as the institutional setting of the research group, but also address topics that are hardly accessible by a standardized approach. This concerns perceived career obstacles, networking and gender aspects, motives for (im)mobility, perceived degree of independence, work-life-balance, etc. At the very beginning of the project, 10 approved Starting Grant applicants (2007 cohort) from different EU countries were interviewed. These first explorative interviews carried out in summer 2009 aimed at getting in-depth information about specific problems, institutional settings, experiences with the project management and specific national conditions young researchers were facing. The analysis of the first interviews provided valuable insights into the perception of the Starting Grant recipients and on the status ascribed to the ERC funding. In addition, results of these interviews were used for constructing the questionnaire of the online survey (1st wave) and compiling the guideline for the qualitative main study. For the qualitative main study, 40 applicants (29 accepted and 11 rejected) of the 2009 Starting Grants call were interviewed in Spring and Summer 2011. By analogy with the qualitative pre-study the results of the main study have been used as source of information for constructing questionnaires for the online panel survey (intermediate and 2nd wave questionnaire).

Online Panel Survey
The evaluation of the Starting Grant program's effects requires a comprehensive collection of individual and aggregate data and their change over time. Therefore we establish a valid and sustainable database on the career development of the researchers who responded to a Starting Grants call. For this purpose, two waves of online surveys are conducted: The first wave is conducted approximately one year after the ERC funding decision, the second wave is conducted 2.5 years later. A short intermediate inquiry is carried out between the 1st and the 2nd wave of the online survey. In the 1st wave, the online survey focuses on the assessment of the ERC application and evaluation process, the professional background of the applicants (education and work experience), their research environment, publication activities, and career expectations. In terms of the approved applicants, we are – amongst other things – interested in the financial and personnel endowment of the research group, the integration into the existing institutional setting and the working conditions at the ERC host institutions after the implementation process. The short intermediate survey primarily serves panel maintenance purposes (check-up of contact data) but also reports on career development in terms of changes of the institution, status or position and gives first indication about working conditions and the overall satisfaction with the funding scheme. The 2nd wave provides information on the Starting Grant recipient's skills and competencies generated during the ERC funding period, the (perceived) influence of the funding on the scientific career, experiences of international mobility, career aspirations as well as the overall satisfaction with the ERC funding scheme. Beyond that, the 2nd wave survey enables us to observe changes over time with regard to satisfaction, career aspirations and especially career development by tracing changes of the position or the host institution. Up to now, within the framework of our online panel six field phases have been finalized successfully. Whereas the 1st wave survey has been conducted for the applicants of the Starting Grants calls 2009, 2010 and 2011, the intermediate survey is available for the calls 2009 and 2010. In summer 2013, applicants from the 2009Starting Grants calls completed the 2nd wave survey. Thus, this is the first Starting Grants cohort which fully completed our panel survey.

Bibliometric analysis
To evaluate the ERC selection process of the "Starting Grants" program, a publication and citation analysis of funded and rejected applicants of the 2007 and 2009 Starting Grants calls is carried out. In Spring and Summer 2011, the iFQ and the Bielefeld University carried out the investigation of publications and citations for these two cohorts. The analysis of data is mainly carried out by the iFQ.

Consortium and cooperation
MERCI consortium partners are the Humboldt-University of Berlin (HU Berlin), the University of Dortmund (TU Dortmund) and the Bielefeld University. The administrative supervision of the project partners and the cooperation with the European Commission resides with the HU Berlin. The iFQ is responsible for the project coordination, the report preparation as well as for the conceptualization and analysis of the online surveys. Prof. Michael Meuser (TU Dortmund) is responsible for the conceptualization, implementation and analysis of the semi-structured interviews. The bibliometric study is conducted both by Dr. Matthias Winterhager (Bielefeld University) and the iFQ. We cooperate with PREST Manchester, which carried out another evaluation study of the Starting Grants Program (EURECIA; "Understanding and Assessing the Impact and Outcomes of the ERC Funding Schemes") and with the VU University Amsterdam, which is currently evaluating the Starting Gants Program with regard to gender issues (ERCAREER; "Capturing career paths of ERC grantees and applicants").

International Advisory Panel
MERCI is accompanied by an international group of experts (International Advisory Panel, IAP). The primary aim of this panel is to assure intercultural validity of the MERCI sub-studies and to bring in expertise about different national research systems. The first meeting of the IAP took place in Berlin in September 2009. The central question discussed was the attractiveness of the ERC funding for scientists against the background of the characteristics of the various national research systems and framework conditions for young researchers in Europe.

MERCI in the context of iFQ-projects
MERCI is part of the iFQ subject area "Young researchers and careers" and builds upon the evaluation of the Emmy Noether program. With the aid of MERCI, the iFQ further develops its expertise with regard to program evaluation, triangulation of methods and online panel survey. Furthermore, the project significantly expands the data base for processing central topics of the institute. In particular this refers to the situation of young researchers in the postdoctoral period, decision-making processes in research funding, measurement of research services (qualitative and quantitative indicators), as well as governance of science.

Locating MERCI in the scientific debate
So far, long-term studies of funding instruments for post-docs are scarce. However, in the course of the consolidation of ERA, the demand for scientists is not only expected to increase but simultaneously the role of post-docs will change. In their function as Principal investigators they more frequently take leading positions and act as independent researchers already at an (relatively) early stage of their career. In view of this development, the interest in information about the performance and effectiveness of funding programs for young scientists will increase in the future. In this respect, our project does not only address fundamental issues of the promotion of young talents but will also generate a valuable input to an important science policy discussion and improve the empirical data base in this subject area.