PRACE Key Performance Indicators
Given the scale of the computational power in the PRACE portfolio, PRACE related statistics are becoming increasingly important to highlight the impact of PRACE on HPC based research, HPC know-how in Europe, and European Industry engagement in HPC.
In 2014, the PRACE Council approved a set of Key Performance Indicators (KPI) that facilitate the analysis and evaluation of PRACE achievements and successes.
Source: PRACE 2015 Annual Report
PRACE’s impact on evolving research
- The number of project applications received via PRACE Calls for Proposals for Project Access exhibit a clear overall upward trend. This is particularly evident up to the 8th Call, with a large sustained increase between the 6th and 8th Call, followed by a slight decrease (Figure 1). A downward trend of rejected projects below the technical quality threshold is noted, displaying the maturation process of proposal submissions, in which researchers put more effort into the quality of their proposals. Moreover, the evolution reflects the positive outcomes of PRACE Preparatory Access Calls (including access type C) that enable prior technical support for application and scalability tests. Figure 1 also highlights an increase in rejected projects above the scientific threshold, particularly after the 6th Call. This is correlated with the increase in total applications.The overall trend highlights the increasing importance and impact of PRACE Tier-0 service for research.
- PRACE also keeps track of the submission of Project proposals by recurrent Principal Investigators (PIs) (Figure 2). The ratio of recurrence is relatively low: over 41% of PIs who submitted the most recent proposals were recurrent applicants to a PRACE Call for Proposals for Project Access. The upward trend of the ratio of recurrence is visible, particularly from the 6th Call onwards (reaching 57% at the 11th Call), influenced by the downward trend on awarded projects.
- A total of 60% of the projects that are awarded with PRACE resources, and two-thirds (63%) of those resources are awarded to “foreign projects” (i.e. projects with PIs from a different country (recorded as the primary institution of the PI) than the machine on which the research is executed). The ratio of awarded foreign projects remains rather stable over time (Figure 3). This suggests that the nationality of the PIs institution does not impact the chances of a project being awarded and reflects that the PRACE Peer Review Process works with scientific excellence as its main criterion.
- National and international co-funding for PRACE-awarded projects show a downward trend (Figure 4), despite the clear increase in the 10th Call for national co-funding. National co-funding represents the most prevalent form of self-support for the projects awarded by PRACE, overlapped only by EC co-funding in the 9th Call. EC co-funding exhibits a gentle overall upward trend.
- The increase of EC support for the projects awarded by PRACE illustrates the outcomes of EC funding policies, aligned with the support to HPC, as key enabler technology.
PRACE’s impact on scientific production
- Considering that it is only possible to measure the impact of PRACE-supported projects via their scientific production one year after the end of the project, Figure 5 only presents the evolution of scientific production supported by PRACE up to and including the 6th PRACE Call for Proposals for Project Access (mid-2014).
Up to and including the 6th PRACE Call for Proposals for Project Access, PRACE supported 275 MSc and PhD theses, 746 publications and 1363 scientific talks. On top of that, two patents resulting from projects supported by PRACE have been filed. The data reflect an increasing trend in all types of scientific production supported by PRACE.The increase in scientific production (Figure 5) is partly related to the increase in PRACE HPC resources made available and therefore the possibility of awarding a larger number of project applications. Of course, a higher number of publications raise the chances of a higher impact in science, however a future analysis of the impact-factor of published papers will reveal quantitatively whether the quality of scientific production also improves over time.
PRACE’s impact on growing know-how in Europe
- Since 2008 PRACE has engaged to provide top-class education and training for computational scientists in Europe through the PRACE Advanced Training Centres (PATC), the International HPC Summer School, and PRACE Seasonal Schools, with a clear increase of participants registered (Figure 6). A slight decrease is observed between 2014 and 2015.
PATC courses, Seasonal Schools and the International HPC Summer School are offered free of charge to eligible participants.
- Between August 2008 and December 2015, PRACE has provided 23 783 person-days of training through attendance-based courses, with an upward attendance trend. PRACE courses were attended by 7 350 unique individuals. This shows the effectiveness of PRACE in attracting, training and retaining competences.
The six PRACE Advanced Training Centres are Barcelona Supercomputing Centre (Spain), CINECA – Consortio Interuniversitario (Italy), CSC – IT Center for Science Ltd. (Finland), EPCC at the University of Edinburgh (UK), Gauss Centre for Supercomputing (Germany) and Maison de la Simulation (France).
The average rate of recurring participation in training is of 30%. This excellent ratio proves that PRACE trainings are not a closed circuit where the majority of attendees are the same people attending repeatedly. It also shows enough recurrence to indicate the attractiveness of PRACE training courses.
In 2015 the number of participants registered in PATCs courses was 1628 (1366 from academia and 262 from non-academia affiliation). More than 83% of participants attending PATCs trainings days have academic affiliation (1366), illustrating the impact of such event on research and scientific communities, in particular for early stage researchers and PhD students.
A clear difference of attendance is observed between the first and second semester of 2015. As observed in Figure 7, the total number of attendances registered in the first semester (first and second quarters) is significantly higher than during the second semester (third and fourth quarters). This is the result of the varying number of training days and courses available during these periods.
PRACE’s impact on attracting the industrial sector
- The interest of industry in PRACE at high-level international events has increased steadily over the past years (Figure 8). The total number of industrial participants showing interest in PRACE during the two main HPC events (Supercomputing (SC) in the USA, and the International Supercomputing Conference (ISC) in Germany) between 2008 and 2015 was 1 233 unique individuals. More than half of the companies that visited the PRACE booth at ISC’15 and SC15 were first-time visitors. This indicates that industrial interest in PRACE is growing on both sides of the Atlantic.
- The average participation of industry in PATC trainings is 12% between 2012 and 2015 (15% in 2015). The increasing interest from industry in participating in HPC training is visible in Figure 9. Over 260 industrial participants were trained by PRACE. Eligible industrial participants enjoy the same service as academic trainees and can attend PATC courses free of charge.
Industrial use of PRACE HPC resources
- PRACE opened its Calls for Proposals to industrial applicants in mid-2012. Industrial participation can take the form of a project led by a principal investigator coming from an industrial enterprise, or a researcher from industry collaborating in an academia-led project. The reduction and stabilisation of projects awarded after the 7th Call has a strong impact on the number of projects awarded with industrial participants (Figure 10).
- Regarding SHAPE pilot, PRACE can report 10 success stories of SMEs from 6 different countries benefitting not only from PRACE HPC resources but more importantly, from the know-how in the PRACE centres. The Second SHAPE Call awarded 11 more projects and a Third SHAPE Call was opened in November 2015.