PRACE Key Performance Indicators
Given the scale of the computational power in the PRACE portfolio, PRACE related statistics are becoming increasingly important to highlight the impact of PRACE on HPC based research, HPC know-how in Europe, and European Industry engagement in HPC.
In 2014, the PRACE Council approved a set of Key Performance Indicators (KPI) that facilitate the analysis and evaluation of PRACE achievements and successes.
Source: PRACE 2016 Annual Report
PRACE’s impact on evolving research
Number of projects
During the initial phase of PRACE (dubbed PRACE 1), the number of project applications received via PRACE Calls for Proposals for Project Access exhibited a clear overall upward trend. The phasing in of PRACE 1 naturally incited an increase in demand for Tier-0 resources. This is particularly evident up to the 8th Call, with a large sustained increase between the 6th and 8th Call, followed by a slight decrease (Figure 1).
A downward trend of rejected projects below the technical quality threshold is noted, displaying the maturation process of proposal submissions, in which researchers put more effort into the quality of their proposals, as a reaction to increased competition. Moreover, the evolution reflects the positive outcomes of PRACE Preparatory Access Calls (including access type C) that enable prior technical support for application and scalability tests. Figure 1 also highlights an increase in rejected projects above the scientific threshold, particularly after the 6th Call. This is correlated with the increase in total applications.
During the phasing out of PRACE 1, the number of available core hours dropped, and this decreased the demand, as researchers anticipated an even stronger competition for the remaining resources. This trend was mitigated in Call 12 and 13, when PRACE Hosting Members made additional core hours available while preparing the PRACE 2 programme (see pages 24 to 27 for more information on PRACE 2.
Despite the tough competition, demand for PRACE resources remains high, indicating that scientists consider Tier-0 access an essential asset to their work. This was also underlined by the PRACE Scientific Steering Committee (SSC) during the preparation of PRACE 2 (see page 24 for the article by the Chair of the SSC).
Figure 1 –Total number of Projects awarded and rejected.
PRACE also keeps track of the submission of Project proposals by recurrent Principal Investigators (PIs) (Figure 2). This KPI is created by checking for each call if a PI is new to PRACE, and if not, verifying how many times he / she has submitted in previous calls. The ratio of first-time users is relatively high: only about 41% of PIs who submitted the most recent proposals were recurrent applicants to a PRACE Call for Proposals for Project Access, meaning that more than half of the project proposals are submitted by new users. This indicates that PRACE is attracting new PIs continuously, while remaining an essential support for existing users. The upward trend of the ratio of recurrence is visible, particularly from the 6th Call onwards (reaching 57% at the 11th Call), influenced by the downward trend on awarded projects. PRACE is implementing, under the PRACE 2 programme, measures to push proposals improvement with resubmission: proposals not reaching a minimum scientific excellence threshold according to PRACE peer-review will not be allowed to re-submit the proposal for a year.
Figure 2 – Ratio of Principal Investigator inter-Call recurrence and related trend line.
Two-thirds (63%) of the resources awarded under the Early Access Call through the 13th Call, are awarded to “foreign projects” (i.e. projects with PIs from a different country (recorded as the primary institution of the PI) than the machine on which the research is executed). The ratio of awarded foreign projects remains rather stable over time (Figure 3). This suggests that the nationality of the PIs institution does not impact the chances of a project being awarded and reflects that the PRACE Peer Review Process works with scientific excellence as its main criterion. It also demonstrates PRACE’s impact in the enhancement of European and International collaboration.
Figure 3 – Ratios of awarded ‘foreign’ projects (blue) and resources for awarded ‘foreign’ projects (orange).
National and international co-funding for PRACE-awarded projects show a downward trend (Figure 4), despite the clear increase in the 10th Call for national co-funding. National cofunding represents the most prevalent form of self-support for the projects awarded by PRACE, overlapped only by EC cofunding in the 9th Call. EC co-funding exhibits a gentle overall upward trend.
The increase of EC support for the projects awarded by PRACE illustrates the outcomes of EC funding policies, aligned with the support to HPC, as key enabler technology.
Figure 4 – Ratios of awarded projects with National, EC and International support; and related trend lines.
PRACE’s impact on scientific production
Since 2008 PRACE has engaged to provide top-class education and training for computational scientists in Europe through the PRACE Advanced Training Centres (PATC), the International HPC Summer School, and PRACE Seasonal Schools, with a clear increase of participants registered (Figure 5).
An overall increase can be seen, plateauing in 2016. As this indicates that the maximum capacity of the current PRACE training offering has been reached, preparations are underway to open PRACE Training Centres in 4 PRACE Member Countries, as a second layer of training and education, below the PATCs.
PATC courses, Seasonal Schools and the International HPC Summer School are offered free of charge to eligible participants.
Figure 5 – Number of person-days registered at PRACE Training days between 2008 and 2016
Between August 2008 and December 2016, PRACE has provided 29 595 participant-days of training through attendance-based courses, with an upward attendance trend. PRACE courses were attended by 7 350 unique individuals. This shows the effectiveness of PRACE in attracting, training and retaining competences.
The six PRACE Advanced Training Centres are Barcelona Supercomputing Centre (Spain), CINECA – Consortio Interuniversitario (Italy), CSC – IT Center for Science Ltd. (Finland), EPCC at the University of Edinburgh (UK), Gauss Centre for Supercomputing (Germany) and Maison de la Simulation (France).
The average rate of recurring participation in training is of 30%. This excellent ratio proves that PRACE trainings are not a closed circuit where the majority of attendees are the same people attending repeatedly. It also shows enough recurrence to indicate the attractiveness of PRACE training courses.
In 2016 the number of participants attending PATCs courses was 1670 (1317 from academia and 353 from non-academia affiliation). More than 78% of participants attending PATCs trainings days have academic affiliation (1317), illustrating the impact of such event on research and scientific communities, in particular for early stage researchers and PhD students.
A clear difference of attendance is observed between the first and second semester of 2016. As observed in Figure 6, the total number of attendances registered in the first semester (first and second quarters) is significantly higher than during the second semester (third and fourth quarters). This indicates that the bulk of the training offering occurs in the first semester, but the rate of attendance remains high throughout the year.
Figure 6 – Number of person-days registered at PRACE Training days in 2016
PRACE’s impact on attracting the industrial sector
Industrial visitors of the PRACE booth at ISC and SC
The interest of industry in PRACE at high-level international events has increased steadily over the past years (Figure 7). The total number of industrial participants showing interest in PRACE during the two main HPC events (Supercomputing (SC) in the USA, and the International Supercomputing Conference (ISC) in Germany) between 2008 and 2016 was 1 243 unique individuals.
Figure 7 – Number of industrial attendees that made contact with the PRACE booth at ISC and SC; and related trend line.
More than half of the companies that visited the PRACE booth at ISC’16 and SC16 were first-time visitors. This indicates that industrial interest in PRACE is growing on both sides of the Atlantic.
Industrial participants in PATCs
The average participation of industry in PATC trainings is 14.24% between 2012 and 2016 (22% in 2016). The increasing interest from industry in participating in HPC training is visible in Figure 8. Over 260 industrial participants were trained by PRACE. Eligible industrial participants enjoy the same service as academic trainees and can attend PATC courses free of charge.
Figure 8 – Industrial participation in PATCs training days; and related trend line.
Industrial use of PRACE HPC resources
PRACE opened its Calls for Proposals to industrial applicants in mid-2012. Industrial participation can take the form of a project led by a principal investigator coming from an industrial enterprise, or a researcher from industry collaborating in an academia-led project. The reduction and stabilisation of projects awarded after the 7th Call has a strong impact on the number of projects awarded with industrial participants (Figure 9). In other words: industry suffers more from the increased competition for PRACE resources than academia.
Regarding SHAPE pilot, PRACE can report 10 success stories of SMEs from 6 different countries benefitting not only from PRACE HPC resources but more importantly, from the know-how in the PRACE centres. The Second SHAPE Call awarded 11 more projects and the Third SHAPE Call awarded another 8, and the SHAPE Fourth Call an additional 4. The SHAPE Fifth Call opens on 31 March 2017.
Figure 9 – Industry participation in PRACE.