PRACE KPI

PRACE Key Performance Indicators

Given the scale of the computational power in the PRACE portfolio, PRACE related statistics are becoming increasingly important to highlight the impact of PRACE on HPC based research, HPC know-how in Europe, and European Industry engagement in HPC.

In 2014, the PRACE Council approved a set of Key Performance Indicators (KPI) that facilitate the analysis and evaluation of PRACE achievements and successes.

Source: PRACE 2017 Annual Report

PRACE’s impact on evolving research

Offer and demand of resources

Figure 1 shows the evolution of PRACE resources offered andrequested in Project Access Calls. PRACE first provided HPCservices in 2010 with contributions from GermanTier-0 systems.France, Italy and Spain added their contributions gradually. Thisis reflected in the constant increase of HPC resources offeredby PRACE to the scientific community until the 6 th Call, where astable regime was reached. The phasing out of the initial phase ofPRACE (known as PRACE 1) started in the 10 th Call, while the secondphase of PRACE (known as PRACE 2) only started in Call 14. This isreflected in the valley from the 10 th Call to the 13 th Call. With PRACE2, a substantial increase in the amount of available resources canbe seen, thanks to the renewed contributions of all the originalPRACE Hosting Members and Switzerland as a new hostingmember. Since the 14 th Call an average of 1.9 thousand million corehours has been offered to the HPC scientific community.

The demand for HPC resources has always exceeded the capacityof PRACE to provide them. The average oversubscription of PRACECalls is 3:1, reaching a 5:1 ratio during the phasing out of PRACE1. This constant interest in PRACE resources has been driving theperiodic upgrades and additions of new systems that PRACE offers,specifically with regards to the PRACE 2 programme.

Figure 1 – Core hours offered and requested in each PRACE Call

Number of projects

During the initial phase of PRACE (known as PRACE 1), the number ofproject applications received via PRACE Calls for Proposals for ProjectAccess exhibited a clear overall upward trend. The phasing-in ofPRACE 1 naturally incited an increase in demand for Tier-0 resources.This is particularly evident up to the 8 th Call, with a large sustainedincrease between the 6 th and 8 th Call, followed by a slight decrease(Figure 2).

A downward trend of rejected projects below the technical qualitythreshold is noted, displaying maturity of submitted proposals,in which researchers put more effort into the quality of theirproposals, as a reaction to increased competition. Moreover, theevolution reflects the positive outcomes of PRACE PreparatoryAccess Calls (including access type C) that enable prior technicalsupport for application and scalability tests. Figure 2 also highlightsan increase in rejected projects above the scientific threshold,particularly after the 6 th Call. This is correlated with the increase intotal applications.

During the phasing-out of PRACE 1, the number of available corehours dropped (Figure 1), and this decreased the demand, asresearchers anticipated an even stronger competition for theremaining resources. This trend was mitigated in the 12 th and 13 thCall, when PRACE hosting members made additional core hoursavailable during the preparation of the PRACE 2 programme, whichstarted in the 14th Call.

With the start of PRACE 2, there has been an increase in the numberof projects awarded, which combined with the increase of resourcesthat started in the 14 th call (Figure 1) shows the clear success ofthe second phase of PRACE. In this second phase, the scientificobjectives of PRACE have been updated towards an increase ofthe scope and excellence of the projects awarded. The minimumsize of allocations has been increased three-fold, and the scientificthreshold as well. The decrease in proposals submitted and theapparent decrease in their quality in the 15 th Call is a positive sign ofthe success of this scientific update.

Figure 2 –Total number of Projects, awarded and rejected

Despite the competition, demand for PRACE resources remains highand all PRACE calls are oversubscribed (Figure 1), indicating thatscientists consider Tier-0 access an essential asset to their work.This was also underlined by the PRACE Scientific Steering Committee(SSC) during the preparation of PRACE 2 (see page 8 for the article bythe Chair of the SSC)

Recurring users

PRACE also keeps track of the submission of Project Accessproposals by recurrent Principal Investigators (PIs) (Figure 3). This KPI is created by checking for each call if a PI is new to PRACE.The ratio of first-time applicants is relatively high – roughly 30per cent of PIs who submitted to the two Project Access Callsin 2017 were recurrent applicants to a PRACE Call for Proposalsfor Project Access. This means that more than two thirds ofproject proposals are submitted by new users. This indicatesthat PRACE is continuously attracting new PIs, while remainingan essential support for existing users. The upward trend of theratio of recurrence is visible, particularly from the 6 th Call onwards,influenced by the downward trend on awarded projects, butrecovered with the onset of PRACE 2.

Figure 3 – Ratio of Principal Investigator inter-Call recurrence and related trend line

International and transnational cooperation

Two-thirds (63%) of resources awarded under the Early Access Callthrough to the 15 th Call are awarded to “foreign projects”. Foreignprojects are defined as projects with Principal Investigators (PIs)from a different country (recorded as the country of the PI’s primaryinstitution ) than the machine on which the research is executed.The ratio of awarded foreign projects remains rather stable over time (Figure 4). This shows that the nationality of the PI’s institution doesnot influence the chances of a project being awarded, as another signof the scientific excellence selection criterion of PRACE Project Access.

It also demonstrates PRACE’s impact in the enhancement of Europeanand International collaboration.

Figure 4 – Ratios of awarded ‘foreign’ projects (blue) and resources for awarded ‘foreign’ projects (orange)

Co-funding

PRACE awards are normally developed within larger scientific initiatives,where HPC resources are part of the needs of the project. PRACE asksProject Access awardees to declare such synergies.

On average, 75% of PRACE users have declared that their awards are complemented with EC, national or international funds (Figure 5). The major fraction corresponds to nationalprojects, whichhas slowly been showing a downwardtrend since the 10 th Call. EC funding shows an increasingtrend, coinciding with the implementation of the H2020programme. International funding remains low, with 8%being the average contribution.

Figure 5 – Ratios of awarded ‘foreign’ projects (blue) and resources for awarded ‘foreign’ projects (orange)

PRACE’s impact on growing know-how in Europe

Since 2008, PRACE has been engaged in providing top-classeducation and training for computational scientists in Europe throughthe PRACE Advanced Training Centres (PATCs), the International HPCSummer School, and PRACE Seasonal Schools, with a clear increaseof participants registered (Figure 6).

Six PATCs were first established, and these are CSC – IT Center forScience Ltd. (Finland), Maison de la Simulation (France), Gauss Centrefor Supercomputing (Germany), CINECA – Consortio Interuniversitario(Italy), Barcelona Supercomputing Centre (Spain), and EPCC at theUniversity of Edinburgh (United Kingdom).

After the rapid increase between 2010 and 2012, a plateau isevident since 2012. As this indicated that the maximum capacityof PRACE training offerings had been reached, four new PRACETraining Centres were opened in 2017 – in IT4Innovations (CzechRepublic), GRNET (Greece), ICHEC (Ireland) and SURFSara(Netherlands). This acts as a second layer of training andeducation, below the previous PATCs.

PATC / PTC training events, Seasonal Schools and the InternationalHPC Summer School are offered free of charge to eligible participantsfrom academia and industry.

Figure 6 –Number of person-days registered at PRACE Training days between 2008 and 2017

Between August 2008 and December 2017, PRACE provided 36 273participant-days of training through attendance-based courses, withan upward attendance trend. PRACE courses were attended by over 11500 individuals. This shows the effectiveness of PRACE in attracting,training and retaining competences.

In 2017 the number of participants attending PRACE courses was 1858 (1 487 from academia and 371 from non-academia affiliation).More than 80% of participants attending trainings days haveacademic affiliation (1487), illustrating the impact of such event onresearch and scientific communities, in particular for early stageresearchers and PhD students.

A clear difference of attendance is observed between the firstand second semester of 2017. As observed in Figure 7, the totalnumber of attendances registered in the first semester (Q1 and Q2)is significantly higher than during the second semester (Q3 andQ4). This indicates that the bulk of the training offered occurs in thefirst semester, with a notable drop in attendance during Q3 whichcorresponds with the summer vacation period.

Figure 7 –Number of person-days registered at PRACE Training days in 2017

PRACE’s impact on attracting the industrial sector

The interest of industry in PRACE at high-level international eventshas increased steadily over the past years (Figure 8). The totalnumber of industrial participants showing interest in PRACE duringthe two main HPC events (Supercomputing (SC) in the USA, andthe International Supercomputing Conference (ISC) in Germany) between 2008 and 2017 was 1 707 individuals.

More than half of the companies that visited the PRACE booth at ISC17and SC17 were first-time visitors. This indicates that industrial interestin PRACE is growing on both sides of the Atlantic.

Figure 8 – Number of industrial attendees that made contact with the PRACE booth at ISC and SC; and related trend line

Industrial participants in PTCs

The average participation of industry in PTC trainings is 15.68%between 2012 and 2017 (19.7% in 2017). The increasing interest fromindustry in participating in HPC training is visible in Figure 9. Over 360 industrial participants were trained by PRACE in 2017. Eligibleindustrial participants enjoy the same service as academic traineesand can attend PTC courses free of charge.

Figure 9 – Industrial participation in PTCs training days, and related trend line

Industrial use of PRACE HPC resources

PRACE opened its Calls for Proposals to industrial applicants mid-2012. Industrial participation can take the form of a project ledby a Principal Investigator coming from an industrial enterprise,or a researcher from industry collaborating in an academia-ledproject. The reduction and stabilisation of projects awarded afterthe 7 th Call has had a strong impact on the number of projectsawarded with industrial participants (Figure 10). In other words,industry suffers more from the competition for PRACE resourcesthan academia. This trend is starting to change with the start ofPRACE 2.

Figure 10 – Industry participation in PRACE

Regarding the SHAPE pilot, PRACE can report 10 successstories of SMEs from six different countries benefitting notonly from PRACE HPC resources but, more importantly, fromthe know-how in the PRACE centres. The second SHAPE Callawarded 11 more projects, the third SHAPE Call another eight,the fourth SHAPE Call an additional four, and the fifth SHAPECall (which opened in March 2017) another six.