PRACE Preparatory Access – 38th cut-off evaluation in September 2019

Find below the results of the 38th cut-off evaluation of 3 June 2019 for the PRACE Preparatory Access.

Projects from the following access types:

 

Turbulence-resolving simulations of wind farm aerodynamics

Project Name: Turbulence-resolving simulations of wind farm aerodynamics
Project leader: Dr Sylvain Laizet
Research field: Engineering
Resource awarded: 50 000 core hours on MareNostrum hosted by BSC, Spain
Description

In this proposal for Preparatory Access the scalability and the performance of the Computational Fluid Dynamics solver called Incompact3d (www.incompact3d.com) are investigated on MareNostrum, hosted by Barcelona Supercomputing Center (Spain). This high-order flow solver is able to perform turbulence-resolving simulations of turbulent flows based on the incompressible Navier-Stokes equations. It is a finite-difference solver which is based on a powerful 2D domain decomposition (http://www.2decomp.org/). It has lead to excellent parallel efficiency and it is used routinely for production runs on several thousands of cores in France, in Brazil and in the UK. The aim of this project is to generate turbulence-resolving simulations of a full-scale wind farm during operation.
top

Monte Carlo Simulation for the ATLAS Experiment at the CERN LHC (II)

Project Name: Monte Carlo Simulation for the ATLAS Experiment at the CERN LHC (II)
Project leader: Dr. Andres Pacheco Pages
Research field: Fundamental Physics
Resource awarded: 50 000 core hours on MareNostrum hosted by BSC, Spain
Description

The ATLAS experiment at the CERN Large Hadron Collider is currently collecting large sample of data. The rate is expected to increase significantly in the future. The analysis of the data relies on the precise simulation of the proton-proton collisions, as well as a detailed simulation of the response of the large high-granularity ATLAS detector. Large samples of simulated data are required for precise analysis capable of separating small signals from large backgrounds. In this project, we will install the ATLAS mandatory software packages and produce test samples relevant for analysis carried out by the IFAE ATLAS group. We will test that the connection between BSC and IFAE’s Worldwide LHC Computing Grid infrastructure for ATLAS is operative for software download, database configuration and upload of the results.
top

Metals in Proteins

Project Name: Metals in Proteins
Project leader: Dr Marco Pagliai
Research field: Chemical Sciences and Materials
Resource awarded: 100 000 core hours on MARCONI hosted by CINECA, Italy, 50 000 core hours on MareNostrum hosted by BSC, Spain and 100 000 core hours on Piz Daint hosted by CSCS, Switzerland
Description

Metalloproteins are essential for life and are involved in several biological processes with a fundamental role in both human health and diseases. Therefore, the study of both the structure and the dynamic and thermodynamic properties of metalloproteins is an essential aspect to elucidate the structural, regulatory, or catalytic roles of metals in biological systems and to provide useful information in medicine. Molecular dynamics (MD) simulations are a valuable computational tool to achieve this information at atomic level, but their accuracy is strongly related to the availability of a force field (FF). A new parametrization strategy has been developed to determine the FF for zinc-proteins [1] which allows to obtain structural properties in agreement with experiments and to compute with success the dissociation constants. The latter are a property of essential importance to understand the metal selection and speciation in proteins. The aim of this project is to develop and to assess the accuracy of the FF for a series of other metal cations of physiological importance in determining both structural properties and dissociation constants of metalloproteins. The project is extremely ambitious and requires a huge amount of quantum mechanical calculations and molecular dynamics simulations, which are planned to be performed during a PRACE Project Access. To properly select the HPC system, it is essential to carry out several tests employing computational chemistry programs for both quantum mechanical (CP2K and NWChem) and classical (GROMACS and ORAC) methods on systems characterized by a different hardware architecture. [1] M. Macchiagodena, M. Pagliai, C. Andreini, A. Rosato, P. Procacci, J. Chem. Inf. Model, submitted for publication.
top

Algebraic multigrid domain decomposition (AMG-DD) on GPUs

Project Name: Algebraic multigrid domain decomposition (AMG-DD) on GPUs
Project leader: Dr. Wayne Mitchell
Research field: Mathematics and Computer Sciences
Resource awarded: 100 000 core hours on Piz Daint hosted by CSCS, Switzerland
Description

Algebraic multigrid (AMG) is a widely used scalable solver and preconditioner for large-scale linear systems resulting from the discretization of a wide class of elliptic PDEs. While AMG has optimal computational complexity, the cost of communication has become a significant bottleneck that limits its scalability as processor counts continue to grow and as GPU acceleration becomes more prevalent on modern machines. This project aims to examine the scalability of a novel, communication-avoiding algorithm, Algebraic Multigrid Domain Decomposition (AMG-DD). The goal of AMG-DD is to provide a low-communication alternative to standard AMG V-cycles by trading some additional computational overhead for a significant reduction in communication cost. Previous work on CPU clusters has shown that AMG-DD converges with similar accuracy but significantly less communication (both in number of messages and total volume of data) compared with AMG. Actual observed speedup was modest at best, however, due to the dominance of computational cost compared to communication cost on the small- to medium-scale CPU clusters used. AMG-DD is expected to yield significantly better speedup on large GPU clusters such as Piz Daint, which have far greater computational capacity and thus are more communication limited. The scaling studies accomplished in this project will be used to determine the regimes in which AMG-DD outperforms AMG and to guide further algorithm development and optimization of AMG-DD.
top

ParStoBig: ParSMURF Scaling to Big data

Project Name: ParStoBig: ParSMURF Scaling to Big data
Project leader: Prof. Giorgio Valentini
Research field: Biochemistry, Bioinformatics and Life sciences
Resource awarded: 100 000 core hours on MARCONI hosted by CINECA, Italy
Description

Several prediction problems in Computational Biology and Genomic Medicine are characterized by both big data as well as a high imbalance between examples to be learned. For instance, deleterious or pathogenic variants are overwhelmed by the sea of neutral variants in the non-coding regions of the genome: as a consequence the prediction of deleterious variants is a very challenging highly imbalanced classification problem, and classical prediction tools fail to detect the rare pathogenic examples among the huge amount of neutral variants or undergo severe restrictions in managing big genomic data. To overcome these limitations we developed and implemented parSMURF: “Parallel SMote Undersampled Random Forests”, a method that adopts a hyper-ensemble approach and oversampling and undersampling techniques to deal with imbalanced data, and parallel computational techniques to both manage big genomic data and significantly speed-up the computation. The synergy between Bayesian optimization techniques and the parallel nature of parSMURF enables efficient and user-friendly automatic tuning of the hyper-parameters of the algorithm, and allows specific learning problems in Genomic Medicine to be effectively fit. Moreover, by using MPI and machine learning ensemble techniques, parSMURF can manage big data by partitioning them across the nodes of a High Performance Computing cluster. Preliminary results, using the Marconi A1 cluster at CINECA, with synthetic data, with Mendelian diseases data and with GWAS hits, involving millions of examples, show that parSMURF achieves state-of-the-art results and a speed-up of 80× with respect to the sequential version. A limitation of this initial approach is that we restricted ourselves to a relatively small number of features (26 in the case of Mendelian diseases) associated with single nucleotide variants (SNV) potentially responsible or associated with a specific pathology. In this project we would like to significantly enlarge the space of potentially informative features associated with SNVs, using omics features freely downloadable from the ENCODE project or using the public portal of the International Human Epigenomic Consortium. In this way we think to enlarge by two orders of magnitude the space of the features to be investigated and used for the prediction of pathogenic SNVs. This in turn requires the application of advanced Machine Learning tools able to deal with big data and to scale-up with data characterized by millions of examples and thousands of features. The main objectives of the project are the following: 1) Application of parSMURF to big omics data, where a large number of features will be investigated and used to predict pathogenic variants. We expect to obtain breakthrough models able to achieve a significant advance in state-of-the-art prediction of pathogenic variants in Mendelian and complex genetic diseases. 2) The release of a highly parallel parSMURF application able to scale with big data and to fully exploit the High Performance Computing architectures available at CINECA for relevant prediction problems in the context of Personalized and Precision Medicine. These objectives require relevant computing resources and to actually quantify them we need to test parSMURF code with a realistic set-up and an adequate large number of nodes and cores.
top

CoopCat: developing theoretical tools and concepts to understand cooperative effects in catalysis

Project Name: CoopCat: developing theoretical tools and concepts to understand cooperative effects in catalysis
Project leader: Dr Adam Kubas
Research field: Chemical Sciences and Materials
Resource awarded: 100 000 core hours on MARCONI hosted by CINECA, Italy
Description

In the CoopCat group we will aim to provide theoretical tools and concepts to be used in the growing field of cooperative catalysis that due to its borderline character require a unique holistic approach. The use of high-level quantum chemical methods will enable us to provide quantitative data of controllable accuracy that will allow to in silco design of efficient catalysts. The key area of our interests are as follow: embedding techniques for metallic surfaces, cooperativity in homogeneous systems and active site-protein cooperativity.
top

Large-scale dynamos at high magnetic Reynolds number and role of magnetic helicity fluxes.

Project Name: Large-scale dynamos at high magnetic Reynolds number and role of magnetic helicity fluxes.
Project leader: Dr Pallavi Bhat
Research field: Universe Sciences
Resource awarded: 50 000 core hours on Joliot Curie (SKL) hosted by GENCI at CEA, France and 50 000 core hours on SuperMUC hosted by GCS at LRZ, Germany
Description

Astrophysical systems like the Sun, stars and galaxies exhibit large-scale magnetic fields, which are thought to be generated by turbulent dynamo action. Understanding the large-scale spatio-temporal organization of these magnetic fields remains a challenge. An essential ingredient to obtaining large-scale coherence in the magnetic fields is kinetic helicity in the underlying turbulent flow. Such helical turbulent flows may then generate helical large-scale magnetic fields. Additionally, in the presence of shear, the system generates magnetic dynamo waves which are typically used to model the solar cycle. An important feature of astrophysical turbulence is that it operates at extreme parameters, i.e. magnetic Reynolds numbers (Rm). For closed systems with large Rm, magnetic helicity is a well conserved quantity and this implies that the helical large-scale fields evolve only on long resistive timescales which would not match with astrophysical observations, such as that of the 11 year variation of the large-scale fields during the solar cycle. One possible resolution to this issue involves allowing for magnetic helicity fluxes out of the domain. Despite much theoretical discussion, the role played by helicity fluxes in the nonlinear evolution of large-scale dynamos remains poorly understood. Numerical investigations at high Rm are critical to make progress. Though computationally expensive, we will run such high Rm simulations of helical turbulence along with shear in a domain which has open boundaries. We will be assessing the role of fluxes in the dynamical equation for magnetic helicity to understand what effects are important to obtain rapid coherence and organization in the magnetic fields in such a turbulent system. We will also study how these magnetic helicity fluxes affect turbulent transport coefficients which govern the dynamo wave cycle timescales.
top

Type B: Code development and optimization by the applicant (without PRACE support) (3)

Machine Learning Approaches for Diagnosis Breast Cancer

Project Name: Machine Learning Approaches for Diagnosis Breast Cancer
Project leader: Dr. Buse Melis Ozyildirim
Research field: Mathematics and Computer Sciences
Resource awarded: 100 000 core hours on MareNostrum hosted by BSC, Spain
Description

The aim of the study is diagnosis breast cancer by using machine learning approaches on MRI data. By the improvements on technologies, deep learning has become important approach for segmentation and classification. However, deep learning approaches require more training data and the main problem is having small number of labeled data. In this study, both unsupervised and supervised techniques will be used for solving labeling issue, segmentation, and classification. While U-Net approach is utilized for segmentation and classification, small number of labeled data will be used to label unlabeled data by using clustering approaches. U-Net is one of the efficient segmentation approaches used for biomedical images. It consists of autoencoder and convolutional layers. It provides class based segmented images.
top

McSAFESC

Project Name: McSAFESC
Project leader: Dr Eduard Hoogenboom
Research field: Engineering
Resource awarded: 200 000 core hours on Joliot Curie (SKL) hosted by GENCI at CEA, France
Description

McSAFESC is the SuperComputing part of the European Union research project McSAFE (No. 755097; “High-Performance Monte Carlo Methods for reactor SAFEty Demonstration- From Proof of Concept to realistic Safety Analysis and Industry-like Applications”) which aims to move the Monte Carlo (MC) based stand-alone and coupled solution methodologies for nuclear reactor calculations (advanced depletion, optimal coupling of MC-codes to thermal-hydraulic solvers, time-dependent Monte Carlo and methods and algorithms for massively parallel simulations) to become numerical tools for realistic reactor core design, safety analysis and industry-like applications of light-water reactors (LWRs). This involves not only the accurate determination of the power distribution in the reactor with coupling of the MC code to a thermal-hydraulics (TH) code for iteration to the effective temperature distribution in fuel and coolant of the core, but also the time-dependence of the power distribution due to control rod movements in seconds and minutes scale, taking into account the generation and decay of delayed neutron precursors, and the depletion of the nuclei in the fuel on a day and year scale. The stochastic Monte Carlo method handles the neutron transport in an exact way, in contrast with currently applied deterministic codes for reactor calculations, but requires, especially for time-dependent calculations and burnup calculations, even with many variance reduction techniques, simulation of very large quantities N of neutron histories for statistically accurate results, as the standard deviation in each value of the power distribution behaves inversely proportional to SQRT(N). The aim of the SuperComputing project is to optimise the parallel execution of the Monte Carlo codes in order to calculate the detailed power distribution with 0.5 % accuracy (standard deviation) within 1 day wall clock time on a massively parallel computer.
top

Calculation of atomic data for diagnostics of cold astrophysical plasmas.

Project Name: Calculation of atomic data for diagnostics of cold astrophysical plasmas.
Project leader: Dr Luis Fernandez Menchero
Research field: Fundamental Physics
Resource awarded: 100 000 core hours on SuperMUC hosted by GCS at LRZ, Germany
Description

We study modelling and diagnostics of cold astrophysical plasmas, such as nebulae or remnants of supernovae. Low-energy atomic processes are the dominant in those cold plasmas for example photoionization, photoexcitation or dielectronic recombination. Data for such processes are lacking in the principal atomic data bases.
top

 

Type D: Optimisation work on a PRACE Tier-1 (2)

Development of a parallel algorithm ILST for the iterative reconstruction of 3D image based on the MPI system

Project Name: Development of a parallel algorithm ILST for the iterative reconstruction of 3D image based on the MPI system
Project leader: Prof. Sergei Zolotarev
Research field: Mathematics and Computer Sciences
Resource awarded: 150 000 core hours on Tier-1 resources
Description

Effective synchronous parallel computational algorithms will be developed and program codes in the C ++ programming language for industrial tomographic reconstruction under scanning of a controlled object in a conical beam for a circular data acquisition scheme. Debugging of the program code and numerical calculations should be performed on a supercomputer in the MPI system.]
top

Next steps for scalable Delft3D FM for efficient modelling of shallow water and transport processes

Project Name: Next steps for scalable Delft3D FM for efficient modelling of shallow water and transport processes
Project leader: Dr Menno Genseberger
Research field: Earth System Sciences
Resource awarded: 150 000 core hours on Tier-1 resources
Description

Forecasting of flooding, morphology and water quality in coastal and estuarine areas, rivers, and lakes is of great importance for society. To tackle this, the modelling suite Delft3D, was developed by Deltares (independent non-profit institute for applied research in the field of water and subsurface). Delft3D is used worldwide. Delft3D has been open source since 2011. It consists of modules for modelling hydrodynamics, waves, morphology, water quality, and ecology. In two earlier (small) PRACE projects [1, 2] and the FP7 Fortissimo experiment Delft3D as a Service (see for instance example in [3]) steps have been taken (a.o. with SURFsara and CINECA) to make Delft3D modules more efficient and scalable for high performance computing. Currently, for Delft3D there is a transition from the shallow water solver Delft3D-FLOW for structured computational meshes to D-Flow FM (Flexible Mesh) for unstructured computational meshes. D-Flow FM will be the main computational core of the Delft3D Flexible Mesh Suite. For typical real-life applications, for instance for highly detailed modelling and operational forecasting, there is urgency to make D-Flow FM also more efficient and scalable for high performance computing. As the solver in D-Flow FM is quite different from the one in Delft3D-FLOW some major steps have to be taken. Also for the modules for modelling waves, morphology, water quality, and ecology that connect to D-Flow FM. Aim of the current project is to make significant progress towards Tier-0 systems for the shallow water and transport solvers in the Delft3D Flexible Mesh Suite. This project is a continuation of a previous preparatory access type D project [4] carried out by SURFsara, CINECA, and Deltares. Earlier testing did not go beyond a few hundred MPI processes, now we successfully ran representative simulations on up to several thousands of MPI processes. As a rule-of-thumb, mainly due to the PETSc solver that is used, partitions should have more than ~25000 cells to get good parallel efficiency. The scalability depends on the architecture and mesh size of the model, we measured some speedup up to a few thousands of MPI processes and a good efficiency up to a few hundred MPI processes for large meshes. It was very useful to have access to two systems at CINECA and SURFsara with different architectures to compare the behaviour of D-Flow FM, get a good overview of its performance and identify the bottlenecks on Tier-1 and Tier-0 systems. This yielded new insights and it also gave us the opportunity to test possible improvements in D-Flow FM for real-life applications. For the selected test cases from the previous preparatory access type D project, it has become clear which further steps have to be taken to be able to run the software efficiently on the Tier-0 systems. That will be subject of the current preparatory access type D project. [1] http://www.prace-project.eu/IMG/pdf/wp100.pdf [2] http://www.prace-project.eu/IMG/pdf/wp177.pdf [3] https://ir.cwi.nl/pub/24648 [4] http://www.prace-ri.eu/IMG/pdf/WP284.pdf
top