PRACE Preparatory Access – 15th cut-off evaluation in December 2013

Find below the results of the 15th cut-off evaluation of December 2013 for the PRACE Preparatory Access Call.

Type A – Code scalability testing

Project name: Optimization and Scalability testing of a new OpenFoam application

Project leader: Prof. Bejo Duka; University of Tirana, ALBANIA
Collaborators: MSc. Klaudio Peqini, MSc. Ergys Rexhepi; University of Tirana – AL
Research field: Engineering and Energy
Resource awarded: 100,000 core hours on Fermi;



Abstract: OpenFoam is an open source CFD software package used to solve numerically not only different problems of Fluid Dynamics and Heat Transfer but a wide range of problems governed by PDE systems. OpenFoam uses the Finite Volume Method that approximates the PDE into a system of algebraic equations in terms of discrete quantities defined at specific locations of the domain (mesh). OpenFOAM frequently needs to store large sets of data. By running OpenFoam in existing supercomputers, it is possible to obtain accurate results for many kinds of problems.

We have used OpenFoam and the visualization software named “Paraview” to simulate different kinds of fluid flow, where different models of turbulence have been considered.

Actually, the OpenFoam comes with different known application cases ready to be used in parallel computing. We have modified one of these cases named “HotRoom” and used the “BuoyantBoussinesqPimpleFoam” solver for numerically solving the problem of Natural Convection in cylindrical and spherical annuli. We have tested this modified application and studied the scalability in a cluster with limited capacities. By running the application for different time step and cell size values of discretization, in a large system such as CINECA center (Fermi Machine), the project-proposal aims to define the appropriate values of discretization for a high accuracy of numerical results. This would be the process of the optimization of our application. The second aim of the project-proposal is to test the scalability and to obtain the scalability plots which will be used as supporting information for a future PRACE project.

Project name: Large scale simulation of light propagation in disordered media

Project leader: Prof Claudio Conti; University Sapienza, ITALY
Research field: Fundamental Physics
Resource awarded: 100,000 core hours on Fermi; 100,000 core hours on Juqueen; 100,000 core hours on SuperMUC;



Abstract: We will solve by a Finite Difference Time Domain parallel code the fully vectorial nonlinear Maxwell equations in large scale disordered systems obtained by assembly of dielectric particle. The aim is to simulate structure with spatial size of the order of tens of micros for a time scale of the order of the nano-second at optical frequencies. This will allow to test the onset of the Anderson localization of light in three dimensions and the way it is affected by optical nonlinearity.

Project name: Modeling large scale conformation changes in membrane with a multiple walker algorithm.

Project leader: Dr. Christophe Chipot; CNRS – Université de Lorraine, FRANCE
Collaborators: Dr. Jeffrey Comer; CNRS – Université de Lorraine – FR
Research field: Chemistry and Materials
Resource awarded: 50,000 core hours on Curie Hybrid Nodes; 50,000 core hours on Curie Thin Nodes (TN); 100,000 core hours on SuperMUC;



Abstract: Large scale movements in complex biological systems usually span time scales that are not currently amenable to standard brute force molecular-dynamics simulations. To access the molecular details underlying large conformational changes in proteins, a popular strategy consists in biasing the trajectory along a given reaction coordinate (distance, dihedral, vector field, root-mean-square-deviation, …), the latter being either mono- or multi-dimensional. Among different approaches, the Adaptive Biasing Force (ABF) has proven to be effective in determining potentials of mean force (PMF) associated to various biological processes. When transformation are slow and complex, full sampling remains however difficult to achieve, obliterating the proper convergence of PMFs. To overcome this issue, a multiple walker strategy has been recently implemented by our group in NAMD. In this project, we aimed at benchmarking this new facility of NAMD to handle conformational transitions associated to the function of membrane proteins such as carriers.

Project name: Large scale MD simulations of super mitochondrial complexes.

Project leader: Dr François Dehez; CNRS – Université de Lorraine, FRANCE
Collaborators: Mr. Daniel Bonhenry; CNRS – Université de Lorraine – FR
Research field: Chemistry and Materials
Resource awarded: 50,000 core hours on Curie Hybrid Nodes; 50,000 core hours on Curie Thin Nodes (TN); 100,000 core hours on SuperMUC;



Abstract: Molecular-dynamics (MD) simulations have proven to be a powerful technique to complement structural information gain by experimental methods (X-ray crystallography, NMR, Electron Microscopy) by allowing the study of the dynamics of large biomolecular assemblies in physiologically relevant environments. The project aims at benchmarking the popular MD package NAMD for modeling ultra large membrane protein assemblies composed of several million atoms on Tier0 machines.

The main aim here is to obtain scalability plots, which compare the performance of hybrid CPU/GPU and thin nodes architectures, and can be used as supporting information when applying to future PRACE project.

Project name: Translocation of antibiotics through bacterial porins

Project leader: Dr Susruta Samanta; University of Cagliari, ITALY
Collaborators: Prof. Matteo Ceccarelli; University of Cagliari – IT
Research field: Medicine and Life Sciences
Resource awarded: 50,000 core hours on Curie Thin Nodes (TN); 100,000 core hours on SuperMUC;



Abstract: The urgent need of new antibiotics to combat multidrug resistant bacteria and re-emerging pathogens demands a new method to develop/identify a new generation of antibiotics. While gram-n
egative bacteria can resist to antibiotics decreasing their permeability, a reliable technique to screen molecules for their penetration efficiency does not exist. Then, the identification of physical/chemical properties of molecules for an efficient permeability through porins is a key issue to define new scaffolds.

The proposed study is a part of our efforts to understand the translocation of antibiotics through the outer membrane of gram negative bacteria using all atom molecular dynamics simulations. The goal of the preparatory stage is to test the efficiency of the molecular dynamics (MD) simulation package GROMACS/PLUMED on large supercomputers, and to optimize the distribution of computational power fit for the proposed simulation systems.

As a benchmark problem, we choose to study the dynamics of a few specific porins found in the outer membrane of the gram negative bacteria Pseudomonas aeruginosa using the “Replica Exchange Molecular Dynamics” (REMD) simulation method coupled with metadynamics. REMD method demands substantial amount of computational power since it includes running several parallel simulations at different temperatures for the proper sampling of the protein dynamics. On the other hand, when coupled with metadynamics, it allows the sampling of the many degrees of freedom not explicitly declared in the metadynamics.

In the preparatory stage we will benchmark the multilevel parallelization approach and optimum computational protocol for our systems on a large supercomputer. The outcomes of this stage will support a PRACE request in a regular call.

Project name: First principles investigations of multiferroic magnetic oxides

Project leader: Dr. Jose Lorenzana; ISC-CNR, ITALY
Collaborators: Mr Marcello Balestieri, Dr Valentina Brosco, Dr Johan Hellsvik, Mr Andrea Rovere, Mr Daniel Bucheli, Mr Matteo Capati; ISC-CNR – IT, Sapienza University of Rome – IT
Research field: Chemistry and Materials
Resource awarded: 100,000 core hours on SuperMUC;



Abstract: Materials involving active d-orbitals, like magnetic oxides, are often in an intermediate coupling regime where neither the Coulomb interaction nor the kinetic energy of the electrons dominates. This leads to a variety of complex phases with interesting phenomena like colossal magneto-resistance, high-temperature superconductivity and magneto-optic activity. During the last decade interest in multiferroics and magnetoelectricity gave yet another boost to research on these materials.

For first-principles calculation of multiferroic oxides, the Vienna Ab initio Simulation Package (VASP) has established itself as a widely used and general tool. Efficient structural relaxation of the internal coordinates, and optionally also of the lattice vectors, are important in investigating polar zone centre phonons which can contribute to the ferroelectric polarization in a multiferroic compound. In comparisons with full potential all-electron codes, VASP enables such calculations to be performed at a much lower computational effort, but yielding results which are of close to equal accuracy.

The individual VASP calculations that we want to pursue will be intermediate to large in size, with supercells ranging from 60 up to 200 atoms. The smaller cells are sufficient for collinear spin configurations of undoped compounds. The larger cells are necessary for two reasons: 1. The case of spin configurations incommensurate with the crystal lattice. For the family of DFT programs to which VASP belongs, one has to restrict the spin-configurations to be commensurate with the supercell. In order to approximate the spin spiral it can therefore be necessary to extend the supercell along the propagation vector of the spiral (or the closest lying lattice vector). 2. Addressing the low concentration limit of chemical doping requires large cells.

The purpose of the present application is to explore and tune the performance of VASP on the SuperMUC system, which we believe has a suitable architecture for this code. Whereas parallelization over bands and plane wave coefficients have been available in VASP since long time back, k-point parallelization was implemented more recently. The emphasizes of our benchmarks will be to find the best trade off between different parallelization for the type of representative problems described above. The benchmarks results will guide us in the choice of target system for the upcoming 9th PRACE project access call.

Project name: Scalability of turbulent jet simulations

Project leader: Prof. Carlos B. da Silva; Instituto Superior Técnico, PORTUGAL
Research field: Engineering and Energy
Resource awarded: 50,000 core hours on MareNostrum; 5,000 core hours on MareNostrum Hybrid Nodes; 50,000 core hours on Hermit; 100,000 core hours on Juqueen;



Abstract: The project aims at assessing and improving the scalability of a recently upgraded code for the simulation of spatially developing turbulent jets using direct numerical simulations – DNS. Recent research work published by the applicant in several journals (e.g. Journal of Fluid Mechanics, vol. 685, 2011, vol. 695, 2012, and Physics of Fluids, vol. 20, 2008, vol. 21, 2009, vol. 22, 2010, vol. 25, 2013) used temporal simulations of turbulent jets. Since spatial simulations are more realistic it would be desirable to move to this type of simulation. A spatially simulation code already exist for this. This code is a major upgrade of a similar code used before by the applicant and detailed in several journal publications (e.g. Journal of Fluid Mechanics, vol. 473, 2002). The applicant has experience in running the present code in the Lonestar and Ranger machines of the Texas Advanced Computing Centre (TACC), but more information is needed in order to improve the code the carry out massive new simulations. For this purpose the applicant needs to test the scalability in several supercomputer architectures such as Cray, IBM, INTEL XEON and XEON PHI. The aim is to run the code in simple/very short simulations using a large number of cores in order to make the scalability test/assessment. If weaknesses are detected in the present code, the algorithm will be improved.

Project name: Accurate computational prediction of lipophilic fluorophores

Project leader: Dr. Himanshu Khandelia; University of Southern Denmark, DENMARK
Collaborators: Dr. Lukasz Cwiklik; Academy of Sciences of the Czech Republic – CZ
Research field: Medicine and Life Sciences
Resource awarded: 50,000 core hours on Curie Hybrid Nodes; 50,000 core hours on Curie Thin Nodes (TN);



Abstract: The goals of this project are to (1) accurately predict the localization and orientation of fluorescent lipophilic membrane probes in complex lipid mixtures (2) calculation of the emission behavior of the probes taking into account the local environment and (3) design new probes with improved environment sensitivity.

A large portion of our current knowledge about the structure, organization, dynamics, phase behavior and phase separation in mod
el and cell membranes composed of phospholipids comes from detection of probe fluorescence in bilayers. Most of these lipophilic fluorescent probes are environment sensitive. Laurdan detects a local change in hydration (or polarity) and therefore can distinguish between ordered and disordered phases in bilayers 1. Recently, a new laurdan derivative was used to visualize raft domains 2, 3. Similarly, PRODAN4, DiIC18 and Bodipy-PC are often used to label disordered and ordered phases in model giant liposomes. The emitted fluorescence depends both on the local molecular environment of the probe and therefore it’s partitioning into the two lipid phases. The accurate interpretation of the emission response of the probes in bilayers depends on two factors that are nearly impossible to obtain from experimental measurements alone. Firstly, the probe’s emission depends on the orientation and partitioning of the probe in the membrane bilayer, in a specific (ordered versus disordered phase), because this determines the local probe environment and in particular, its hydration level. Secondly, because the probe emission depends on its environment, the emission intensity and wavelength can only be predicted from excited-state ab-initio calculations taking into account the local environment. In this project, we will employ large-scale all-atom molecular dynamics (MD) simulations of different fluorescent probes in model phase-separating lipid mixtures to accurately predict the orientation and partitioning of the probes in different phases. We will also employ state-of-the-art QM/MM calculations to determine fluorescent behavior of the probes taking into account the influence of the local environment including lipids and water.

Successful implementation of the simulations will (1) significantly enhance our knowledge of lipid phase behavior in model lipids mixtures and eventually of lipid reorganization in cell membranes and (2) will enable the design of more sensitive and improved fluorescent probes.


  1. T. Parasassi, G. De Stasio, A. d’Ubaldo and E. Gratton, Biophys. J., 1990, 57, 1179-1186.
  2. H. M. Kim, H. J. Choo, S. Y. Jung, Y. G. Ko, W. H. Park, S. J. Jeon, C. H. Kim, T. Joo and B. R. Cho, ChemBioChem, 2007, 8, 553-559.
  3. E. Sezgin, H. J. Kaiser, T. Baumgart, P. Schwille, K. Simons and I. Levental, Nature protocols, 2012, 7, 1042-1051.
  4. P. L. Chong, Biochemistry, 1988, 27, 399-404.

Project name: IMI ND4BB- Translocation

Project leader: Mr. Tommaso D’Agostino; University of Cagliari, ITALY
Collaborators: Mrs Silvia Acosta Gutiérrez, Prof. Matteo Ceccarelli; University of Cagliari – IT
Research field: Medicine and Life Sciences
Resource awarded: 50,000 core hours on MareNostrum; 50,000 core hours on Curie Thin Nodes (TN); 50,000 core hours on Hermit; 100,000 core hours on SuperMUC;



Abstract: In the last years, many well-known gram negative pathogens of the enterobacteriaceae family (Escherichia Choli, Klebsiella Pneumoniae, Providencia Stuartii) are showing reduced antibiotic susceptibility, resulting in increased resistance to medical treatments. One of the main mechanisms for increasing resistance is to change the permeability of the outer bacterial membrane, by reducing or increasing the expression of different porins that share high sequence identities but are less likely to be permeated by antibiotics. As part of a large project (ND4BB- IMI), we are studying the permeability and characterizing the dynamics of small molecules that undergo translocation through membrane protein channels of pathological bacteria. The aim of the project is to use simulations in order to characterize permeation of different chemical compounds through porins of gram-negative bacteria: by comparing the results for various known antibiotics we can predict the characteristics that a molecule has to have for successful translocation in specific porins. We can use this data for establishing new rules to develop a new generation of drugs with enhanced permeability to overcome the increasing bacterial resistance that many pathogens are showing with respect to the outer membrane

We plan to test Gromacs code on OmpSt1 from P.Stuartii, as well as some other porins, modeled in an explicit phospholipid bilayer. We will investigate the translocation of various chemical compounds to test the scalability of the approach.

Contribution of our group to this topic:
- K. R. Mahendran, E. Hajjar, T. Mach, M. Lovelle, I. Sousa, A. Kumar, E. Spiga, H. Weingart, P. Gameiro, M. Winterhalter, M. Ceccarelli: Molecular basis of enrofloxacin translocation through a bacterial porin: When binding does not imply translocation, J. Phys. Chem. B 114, 5170-5179, 2010
- E. Hajjar, A. Kumar, P. Ruggerone, M. Ceccarelli: Investigating reaction pathways in rare events simulations of antibiotics diffusion through protein channels, J. Mol. Mod., 16: 1701-1708, 2010
- A. Kumar, E. Hajjar, P. Ruggerone and M. Ceccarelli: Molecular Simulations Reveal the Mechanism and the Determinants for Ampicillin Translocation through OmpF, J. Phys. Chem. B, 114, 9608-9616, 2010
- E. Hajjar, A. Bessonov, A. Molitor, A. Kumar, K. R. Mahendran, M. Winterhalter, J.-M. Pagès, P. Ruggerone, M. Ceccarelli: Towards screening for antibiotics with enhanced permeation properties through bacterial porins, Biochemistry, 49: 6928-6935, 2010
- H. Lou, M. Chen, S. S. Black, S. R. Bushell, M. Ceccarelli, T. Mach, K. Beis, A. Low, V.A. Bamford, I.R. Booth, H. Bayley and J.H. Naismith, Altered antibiotic transport in OmpC mutants isolated from a series of clinical strains of multi-drug resistant E.coli, PLOS ONE, 6: E25825, 2011
- P.R. Singh, M. Ceccarelli, M. Lovelle, M. Winterhalter, K. R. Mahendran, Antibiotic Permeation across the OmpF Channel: Modulation of the Affinity Site in the Presence of Magnesium, J. Phys. Chem. B, 2012, 116: 4433-44383

Project name: Towards viscoelastic anisotropic seismic imaging

Project leader: Dr Benjamin Pajot; Université Joseph Fourier, FRANCE
Research field: Earth Sciences and Environment
Resource awarded: 50,000 core hours on Curie Thin Nodes (TN);



Abstract: Seismic imaging has many applications in the fields of civil engineering, risk study, resources exploration, waste storage, and fundamental knowledge of geodynamic processes at various scales. Full Waveform Inversion (FWI) is a particular seismic imaging method taking advantage of the whole information contained in the seismic data. It allows to recover subsurface parameters, such as P and S wave velocities, density, attenuation and anisotropy.

Performing FWI requires to solve a non-linear minimization problem in a way that needs a number
of modelling of the wave propagation proportional to the number of seismic sources generating the data. For 3D real cases, the size of the models and the number of seismic sources to consider make that most of the applications are performed in the isotropic acoustic approximation.

In the end, we want to benefit from the computational power increase and the last generation of processors to reach viscoelastic anisotropic multi-parameter inversion for real cases. We will tackle such cases in the next PRACE calls.

Project name: Analysing the effect of mutations on the energy landscape of influenza fusion peptide using bias-exchange metadynamics simulations

Project leader: Dr Cláudio Soares; Instituto de Tecnologia Química e Biológica – Universidade Nova de Lisboa, PORTUGAL
Collaborators: Dr Diana Lousa; Instituto de Tecnologica Química e Biológica-Universidade Nova de Lisboa – PT
Research field: Medicine and Life Sciences
Resource awarded: 50,000 core hours on MareNostrum;



Abstract: Influenza is one of the most devastating human pathogenic viruses. This virus is covered by a lipid bilayer and it must fuse the viral and host membranes to infect cells. This is accomplished through the action of hemagglutinin, a glycoprotein, which is attached to the viral bilayer. Fusion is a crucial step in influenza infections and inactivating this process is one of the most promising strategies in the development of antiviral drugs. Thus, there is a tremendous interest in elucidating the molecular details of the influenza fusion process.

This project focuses on the analysis of the influenza fusion peptide (FP), which is located on the N-terminal region of HA and comprises 20 aa residues. The FP plays a major role in the fusion process, because it inserts into the host membrane destabilizing it. This peptide is highly conserved and several mutations have been shown to affect its activity (Cross et al., 2009, PPL, 16, 766-778). To understand how mutations affect the behaviour of this peptide, we are using a molecular simulation approach, together with experimental studies, to study the wild type and several HA FP mutants.

Given that we are not able to properly sample these systems with standard MD, we are performing bias-exchange metadynamics (BE-MetaD) simulations (Piana et al., 2007, JPCB, 111, 4553-4559), which considerably enhance sampling, both in water and in a DMPC membrane. This will enable us to compare the energy landscape of the mutant and wild type peptides. The BE-MetaD simulation studies are being performed in collaboration with Professor Alessandro Laio, from SISSA, Italy, who was one of the developers of this method. Our preliminary BE-MetaD simulations of the FP in a DMPC membrane indicate that sampling this system is very difficult, even using BE-MetaD, due to the large viscosity of the membrane. A possible solution to this problem is to use a large number of CVs (and, thus, a large number of replicas) because this enables a much more efficient exploration of the conformational space. Using PRACE resources will enable us to use a large number of replicas, with several cores per replica, which will allow the sampling of the energy landscape of this system in a reasonable time frame. Thus, we are submitting this proposal to test the scaling of BE-MetaD simulations of the FP in a membrane with 528 DMPC molecules.

The simulations will be performed with the GROMACS package, version 4.0 (Hess et al., 2008, JCTC, 4, 435-447), with a patch to PLUMED 1.3 (Bonomi et al., 2009, CPC, 180, 1961-1972). We aim to make tests with different numbers of replicas (from 10 to 30) and using different numbers of cores per replica, in order to analyse the performance and test the convergence of the simulations in these conditions.

Project name: Drug delivery controlled release by electric fields

Project leader: PhD Francesca Apollonio; University Sapienza of Rome, ITALY
Research field: Engineering and Energy
Resource awarded: 100,000 core hours on Fermi;



Abstract: The mechanism through which a highly intense electric field acts on polymeric nanocarriers loaded with a cargo inside them, is a new research issue. The project will provide the physico-chemical basis to comprehend how an electric field of high intensity and short duration in time may modify the molecular conformation of the carrier, determining the exit of the payload carried inside, thus configuring a controlled drug delivery. The approach used is based on classical molecular dynamics (MD) techniques. The final aim is to provide the threshold for electric field intensities necessary to achieve drug delivery controlled by nsPEF.

The idea to combine the use of nanocarriers, suitably designed to respond to external electric fields, and the technology based on nsPEFs, that are high in power and low in energy, arises from a long standing research on a new challenge for drug delivery that is stimuli-response release.

The overwhelming majority of reports in the literature describe stimuli-responsive systems that are sensitive to only a few common triggers, including changes in pH, temperature, and electrolyte concentration [de las Heras Alarcon C. et al., Chem. Soc. Rev., 2005, 34, 276285, 2005].

More recently attention has been focused on recent results and future trends that exploit stimuli that have not yet been as heavily considered [Roy D. et al., Progress in Polymer Science, Volume 35, Issues 1-2, 278-301, 2010]. On this basis, an extremely interesting perspective for controlled drug delivery is based on tailoring smart carriers sensitive to external electric fields, in particular nsPEFs, similar to those which produce reversible electroporation on cells [R. Nuccitelli, et al., Int. J. Cancer, vol. 27, pp. 17271736, 2010].

The activity will be divided in three steps: a first characterization of the molecular target chosen, an aqueous zwitterionic micelle formed by monomers of an alkyl-N,N-dimethylamine N-oxide(CH3-(CH2)n-NO(CH3)2, ADAO (990 atoms), about 4nm diameter . The micelle will be studied adding more than 10.000 SPC water molecole in order to have a dilute system. The aim of this first step will be to extrapolate geometrical properties of the target together with stability behaviour.

Successively a typical nanosecond pulse of electric field will be applied to the minimized configuration of the micelle in water. The duration of the pulse will be the one used for typical nsPEF on cells: 60-100 ns. The analysis for this case will refer to possible modifications of the geometrical shape due to the presence of the field.

Finally, a chemotherapy drug, the fluorouracil (5-FU) will be inserted inside the vesicle and again 60-100 ns simulations will be repeated with varying electric field intensities. Fluorouracil belongs to the class of chemotherapy drugs known as anti-metabolites. It interferes with cells making DNA and RNA, which stops the growth of cancer cells. Diffusion coefficients will be evaluated trying to understand if the electric field applied may be able to modify the way out of the solute from the vesicle, thus performing a controlled delivery.

Project name: Numerical Investigation of Active Scalars in Turbulent Flows at High Schmidt Number


Project leader: Prof. Vincent E. Terrapon; University of Liege, BELGIUM
Research field: Fundamental Physics
Resource awarded: 100,000 core hours on Fermi; 50,000 core hours on Hermit; 100,000 core hours on Juqueen; 100,000 core hours on SuperMUC;



Abstract: Turbulent flows are prevalent in many natural systems and often include additional complex physics (e.g. heat transfers, combustion, pollutants dispersion, magneto-hydrodynamics (MHD), multi-phase flows, non-Newtonian rheology, etc.). This additional physics is typically modeled through additional transport equations of some scalars supplementing the Navier-Stokes equations (e.g. temperature in natural convection).

Of particular interest in this project is the case where the scalar diffusion is much lower than the momentum diffusion by viscosity (i.e. for a high Schmidt/Prandtl number). In this case, length scales smaller than Kolmogorov scale (i.e. the smallest length scale of the velocity field) are created by the advection of the flow. Furthermore, if the scalar is active (i.e. it feeds back into the momentum transport equation), these sub-Kolmogorov scales of the scalar can modify the flow field and have an important impact on the large-scale dynamics of the flow.

For instance, the phenomenon of early transition observed in some polymeric solutions is induced by small-scale polymer instabilities. These small scales instabilities induce a transition to turbulence at lower Reynolds number than in Newtonian flows and thus impact the large-scale dynamics of the flow. Therefore, predicting accurately these small scales and their dynamics is of major importance to accurately compute the flow field and the related quantities of interest. Numerous applications of active scalar advection at high Schmidt number can be found, such as MHD, which models the interactions between a magnetic field and a viscous incompressible fluid of electrically charged particles, viscoelastic turbulence and polymer drag reduction, which deals with nonlinear fluid rheology or natural convection.

However, the impact of the sub-Kolmogorov scales of active scalars on the large-scale flow dynamics is still not well understood, and there is no general theory similar to Batchelor’s theory for active scalars. Moreover, the creation of the sub-Kolmogorov scales increases the already stringent resolution requirements for computing turbulent flows, rendering the simulation of such flows very challenging.

The aim of the research project is to better understand the characteristics of sub-Kolmogorov scales and how they impact the large-scale dynamics of the flow. More precisely, we would like to answer following questions: how is the flow field modified by the scalar sub-Kolmogorov scales? what is the exact process of energy transfer and interaction between the different scales of the scalar and the velocity? are sub-Kolmogorov scales created in the velocity and/or pressure fields? Is there any universality in this process across different types of active scalar, i.e. different physics?

Project name: Direct numerical simulation of adverse pressure gradient boundary layer control

Project leader: Asistant Professor Ayse Gungor; Istanbul Technical University, TURKEY
Collaborators: Dr Omid Amili, Dr Callum Atkinson, Dr Vassili Kitsios, Professor Julio Soria, Mr Paul Stegeman, Asist. Prof. Ayse Gungor; Monash University – AU, Istanbul Technical University – TR
Research field: Engineering and Energy
Resource awarded: 50,000 core hours on Hermit; 100,000 core hours on SuperMUC;



Abstract: This research project will investigate the structure and dynamics of wall-bounded turbulence in adverse pressure gradient (APG) environments using Direct Numerical Simulation (DNS). An APG is a pressure gradient that decelerates the flow leading to potential flow separation from the aerodynamic surface. The DNS program will be complimented by both theoretical studies, and by experiments in the water tunnel at the Laboratory for Turbulence Research in Aerospace and Combustion water tunnel at Monash University. The target Reynolds numbers based on the momentum thickness is 2000.

The long term objectives of this research are to revolutionize the design of energy generation and transport platforms that operate in APG environments, and to develop active flow control systems that increase the operational envelope. This will improve the energy efficiency over a wide range of operating conditions. Real world examples include the flow over aircraft wings, wind turbine blades, and any form of turbo-machinery. Improvements in the performance of such systems will lead to more efficient and cleaner power generation, a reduction in fuel consumption, and minimization of CO2 emissions. In the present study our efforts are focussed on the canonical flow configuration of a flat plate turbulent boundary layer (TBL) subjected to an APG.

The research program involves firstly the DNS of a self-similar equilibrium APG TBL at the verge of separation as described in Skate & Krogstad (1994) up to a Reynolds number of 2000. This will require the careful application of the farfield boundary conditions to ensure we achieve the desired equilibrium state. The sensitivity of this state to a wall mounted jet perturbation will be determined to ascertain the physical mechanisms of wall turbulence. The optimal jet forcing geometry, location and operating condition that retains an attached TBL using minimal energy will be determined. The results will be post-processed to study the evolution of the coherent structures, with attention paid to their dynamics in the outer and middle region, and how they interact with structures in the inner region layer. Each region is defined by its distance from the wall.

The research program will address the following questions:

  1. How are the middle and outer regions influenced by the structures of the inner region in APG wall bounded flows? This is important for wall mounted sensing and control of the APG TBL.
  2. If there is an influence of the middle and outer flow by the structures of the inner region, what is the mechanism? Is it possible for actuators to affect the middle and outer region by acting upon near wall structures?
  3. What is the origin, structure and evolution of coherent structures in the APG equilibrium TBL at the verge of separation? Are local instabilities due to inflectional mean velocity profiles present? If so can they be used to optimise control of separation?

The preparatory access will focus on setting up the initial DNS of the self-similar APG TBL.

Skare,P.E. & Krogstad, P.-A., 1994, “A turbulent equilibrium boundary layer near separation”, J. Fluid. Mech., Vol. 272, pp319-348.

Project name: Computational Studies of the MECHANICAL PROPERTIES of Actin Filaments

Project leader: Dr. Frauke Graeter; Heidelberg Institute for Theoretical Studies, GERMANY
Research field: Medicine and Life Sciences
Resource awarded: 100,000 core hours on Fermi; 100,000 core hours on SuperMUC;



Abstract: The cytoskeleton constitutes a complex network of scaffolding elements that insure the structural integrity of
cells through the formation of protein assemblies. Actin plays a crucial role in promoting vital cellular functions as it is assembled in filaments by specialized proteins, which are thought to function as molecular motors and to catalyze actin polymerization through the addition of actin monomers at one end of the nascent filament, defined as barbed end. Such proteins are called formins and provide a remarkable example of coupling the function of structured and unstructured protein domains. Indeed, formins mainly consist of two domains, FH1 and FH2. The FH1 domain is intrinsically disordered and binds actin-cargo proteins, such as profilin, that transport new soluble actin monomers to the barbed end of the actin filament. On the other hand, the FH2 domain is fully structured and binds the barbed end of the actin filament adding the actin monomers transported by the FH1 domain. The FH2 domain is shaped as a toroid that surrounds the actin filament and moves along the filament after the stepwise addition of actin monomers1. X-ray crystal structures have demonstrated that the FH2 domain can exist in two separate conformations: a closed conformation, in which the protein is incapable of adding actin monomers to the filament, and an open conformation, in which the two monomers composing the domain are rotated of 33° with respect to each other. Interestingly, it has been shown that the ability of formins to promote the oligomerization of new actin filaments is influenced by the application of an external force, in line with the observation that the assembly of the cellular cytoskeleton needs to occur in relation to changes in the microenvironment surrounding the cell. Therefore, such molecular machineries must work as mechano-sensors. Nevertheless, it remains difficult to identify the mechanism through which force influences the process of actin polymerization. Remarkably, predictions about the behavior of formins under the effect of applied force have been completely contradicted by experimental findings. As an example, it has been found that the application of shear force on the FH1/FH2-actin complex inhibits actin polymerization, suggesting that force shifts the conformational equilibrium of the FH2 domain towards the closed state.

The proposed project aims at elucidating the mechanism through which the effect of force can inhibit actin polymerization by switching conformational equilibrium of the FH2 domain towards a closed state. By means of Molecular Dynamics simulations, an actin filament and an actin dimer bound to the full FH1 and FH2 domains of the FMLN3 formin from rabbit will be subjected to shear and pulling forces. Simulations will be of crucial importance to determine the mechanism by which formins can be regulated by the effect of external force, and will provide details into how this occurs at an atomic scale. Our proposed mechanism will be testable by experiments and will critically guide future experiments to decipher the mode of function of this intriguing molecular machinery.

Project name: Computational Investigation of Binding- Release Dynamics in Intracellular Iron Carrier Proteins

Project leader: Prof. Canan ATILGAN; Sabanci University, TURKEY
Collaborators: Miss. Haleh Abdizadeh; Sabanci University – TR
Research field: Chemistry and Materials
Resource awarded: 100,000 core hours on Juqueen;



Abstract: The unique ability of iron to serve both as an electron donor and acceptor renders this metal irreplaceable for various physiological and metabolic pathways. Vital biochemical activities, including oxygen transport, energy production and cellular proliferation depend on iron-containing cofactors, such as heme, or iron-sulfur clusters. However, excess iron may become toxic due to its ability to catalyze the generation of free radicals and may damage cellular macromolecules. Transferrins (Tf) form a family of glycoproteins whose main function is to control the level of free iron in physiological fluids by tightly binding this element, but readily releasing it at endosomal pH[1]. Bacteria, in return, target Tf specifically and liberate iron from Tf at neutral pH[2]. In particular, bacterial ferric binding protein (FBP) is a structural homolog of a single lobe of human serum Tf and coordinates iron with a similar local motif. In this project, we propose to study the dynamics of these two proteins which, upon iron binding, undergo structural changes compared to their unbound forms. We aim to gain insight into these processes with a range of computational tools.

To this end, one needs to specify the conditions under which iron is bound or released. Using computational approaches, we shall define the mechanisms that are active in binding-release of iron under different environmental conditions such as pH and ionic strength. At the scale of the protein, the effect of the environmental conditions on protein conformations shall be determined via extensive molecular dynamics (MD) simulations. We select a series of initial conditions where ionic strength and pH are varied, and/or point mutations are introduced.

We employ the Perturbation-Response Scanning (PRS) methodology, developed in our group, to determine key residues whose perturbation lead to the expected conformational change[3]. This method characterizes the response of proteins to a given perturbation on systematically selected residues. The approach relies on coarse-graining protein structures as residues represented by their C_alpha atoms and employs linear response theory to record relative changes in the residue coordinates in response to sequentially applied directed forces on single-residues. Through PRS we determine the regions where point mutations shall be introduced, and study if the conformational change is actually induced by acting on remote sites in all-atom MD simulations. We have previously studied point mutants of calmodulin, a 148 residue protein serving a plethora of functions. Therein, structurally important residues were determined by PRS, and alanine mutations were shown to affect conformation distributions[4]. Here, we wish to extend this approach to much larger, multi-domain proteins.

[1]AN Steere et al., “Kinetics of Iron Release from Transferrin Bound to the Transferrin Receptor at Endosomal pH,” Biochimica et Biophysica Acta, 1820, 326-333(2012).

[2]N Noinaj et al., “Structural Basis for Iron Piracy by Pathogenic Neisseria,” Nature, 483, 53-58(2012).

[3]C Atilgan, AR Atilgan, “Perturbation-Response Scanning Reveals Ligand Entry-Exit Mechanisms of Ferric Binding Protein,” PLoS Comput. Biol., 5, e1000544(2009).

[4]AO Aykut, AR Atilgan, C Atilgan, “Designing Molecular Dynamics Simulations to Shift Populations of the Conformational States of Calmodulin,” PLoS Comput. Biol., accepted(2013).

Project name: Interactions between internal tides and mesoscale dynamics and impact for future high resolution satellite altimetric missions.

Project leader: Dr. Aurélien Ponte; Ifremer, FRANCE
Collaborators: Dr. Patrice Klein, Mrs Sylvie Le Gentil; Ifremer – FR
Research field: Earth Sciences and Environment
Resource awarded: 50,000 core hours on Curie Thin Nodes (TN);



Abstract: Oceanic mesoscale eddies and submesoscale structures (generated by instabilities of the large scale ocean circulation), with horizontal scales of 1 to 200 km, are known to control exchanges
between the interior and the surface of the ocean, to contribute to ocean mixing, and to play an important role for the physical, climatic and biochemical functioning of oceans. More specifically, the recent ability to perform numerical simulations with a resolution never attained before has led to highlight and quantify the significant dynamical impact of submesoscales (1 km- 50 km) on larger oceanic scales and on the physical-biological interactions. This is one of the major advances in physical oceanography of the last 8 years, a field in which our group has contributed (in cooperation with scientists from IPSL (Paris) and JAMSTEC (Japan)) through more than 30 publications in international journals.These new numerical and theoretical results are one the main arguments for the development of SWOT, a wide-swath altimeter satellite mission led by the Centre National d’Étude Spatiale (CNES, France) and NASA, that should capture the sea surface height (SSH) with a resolution ten times higher than the conventional altimeters.This satellite mission will provide an unprecented view of the fine scale ocean dynamics and its impact on the larger-scale dynamics. Our project concerns internal tides, another class of motions whose range of scales is close to that of mesoscale and submesoscale turbulence. While progresses have been made to understand each of these two classes of motions separately, a fundamental question remains: how do both classes of motions interact ? This scientific question has rarely been addressed even though it has recently become a priority for the development of SWOT. The interactions between internal tide and mesoscale and submesoscale dynamics represent a difficulty that needs to be overcome. The goal of this project is to provide a better understanding for these interactions and to assess their signature on the SSH. One phenomenon of particular interest is the intermittency of internal tides, observed but unexplained, that directly affects our ability to predict the internal tides and to estimate its signature on altimetric observations. This project will contribute at explaining this intermittency and in particular how the interactions between the internal tide and mesoscale and submesoscale fluctuations affect this intermittency and the larger oceanic scales.

Project name: Analysis of cycle variations in a multi-cylinder Diesel engine with LES and detailed chemistry

Project leader: Mr Olivier Davodet; PSA Peugeot Citroen, FRANCE
Collaborators: Dr Wolfgang Schwarz, Dr Marc Chauvy, Mr Patrick Madea, Mr Joël Perou; AVL France SA – FR, PCA – FR, PCA – Peugeot Citroen – FR
Research field: Engineering and Energy
Resource awarded: 50,000 core hours on Curie Thin Nodes (TN);



Abstract: 3D Combustion Simulations provide a potential route to increase the understanding of cycle-to-cycle variations (CCV) due to combustion instabilities.The main factors underlying the CCV are generally considered to be the dilution rate and intake port induced flow dynamics (which accelerate combustion but generate pressure losses induced by the swirl ducts and therefore reduce the volumetric engine efficiency). However the large number of involved phenomena (turbulence, chemistry, gas-dynamics) and their coupling make these CCV difficult to quantify. Because of the complexity of the relation between all these phenomena, Large Eddy Simulations (LES) is generally used.

In addition, conclusions of previous works on this subject have highlighted two important points for modeling the CCV. On the one hand, the number of cycles is an important factor in order to catch the cyclic variations: it has been shown that for an instable point, the statistics of the flow are obtained within about 50 engine cycles. On the other hand, a more detailed chemistry like ECFM3Z (Extended Coherent Flame Model) or the use of resolution of detailed chemistry are needed for the modeling of CCV in combination.

Thus, to produce realistically simulations extreme computing resources are required. In this preparatory project we will explore the ability of the commercial code Fire (from AVL) to simulate the complete loop of a four piston Diesel engine on the Curie system. In other way, we target to evaluate the scalability of the framework in order to prepare a future PRACE regular call submission.

We emphasize that the LES approach with solving detailed chemistry and using complete EGR loop allows a complete engine model (there are currently very few complete models like this, especially with the EGR loop) and get answers about the understanding of CCV responses (we do not have numerical models sufficiently complete to have a clear understanding of the CCV problem at the present time). Generally this approach is not used due to the large necessary amount of CPU resources.

Type B – Code development and optimization by the applicant (without PRACE support)

Project name: iTesla (Innovative Tools for Electrical System Security within Large Areas)

Project leader: Mr Christian Lemaître; RTE Réseau de Transport d’électricité, FRANCE
Collaborators: Mr Olivier Bretteville,Mr Jean-Baptiste Heyberger, Mr Geoffroy Jamgotchian; RTE Réseau de Transport d’électricité – FR
Research field: Engineering and Energy
Resource awarded: 200,000 core hours on Curie Fat Nodes (FN);



Abstract: The iTesla project is a collaborative research project carried out by a consortium of 20 European partners and supported in large part by EU funding in the framework of the FP7 programme.The consortium members are the following:
- 6 national operators of electricity transmission systems: RTE (France), Elia (Belgium), National Grid (UK), REN (Portugal), Statnett (Norway) and IPTO (Greece),
- 1 Regional coordination service centre: Coreso (Belgium),
- 6 universities and research centres: Imperial College (UK), INESC Porto (Portugal), KTH (Sweden), KU Leuven (Belgium), RSE (Italy), DTU (Denmark),
- 7 industrial R&D providers: AIA (Spain), Artelys (France), Bull (France), Pepite (Belgium), Quinary (Italy), Tractebel Engineering (Belgium), Technofi (France).The project is coordinated by RTE, the French transmission system operator (TSO). It started in January 2012 for a period of 4 years.

Security issues of the pan-European electricity transmission system are likely to become more and more challenging in the coming years due to:
- the growing contribution of less predictable and intermittent renewable energy sources (wind and photovoltaic generation),
- the introduction of new controllable devices such as HVDC lines,
ht=’11’ class=’puce’ alt=”-” style=’height:11px;width:8px;’ /> a partially controllable electricity demand,
- the increasing difficulty to build new overhead transmission lines,
- the progressive construction of a single European electricity market.

These new constraints but also new opportunities will result in more complex system operation, a grid working closer to its operational limits and therefore a need for a major revision of operational rules and procedures. In this context, it is clear that currently available tools for security assessment will no longer be suitable for network operators to take the right decisions. Furthermore, coordinated operation initiatives have already emerged for different regions of the pan-European transmission system (such as Coreso for instance). These coordination initiatives will not be fully efficient without a new generation of tools allowing the different TSOs to increase coordination.

The main goal of the iTesla project is to develop a novel toolbox able to support the operation of the pan-European grid in the coming years and to validate the different functionalities of this toolbox with datasets of increasing complexity and size. More precisely, this toolbox will support the decision-making process from two-days ahead to real time and will take up the three main following challenges:
- to perform accurate security assessment taking into account the dynamics of the system using time-domain simulations,
- to provide a risk-based assessment taking into account the different sources of uncertainties (in particular, those brought by intermittent power generation), the probabilities of contingencies and the possible failures of corrective actions,
- to provide operators with relevant proposals of preventive and curative actions to keep the system in a secure state (such as generation redispatching, change in transformer tap position, topology of substations, set point values of HVDC lines or phase shift transformers).

This toolbox will be designed to be used by a single TSO, by a coordination center such as Coreso or by a group of TSOs working in a coordinated way.

Project name: Brainomics

Project leader: Dr Edouard Duchesnay; CEA, FRANCE
Collaborators: Mr. Benoit Da Mota; AS+ – Groupe EOLEN – FR
Research field: Medicine and Life Sciences
Resource awarded: 100,000 core hours on Curie Hybrid Nodes;



Abstract: The last two decades have seen tremendous advances in our understanding of human brain structure and function, particularly at the level of systems neuroscience, where neuroimaging methods have led to better delineation of brain networks and brain modules. Even more striking advances have been reported in molecular genetics research, where the Human Genome Project has provided a first estimate of the human genome, including the total number of genes and their chromosomal locations, and with the development of functional genomics. Yet, despite important progress in both molecular genetics and neuroimaging research, there has been relatively little integration of the two fields (Hariri and Weinberger, 2003).

Clearly, this integration of genetics and functional genomics information into neuroimaging methods promises to significantly improve our understanding of a given brain disease. It should lead to the development of biomarkers and in the future personalised medicine.

However, a successful investigation requires several challenges to be met:

  1. manage complex, high dimensional, and large data,
  2. develop the statistical methods that are needed to extract the relevant information,
  3. develop the software components that will permit large computation to be done, and
  4. organise the leading by specific applications with expert clinical partners.

These four components are challenges addressed by Brainomics project.

We gather academic and industrial partners to tackle these four aspects of imaging genetics integration. We believe that imaging genetics will make fast and significant advances only if appropriate tools are designed, constructed, and proposed to the scientific community. These tools will also be of relevance for pharmaceutical companies.

We therefore plan to develop the following components. First, a data management system will integrate neuroimaging, phenotypic clinical or behavioural- information, and genetics or genomics data. In our experience, it is not enough that these complex and large data be well organized on the disks: a relevant query system should be proposed to help researchers quickly access the data on which statistical analysis has to be performed. Access to the genetic / genomic information available as web resources (NCBI, EBI, Kegg etc.) is also essential to link with the data available from a specific cohort. Once the relevant data are extracted and that often requires several complex queries they have to be analysed with the relevant statistical methods, which often involve cross validation or permutation techniques and therefore large computing resources. Last, these analyses have to be tailored and put in the context of a specific research (e.g. Addiction, Schizophrenia, brain tumours).

Project name: T7.1C-3IP: Final enabling of the applications associated with Safe, Environmental-friendly energy production and Future aircraft transportation.

Project leader: Mr Maciej Szpindler; ICM University of Warsaw, POLAND
Research field: Engineering and Energy
Resource awarded: 100,000 core hours on MareNostrum; 200,000 core hours on Curie Fat Nodes (FN);



Abstract: The project is the final enabling phase for applications associated with socio-economic challenges in Energy and Engineering. This is PRACE WP7-3IP task performed by PRACE partners. Challenges addressed are “Safe and Environmental-friendly energy production” and “Future aircraft transportation”. First effort is enabling of the application suite supporting optimal placement of marine and wind turbines for energy production. The focus is on the suite applications parallel coupling for improved overall performance and scaling for selected Tier-0 systems. Second approach is to improve fluid-structure simulations for aircraft designs. Final enabling will include benchmarking of the improved solver, removal of the identified bottlenecks and industrial show-cases preparation.

Project name: T7.1C-3IP: Final enabling of the applications associated with Rational drug design, Sustainable food supply and Multiscale modelling of the human cells and organs.

Project leader: Mr Maciej Szpindler; ICM University of Warsaw, POLAND
esearch field:
Medicine and Life Sciences
Resource awarded: 200,000 core hours on Curie Fat Nodes (FN);



Abstract: The project is addressing final enabling phase for applications associated with socio-economic challenges in Life Sciences, Cardiac Modelling and Genetics. This is PRACE WP7-3IP internal activity performed by PRACE partners. Challenges addressed are “Rational drug design”, “Sustainable food supply” and “Multiscale modelling of the human cells and organs”. Project focus is on either application scaling improvement for large complex molecules (drug design) or on multi-discipline coupling in one application with two or more different solvers targeting scalability for Tier-0 systems. Another approach for genetic data processing (clustering, alignment, sequencing) is to develop application work-flow for large number of genomes scalable and applicable for a large HPC systems.

Project name: T7.1C-3IP: Final enabling of the application tools for “Big data” management and processing.

Project leader: Mr Maciej Szpindler; ICM University of Warsaw, POLAND
Research field: Mathematics and Computer Science
Resource awarded: 250,000 core hours on Juqueen;



Abstract: The project is addressing the final enabling phase for applications associated socio-economic challenges. This is PRACE WP7-3IP internal activity performed by PRACE partners. Challenge addressed is “Big data management and processing”. The goal of the project is to enable Peta-scale data mining with MapReduce paradigm on HPC systems and show its usability.

Project name: T7.1C-3IP: Final enabling of the applications associated with Understanding of climate change and Natural environment protection.

Project leader: Mr Maciej Szpindler; ICM University of Warsaw, POLAND
Research field: Earth Sciences and Environment
Resource awarded: 250,000 core hours on Fermi; 100,000 core hours on Curie Hybrid Nodes;



Abstract: The project aim is the final enabling phase for applications associated socio-economic challenges in Climate and Environment Studies. This is PRACE WP7-3IP internal task performed by PRACE partners. Challenges addressed are “Understanding of climate change” and “Natural environment protection”. The project focus is on enabling climate chemistry code with a GPU accelerated chemistry solvers that would greatly improve the performance of climate chemistry modelling for Climate Change studying. Second application to be enabled on PRACE Tier-0 systems is addressing interactive Lake Modelling for natural environment protection.

Project name: Optimization of the NEMO oceanic model in the GLOB16 configuration

Project leader: Prof. Giovanni Aloisio; Euro Mediterranean Center on Climate Change, ITALY
Collaborators: Ph.D. Silvia Mocavero, Ph.D. Italo Epicoco; Euro Mediterranean Center on Climate Change – IT, University of Salento – IT
Research field: Earth Sciences and Environment
Resource awarded: 100,000 core hours on MareNostrum; 250,000 core hours on SuperMUC;



Abstract: The NEMO oceanic model is based on the Navier-Stokes equations along with a nonlinear equation of state, which couples the two active tracers (temperature and salinity) to the fluid velocity. The partial equations involved are solved through the finite difference numerical method. Two elliptic solvers are implemented for computing the free surface pressure. All of the numerical solvers are encapsulated in the code, without using external libraries. The code is written in Fortan 90 and parallelized using MPI. The low resolution of the global ocean models used today for climate change studies limits the prediction accuracy. NEMO has a maximum horizontal resolution of 1/12° with 50 vertical levels. To overcome this limit, a new high-resolution global model, based on NEMO, simulating at 1/16° and 100 vertical levels has been developed at CMCC. The model is computational and memory intensive, so it requires many resources to be run. Goal of the project is to improve the scalability of the parallel code, optimizing its execution time. The optimization methodology requires a preliminary analysis to highlight scalability bottlenecks.

Preliminary insights on the code scalability has been performed on a SandyBridge architecture at CMCC. An efficiency of 48% on 7K cores (the maximum available) has been achieved. The project aims at defining suitable strategies to overcome the scalability bottlenecks to improve the parallel efficiency. The analysis has been also carried out at routine level, so that the improvement actions could be designed for the entire code or for the single kernel. The analysis highlighted for example a loss of performance due to the routine used to implement the north fold algorithm (i.e. handling the points at the north pole of the 3-poles Grids): indeed an optimization of the routine implementation is needed. However, the parallelization strategy could be also reviewed to better exploit the new emerging manycore architectures, where a hybrid parallelization approach could provide more benefits. Moreover, an alternative I/O strategy could be investigated, taking into consideration parallel I/O techniques as the use of the XIOS library for overlapping computation with I/O operations.

Project name: Development and scalability analysis of multilevel domain decomposition codes on IBM BG/Q

Project leader: Prof. Santiago Badia; CIMNE (Universitat Politecnica de Catalunya), SPAIN
Collaborators: Postdoctoral researcher Alberto F. Martin, Prof. Javier Principe; CIMNE (Universitat Politecnica de Catalunya) – ES
Research field: Mathematics and Computer Science
Resource awarded: 250,000 core hours on Juqueen;



Abstract: The design of breeding blankets in a fusion reactor is one of the most challenging technological problems for the development of fusion energy. Unfortunately, experimental set-ups will not be at our disposal during the next decade, and numerical tools that provide realistic simulations are necessary. In this proposal, we aim at developing efficient large-scale magnetohydrodynamic solvers for the forthcoming many-core supercomputers.

This preparatory access can be considered as a final step on the road to a fully distributed-memory, highly scalable implementation of the multilevel BDDC solver within FEMPAR (the numerical software being developed at our group), and its initial performance and scalability assessment on the IBM BG/Q massively parallel processor JUQUEEN.

In order to reach such objective, a clear development plan, which is currently in a quite advanced stage, is being followed. This s
chedule specifies at different levels of abstraction, how the design/implementation of the distributed-memory data structures and codes has to be carried out in order to support both sub-communicator awareness and recursion. Both techniques are basic to extend our novel implementation ideas of the two-level BDDC preconditioner to a multilevel setting. Indeed, we expect that a 3-level (at most 4-level) BDDC, combined with our novel implementation ideas, will be able to efficiently exploit the high degree of parallelism available on JUQUEEN.

Project name: Latency-centric communication metrics for unsymmetric sparse-matrix vector multiplies

Project leader: Prof. Cevdet Aykanat; Bilkent University, TURKEY
Collaborators: Mr Reha Oguz Selvitopi; Bilkent University – TR
Research field: Mathematics and Computer Science
Resource awarded: 100,000 core hours on MareNostrum; 50,000 core hours on Hermit; 250,000 core hours on SuperMUC;



Abstract: Various iterative algorithms consist of repeated sparse-matrix vector multiplies (SpMxV) that include sparse, unsymmetric square or rectangular matrices. These computations are performed either on the matrix itself or its transpose. For instance, the iterative methods CGNE (conjugate gradient normal equation error), GGNR (conjugate gradient residual method) and QMR (Quasi-Minimal Residual Error) perform repeated SpMxVs that involve computations on an unsymmetric square matrix and its transpose. The LSQR (least squares method) and the Lanczos method (used for computing singular value decomposition) also require the same type of computations on a rectangular matrix. These iterative solvers are widely used kernel operations in solving problems from different domains.

For parallelization of these solvers, generally one-dimensional (1D) intelligent partitioning methods are adopted. These methods utilize either rowwise or columnwise partitioning techniques which usually aim to minimize total/maximum message volume. The literature overlooked the importance of other communication metrics. According to our extensive analysis on various modern supercomputers, these 1D techniques do not scale beyond 256/512 processors, which makes them infeasible for solving very large sparse linear system of equations. Our findings indicate that the fundamental factor that hinders scalability of these solvers is actually the message latency, and not the message volume. We observed that the message latency becomes the dominant factor in the communication performance of the algorithm, especially with increasing number of processors.

This project aims to scale the above-mentioned iterative solvers that specifically utilize unsymmetric matrices (square or rectangular). The literature lacks combinatorial models and extensive experiments that center around minimizing message latency of the parallel algorithms designed for solvers that perform repeated SpMxVs with unsymmetric matrices. We already have devised models and methods to reduce the latency of the parallel algorithms that use 1D partitioning techniques. Our method consists of two stages for minimizing multiple communication-cost metrics. Basically, our approach strives for minimizing message latency in the later stage without much increasing the message volume. Besides minimizing total message latency, we also have models and techniques ready to minimize other crucial metrics such as maximum message latency and maximum message volume.

We wish to test our approaches on different architectures to show their validity. We believe that the scientific codes that utilize the mentioned solvers can greatly benefit from the proposed algorithms for partitioning the matrices used on these solvers and scale their code without hassles since our method only incurs a low runtime preprocessing phase and requires no changes to the solvers themselves.

Project name: Scalable state-of-the-art DFT and Green’s function methods to assess pseudo-magnetic effects in strained graphene.

Project leader: Dr Simon Dubois; Université catholique de Louvain, BELGIUM
Collaborators: Dr. Andrés Botello-Méndez, Prof. Jean-Christophe Charlier,Dr Simon Dubois,Prof. Peter Haynes,Dr. Chris-Kriton Skylaris; Université catholique de Louvain – BE, Imperial College London – UK, University of Southampton – UK
Research field: Chemistry and Materials
Resource awarded: 200,000 core hours on Curie Fat Nodes (FN); 250,000 core hours on Juqueen;



Abstract: The coupling between the mechanical and electronic properties of graphene is of interest both from the point of view of fundamental physics and applications. Based on a continuum model within a Dirac Hamiltonian for graphene, it has been predicted that deformations could lead to Landau-like levels in the absence of a magnetic field. These arise from the breaking of degeneracy of the sub-lattice symmetry of graphene, referred to as pseudo-spin, and results in a effective vector potential that mimics the effect a magnetic field.

The experimental observation of such a phenomenon encourages going beyond the simple continuum model from which it was predicted. Instead of analytical and/or semiempirical solutions, a more realistic description of the mechanical-electronic coupling should be achieved through first principles (e.g., Density Functional Theory – DFT). However, to tackle such a problem using common first-principles approaches is difficult because of the number of atoms involved (i.e, tens of thousands of atoms). The need of both state-of-the-art linear scaling methods and access to massively parallel computers are therefore essential.

In practice, the rationalization of the effect of pseudo-magnetic fields on the electronic properties of graphene is carried out in two steps. In a first step we use the ONETEP code in order to estimate the ground-state Hamiltonian of strained graphene devices. This method allows us to get electronic and structural properties in very good agreement with experiment. In a second step, we assess the impact of the strain-induced vector potential on the transport properties of the considered devices within the framework of the Landauer-Buttiker formalism. This requires the estimation of the retarded Green’s function associated with the ground-state Hamiltonian over a restricted energy range corresponding to low-energy carriers.

The aim of this project is to develop and test efficient and scalable algorithms to estimate the retarded Green’s function of very large systems (e.g., 2000 to 60000 atoms). Algorithms under consideration include recursive Green’s functions, Krylov based, and Gauss decimation methods.

Project name: Investigation of the Applicability of Map/Reduce Paradigm on Supercomputing Systems for Multi-Petascalability

Project leader: Mr Kadir Akbudak; Bilkent University, TURKEY
Collaborators: Prof. Cevdet Aykanat; Bilkent University – TR
Research field: Mathematics and Computer Science
Resource awarded: 250,000 core hours on Juqueen;



Abstract: In
this PRACE-3IP project of Task 7.2, we aim to investigate the applicability and scalability of Map/Reduce parallel programming paradigm on supercomputers. We target at scalability analysis of PageRank algorithm, which has attracted interest of the European scientific and engineering research community. We use MR-MPI library, which is a lightweight Map/Reduce implementation developed in C++ and it uses the MPI library for inter-process communication. We obtain super linear speedup for 512 cores due to increase of total available memory. If more nodes are used, then it is possible to fit the partitioned data to local memory available for each CPU core. Moreover, between the 512 and 2048, cores almost linear speedup is achieved which shows the efficiency and the scalability of the MR-MPI implementation of the PageRank algorithm. Our goal is to show the scalability of PageRank algorithm, which uses MR-MPI library, beyond 2048 cores. Our work will be reported in D7.2.2 and will also be available to the European research community in the form of PRACE whitepapers.

Project name: Swept wing simulation in a virtual wind tunnel

Project leader: Dr Philipp Schlatter; Linné FLOW Centre, SWEDEN
Collaborators: Dr. Ismaël Bouya,Dr. Matthew de Stadler,Dr. Ardeshir Hanifi,Prof. Dan Henningson,; Linné FLOW Centre – SE
Research field: Engineering and Energy
Resource awarded: 250,000 core hours on Fermi; 100,000 core hours on MareNostrum; 200,000 core hours on Curie Thin Nodes (TN); 50,000 core hours on Hermit; 250,000 core hours on Juqueen; 250,000 core hours on SuperMUC;



Abstract: Flow past an airplane wing in steady and level flight exhibits complex flow phenomena including laminar-turbulent transition, flow separation and turbulence on the wing surface and in the wake of the wing. Additional challenges include understanding how turbulence in the background influences turbulence on the wing and the interaction between transition, turbulence on the wing, and flow separation. Accurate prediction of these processes for engineering design remains a major challenge to this day.

In the present project we will extend the state of the art for direct numerical simulation of flow past a swept wing at Re=O(10^6). A Reynolds number of 10^6 is low for an airplane in flight but is a typical value for university-run experimental studies using a wind tunnel. One of the aims of this study is to develop a virtual wind tunnel capable of running cases comparable to what can be done experimentally. A virtual wind tunnel has the advantage of allowing simpler, more accurate and less expensive wing testing. Additionally, it will provide access to previously inaccessible flow data.

The open source spectral element solver nek5000 will be used for the simulation. Nek5000 combines high spatial accuracy with scalability to over one million ranks. The aim of the preparatory access project is to extend the present methodology from its current capability of simulating flow past the upper half of the wing to a domain covering upstream of the wing, the wing itself and downstream of the wing to capture the full flow from upstream conditions to the wake. This requires development and optimization of a high quality numerical grid around the wing. Additionally, extension and modification of boundary conditions are required to allow for upstream flow with carefully controlled perturbation characteristics together with a proper outflow boundary condition to allow vortical structures to smoothly propagate out of the computational domain without reflection.

The present project will require significant computational resources to resolve all the relevant length, velocity and time scales. The present computational tool nek5000 has been shown to scale to over one million ranks. In this project, the major effort will be to design a proper computational mesh and simulation settings to allow us to perfom a full scale proof of concept simulation in the next PRACE call.

Project name: Study of failure criteria for crash automotive simulations

Project leader: Mrs GERALDINE GRAFF; PSA Peugeot Citroen, FRANCE
Research field: Engineering and Energy
Resource awarded: 200,000 core hours on Curie Fat Nodes (FN);



Abstract: Automotive regulations impose levels of CO2 emission lower and lower. One way to reach the targets is to decrease mass structures, by optimizing design and using new materials such as high strength steels.To introduce such new materials PSA design rules and numerical tools must evolve. In the automotive development scheme the physical tools and parts are available late, that impose to make decisions according to simulation results.

The material failure criteria used today in crash simulations are not accurate enough and can lead to an oversizing of structural parts. To improve its knowledge on that field PSA joined the Fracture Consortium managed by MIT (ICL) in partnership with Ecole Polytechnique (LMS).

That is why PSA wants to evaluate the robustness of this advanced failure criterion for steel sheets in the context of industrial computing (shock subsystem) with RADIOSS, its standard code for crash simulations. The main difficulty is the need to describe highly localized deformation fields (because failure is a local phenomenon) in a whole vehicle crash simulation. The researchers of fracture consortium recommend the use of solid elements of 75µm size to obtain accurate predictions of the onset of fracture using their recently developed criterion.

The use of solid element meshes with 75µm edge length is not yet realistic in industrial environment. However, in view of developing simplified methodologies, it would be essential to know the “exact” solution of a car crash simulation, i.e. the solution for a full vehicle respecting the modeling guidelines for accurate fracture predictions.

The project will be executed in three major steps:

  1. The first step requires “preparatory access” to ensure the capability of the tools (handling models, robustness of the management interfaces, visualization of results), to validate capability to represent and manage failure (for qualitative point of view) and the consistency of the sequence of calculations stamping then crash.
  2. The second step requires “Regular Access” to realize the simulation “side crash to failure” taking into account the stamping effect, the accurate material laws, elements deactivation at failure in order to qualify the performance of the failure criterion.
  3. The third step (after the project) will aim at proposing a strategy that fits industrials constraints with regards to time restitution, model preparation and post-processing.

Note that the regular access only makes sense if the results of the first step are satisfying.

Project name: Scaling coupled Delft3D-SWAN simulations for fast-response lake design

Project leader: Dr. John Donners; SARA, NETHERLANDS
Collaborators: Dr. Mennno Genseberger; Deltares – NL
Research field: Earth Sciences and Environment
Resource awarded: 200,000 core hours on Curie Fat Nodes (FN); 50,000 core hours on Hermit;



Abstract: The design process of a lake and its environment asks for an interactive approach in which different aspects (economical, engineering, recreational, safety for flooding, ecology) from different stakeholders can be combined. For this purpose, for Lake Marken in the Netherlands, a multidisciplinary coupled model exists. However, the current runtime for a scenario with the model is four days, so interactive sessions (that combines drawing measures with calculations effects with the model with stakeholders) are not feasible yet.

This project is part of the EU FP7 PRACE3IP project, task 7.1c ’Socio-economic applications’. I require only a modest amount of computing time: 10.000 hours on the Curie xlarge partition and 5.000 hours on Hermit would already suffice.

Project name: Profiling and Scalability Analysis of Linear Scaling DALTON Code for Large Molecular Simulation of Biological Interest

Project leader: Dr Soon-Heum Ko; SNIC-LiU, SWEDEN
Collaborators: Dr. Thomas Kjærgaard, Dr. Simen Reine; Aarhus University – DK,University of Oslo – NO
Research field: Chemistry and Materials
Resource awarded: 200,000 core hours on Curie Thin Nodes (TN);



Abstract: In this project, we will evaluate the performance of the new tensor structure in LSDALTON code, which is being undertaken through PRACE-3IP wp7.1.c project on socio-economic applications. The DALTON family of codes has been renowned as an accurate tool for electronic-structure calculations, which results in a wide use in life-science community. The Linear Scaling version of DALTON code, called LSDALTON, is implemented under AO (Atomic Orbital) basis so that the code can provide the good scalability on large-scale simulations at Tier-0 (thousands of MPI ranks) scale. Its Density Functional Theory (DFT) calculation runs much faster with the density fitting approach for Coulombic integral evaluation than the exact Coulombic calculation. On the other hand, density fitting technique also requires much more memory consumption than the exact integral evaluation, so that it is not applicable to large molecular simulation such as Insulin (787 atoms) calculation. It motivates us to reconstruct the the tensor data structure in the LSDALTON code to a light-weight formulation. The new tensor design should satisfy the reduction in memory consumption without much loss of performance, which could be verified by the intensive profiling and scalability analysis. In this project, we will conduct profiling and scalability runs on different molecular structures of various sizes, in both pure MPI and MPI+OpenMP parallel environments. These experiments will evaluate the performance of new tensor structure at LSDALTON.

Type C – Code development with support from experts from PRACE

Project name: CAPITOLE-HPC+

Collaborators: Dr José Maria Tamayo Palau; ENTARES ENGINEERING – FR
Research field: Engineering and Energy
Resource awarded: 200,000 core hours on Curie Fat Nodes (FN);



Abstract: ENTARES Engineering is a French SME, subsidiary of Nexio Group, developing an electromagnetic simulation software to study the electromagnetic behavior of any product during the design process, before the manufacturing phase.

Among the different applications of the software, the solver can be used to design an antenna and study its performances. Furthermore, it can help to optimize the placement of an antenna on its supporting structure (such a car, an airplane, a frigate, etc.). It might be used as well to analyze interferences between equipment to meet EMC standards.

This project is in the framework of the SHAPE pilot programme for which, ENTARES Engineering is supported by GENCI. The project aims to improve the parallel efficiency of an electromagnetic solver based on the concept of compressed “low-rank” matrix.

Project name: Testing LES turbulence models in race boat sail – SME HPC Access Programme in Europe (SHAPE)

Project leader: Mr Gonzalo Kouyoumdjian; Juan Yacht Design, SL, SPAIN
Collaborators: Dr Matias Avila,Mr Hadrien Calmet,Dr Herbert Owen Coppola, Dr Mariano Vazquez; Barcelona Supercomputing Center – ES
Research field: Engineering and Energy
Resource awarded: 250,000 core hours on Fermi; 100,000 core hours on MareNostrum; 250,000 core hours on Juqueen; 250,000 core hours on SuperMUC;



Abstract: Juan Yatch Design SL (JYD) is a Spanish company that specializes in the design of sail boats. Its main market is race boats for the most important competitions in the world: America’s cup and Volvo Ocean Race. JYD’s market is clearly a very competitive one. In order to stay ahead the company needs to have access to the latest technological developments in its field. The early adoption of RANS CFD has been a key competitive advantage that has contributed to a leading position in the market and important victories for its clients. Nowadays commercial RANS CFD codes have become more accessible to the competence and stepping to more innovative simulation tools would help JYD maintain its leadership. RANS models work well for most problems but their accuracy is reduced when there are important regions of separated flow. This happens at the boat sails for certain wind directions. Large Eddy Simulation (LES) turbulence models are needed for such flows. These models require significantly higher computational resources than RANS models. In order to make LES models useful at the non-academic (industrial) level highly parallel codes and HPC resources are mandatory. The commercial software licenses are charged per number of cpus used. Therefore, even if problems with the parallelization of the code can be solved, the cost of the licenses makes their use unfeasible for LES simulations. The Computer Applications in Science & Engineering (CASE) department at BSC-CNS develops its own HPC CFD code, Alya, that already includes LES turbulence models. The proposed HPC solution would involve applying Alya to solve the flow around the sails for challenging wind directions where the results obtained with RANS models have provided poor accuracy. Through this project, the companypretends to evaluate a drastic change in the way they approach CFD: from many simulations using RANS models (with limited accuracy) to LES simulations requiring hundreds or thousands of CPU’s (with more accuracy). The objective is therefore to provide the company with a tool that consists of a new competitive advantage and prove the advantages of the use of real HPC power. Moreover it will open the pos
sibility of exploring future collaborations.

Project name: High performance computation for short read alignment

Project leader: Dr Paul Walsh; NSilico Life Science Ltd, IRELAND
Collaborators: Dr Simon Wong, Mr Xiangwu Lu; Irish Centre for High-End Computing – IE, NSilico Life Science Ltd – IE
Research field: Medicine and Life Sciences
Resource awarded: 100,000 core hours on MareNostrum; 20,000 core hours on MareNostrum Hybrid Nodes;



Abstract: We are investigating high performance computational techniques for the analysis of ribosomal RNA, which is the mechanism that cells use to translate an organism’s DNA into protein. Next generation sequencing techniques are enabling the capture of vast amounts of data on the ribosomal RNA characteristics of cells in varying conditions. The ability to monitor the identity and quantity of proteins that a cell produces would inform all aspects of biology. However reads for such RNA fragments are smaller than those typically encountered in sequencing projects, hence most alignment algorithms are optimised for longer reads. Thus there is a significant opportunity to address limitations of current alignment software and downstream analysis. We propose to develop high performance RNA analysis tools optimised for parallel computing architectures to address current gaps, in collaboration with PRACE experts as part of a SME HPC Adoption Programme in Europe (SHAPE) pilot project.

Project name: Optimizing mesh and solver parameters for clean room airflow simulations with OpenFOAM (SHAPE pilot project)

Project leader: Mr. Ralph Eisenschmid; OPTIMA pharma GmbH, GERMANY
Research field: Engineering and Energy
Resource awarded: 50,000 core hours on Hermit;



Abstract: OPTIMA pharma GmbH (Schwäbisch Hall, Germany) produces and develops filling and packaging machines for pharmaceutical products (sterile and non-sterile liquids and powders) and pharmaceutical freeze-drying systems as well as isolator and containment technology.

Sterile filling lines are enclosed in clean rooms and are operated by persons from outside the clean rooms. As the pharmaceutical products may be toxic the clean rooms have to be vented and purged. A detailed and reliable knowledge of the airflow in our clean rooms would enhance the design of the filling machines according to our customers’ requirements. For example, the airflow around the vials and cartridges in the filling line should prevent the remaining dust and impurities from reaching the sterile materials. And the contaminated, outgoing air should leave the clean rooms through the ventilation slots without endangering the staff. Furthermore, turbulences and flow detachments, especially in corners and on filling devices, should be avoided.

In the future airflow simulations will help to replace most of expensive smoke studies in production machines at the customer’s site. They will also attend the design process in an early state and support the whole CAE job. Trial and error cycles and expensive redesigns and will be minimized.

The goal of this SHAPE pilot application coached by PRACE-3IP WP5 is to find a suitable CFD (Computational Fluid Dynamics) model capable of simulating the airflow in a whole clean room and meeting the requirements of industrial production.

We will work with the open source CFD software package OpenFOAM using standard solvers for incompressible flow and the utility snappyHexMesh for the mesh generation. Our preparatory work on a PC cluster indicates that we may need a mesh with 500 million cells in order to resolve the flow and the details of the clean rooms’ interior and that the biggest challenge will be the decomposition and reconstruction of the mesh and the fields and the parallel mesh generation itself.

Project name: Development of a package for computer aided drug design

Project leader: Dr Miroslav Rangelov; National centre for supercomputing applications, BULGARIA
Collaborators: Prof. Stoyan Markov,Dr Miroslav Rangelov, Dr Peicho Petkov; National centre for supercomputing applications – BG, University of Sofia – BG
Research field: Medicine and Life Sciences
Resource awarded: 50,000 core hours on Hermit; 250,000 core hours on Juqueen;



Abstract: We are in development phase of a software package for calculation of molecular mechanics force field parameters of drug candidates. It uses CP2K packages for needed quantum mechanics calculations and GROMACS for molecular dynamics simulations of drug candidates with a target.

The first program module reads XYZ file with structure of drug candidate molecule and generates CP2K input file adding necessary keywords for proper structure optimization. The second one reads optimized CP2K geometry and writes an input file with necessary keywords for frequency analysis and electron density map calculations. The next module prepares input file for g_x2top program from GROMACS package using optimized geometry of drug like candidate and run it.

Next Module uses new heuristic algorithm based on set of rules to determine atom types for GROMACS topology file.

The most important module reads hessian matrix, masses, optimized structure and created by g_x2top topology and calculates all necessary force field constants. For parameter calculation we used the method published in [JORGE M. SEMINARIO, “Calculation of Intramolecular Force Fields from Second-Derivative Tensors”, International Journal of Quantum Chemistry: Quantum Chemistry Symposium 30, 1271 -1277 (1996)]. The hessian matrix calculated with CP2K frequency analysis run is used to determine intramolecular force constants. We implemented formulae from equations (10) to (19) from the article in the module. It also writes a gromacs topology file with calculated force constants. The last module generates input files for molecular dynamics simulations and runs the GROMACS.

The goal of the project is to test the developed package with real models (see “Scientific case of the project”) on supercomputing platforms.

We need 100 000 core.hours to complete the project.

Project name: Very high resolution Earth System Model (CESM) energy flux and wind stress sensitivty experiments.

Project leader: Prof. Markus Jochum; University of Copenhagen, DENMARK
Collaborators: Dr. Brian Sørensen; University of Copenhagen – DK
Research field: Earth Sciences and Environment
Resource awarded: 250,000 core hours on Fermi; 250,000 core hours on Juqueen; 250,000 core hours on SuperMUC;



Abstract: The stress between the atmosphere and ocean depends primarily on the state of the lower atmosphere, but also on the state of the ocean surface. The latest gen
eration of satellite observations suggests that the computed stress depends on which processes are resolved. Here we propose to quantify the relative importance of the various scales for the air-sea flux of momentum by integrating the ultra-high-resolution version of the Community Climate System Model and performing a spatial and temporal decomposition of the relevant output fields. By implementing dynamic load balancing at runtime where the processes of one climate component may “steal” work from other components based on current and future workloads. This, combined with general communication latency hiding and static load balancing, will enable good scalable performance of CESM over a broad range of simulation workloads.

Project name: Improvement of hydraulic turbine design through HPC (SHAPE Pilot activity)

Project leader: Dr Roberto Vadori; Thesan S.p.A., ITALY
Research field: Engineering and Energy
Resource awarded: 250,000 core hours on Fermi;



Abstract: The aim of the project is to optimize the design of a volumetric machine. The machine is under active development, and a prototype is already working and fully functional. This prototype operates under controlled conditions on a workbench, giving as an output the efficiency of the machine itself. Main goal is to obtain an increased efficiency through the design and realization of the moving chambers in which fluid flows. In order to obtain such a task, an extensive CFD modelisation and simulation is required in order to perform virtual tests on different design solutions to measure the physical quantities assessing the performance of a given geometry. The final goal is to design a better geometry of the different components, mainly the supply and exhaust chambers, cutting down time and resources needed to realize a physical prototype and to limit the physical realisation only on a single geometry of choice.

Project name: Improving the scalability of the overlapping fragments method code for electronic structure of organic materials

Project leader: Dr Nenad Vukmirovic; Institute of Physics Belgrade, SERBIA
Collaborators: Mr Marko Mladenovic; Institute of Physics Belgrade – RS
Research field: Chemistry and Materials
Resource awarded: 200,000 core hours on Curie Thin Nodes (TN); 50,000 core hours on Hermit;



Abstract: Overlapping fragments method is a method for solving the eigenvalue problem of the Hamiltonian of large physical systems, such as disordered organic polymer materials. It is based on the division of the system into fragments and representation of the Hamiltonian in the basis of orbitals of these fragments. The method is inherently parallel and can be used to calculate the electronic structure of systems with thousands of atoms. In this project, we plan to enhance the scalability of the code by improving the communication in the parts of the algorithm where orbitals of the fragments are redistributed among different nodes.