Use Case: The human lung

Graphic, three-dimensional image of the human lung. Graphic: Chair of Numerical Mechanics/TUM

Ever tried to blow up a figure balloon - with ears, legs, paws and more? It doesn't always go well: sometimes only one ear fills up, sometimes legs remain small or limp, sometimes the thing bursts. Ventilation of the lungs is very similar. The only difference is that intensive care physicians are faced with the delicate task of evenly filling millions and millions of small ears or, better, filigree bubbles with air. Some of them, however, are already full of water or mucus and can therefore only absorb a small amount of air; in others, the tissue is stiff or brittle, and they need be filled with extreme caution. Last but not least, doctors are not only working under extreme time pressure during ventilation, they also cannot see how which parts of the lungs are filling up and where there may be a risk of excessive pressure. Ventilation is a complex task and it comes with high risks: According to experience from hospitals, up to 50 percent of ventilated patients still lead to death.

A solution to this intolerable dilemma is now being provided by the academic field of mechanics and its spin-off, numerical mechanics. These disciplines have long been devoted not only to machines, technology or production processes, but increasingly to organ functions or even living beings: "All physical, biological, chemical phenomena can be described by mathematical equations," explains Professor Dr. Wolfgang Wall, head of the Chair of Numerical Mechanics (LNM) at the Technical University of Munich (TUM). "And mechanics plays a very essential, though long underestimated, role in living systems." Mechanics focuses on interactions of forces and motions as well as deformations of solid, liquid or gaseous bodies and substances and their calculations; numerical mechanics, in turn, develops models, equations, even computer programs to digitally reproduce flows and deformations or even organs. In recent years, Wall's department has developed a highly regarded, extremely accurate model of the human lung, which breaks with traditional ideas, helps medicine to better understand the processes involved in ventilation and, in any case, to gently and individually fill the millions and millions of bubbles of what is probably the most complex figure balloon today.

Interdisciplinary development of better models

High-performance computing (HPC) and supercomputers are part of the tools of the trade. The LNM is therefore one of the users of the Leibniz Supercomputing Centre: "We conduct application-motivated basic research," says Wall, describing the work at the chair. "In my field of science, the relevance of HPC is often still underestimated, but it holds enormous potential. The proximity to the supercomputers at the LRZ helps us. We can use great computing power there and find the specialists who can help us optimize algorithms and implement programs. And with the knowledge we gain from this collaboration, we can develop new, better models and methods." The LNM specializes in the computation and modeling of complex multi-field and multi-scale problems, which are biological and technical phenomena in which, for example, electromagnetic or chemical fields alter the motions of solid substances or in which solid, liquid, gel-like or gaseous substances interact. About 30 employees work here with engineers, biologists, physicians, and recently even physio-therapists to develop ideas, hypotheses, algorithms, and simulations. Mathematics plays a special role here: "Our central task," specifies mathematician and computer scientist Dr. Martin Kronbichler, "is to develop the numerical tools and algorithms that can be used to model concrete problems from other disciplines as well."

    

Prof. Dr. Wolfgang Wall, Chair Numerical Mechanics, and Prof. Dr, Martin Kronbichler, University Augsburg

The digital model of the lung, which has been worked on since about the year 2000, is a good example of the interdisciplinary work at the LNM. Over the years, it has been expanded to include more and more criteria and details, resulting in growing amounts of data that only supercomputers can handle today. "To develop such a complex model, you have to analyse and understand the organ as comprehensively as possible," Wall says, "and sometimes get rid of misconceptions." For example, the team realized that the approximately 500 million alveoli, or aveoli, of the lungs are not attached to the bronchi or bronchioles in a grape-like fashion as depicted in textbooks, but rather form a spongy, elastic tissue of enormous size. "In medicine, there are few ways to measure or take pictures of essential processes in the body or bodily functions," Wall says. "Despite state-of-the-art methods, morphological and histological knowledge, for example, little is known to date about vital processes deep in the lungs." Digital models better illustrate ways of functioning, leading to insights or new therapies. In ventilation, adapted treatment methods can prevent lung disease and reduce mortality risks.

Data + mathematics + models + computer science = simulation

Wall's research group has now developed a number of models of the digital lung that answer different questions. They are based on static views from computer tomographs and X-ray machines, and also on image data from extensive experiments with tissue samples from microscopes. Available anonymized patient data and measurements were also used for the model. Step by step, the trachea and bronchial system, the alveoli and tissue of the lungs, as well as the parenchyma, where breathing takes place, were modeled and finally combined in a simulation. A wide variety of mathematical equations and numerical methods were used: for example, highly efficient discretization methods based on the discontinuous Galerkin method as well as algebraic multigrid methods to calculate the flow of gases and liquids in the lungs, including the turbulence effects that also occur there, using Navier-Stokes equations.

Each modeling step is critically scrutinized, checked against measured values, observations, experience and logic. It's an advantage that so many different schools of thought and research directions are represented on the LNM team: "You can't prove hypotheses and theories, you can only try to disprove them. That's what we try to do here every day," says Wall, and Kronbichler adds, "The great thing about simulations is that the results can be looked at closely and recalculated in detail over and over again." That, however, demands computing capacity, especially when multiple models are merged into a more detailed one, causing the calculations, like the data sets, to grow exponentially. In this case, a particular challenge for the SuperMUC-NG was the resolved description of the air flows in the lungs. Together with the LRZ specialists, the simulation code exaDG recommended for this purpose was optimized; as a highly scalable solver, it can now also be used to simulate other turbulent flows. Matrix-free operator evaluations accelerated the calculations and memory accesses, interventions in the computer system improved cache utilization and vectorization and led the SuperMUC-NG to higher performance.

Bringing research results into everyday life

"To advance a topic like the lung model, we need different skills and knowledge, but above all we need enthusiastic people who want to understand how processes really work and keep questioning them," Wall says. They are already working on new mechanical problems at the LNM in Garching–both technical and medical. For example, they are developing models to capture the human shoulder, to describe the growth of tumors, or to improve therapies or approaches in nanomedicine. In the technical field, battery systems for e-vehicles are being developed, or innovative processes for 3D printing of metal components. The digital lung, which is now available, has received several awards, and the approaches for simulating air flows were even nominated for the international Gordon Bell Award for outstanding work in HPC in 2021.

Much more important to the team than the awards is the fact that this will improve ventilation and make it safer: Whereas ventilation was previously often based only on general information and a few values from patients, doctors can now adjust these individually on the digital model and, above all, try out how the planned pressure curve of the breathing air, the frequency of breaths or the oxygen content can be affected and improved. This saves lives and prevents lung damage, for example in the case of corona diseases or even in the case of delicate premature babies who are ventilated in their first weeks of life. The lung model also provides a good basis for exploitation and thus for business: After AdCo Engineering, which has been advancing simulation methods in medium-sized companies and helping to improve engineering services since 2011, Ebenbuild is the second company to be founded from the chair. Ebenbuild has just completed a seven-figure round of funding for its market launch. "At SuperMUC-NG, we developed important foundations for the lung model," Wall reports. "From that, we were able to derive simpler models and, in turn, create software for clinics and physicians to quickly process and provide individual disease data." (vs)

In the picture above, a computer tomograph shows a top view of a human chest. The curve shows the pressure during ventilation over about 85 seconds, the digital lung model illustrates this process. Fig.: Chair of Computational Mechanics/TUM

 

"Codes must be modernized and adapted to higher HPC complexity"

The Institute for Computational Mechanics is located less than 500 meters from the Leibniz Supercomputing Centre (LRZ): this neighborhood promotes the search for common solutions. In addition to the remarkable lung model, software for the calculation of turbulent flows, which occur not only in the lungs but also in nature, in space as well as in many areas of technology, has been developed in recent years. Momme Allalen holds a doctorate in theoretical physics, is one of the specialists for computational fluid dynamics at the LRZ and, together with his team, supported the scientists around Professor Wall in optimizing the software.

How long has the HPC team at LRZ been working on Prof. Wolfgang Wall's lung project? Dr. Momme Allalen: The first, closer contact came about in 2015 through a KONWIHR project. Professor Wall and his team wanted to develop new algorithms for finite element discretizations for their lung model, and also to increase the scaling of their own codes and software in order to simulate a larger model with more parameters at the then SuperMUC with more and preferably all compute nodes. With support from the Computational Fluid Dynamics or CFD Lab team at the LRZ, the group was able to solve porting and performance issues to take advantage of all the capabilities of our supercomputer. Since then, we have collaborated regularly. Professor Wall's Institute for Computational Mechanics benefits from our technical know-how in modernizing algorithms, and conversely we learn more fields of application of and requirements for software for flow simulations.

What do you do specifically, what are the tasks of the HPC and CFD team? Allalen: Our job is to help LRZ users adapt codes so that they run faster and more efficiently on HPC hardware. We use a variety of techniques to do this, and the process goes like this: We first analyze and identify patterns in the algorithms that provide better performance at the node level. Then, together with researchers, we optimize and modify the codes in such a way that, first, the computing time and resources of the SuperMUC-NG are better utilized and, second, as many computing nodes as possible are addressed in the process. The background to this is that many researchers are still working with older codes, but want to use them on current computing systems such as multi-core CPUs, Xeon Phi and GPUs. HPC systems are evolving very fast, CPU parallelism is increasing with new memory hierarchy technologies. There are about three years between the construction of SuperMUC Phase 2 and SuperMUC-NG, but the architecture of both systems is very different because the technology has improved in the meantime. Consequently, codes need to be modernized and adapted to higher HPC complexity. Our service supports the computational side of a research project, but has rather little to do with the scientific question.

Which codes are important for the lung model? Allalen: To simulate a complex geometry of the lung system and respiratory functions, we first looked for a reliable meshing algorithm to convert a geometric representation into a mathematical formula that algorithms for calculating turbulent flows can work with. Professor Wall's team works mainly with their own BACI code, but also with codes based on the Deal.II library. We are investigating their behavior on the SuperMUC, using single-instruction multiple-data or SIMD vectorization and shared-memory parallelization. These two components are single-node optimizations that form the basis for hybrid codes based on the MPI programming scheme that span multiple nodes. For shared memory parallelization, we rely on Intel's Threading Building Blocks or TBB library.

In addition to the remarkable lung model, a paper nominated for the Gordon Bell Prize was developed in collaboration with the Technical University of Munich, the Helmholtz-Zentrum Geesthacht, the Institute for Computational Mechanics and the LRZ. What was developed there? Allalen: For the lung project, a highly scalable HPC program was developed for the numerical calculation of human respiration or, more generally, turbulent flows. This solver is based on different equations, so-called high-order Galerkin discretizations as well as Navier-Stokes equations, and shortens the computation time. The paper describes the mathematical elements, how the solver works, and also how it is implemented and scales on current supercomputers. The paper and the program could be useful to researchers who want to compute other turbulent flows, and it also shows that innovations in supercomputing are mostly based on the cooperation of specialists from different scientific fields.

Dr. Momme Allalen, CXS-Lab LRZ