"MillenniumTNG is a kind of treasure chest with which we can explore many questions"
The graphic from the MillenniumTNG simulations shows neutrinos in the baryonic matter of our universe.
Our understanding of the universe, the formation of the universe and the hypothetical collision of galaxies has made a giant step forward. An international research team led by Prof. Dr. Volker Springel, astrophysicist and director of the Max Planck Institute for Astrophysics in Garching, has presented what is so far the largest high-resolution dark matter simulation to date, covering a region of almost 10 billion light years. It was carried out on the SuperMUC-NG supercomputers of the Leibniz Supercomputing Centre (LRZ) and on the University of Durham's Cosma8. The modelling of dark matter and galaxy formation is part of MillenniumTNG, which comprises a series of large-volume simulations of our universe. These simulations are now being analysed to answer a range of questions and have so far prompted nearly 50 analysis projects. Ten of them have already published results in the journal Monthly Notices of the Royal Astronomical Society. For the first time, a simulation also takes into account neutrinos, ghost particles in the universe that have only a tiny mass: "The cosmological data is now so precise that even very small units can be measured. We can now learn more about neutrinos, their properties and perhaps even their mass," Springel enthuses in the interview. "We are trying to use numerical models to recreate the evolution of the universe after the Big Bang." And while MillenniumTNG offers a model for exploring further phenomena for cosmology, astrophysics and physics, the simulation codes are now being prepared for acceleration by Graphics Processing Units (GPU) and for the use of Artificial Intelligence (AI) methods: "We want to exploit different GPU technologies in the most flexible way possible, and it is not yet clear whether all our solvers can be accelerated at all," Springel says, specifying the challenges. "GPUs require different algorithms and thus a considerable development effort."
You research the cosmos and the formation of galaxies – let me ask you a provocative question: Of what use are your findings to us? Prof. Dr. Volker Springel: Certainly of no commercial use, it is pure basic research, driven by scientific curiosity. Many people are fascinated by the universe, by the way our world came into being, and they would like to understand how things have developed and will continue to develop. Astrophysical, cosmological research provides answers to these existential questions. It has the added effect of awakening enthusiasm for the natural sciences and for basic research. This is useful even if one deals with more pragmatic issues after university. Last but not least, astrophysics and astronomy drive technical development, for example with regard to observation telescopes and satellites.
The latest series of large-volume simulations, MillenniumTNG, now presents the galaxy and matter distribution for the first time in a physically stable way and also on a scale capable of representing the entire universe – who can use these simulations? Springel: Researchers in cosmology and theoretical astronomy can use them. We use this model to test our theories about the universe and to reconcile them with new observational data. At the moment, a lot of observational missions are providing information about the early events in the Universe. The James Webb Space Telescope is able to look back in time at the infancy of our cosmos, and the recently launched Euclid satellite will provide very precise measurement data of a billion galaxies, producing vast new maps of the Universe. To evaluate this statistical data, we need theoretical models and computers to validate our ideas about what is happening in space.
You deal with numerical calculations, develop the formulas and models with which measurement and observational data are processed. First Millennium, then IllustrisTNG and now MillenniumTNG – how do these models differ from each other? Springel: These models build on each other and have become more and more detailed and complete over more than ten years. The underlying idea is this:: We try to use numerical models to reconstruct the development of the universe after the Big Bang, a period of more than 13.5 billion years. If we realise the initial conditions on the computer and calculate the correct physical laws forward in time, the result should be a universe as we can see it today. This is how we reconcile theory and observation. In 2005, we were able to calculate dark matter using the Millennium Simulation, which was already a groundbreaking achievement. But it was only with IllustrisTNG that we were able to factor in normal matter comprising helium and hydrogen, what is known as baryonic matter.. To do this, however, we had to understand various physical processes even better and we also needed significantly higher computing capacities. With MillenniumTNG, we can now simulate baryonic matter in great detail, as well as hydrodynamics, star formation and the physics of black holes. The series of simulations that we have calculated on the SuperMUC-NG and the Cosma8 is not as big a step forward as to IllustrisTNG, but it is a significant one. It takes into account a new component, the neutrinos. These are ghost particles that make up only one to two per cent of the density of matter, which is why they used to be neglected for the sake of simplicity. Now, however, the cosmological data is so precise that these small units can be measured. This makes the simulation particularly interesting - we can now learn more about neutrinos, their properties and perhaps even their mass.
MillenniumTNG is based on the programmes Gadget4 and Arepo – why did you need them? Springel: Both are two very large, parallel cosmological simulation programmes developed here at the Max Planck Institute for Astrophysics that calculate gravity over large cosmological distances in an expanding universe. Gadget is about 25 years old, Arepo is more than 10 years old and builds on Gadget. Arepo uses moving grids and calculates hydrodynamics and especially turbulent flows very accurately. The fourth version of Gadget, in turn, helps us simulate neutrinos. Both codes complement each other; they allow us to calculate gravity in particular with extreme precision and also very quickly.
Can MillenniumTNG be applied to other scientific studies? Springel: The MillenniumTNG model and the corresponding simulations can be used for many scientific research purposes. The first ten scientific papers from various fields of astrophysics based on the simulation series have just been published, and about 45 further analysis projects are currently being planned. MillenniumTNG can be used not only to calculate cosmological aspects , but also to explore astrophysical questions about the formation and development of galaxies , and to carry out intensive studies of the cosmic web . The range of scientific studies that can be carried out with the help of these simulations is quite wide. By the way, the simulation codes themselves are not only used for the universe; we use Arepo, for example, to reproduce collisions of stars or galaxies or to describe protoplanetary disks and the formation of stars and planets. The improvements we have made in the codes for MillenniumTNG also have an effect on these other areas of application and offer more possibilities there.
The largest simulation of our universe was calculated on the SuperMUC-NG and on the Cosma8 at Durham University – what is the difference between these two computers? Springel: SuperMUC-NG, with its 311,040 computing nodes, is the significantly larger and faster system of the two. We used it for the main calculation, our hydrodynamic simulation, which also models the formation of stars, supernova explosions and black holes. This requires an enormous amount of computing or CPU time, and the SuperMUC-NG is a very powerful and fast computer. But we didn't just run one simulation, we ran several, including a second big one just with dark matter on the Cosma8. This simulation is physically somewhat simpler, but still very comprehensive because it covers a very large volume. Overall, the simulation is not quite as demanding in terms of CPU time, but it needs a lot of memory. The Cosma8 has a lot of memory per compute node. Even though the computer is somewhat slower, we were therefore able to calculate this relatively large volume. Therefore, both computers complemented each other very well in the project.
The SuperMUC-NG is a computer that relies mainly on CPU, phase 2 now also includes GPU as an accelerator and for the use of AI processes: will this change your work? Springel: Yes, GPUs are a technology we have to adapt to. We are currently adjusting our simulation codes for GPU acceleration, have already developed the first solvers and are preparing our applications for phase 2 of the SuperMUC-NG and the next computer generation of exascale systems. However, this makes parallel programming a lot more demanding. We want to exploit different GPU technologies as flexibly as possible, and it is not yet clear whether all of our solvers can even be accelerated with them. Our relatively complex algorithms have been optimised for decades with regard to a serial, massively parallel architecture. In order to be able to efficiently calculate certain problems, GPUs now demand other algorithms and thus a considerable development effort. For our somewhat highly dynamic computations, we also need extremely adaptive lattice methods – putting these on an accelerator that is actually designed to perform many similar or identical operations quickly is not trivial. Presumably, we can only surpass such a large project as MillenniumTNG in the future if we manage to accelerate and adapt simulation codes to it using GPU. GPUs also enable AI processes, and these are also being used more and more in my field. They do not make classical simulation superfluous; after all, we also need training data. But one goal is to generalise simulation data. If we could get not only one simulation from MillenniumTNG, but variants where certain input parameters were in each case changed very slightly, we could achieve even more. This approach is currently being pursued intensively, as is the concept of working with many smaller simulations. However, in order to draw training data from them, you need at least one or more solid models based on detailed physics simulations between which you can interpolate.
AI methods complement your toolbox today, quantum processors probably soon will too. What do you expect for your future? Springel: Quantum computers are a fascinating technology. But it is unclear whether they will be universally programmable. It is clear that a quantum computer can solve certain problems extremely efficiently, but not all problems yet; it is not a universal supercomputer as I understand it. This will remain the case for the foreseeable future, the number of qubits is limited, memory capacity and programmability favour certain classes of algorithms, but not all computing methods. I may be wrong, but for my own work, I don't expect any direct impact yet in the medium term. As I understand it, quantum computers offer some revolutionary opportunities, but for the foreseeable future only for certain computational problems. MillenniumTNG, which involves such a complex calculation, will not be possible to do on a quantum computer in the near future.
What are the next plans for MillenniumTNG? Springel: We want to work on a whole range of scientific topics, such as galaxy clusters. Compared to earlier calculations, the simulation has a very large volume, which makes it possible for the first time to study very heavy, massive galaxy clusters. They are very rare and have a large distance between them. Previously, there was at most one of them in a simulation, but now there are dozens, hundreds of them. So we can finally study the very rare galaxy clusters and the collisions between them. This is very important for cosmological questions. We will be able to study the correlations between different signals, such as gravitational lensing effects, X-ray background radiation and the Sunyaev-Zeldovich effect. MillenniumTNG is a kind of treasure chest that we can use to study and explore many questions and write many papers in the next few years. And hopefully we will discover many things that will surprise us. (Interview: vs, ssc)
Prof. Dr. Volker Springel, astrophysicist