NAMD (Nanoscale Molecular Dynamics)
Overview
Every facet of our physical world is determined by the behaviors, properties, and interactions of the microscopic particles that make up all materials. Perhaps nowhere is this more evident than with the inner workings of biological organisms. Our understanding of all living things originates at the molecular level. Vastly complex arrangements of interconnected chemical systems combine to create the emergent phenomenon that we call life. Sadly, these systems do not always function correctly. They can break down over time, succumb to an invading attacker, or corrupt themselves from within.
The first step in treating any disease is understanding what is going on, and for the most difficult medical challenges that humanity faces today, one key approach is modeling the underlying molecular systems directly using computer simulations. This poses its own challenges: simulating a process effectively could require tracking millions or billions of atoms, over extended durations, with time intervals of important activity significantly more fleeting (nanoseconds) and numerous (billions) than human perception can comprehend. Moreover, interactions between molecules are not evenly distributed over space and this distribution changes over time, leading to dynamically changing load imbalances between the executing processors, in turn slowing down simulations.
All-atom model of the HIV capsid
This video was made with VMD and is owned by the Theoretical and Computational Biophysics Group, NIH Center for Macromolecular Modeling and Bioinformatics, at the Beckman Institute, University of Illinois at Urbana-Champaign.
Simulating billions of timesteps, with “only” millions of atoms requires very fine grained parallelization of the computation. The combination of fine-grained parallelism and dynamic load imbalances presents a challenge to computer simulation architects: How can a real-world computing cluster perform these computations effectively? Fortunately, NAMD is the most longstanding user of the Charm++ runtime system, which allows it to surmount the challenges of such highly dense and dynamic workloads, relieving chemists and biologists of the burden of distributed-systems programming, and bringing these simulations within the realm of feasibility.
Software
NAMD is one of the most popular and best-performing molecular dynamics simulation software, with over 100,000 users worldwide. With collaborative development led by Prof. Klaus Schulten, Bob Skeel, Laxmikant Kale and co-workers at the University of Illinois at Urbana-Champaign in the 1990s, it is one of the most widely used tools for understanding diseases at a molecular level. NAMD can be used on a wide range of devices, from the most portable tablets and laptops, all the way up to the largest supercomputers in the world.
These images were made with VMD and are owned by the Theoretical and Computational Biophysics Group, NIH Center for Macromolecular Modeling and Bioinformatics, at the Beckman Institute, University of Illinois at Urbana-Champaign.
NAMD’s design based on Charm++ objects is one of the reasons its parallel structure has withstood over 25 years of its deployment. In the early 1990s, molecular dynamics programs were “parallelized” using either replication or fixed partitioning of the array of atoms that the sequential versions had; but this was shown to be not scalable. Partitioning based on current coordinates of each atom was more scalable, but also more awkward. For example, typical implementations would require the number of processors to be a cube, because each processor was to be assigned a single sub-cube of the simulation space.
A key insight in NAMD was to break this connection between partitioning (i.e. number of sub cubes) and the number of processors. The size of sub cubes was independently determined based on the distance up to which direct interactions between atoms needed to be explicitly calculated. If this meant multiple cubes were assigned to a processor, so be it. At the limits of scaling, this parallelism was not enough. A second innovation in NAMD was to postulate a separate computational object for each pair of interacting sub-cubes responsible for calculating interactions (i.e. forces) between their atoms. These “compute” objects, shown as diamonds in the figure below, could be assigned now even to a processor different from that of either sub-cubes. Long range interactions can be calculated by another set of objects using a distributed-FFT based PME algorithm.
The resultant design is shown in the figure to the right.
-
The computation can now be expressed solely in terms of objects, with no reference to processors.
-
What cinched the use of Charm++ for NAMD was also its message-driven design. Work on different types of objects could interleave automatically based on availability of data (via messages) without needing an a-priori determined sequence of receive statements, which is both awkward and inefficient.
-
The load of each object can be measured during execution, and objects can be migrated across processors periodically based on these load measurements, under the control of a global load balancing strategy.
NAMD Parallel Design with Migratable Objects
With this design, NAMD was able to scale a simulation with only 92,000 atoms over 64,000 cores (yes, that's less than two atoms per core) of the Blue Gene supercomputer. Of course, on modern supercomputers, with GPGPUs, kernels need to be chunkier. Yet, NAMD has demonstrated high scalability on Summit, one of the largest supercomputers in the world, to about half the size of the whole machine (i.e. 2048 very powerful nodes) for a 220+ million atom simulation, roughly the size of some of the virus simulations. More recently, NAMD scaled to a record 86,400 cores on Azure.
The design of both Charm++ and NAMD has evolved together over the years. NAMD was one of the first use cases that motivated the concept of Chare arrays in Charm++. As a consequence, and also because NAMD is a full-fledged application with all its bells and whistles, it is complex. If you want to see this basic design in a simpler setting, check out the LeanMD MiniApp.
NAMD is most well-known for its use in modeling the first HIV capsid, which has helped identify new ways to create life-saving therapies for HIV/AIDS patients. As described in Nature in 2013, the HIV capsid contains genetic material for the virus as it enters the cell. When cells are infected, the capsid bursts open, allowing the virus to begin replicating. By understanding the structure and mechanisms of the HIV capsid, scientists can develop ways to prevent the capsid from releasing genetic material, effectively stopping the virus in its tracks. Achieving this lofty goal requires significant computing power to simulate the millions of atoms that make up the HIV capsid, as well as an effective framework to attain efficient parallel performance. The object-oriented, highly scalable parallel programming framework of Charm++ was the perfect fit to achieve these goals.
More recently, NAMD has been used for groundbreaking coronavirus simulations as part of a world-wide effort to stymie the spread of COVID-19. The Amaro Lab of UC San Diego is using NAMD to build the first complete model of the SARS-COV-2 coronavirus envelope, to help scientists understand how the virus interacts with receptors within the host cell membrane. The simulation for this model, which is anticipated to contain 200 million atoms, ran on 4,000 cores of the Frontera supercomputer in March of 2020. These simulations can help scientists better understand how to design new drug treatments to more effectively treat those infected with the virus. These researchers have since been awarded the Gordon Bell Special Prize for this invaluable high performance computing-based COVID-19 research. Animations of these simulations have also been featured in the New York Times.