OFFICE OF ADVANCED SIMULATION AND COMPUTING AND INSTITUTIONAL R&D PROGRAMS (NA-114)

Welcome to the fourth 2023 issue of the NA-114 newsletter - published quarterly to socialize the impactful work being performed by the National Nuclear Security Administration’s (NNSA’s) laboratories and our other partners.  This edition begins with a highlight from Los Alamos National Laboratory (LANL) illuminating the routine use of ASC high performance computing (HPC) systems in the analysis of lot-to-lot variation and qualification of PBX 9502 high explosive for future warheads.  Other highlights include: 

• Significant progress in the delivery of hardware for the El Capitan system at Lawrence Livermore National Laboratory (LLNL) - NNSA’s first exascale-class supercomputer.  The image in the banner above shows the summer 2023 Center of Excellence “Hackathon” at LLNL with participants collaborating on porting and tuning application codes in preparation for El Capitan availability in 2024.
• A May 2023 Tri-lab L2 milestone completion review at Sandia National Laboratories in New Mexico (SNL-NM) with results demonstrating performance, porting, and scaling of complex combined hydrodynamics, plasma physics, and radiation transport problems on the Early Access System-3 (EAS-3), precursor hardware to El Capitan. 
• SNL’s Accelerated Digital Engineering Initiative-related improvements streamlining DetNet (a software-as-a-service application) for weapon system designers and analysts.

Please join me in thanking the professionals who delivered the achievements highlighted in this newsletter and on an ongoing basis, all in support of our national security mission. 
 

Thuc Hoang
NA-114 Office Director

Sierra, Crossroads, and El Capitan supercomputers will enable routine billion-atom simulations at LANL examining large pore and multi-pore interactions in PBX 9502 high explosive to inform the understanding of lot-to-lot variations and strengthen design and qualification of future warhead systems.

Details of how high explosives perform are important for the safety and reliability of nuclear weapons.  These details can be sensitive to subtle changes in manufacturing.  There is notable lot-to-lot variation in PBX 9502 (HE with triaminotrinitrobenzene (TATB)), but how grains come together to make voids and how these voids lead to variations in behavior is only qualitatively understood.  While computational studies have shown how pores collapse, these studies could only examine smaller pores, and therefore have limited ability to inform lot-to-lot variation.

Work at LANL is now taking on a significant challenge in looking at how large pores behave and how multiple pores interact.  This has only recently become possible because of the capabilities of ASC supercomputers.  With machines like Sierra, Crossroads, and El Capitan LANL will be able to conduct routine billion-atom runs, and then process their output so that models can be derived from them.  An example simulation is reflected in Figure 1 where a pore is shown collapsing due to shock loading and the material is heating up to the point where it will chemically react.  If LANL’s work is successful in creating model calibrations to reflect the nuances of various lots in simulations, it will strengthen design and qualification for future warhead systems.  (LA-UR-23-26499)

El Capitan, NNSA’s first exascale-class supercomputer, is progressing towards completion at LLNL: Structural cabinets, high-speed switches, the “Merced” file system hardware, and novel “Rabbit” high-speed storage blades are onsite; compute blade deliveries start later this year.

Figure 2: Initial El Capitan management racks are delivered to the Livermore Computing facility.

The El Capitan system currently being built at LLNL will be the first exascale-class computing resource for the NNSA.  With the recent resolution of the financing terms for the purchase of the system, LLNL procurement has authorized critical El Capitan milestones for performance with the vendor Hewlett Packard Enterprise (HPE).  As of the end of August, all the El Capitan structural cabinets and high-speed switch infrastructure are installed at Livermore Computing (LC).  100 percent of the “Merced” file system hardware, which will support the bulk of data storage for El Capitan calculations, is onsite.  In addition, the novel “Rabbit” high-speed storage blades for the Early Delivery and Initial Delivery system milestones are now onsite.  LC will now proceed with Slingshot interconnect stabilization and testing of software related to the storage systems, in preparation for the delivery of the compute node blades beginning later this year, which will complete the El Capitan hardware.  The user community continues to make productive use of the precursor Early Access Systems (EAS-3), conducting code preparations for El Capitan.  (LLNL-ABS-854843)

Results presented at the May 2023 Tri-lab L2 milestone completion review meeting at SNL-NM demonstrated performance, porting, and scaling of complex combined hydrodynamics, plasma physics, and radiation transport problems on precursor hardware to El Capitan.

Significant contributions to ASC Defense Applications and Modeling (DAM) were presented at the tri-lab L2 milestone completion review meeting at SNL in May 2023 arising from the development of new exascale computing technologies.  LLNL’s Advanced Technology Development and Mitigation (ATDM) subprogram-funded codes exceeded expectations for their final developmental L2 milestone.  Key components of this accomplishment came from LLNL efforts that enable high-order finite element methods for multiphysics applications, including the Modular Finite Element Methods (MFEM) library for discretization support and the RAJA Portability Suite for a software development model that easily supports both central processing units (CPUs) and graphics processing units (GPUs).  Results were demonstrated in multiple application areas on complex problems requiring highly competent teams of applied mathematicians, computer scientists, physicists, and engineers.  The development timeline was quite aggressive, and results came in well within expected timeframes for projects of this high degree of complexity (Post, 2004). 

The presented results fulfilled the objectives of the L2 milestone while simultaneously fulfilling the ATDM subprogram exit criteria and that of the larger Exascale Computing Project (ECP).  A key result of the milestone was the demonstration of scaling results on the Early-Access-System-3 (EAS-3) platforms, which are the precursors to the exascale ASC El Capitan platform being installed at LLNL.  Performance, porting, and scaling in many applications was demonstrated.  This satisfied a key completion result for ECP key performance parameter-2 (KPP-2).  Computational performance on GPUs was demonstrated for complex multiphysics problems that combine hydrodynamics, plasma physics, and radiation transport.  Much of this was built on LLNL open source software technologies that have a community wide impact.  Technologies being enabled include design optimization, machine learning workflows, and 3D uncertainty quantification (UQ) ensemble modeling.  (LLNL-ABS-854847; Additional Reference: D. Post, “Lessons Learned from ASCI”, LANL, LA-UR-04-0388 (2004))

SNL Accelerated Digital Engineering Initiative refactored the DetNet software-as-a-service application, integrating cloud and HPC technologies to avoid manual code changes for each computing platform, simplifying the weapon designer and analyst experience.

The Accelerated Digital Engineering (ADE) Initiative strives to radically streamline the delivery of advanced modeling and simulation capabilities to the Nuclear Deterrence (ND) designer and analyst communities.  One example of this is DetNet, a Software-as-a-Service application that integrates cloud and HPC technologies.  DetNet was originally developed as a Kubernetes application, but this was found to be cumbersome and difficult to deploy. 

To address this issue, DetNet was recently refactored into a Helm application that allows consolidation of all configurable parameters and security keys into one location and removes the need to make manual code changes for each platform.  The result can now be easily deployed with a single command (see Figure 3) which will greatly simplify the user’s experience, thus providing a streamlined alternative for ND designers and analysts.  (SAND2023-04776A)

An efficient and accurate approach to predicting the disruption of signals passing through electrical contacts subject to severe shock and vibration environments is being developed by engineers at SNL in partnership with Texas Tech University for applications across SNL’s Nuclear Deterrence (ND) mission space.

Electromechanical switches that enable electrical current to pass are common in many engineering designs and applications across SNL’s ND mission space.  When switches are subject to severe shock and vibration environments, they are observed to chatter, which refers to the disruption of the passage of the intended electrical signal, which can impact device performance. Modeling and simulating these components is challenging since they are multi-physical and localized within larger and more complex components or subassemblies.  A novel approach to predicting chatter performance of electrical contacts is being developed by engineers at SNL in partnership with research collaborators at Texas Tech University using multi-scale models coupled with reduced order models. The multi-scale dynamic electrical contact resistance (ECR) model has been developed at Texas Tech University to capture the electromechanical contact resistance at the interface where the electrical signal is transmitted.  The combination of the dynamic ECR models with nonlinear reduced order models provides an efficient and accurate approach to predict chatter performance in electrical contacts, enabling enhanced modeling and simulation deployment to predict the occurrence of electrical chatter in switches and components (see Figure 4). (SAND2023-07246N)

LANL published a neural-network-based machine learning approach in Physical Review Materials that produced a one-of-a-kind interatomic potential for tin that can quantitatively replicate phase transitions in this material under high pressure.

Designing materials for purpose is a longstanding grand challenge that would provide great benefits to nuclear security missions. For applications in which materials reach extreme pressures and temperatures, developing new materials depends on an ability to predict the influence of phase changes on materials properties. Atomic-scale modeling is a useful tool for assessing these behaviors. However, with traditional methods accuracy in this modelling can be prohibitively computationally expensive. 

In Physical Review Materials, a team of LANL researchers and their university colleagues present a hybrid potential—one that combines the speed of a conventional embedded atom method with the accuracy of machine learning (ML).  These ASC researchers used a neural-network-based approach to train the ML potential using density functional theory calculations. Tin was selected given its complex phase system and its structural, electrical, and thermodynamic properties—which have interesting technological applications.  The result is a one-of-a-kind potential, which accurately reproduces the pressure-temperature phase diagram of tin—making it the only interatomic potential that can quantitatively replicate phases transitions in this material under high pressure.  

This potential demonstrates the necessity of adding an underlying physics model to the ML potentials to create a stable interatomic potential.  This is different than the general approach which does not use expert judgement or underlying materials science to train potentials.  Using the current approach not only ensures a more accurate potential but also reduces the training data-set size.  
This method paves the way to advances in our ability to design and understand many materials.  It is generally applicable to other elements and multicomponent systems that require a high level of accuracy.  And based on the results for tin, it can be used to gain insights into deformation mechanisms of complex materials. (LA-UR-23-29888)

SNL’s EMPIRE plasma simulation code has been ported to the future El Capitan exascale architecture where it will be used to enhance the understanding of power flow dynamics through simulation with the Saturn and HERMES III accelerators.

SNL’s EMPIRE plasma simulation code will utilize significant computing power to meet the needs of the stockpile stewardship program.  To address this increased need, the NNSA El Capitan supercomputer (Advanced Technology System-4 (ATS-4)) is expected to be the world’s most powerful supercomputer when deployed in 2024.  The EMPIRE team successfully demonstrated readiness to run on the El Capitan platform, using an exemplar problem.  The initial scaling studies demonstrated a significant speedup over traditional central processing unit (CPU) architectures (see Figure 6).  With further tuning and optimization of the electromagnetics solver, this performance scaled to El Capitan will permit higher fidelity stockpile stewardship simulations than previously possible, improving our understanding of the complex physics involved and the credibility of results that we deliver.  This capability will also be leveraged in developing our understanding of power flow dynamics through simulations of both the Saturn and the HERMES III accelerators and will continue to support the refurbishment effort on the Saturn accelerator.  
(SAND2023-08035M)

The International Astronomical Union named two asteroids “Wendycaldwell” and “Plesko” after two LANL scientists for their research on planetary defense, including physics simulations performed on LANL supercomputers; Cathy Plesko previously led the ASC V&V program at LANL.

Figure 7: (On left) Cathy Plesko, LANL PI for Planetary Defense and former ASC V&V Program Lead. (On right) Wendy Caldwell, LANL Planetary Defense Scientist.  For interviews with both Cathy and Wendy, see LANL’s video posted on YouTube: Asteroids named after two Los Alamos National Laboratory scientists.

The planets bear the names of Roman gods, which puts Cathy Plesko and Wendy Caldwell in exalted company, as each of these LANL planetary-defense scientists now has an asteroid named after them.  The Working Group for Small Bodies Nomenclature published the names June 21, 2023 on behalf of the International Astronomical Union, dubbing asteroid 32105 “Plesko” and asteroid 32110 “Wendycaldwell.” Both were discovered in 2000.

Cathy Plesko is the principal investigator for Planetary Defense and former Program Manager for the ASC Verification and Validation (V&V) program at LANL.  In her work, she models asteroid impacts and deflection techniques and leads a team that tests the accuracy of physics simulations done on the laboratory’s supercomputers. 
A mathematician and planetary scientist, Wendy Caldwell leads LANL’s modeling of 16 Psyche, an asteroid that will be visited by a NASA research mission in 2026.  Caldwell’s expertise includes modeling impact and explosion craters in rocky and metal targets.
Plesko and Caldwell, along with others at LANL, modeled Psyche’s two largest impact structures to help determine the asteroid’s composition.  Such information would be useful in understanding how to deflect an asteroid, should the need arise, and could also provide insight into the necessary tools for exploring similar bodies in the solar system.

Both researchers are members of the Double Asteroid Redirection Test (DART) Investigation Team, a multi-agency international collaboration that deliberately crashed a spacecraft into another asteroid in 2022 to test kinetic impact as a deflection technique — it worked better than expected.  The team is preparing a research paper based on the data from that far-flung mission.  

(LA-UR-23-27037, read more in the LANL news release)

WELCOME ABOARD...

SNL ASC program

Cheryl Lam has been supporting the SNL ASC program since July 2002.  In August 2023, she became the Sub-Program Lead for the SNL ASC Verification & Validation (V&V) subprogram.  She is also Senior Manager of Group 8750, Computational Science & Analysis.  Although Cheryl has supported the SNL ASC program since 2002, she has also worked in Systems Engineering, Weapon Reliability, Advanced & Exploratory (A&E) projects, and various modeling and simulation roles within the Nuclear Deterrence program.  Her educational background includes a PhD in Mechanical Engineering, with research focused on Computational Plasma Physics.  Away from SNL, Cheryl enjoys cooking, eating, beach vacations, and looks forward to more world traveling in her future.

Cole Yarrington started supporting the SNL ASC program in June 2011.  In August 2023, he became the Subprogram Lead for the SNL ASC Physics & Engineering Modeling (PEM) subprogram.  He also is Senior Manager of Group 1510, Thermal, Fluid & Aerodynamic Sciences.  Before returning to SNL, Cole worked at NNSA as a Senior Technical Advisor in NA-115 within the Office of Engineering and Technology Maturation.  His educational background includes a PhD in Mechanical Engineering.  Away from SNL, Cole enjoys family time with four children, and outdoor activities such as running, mountain biking, hiking, skiing, and surfing.  He also enjoys video and board gaming.

LANL ASC program

Kierstyn Paschke is the new LANL ASC Deputy Facility Operations and User Support (FOUS) Program Manager.  Kierstyn earned a BS in Computer Science from Montana State University in late 2017 and subsequently joined LANL as a post bachelor student on the HPC Platforms team in 2018, converting to staff within the year.  In 2020, she joined the HPC Filesystems team until she stepped into the role of Institutional Computing Program Manager at LANL in October 2022.  In her free time, Kierstyn enjoys running, reading, hockey, and working on projects with her husband. 

LLNL ASC program

Latisha Sumardy joined LLNL in February 2023 as a Software Developer in the Applications, Simulations, and Quality (ASQ) Division.  She works on the Deterministic Transport code team in the Weapons Simulation and Computing (WSC) program, primarily focusing on C++ development for the Ardra code and on Spack development for the Ardra and Fed-Tra codes.  Latisha has a B.S. degree in Computer Science from UCLA.  Prior to joining Deterministic Transport, she interned at LLNL through the Defense Science and Technology Internship (DSTI) summer program, working with Paul Minner on the Opacity Library.

Lindsey Whitehurst rejoined LLNL in June 2023 as a Cybersecurity Software Engineer after some time in private industry.  She has experience in engineering management, software development, and cloud architecture design, and implementation.  She focuses her time with Livermore Computing on low level system software security.  Lindsey previously worked in Livermore Computing on the Security Technologies Team, supporting authentication and logging services for the center.  She has a B.S. in Electrical Engineering from Boston University and a M.S. in Computer and Information Sciences from the University of South Alabama.

NNSA LDRD/SDRD Quarterly Highlights

LLNL LDRD: Understanding the plasticity of diamond for improved fusion ignition.

Alex Li, a LLNL summer student in the Computational Chemistry and Materials Science Summer Institute, recently led a study published in the journal Matter to investigate the evolution of plasticity in diamond along different loading orientations and the effects that voids (pores) within the material can have on stresses within the diamond. This team’s research was supported by LLNL’s LDRD program.

While diamond carbon is one of nature’s strongest naturally occurring materials, it is known to undergo irreversible plastic deformation when loaded at high rates.  Rob Rudd, LLNL scientist and Li’s mentor, notes, “We usually think of diamond as brittle, strong and unyielding until it cleaves.  Shocked diamond is different.  Alex did a fantastic job working out the complex and unexpected details of the deformation.”

The nature of this deformation is important for high-energy-density experiments on high-energy laser systems such as the National Ignition Facility (NIF), as well as for further evolving scientific understanding of carbon-rich exoplanets.  At NIF, diamond carbon is used as an ablator and capsule material for producing the extremely high pressures needed to cause nuclear fusion reactions that are being intensively investigated as a source of energy.

In many shock-compression experiments, diamond has shown little to no plastic behavior until reaching extreme pressure and temperature conditions.  Because diamond exhibits strong anisotropic behavior — that is, has different properties in different locations, depending on the direction of applied stress — it can be difficult to anticipate how diamond will react under such extremes.  Defects present within the diamond capsules that hold the fusion fuel can cause imperfect compression, resulting in a failure to ignite.  (Read more in the full article on the LLNL website)

SNL LDRD: Integrating microscale optical devices on silicon microchips.

SNL has been awarded a patent for its new method of integrating many different materials onto silicon — the same starting material semiconductor fabrication plants use to make microchips.  This method enables SNL to build high-bandwidth, high-speed optical devices, including indium phosphide lasers, lithium niobate modulators, germanium detectors and low-loss acousto-optic isolators — all critical components for high-power optical systems. Building a laser on silicon is a challenging and unusual feat that could extend America’s leadership in semiconductor technology.  (For more information, see the SNL news release)

NNSS interns facilitate SDRD’s return to UAS experimentation.

Thanks to the Minority Serving institution Partnership Program-Nuclear Security Science and Technology Consortium (MSIPP-NSSTC), Nevada National Security Sites’ (NNSS’) SDRD program is resuming unmanned aircraft system (UAS) experimentation.  The NNSA-funded consortium was launched three years ago to establish pathways for scientific exchanges between the NNSS and universities, and to provide students from minority backgrounds with exposure to practical applications of the concepts they’re studying in school.  (See the full news release on NNSS.gov)

Questions? Comments? Contact Us.

NA-114 Office Director: Thuc Hoang, 202-586-7050

• Integrated Codes: Jim Peltz, 202-586-7564
• Physics and Engineering Models/LDRD: Anthony Lewis, 202-287-6367
• Verification and Validation: David Etim, 202-586-8081
• Computational Systems and Software Environment: Si Hammond, 202-586-5748
• Facility Operations and User Support: K. Mike Lang, 301-903-0240
• Advanced Technology Development and Mitigation: Thuc Hoang