OFFICE OF ADVANCED SIMULATION AND COMPUTING AND INSTITUTIONAL R&D PROGRAMS (NA-114)

Welcome to the first 2024 issue of the NA-114 newsletter - published quarterly to socialize the impactful work being performed by the National Nuclear Security Administration (NNSA) laboratories and our other partners.  This edition begins with a highlight from Sandia National Laboratories (SNL) on their development and demonstration of a generalized digital engineering framework to design mock hardware for the W93.  Other highlights include: 

• Collaborations formed during the NNSA-University Workshop on Exascale Simulation Technologies (NUWEST).
• Updates to LANL’s Implicit Monte Carlo (IMC) code to allow additional workflows to leverage graphics processing units (GPUs), to significantly improve modeling of radiation flow experiments.
• A 5.5x node-to-node speedup running the Livermore Safety Suite on the Sierra system, enabling rapid turn-around of safety assessments.
• Release of the ASC Artificial Intelligence for Nuclear Deterrence Strategy (2023), available now for download.

Please join me in thanking the professionals who delivered the achievements highlighted in this newsletter and on an ongoing basis, all in support of our national security mission. 

Thuc Hoang
NA-114 Office Director

SNL’s Plato team is developing a generalized digital engineering framework to design mock hardware for the W93, enabling acceleration of the ground and flight test schedule. 

This framework was demonstrated on a W76-0 arming, fuzing, and firing (AF&F) assembly, reducing component count by an order of magnitude.

The Plato team is developing a fast, capable, model-based approach to accelerate decision making.  This Accelerated Digital Engineering (ADE) initiative has developed a generalized digital engineering framework to design mock hardware for the W93, enabling acceleration of the ground and flight test schedule.  The capability was successfully demonstrated on a low-fidelity W76-0 AF&F, closely matching the dynamic behavior with a small number of easy-to-manufacture monolithic parts and reducing the number of components by an order of magnitude.  This ADE initiative aims to integrate ASC capabilities with Sandia’s Digital Engineering efforts to accelerate product realization across the Nuclear Deterrence (ND) development lifecycle.  (SAND2023-11650M) 

NNSA-University Workshop on Exascale Simulation Technologies showcases Predictive Science Academic Alliance Program (PSAAP) software technologies.

The NNSA-University Workshop on Exascale Simulation Technologies (NUWEST) was held on Thursday January 18, 2024, in Albuquerque, NM.  The overarching goal of NUWEST was to share ideas on technologies for facilitating exascale predictive science, by highlighting available technologies, identifying challenges, and initiating further collaborations with NNSA laboratory efforts.  Organized by the University of Illinois’ Center for Exascale-Enabled Scramjet Design (CEESD), NUWEST attracted 61 participants, including 36 NNSA Laboratory staff and representatives from eight out of the nine current PSAAP III Centers (see banner image above).  The one-day event began with plenary presentations from Christian Trott (SNL) and Bill Gropp (UIUC/NCSA) who identified a number of shared challenges and successful approaches to build a community adoption of new standards and technologies, specifically their personal experiences launching the highly successful Kokkos; Message Passing Interface (MPI); and Portable, Extensible Toolkit for Scientific Computation (PETSc) efforts.  Following this session, seven different, emerging technologies (or groups of technologies) from individual PSAAP III Centers were explored in depth, first through brief conceptual overview talks followed by hands-on “code-along” experiences.  NUWEST represented the first post-COVID successor to the Workshop on Exascale Software Technologies (WEST), which was launched during PSAAP II.  For more information, please visit https://illinois-ceesd.github.io/nuwest/.  (LA-UR-24-21423)

Top banner image (top of newsletter): Participants at the NNSA-University Workshop on Exascale Simulation Technologies (NUWEST) held on January 18, 2024, in Albuquerque, NM.

Recent changes in LANL’s Implicit Monte Carlo (IMC) code allow many workflows to leverage GPUs.  This enabled a speed up on Sierra of 3.5-9x over Commodity Technology System-1 (CTS-1), significantly improving modeling of radiation flow experiments.

 A mainstay of the predictive-science ASC tools is fast and accurate thermal radiative transfer (TRT) in both 2 and 3D simulations.  Recent improvements to the LANL’s Implicit Monte Carlo (IMC) code sped up TRT by up to 9x (see Figure 1).  While Monte Carlo transport methods are said to be embarrassingly parallel, simulating IMC particle histories is usually the single most dominant portion of multi-physics simulations.  IMC simulations require hundreds of millions of particle histories for adequate solution fidelity.  Such simulations involve code branching and random data accesses that stress NNSA’s supercomputers.  IMC needs to be performant and compatible with user workflows in order to enable analysis in a timely fashion for many applications (e.g., inertial confinement fusion experiments). 

The transport team has adapted IMC code to use GPUs for particle transport and sourcing. Recent changes in IMC have enabled many user workflows to leverage the GPUs’ superior computing throughput and memory access latency hiding features.  This enabled a speed up on the ASC Sierra system of 3.5-9x over CTS-1 for the IMC portion of the calculation, offering a significant improvement to modeling radiation flow experiments (see Figure 2 below).  (LA-UR-23-32684) 

Running the entire Livermore Safety Suite on the Sierra system achieved a 5.5x node-to-node speedup as compared to running on current NNSA commodity technology systems; this will enable rapid turn-around of safety assessments.

The ability to rapidly simulate ensembles of safety calculations accelerates turn-around of safety assessments.  This fundamentally enables future weapon system development and informs experimental efforts targeted towards weapons safety.  LLNL can now simulate the entire suite in one Sierra hour, i.e., equivalent to using all 4,320 nodes of the machine for an hour.

The safety suite validates LLNL’s common modeling framework against the experimental record.  Simulating the 3D nature and complexity of safety experiments (see the LLNL news: Subcritical experiment captures scientific measurements to advance stockpile safety) requires the computing power of advanced GPU-accelerated platforms.   These simulation codes and platforms have evolved sufficiently such that platforms like Sierra are now able to compute ensembles of simulations allowing uncertainty quantification (UQ) to be performed.   This UQ of the Safety Suite Common Modeling framework is improving safety assessments of our current and future stockpile.  (LLNL-ABS-860761)

SNL developed and demonstrated multi-fidelity analysis capabilities for prediction of the reentry random vibration environment.  These capabilities underpin current and planned modeling and simulation efforts in support of the W87-1 and W93 programs. 

 The Sandia ASC Program has completed an L2 milestone effort as part of the FY23 Stewardship Capability Delivery Schedule (SCDS) Reentry pegpost.  The milestone was to develop and demonstrate multi-fidelity analysis capabilities for prediction of the reentry random vibration environment.  Overall, this work demonstrates how production engineering models for random vibration aerodynamic loadings can be used in concert with targeted high-fidelity simulation efforts to improve the accuracy and reliability of overall environments predictions using modeling and simulation.  The impact of this work enhanced the confidence in SNL’s analysis codes and workflows, showing a responsive, coupled, multi-physics, and multi-fidelity analysis capability to support current and future Nuclear Deterrence (ND) delivery systems.  The demonstrated capabilities, and the associated credibility evidence, underpin current and planned modeling and simulation efforts in support of the W87-1 and W93 programs.  (SAND2023-13842M)

LANL demonstrated a 3D laser ray tracing capability in xRAGE to accurately drive and model complex multi-physics problems found in NIF hohlraums and other inertial confinement fusion applications where the problem is inherently 3D and impossible to accurately simulate in 2D.

 xRAGE is the standard radiation-hydrodynamics code at LANL for inertial confinement fusion (ICF) modeling.  It is one of only three U.S. codes capable of credibly modeling laser-driven hohlraums, which are used to convert a laser drive into a uniform x-ray drive for capsules and other targets.  The 3D laser ray tracing code in xRAGE is an extension of Mazinisin, the 1D and 2D laser ray tracing code originally developed by the Laboratory for Laser Energetics (LLE).  xRAGE is used to accurately predict the ray trajectory and energy coupling between laser beams and a target, in addition to more sophisticated laser-plasma interactions such as cross-beam energy transfer.  One important unanswered question is to what extent the use of 2D modeling has limited predictive capability with respect to laser-driven hohlraums, which can now be investigated with the new 3D laser ray tracing code.

Previously, 2D models of ICF targets were sufficient due to the computational and memory restrictions of a full, 3D problem.  However, as LANL moves to next-generation computing architectures and machines, xRAGE and its related physics packages must be able to deliver 3D results on those systems.  Furthermore, it is well known that many experiments of interest to national security, including those using outputs from igniting capsules, are inherently 3D.  Along with being necessary to enable the design of many new national security-relevant experiments, distinguishing between unmodeled physics and inadequate dimensionality requires this 3D modeling capability.

In addition to the standard analytic test problems, the 3D ray trace was implemented and tested in a variety of ICF and high-energy-density physics (HEDP) problems, such as the target geometry shown in Figure 4.  In this problem, a shock tube is driven by eight laser beams that deposit their energy into an ablator, which in turn compresses and launches a shock into a lattice.  The structured lattice in this problem is complex and impossible to accurately simulate in 2D, highlighting the importance of having a laser ray tracing code that can couple with 3D geometries.  This is also true in direct drive and hohlraum applications, where 3D features that arise from complex geometries, hydrodynamic instabilities, and laser deposition profiles are crucial to predicting subsequent shock progression and timing in an ICF problem.  (LA-UR-23-34084)

LLNL developed a framework that employs Gaussian processes to generate high-fidelity equation of state (EOS) tables that account for data and model uncertainty.  LLNL demonstrated this new approach by generating a new, uncertain EOS for gold showing alignment with published experimental data.

High-fidelity EOS models are indispensable for accurately characterizing material properties under varying temperature and pressure conditions.  Key attributes of the current LLNL EOS model development, such as multiphase equation of state (MEOS) and XEOS software, include precision, consistency with experimental data, and the capacity to extrapolate outside the calibrated domain.  Capturing uncertainties in the available data and EOS functional form is also a critical aspect of EOS development but is often overlooked due to practical limitations of the current EOS modeling technology.  Thus, integrating these uncertainties into an EOS model poses a compelling challenge.

LLNL Physics and Engineering Models (PEM) team has developed a framework that employs Gaussian processes (GPs) to generate high-fidelity EOS tables that account for data and model uncertainty.  They demonstrated this new approach by generating a new uncertain EOS for gold.  GPs offer a robust predictive modeling approach, systematically handling uncertainties in both input parameters (like temperature and density) and output variables (including pressure and other thermodynamic properties).  Scientists collected new first-principles density-functional-theory (DFT) data for gold, observing its properties over maximum density compression (up to 100 g/cc) and extreme temperatures within the plasma region (reaching 100 eV).  Comparison of shock Hugoniot (pressure versus density) from the initial density ρ0 = 19.28 g/cc calculated in this work (U790), L790, Lynx790, and experiments in the literature show the alignment of current results with established benchmarks in the field.  (LLNL-ABS-860810)
 

At LLNL, the Klee shaping package from the Axom software toolkit is accelerating mesh generation in Arbitrary Lagrangian–Eulerian 3D analysis (ALE3D) by up to 100x on Sierra and up to 470x on CTS-2, often greatly reducing user set up time.

The Klee component of Axom is dramatically accelerating the ability of the ASC-supported ALE3D multiphysics code to rapidly assimilate complex geometries, via faster algorithms for shaping materials.   The ASC-supported Axom project provides a toolkit of software infrastructure components that provides high-performance computing (HPC) applications with support for spatial geometry and mesh-based algorithms, among other capabilities.  Multiphysics codes often use shaping, which is the process of overlaying a shape onto a portion of the computational mesh to generate a non-conformal description of that shape as a part.  Axom’s Klee shaping package enables many weapons applications to avoid overly complex and non-ideal computational meshes, yet still contain the representation of complex subcomponents.  This often greatly reduces user set up time for many analysis projects.

 ALE3D users had long been limited by the computation times and workflow issues related to generating and shaping tetrahedral meshes.  These problems are now being addressed via the Axom implementation of algorithms to compute the volume of a hexahedron (with non-planar faces) and the volume of intersection of a hexahedron and tetrahedron more efficiently.  Axom also provides a spatial index that provides even greater performance benefits to the shaping process.  The ALE3D team is finding a 7x to 470x improvement (depending on the specific geometry) over the previous algorithm on CTS-2 machines, and a 21x to 100x speedup on the Sierra-class of machines.  Figure 5 is a 3D rendering of the Axom logo (300k tetrahedral elements generated in Cubit) shaped into ALE3D (17M hexahedral rectilinear element mesh) which exhibited a speedup of 260x in mesh generation time on a CTS-2 system.  For more information, see https://computing.llnl.gov/projects/axom.  (LLNL-ABS-860767)

LANL has applied new models of ductile damage to assess the effects of surface defects on dynamic spall under shock loading.  These models provide insight into dynamic void nucleation and growth in metal components, which is important for stockpile stewardship. 

 Modeling the deformation behavior of metals under shock-loading conditions is important for many stockpile stewardship simulations.  During metal component manufacturing, defects such as scratches or indents may be created on a component’s surface.  These surface defects could alter the subsequent dynamic behavior of the material, leading to greater uncertainty in the overall component response under applied loads.  The ASC program at LANL aims to provide a credible scientific basis for acceptance criteria applied in manufacturing by developing simulation tools to assess the effects of surface defects on dynamic response.  In particular, LANL developed and applied new models of ductile damage to assess the effects of surface defects on dynamic spall under shock-loading conditions.

During shock loading, the material is subjected to dynamic compression, thus increasing the pressure and density behind the propagating shockwave.  When such compression waves reach an interface with air, they reflect off this surface, decompressing the material.  Interactions of these secondary decompression waves pull the material apart, leading to a substantial reduction in material density.  During this process, the material progressively degrades from within through nucleation and growth of microscopic pores, or voids.  The accumulation of porosity in the material may lead to rupture or spallation of the component.
In a long-term effort, LANL has developed and implemented an efficient and reliable computational physics model, called Tepla, in the ASC-developed multi-physics code, FLAG, to study ductile damage under shock-loading conditions.  In the present investigation, simulations of copper targets with surface defects subjected to shock loading were performed using the Tepla model in FLAG.  Various simulation results were compared to the corresponding measurements from flyer plate experiments conducted by the Dynamic Material Properties (DMP) program in the Office of Experimental Sciences (OES) at LANL.  These simulations provide insight into the experiments that could not be attained solely from measurement diagnostics.  The investigation found that surface defects can significantly change the location where rarefaction waves interact within the component and can lead to a substantial change in the magnitude and distribution of voids.  Furthermore, the geometry and size of surface defects affect the significance of these effects.  (LA-UR-23-34084)

SNL ASC completed a high-fidelity digital wind tunnel simulation of hypersonic turbulent flow over a cone.  Hypersonic turbulent flow presents a significant uncertainty in reentry vehicle flight prediction and greater confidence in simulations allows a reduction in expensive test flights.  

 Scientists within the ASC program have completed a digital wind tunnel simulation of hypersonic turbulent flow over a cone using high-fidelity direct numerical simulation (DNS).  Hypersonic turbulent flow presents a significant uncertainty in reentry vehicle flight prediction in surface heating as well as random vibrational loading.  Directly simulating hypersonic turbulent flow in a wind tunnel environment augments data from experiments that scientists are unable to measure that are required to reduce uncertainty in turbulent flow properties and impacts.  An accurate computational turbulent flow simulation capability enables scientists and engineers to improve the accuracy of lower fidelity tools that are used for hypersonic flight prediction and modeling, impacting all systems that have thermal and random vibration requirements.  The resultant impact will be reduced uncertainty in modeling and greater confidence in predictive methods allowing for a reduction in expensive test flight and ensuring weapons systems will meet their requirements across their flight envelope. (SAND2023-13842M)

LANL ASC leadership updates

Scott Doebling is the new Senior Director for the ASC program at LANL.  Scott holds a Ph.D. in Aerospace Engineering Sciences from the University of Colorado Boulder, and M.S. and B.S. degrees in Aeronautical & Astronautical Engineering from Purdue University.  Scott has worked at LANL in weapons physics and engineering for 29 years.  He has been married for 32 years and has four children.  He enjoys reading about personal finance and watching Big Ten basketball and British police dramas.  He's committed to maximizing the opportunities for girls in New Mexico to play ice hockey.

Gowri Srinivasan is the new LANL Verification & Validation (V&V) Program Manager.  Gowri Srinivasan holds a Ph.D. in Applied Mathematics and M.S. degrees in Mechanical Engineering and Applied Mathematics.  Gowri has been involved with the LANL ASC program for over 10 years, modeling complex multi-physics systems and characterizing the uncertainties in the dynamics of these systems.  Her research interests include UQ, machine learning, and reduced order modeling.  In her free time, Gowri enjoys baking, hiking with her dog, and traveling with her family. 

Jimmy Fung is the new LANL Integrated Codes (IC) Program Manager.  Jimmy Fung is a career computational physicist and leader with a broad, comprehensive knowledge of weapons physics and significant experience in mission-critical production software development, weapons applications, and partnerships with the design community.  He came to LANL in 2005 to join X Division and was recently the Deputy Leader for the X-Computational Physics (XCP) Division.  Jimmy is a graduate of the LANL Weapons Physics TITANS Program, and he recently completed the Oppenheimer Science and Energy Leadership Program (OSELP).  Jimmy received his Ph.D. in aeronautics from the California Institute of Technology.  He enjoys spending time with family and church, and is an avid supporter of Arsenal Football Club.

DJ Luscher is the new LANL Physics and Engineering Models (PEM) Program Manager.  DJ earned B.S. (1999) and M.S. (2001) degrees in civil engineering from Montana State University and a Ph.D. (2010) in mechanical engineering with minor in materials science from Georgia Tech. DJ joined LANL in 2000 as a graduate research assistant in the Engineering Sciences and Applications Division and became a technical staff member the following year.  Throughout his career, DJ has contributed to several Defense Programs projects, including ASC-PEM, ASC-V&V, and the W76 and B61 life extension programs, and led the Materials project from 2017-2023.  DJ enjoys skiing, flying airplanes, drumming, and running farther than he should. 

WELCOME ABOARD...

LLNL ASC program

Ramesh Pankajakshan was appointed as the El Capitan Center of Excellence (COE) Deputy in July 2023.  In this role, he coordinates with the vendors, Livermore Computing staff, and code developers to identify, triage, and resolve technical issues arising from the porting of TriLab ASC applications to El Capitan early access systems.  The goal is to ensure that codes are ready to run on El Capitan on Day 1 after acceptance.  Toward this goal, he is also involved in organizing seminars, hackathons, and code days to facilitate knowledge transfer between the vendors and code teams.  Ramesh holds a PhD in Computational Engineering and has over three decades of experience in modelling and simulation on HPC platforms.

Chris White is the new Weapon Simulation and Computing (WSC) DevOps Coordinator.  In this role he will be coordinating software development methodology and implementation across multiple teams to address the ever-growing needs of the program.  He is a Group Leader for the Applications, Simulations, and Quality Division and the Computer Science Lead for the Smith project, a multi-physics simulation code in the LLNL Engineering Directorate. He is also the Lead Developer for BLT, an open-source CMake based build system framework geared towards HPC, as well as a developer on Axom, an open-source Computer Science infrastructure project in the LLNL WSC Program.  He graduated from UC Santa Cruz with a Bachelor of Science in Computer Science and has been at LLNL for 13 years after working in industry.

Available Now: ASC Artificial Intelligence for Nuclear Deterrence (AI4ND) Strategy

Incredible opportunities which are emerging from artificial intelligence (AI) have truly revolutionary possibilities.  This technology can be applied to every aspect of the NNSA nuclear deterrent mission, accelerating the time needed to solve some of the nation’s toughest science challenges, but only if investments are made to re-purpose the technology to apply to the unique challenges posed by the NNSA mission.  The new AI4ND Strategy (finalized at the end of 2023) describes how the ASC program plans to deploy AI methods in the unique high-security and high-consequence environment in which the NNSA must operate.
The AI4ND Strategy is now available for download here.

NNSA LDRD/SDRD Quarterly Highlights

NNSS SDRD: Feasibility Study becomes Strategic Initiative to improve critical skills for Scorpius.

In FY 2023, the Lead for the Accelerator Beam Science and Target Interactions Science & Technology (S&T) Thrust Area, Trevor Burris, proposed an SDRD Feasibility Study to identify critical skill gaps in accelerator science that will be crucial to operating and maintaining the NNSS’ first large-scale linear accelerator, Scorpius.  Working with the scientists and engineers at the Stanford Linear Accelerator Center (SLAC), Trevor and his team identified common topics of interest that are invaluable to NNSA’s mission on Scorpius including electron accelerator operations and maintenance, beam physics, performance optimization, and machine tuning using machine learning/artificial intelligence.  Trevor’s initial study proposed that researchers engage in a mutually beneficial relationship with SLAC where NNSS mentors, SLAC advisors, students, and post-docs will all be aligned and able to collaborate on linear accelerator science.  His proposal was accepted and awarded an SDRD project for FY 2024. 

Trevor will serve as the principal investigator for this project, taking on the first Strategic Initiative for SDRD in many years.  The SDRD Program Director and Chief Scientist reinstated the call for Strategic Initiatives in the FY24 SDRD Broad Site Announcement.  Not only will this project further expand essential skills in accelerator science at the NNSS, but it will also develop accelerator capabilities for future operation and maintenance needs across the Weapons Complex, perform fundamental research to better understand and mitigate beam-target interactions, and establish opportunities to collaborate with DOE national lab partners at SLAC.

LANL LDRD: Machine learning tames huge datasets.

A machine-learning algorithm that was developed at LANL has set a world record for factorizing huge datasets.  This record was set during a test run on Oak Ridge National Laboratory’s Summit system.  The highly scalable algorithm solves hardware bottlenecks that prevent processing information from large data-rich applications that include cancer research, satellite imagery, social media networks, national security science, and earthquake research.  In the record-breaking run by the LANL team, the algorithm processed a 340-terabyte dense matrix and an 11-exabyte sparse matrix, using 25,000 GPUs.  (Read more in the full article on the LANL website, LA-UR-23-29923)

SNL LDRD: Making materials more durable through science.

SNL materials scientist Erica Redline and her team have developed a molecule that helps change the way polymers react to temperature fluctuations, which would make them more durable.  This application could be used in everything from plastic phone cases to missiles.  Polymers, which include various forms of plastics, are made up of many smaller molecules, bonded together.  (See the Sandia LabNews article on the SNL website)

Questions? Comments? Contact Us.

NA-114 Office Director: Thuc Hoang, 202-586-7050

• Integrated Codes: Jim Peltz, 202-586-7564
• Physics and Engineering Models/LDRD/SDRD: Anthony Lewis, 202-287-6367
• Verification and Validation/PSAAP/CSGF: David Etim, 202-586-8081
• Computational Systems and Software Environment: Si Hammond, 202-586-5748
• Facility Operations and User Support: K. Mike Lang, 301-903-0240