OFFICE OF ADVANCED SIMULATION AND COMPUTING AND INSTITUTIONAL R&D PROGRAMS (NA-114)

Welcome to the second 2023 issue of the NA-114 newsletter - published quarterly to socialize the impactful work being performed by the NNSA laboratories and our other partners.  This issue begins with a highlight from Sandia National Laboratories (SNL) in which their collaboration with University of Utah and NASA Langley Research Center led to the application of artificial intelligence (AI) techniques improving continuum damage models used in analysis of additively manufactured (AM) materials.  This issue also “Welcomes Aboard” new staff who have joined the ASC program at SNL, Lawrence Livermore National Laboratory (LLNL), and Los Alamos National Laboratory (LANL).  Other highlights include: 

• LANL’s new Fast Sweeping Detonation (FSD) library accurately matching experimental pressure data for the Cyclops I code validation experiment better than traditional programmed burn models for high explosives (HE).
• The tri-laboratory Remote Computing Enablement (RCE) cross-site team’s demonstration of reciprocal and automated cross-site testing capabilities for code development between the NNSA trilabs.
• LANL’s 8X improvement in solution time for a 6 million cell hydrodynamics (XRage) problem using early-access Intel processors with high-bandwidth memory (HBM), which are to be included in the ASC Crossroads system.
• SNL’s generalization and improvement of the SIERRA Solid Mechanics bolt preloading capability, reducing input file lines on the order of 10:1, and directly benefiting the W80-4 analysis team.

Please join me in thanking the professionals who delivered the achievements highlighted in this newsletter and on an ongoing basis, all in support of our national security mission.

Thuc Hoang
NA-114 Office Director

In collaboration with the University of Utah and NASA Langley Research Center, SNL applied AI techniques to improve outdated continuum damage models and demonstrated their application in SIERRA/Solid Mechanics for additively manufactured materials.

Computational simulation of plasticity and ductile fracture is a core element of SNL’s safety and security nuclear deterrence (ND) mission.  The existing continuum damage models were derived with semi-analytical work in the late 70’s and early 80’s and are limited in their physical fidelity, which can lead to poor predictions for some loadings and some materials.  Through a collaborative investigation with the University of Utah and NASA Langley Research Center, SNL is developing genetic programming with symbolic regression (GPSR) to construct analytical expressions for continuum damage models that better encapsulate materials mechanisms.  These analytical expressions provide two benefits unique to the approach: 1) the opportunity to interpret and understand the results of the algorithms and mechanics; and 2) seamless integration with existing analyst workflows in ASC codes such as SIERRA/Solid Mechanics. The current exemplar focuses on porous additively manufactured (AM) stainless steel, with characterization and validation data from the 3rd Sandia Fracture Challenge (SFC3).  AM represents exciting opportunities for mission agility for ND systems and accurate models are needed for design and qualification. SNL developed training data by direct numerical simulation of pore growth representative of the observed material.  A new pore evolution equation using GPSR was generated and inserted into the existing constitutive model by modifying one line of code.  The team then recompiled SIERRA/Solid Mechanics and ran original finite element input decks with no additional changes.  The result was an interpretable analytical expression with familiar terms, such as stress triaxiality and plastic strain rate, and is summarized in the force versus displacement response plotted in Figure 1. While being developed specifically for this AM material, the GPSR approach was minimally invasive and performed as well as existing models.  Future work aims to improve the GPSR models by addressing stress-state complexity, adding micromechanics, and accounting for inherent materials variability.  (SAND2023-00022O) 

LANL’s new Fast Sweeping Detonation (FSD) library accurately matches experimental pressure data for the Cyclops I code validation experiment better than traditional programmed burn models for HE and can be thousands of times faster.

Modeling the detonation behavior of HE is important for assessing stockpile safety and performance.  For studies of experimental design and accident scenarios it is often necessary to run an enormous number of simulations.  This has been a challenge since modeling HE with past ASC models has been very computationally expensive.  The newly developed Fast Sweeping Detonation (FSD) library provides enormous improvements in both speed and accuracy for simulations with ASC codes.  Figure 2 and Figure 3 show comparisons of the new capability with traditional programmed burn models.  For a simulation of a detonation wave propagating though an arced geometry, the new FSD model is highly accurate and can be thousands of times faster than the traditional models.  For the simulation of Cyclops I (the standard experiment used for code validation), the new capability accurately matches experimental data while the older models predict pressures that are far too high.  The improvements in speed and accuracy with these new models will allow better solutions for a broad range of nuclear security challenges.  (LA-UR-23-22074) 

The tri-laboratory Remote Computing Enablement cross-site Continuous Integration working group demonstrated reciprocal, automated, and remote cross-site testing capabilities for applications code development between LLNL, LANL, and SNL.  

The Remote Computing Enablement (RCE) cross-site Continuous Integration (CI) working group achieved significant building and testing milestones this quarter.  Reciprocal capability for automated, remote, cross-site testing for applications code development was demonstrated between LLNL, LANL, and SNL.  Teams at LANL conducted build and test operations remotely on LLNL’s systems rzAnsel and rzVernal through their local Gitlab CI infrastructure.  In addition to running basal capability tests, the ASC Transport code team built and tested their Draco library on rzAnsel. Other LANL teams are queued up to perform similar testing with their codes. With this addition, cross site CI capabilities have now been proven in multiple directions between the NNSA high performance computing (HPC) tri-labs in this pre-production limited availability mode.  LLNL teams demonstrated builds on LANL’s Trinitite system in November 2022.  In January 2023, the SNL SPARC code team established nightly testing results from rzVernal at LLNL, which automatically report back to their SPARC “cdash” dashboard at SNL.  Full production capability for cross-site CI is expected in April 2023. The RCE Authentication working group also achieved a milestone on March 1, 2023.  Teams from LANL and LLNL completed the final steps to enable a fully realized cross-realm trust between Kerberos authentication services at the two sites.  The LANL Restricted Enclave (RE) network and the LLNL Restricted Zone (RZ) network now honor each other’s Kerberos credentials.  The primary benefit of this is that users from each environment can obtain a local Kerberos credential and use it to seamlessly access the remote equivalent environment without being prompted for a password.  (LLNL-ABS-847762)

LANL demonstrated an 8X improvement in solution time for a 6 million cell hydrodynamics (XRage) problem using early-access Intel Xeon processors with high-bandwidth memory to be included in Crossroads, ASC’s third Advanced Technology system.

Crossroads is the third of the Advanced Technology systems to be deployed by the ASC program.  This supercomputer was designed to simulate some of the most complex physical devices in the world.  These simulations routinely require 1/2 petabyte or more of system memory running on thousands of compute nodes for months at a time.  Improvements in time to solutions for these workloads have a major impact on NNSA mission capabilities. Early results for the Crossroads processor (4th Gen Intel Xeon processors with HBM) show marked improvements over prior generation systems.  Some problems that used to take months can now be completed in appreciably shorter times (see Table 1 and Figure 4).  These findings indicate that Crossroads will be a powerful tool for stockpile stewardship and modernization once it is deployed in the classified environment later this year.  (LA-UR-23-21214)

SNL generalized the SIERRA/Solid Mechanics bolt preloading capability to allow multiple bolts to be independently preloaded, reducing input file lines on the order of 10:1, and directly benefiting the W80-4 abnormal mechanical analysis team by simplifying workflows.

Accurate modeling of initial loads in bolts and fasteners is critical to simulation of ND systems undergoing abnormal mechanical events (e.g., crash, crush, or drop).  A “preload” simulation is typically performed prior to a full abnormal event simulation to model the system state after assembly, including tightened bolts that induce contact between separate parts with resulting material stress.  Previously W80-4 abnormal mechanical simulations had to use numerous separate line commands in the preload simulation input file to specify different preloads on bolts and fasteners.  A group of bolts could be combined only if they shared identical characteristics such as direction, load, size, and shape.  A similar challenge was identified by colleagues at the UK Atomic Weapons Establishment (AWE).  For a system with many fasteners, the extensive list of line commands became difficult and unwieldy. Work in FY23 Q1 generalized SIERRA/Solid Mechanics bolt preloading capabilities to allow multiple bolts to be independently preloaded to a given target using a single line command.  This generalization greatly simplifies workflows and input files, reducing preload lines on the order of 10:1 and helps analysts to be more efficient and avoid errors.  The new generalized capabilities separate bolts in a group by looking for contiguous blocks of material.  Each bolt is then treated as if the analyst had specified its preload individually in the input file, allowing unique loading directions (bolts are automatically preloaded in the appropriate direction based on geometry) and independent measurement of preload.  The ability to preload groups of bolts or fasteners in this way came from a partnership with AWE to extend previous ASC Integrated Codes work on individual bolts and on capabilities to detect disconnected parts (originally developed to detect fragments after severe failure and fracture events).  These needed workflow improvements were identified through direct collaboration with system analysts and are being applied by the W80-4 abnormal mechanical analysis team and their AWE counterparts.  (SAND2023-00019O) 

LANL employs machine learning optimization techniques to develop a new equation of state for gold that provides a standard for experimentalists to calibrate measurements for other materials such as uranium and plutonium.

To predict and design energetic systems, we need a highly accurate understanding of how materials behave at enormous pressures.  Experimentalists rely on well-studied materials to serve as standards against which measurements for other materials are calibrated.  Gold is one of the most important of these standards.  Any errors in our models for gold would propagate into errors in our understanding of materials like uranium and plutonium. To improve ASC simulation capabilities for stockpile modernization, a new equation of state for gold has been developed.  The new equation of state builds on state-of-the-art theoretical calculations and experimental measurements.  It was developed using machine learning optimization techniques that provide a means to minimize systematic differences between theoretical estimates, experimental data, and modeling choices.  The figures show results for the new EOS, which represents a significant advance in this important standard for understanding nuclear materials at high pressures.  (LA-UR-23-21214)

A tri-laboratory ASC/PEM material strength model development effort made crucial progress toward a common understanding of the physics of multi-phase material response, resulting in implementation of the Common Model for Multi-Phase (CMMP) response at all three NNSA laboratories.

A tri-lab effort on material strength has been under way for more than five years, and early successes of the collaboration were recognized with a 2018 Defense Programs Award of Excellence.  While initial efforts focused on tantalum strength, the team has shifted emphasis to tackle the additional complexities associated with strength of multi-phase materials.  A recent success was the publication in early FY23 of a tri-lab article that describes a multi-phase modeling framework that has been implemented at all three of the NNSA laboratories (see source [1]).  This baseline model, called the Common Model for Multi-Phase response (CMMP), includes a range of physics that are important for the dynamic response of materials that undergo phase changes.  Specifically, each phase has its own equation of state, melt and shear modulus model, and strength model.  As the material transforms, phase fractions evolve according to either phase equilibrium or phase kinetics, and the modeling framework tracks the transfer of state descriptors among the phases.  The modeling framework builds from previous contributions in the tri-lab community [2-4], and significant effort has been devoted to verification and validation of the implementations in an ASC setting. While relatively simple given the breadth of physical processes at play, CMMP is effective at capturing observed material behaviors, facilitating the interpretation of focused experiments.  Per-phase strength, phase kinetic rates, and other details of the model have direct manifestations in the observed material response, with Figure 9 giving an example for phase kinetics effects.  Deconvolving the combined effects can be a challenge, and Bayesian model calibration offers a promising path forward [5], with uncertainties in the inferred model parameters providing guidance for further experimental and modeling work. CMMP serves as a useful point of departure for model specialization at the various labs, and a FY21 PEM L2 milestone (MRT 7849, “Multiphase Strength Capability with Information from Multiscale Modeling for Two Phases”) provides an early example of such specialization in per-phase strength model forms.  Future specializations could relate to the coupling of shear and volume change effects during phase transformations, which manifest both in macroscopic response and in microstructural features such as those shown in Figure 10.  All three NNSA laboratories have benefited from this collaborative effort, and the advances in ASC/PEM have been crucial to the progress toward a common understanding of the physics of multi-phase material response.  Figures shown are adapted from source [1].  (LLNL-ABS-847781)

[1]N.R. Barton, D.J. Luscher, C. Battaile, J.L. Brown, M. Buechler, L. Burakovsky, S. Crockett, C. Greeff, A.E. Mattsson, M.B. Prime, and W.J. Schill. A Multi-Phase Modeling Framework Suitable for Dynamic Applications. Metals, 2022. https://doi.org/10.3390/met12111844  
[2]    N. R. Barton, A. Arsenlis, M. Rhee, J. Marian, J. V. Bernier, M. Tang, and L. Yang. A multi-scale strength model with phase transformation. AIP Conference Proceedings, 2012. https://doi.org/10.1063/1.3686570 
[3]    C.W. Greeff. A model for phase transitions under dynamic compression. Journal of Dynamic Behavior of Materials, 2016. https://doi.org/10.1007/s40870-016-0080-4 
[4]    W. Schill, R. Austin, J. Brown, and N. Barton. N. Anelasticity and Phase Transition During Ramp-Release in Tin. Journal of Dynamic Behavior of Materials, 2021. https://doi.org/10.1007/s40870-020-00278-0  
[5]    N.R. Barton,  D.J. Benson, and R. Becker. Crystal level continuum modelling of phase transformations: the α→ϵ transformation in iron. Modelling and Simulation in Materials Science and Engineering, 2005. https://dx.doi.org/10.1088/0965-0393/13/5/006  

Automated design optimization tools are being tested on the ASC Sierra system at LLNL to demonstrate initial physics results ahead of state-of-the-art design exploration studies on El Capitan for future facility upgrade options at the NIF.

The NIF is considering future facility upgrades that could enable robust, high yield Inertial Confinement Fusion (ICF) experiments to be conducted on a routine basis.  Two upgrade options under consideration would enable either higher peak powers than currently available, or higher total energy – but not both.  To determine which upgrade would lead to the highest benefits in ICF research, extensive simulations studies scoping out the potential design spaces are performed for each upgrade option.  While planning to eventually perform state-of-the-art design exploration studies on El Capitan, many of the workflow and optimization tools are being tested on machines like the ASC Sierra system at LLNL to demonstrate capabilities and generate initial physics results. A recent computing campaign using the workflow tools optimized 1D HYDRA hohlraum simulations to find the combination of capsule and laser inputs that lead to robust, high yield performance for a 3MJ, 440TW NIF upgrade, and a 2.6MJ, 600TW upgrade.  Two optimization studies were performed.  The first ran a set of around 500 Latin hypercube sampled simulations that span a 6D parameter space, and then are used to train a neural network to be leveraged in the optimization workflow.  This study optimizes for robust high performance by defining a robustness metric that quantifies the variation in yield due to fluctuations in target and laser delivery (i.e., layer thicknesses that differ from requests, or laser power that differs from requests) and combining it with a performance metric (ITFX ~ yield * rhoR^2) in the cost function to be maximized.  The second study optimizes the neutron yield with HYDRA simulations in the loop, demonstrating the ability to iteratively launch HYDRA with an optimization workflow powered by encore/maestro. Figure 11 illustrates the results of the optimization workflow for a 3MJ 450TW NIF upgrade scenario.  The algorithm is optimizing a combination of performance (ITFX) and robustness (insensitivity to input fluctuations), and converges in about 25 iterations, or 125 simulations.  The next step for this work is to leverage multi-fidelity optimization capabilities currently being developed for the El Capitan system to incorporate 2D hohlraum simulations into the workflow. The multi-fidelity algorithm weighs the cost of each fidelity of simulation (i.e., 1D and 2D) and the expected information gain to selectively run the higher-fidelity simulations at the most promising points in design space, while using the lower-fidelity models to explore the full space for other high-performance regions.  These tools will be used to provide the NIF facility managers a suite of designs that illustrate the performance benefits each upgrade scenario provides.  (LLNL-ABS-847772) 

SNL uses ASC tools to optimize the design of combined thermal-mechanical validation experiments for Delivery Environments to support future nuclear weapon programs.

Commonly, data from experimental tests is used to validate models and add credibility to engineering simulation predictions.  In this work, SNL ASC researchers completed simulation and optimization HPC runs to ensure that the test unit design and the test plan for an FY23 combined mechanical-thermal environment test series will meet desired test performance and safety targets.  The Delivery Environments (DE) testing group requested that two test configurations be considered and prescribed the desired responses for each.  The first is a top mount setup, see configuration (a) in Figure 12, and the second is a bottom mount configuration, see (b) in the figure.  The desired response for the top mount setup is to have the internal rods holding the cylindrical caps fail, altering the thermal pathway for subsequent thermal testing.  The desired response for the bottom mount configuration is for the mount to tear so that the inner can assembly comes into contact with the inner wall of the outer cone.  To achieve these response goals, the simulation team performed targeted design optimization and simulation studies using ASC capabilities and resources.  Ultimately, by utilizing previously validated modeling approaches and material models, it was determined that using a drop speed of 40 ft/s and a modified mounting deck would result in the desired test article response and performance.  These simulation-based analyses verified the design performance (failure/survival of components and modification of thermal pathways with designed loading) of the test articles under consideration and helped the experimentalist team select appropriate test equipment and ensure safe testing conditions.  In turn, it is expected that the combined environments thermal-mechanical results from this test series will be used by ASC researchers later in FY23 to further improve and validate combined environments mechanical-thermal models.  (SAND2023-12372O)

LANL has developed a new algorithm to detect shocks and calculate their speed in engineered systems with combined shocks and ejecta without employing the approximations that limited the predictive range of previous computational methods.

The performance of many systems important for national security depends on the behavior of energetic shocks.  Accurately modeling these systems has been an area of active research since the days of the Manhattan project.  One of the most important longstanding challenges was associated with the interplay between shocks and ejecta.  Previous computational methods have either had significant uncertainties or employed approximations that limited the range over which performance could be predicted. A new two-part algorithm was developed to strengthen the ability for predicting engineered systems that have both shocks and ejecta.  The first part involves detecting a shock as it travels throughout the computational domain.  The second part accurately determines the shock speed through a derived expression as a function of pre- and post-shock densities and pressures.  This is all done dynamically as the simulation proceeds without any post-processing, making it possible to use in conjunction with advanced ejecta models.  Figures 13 and 14 illustrate the improvements in ASC code capabilities for nuclear security.  (LA-UR-23-21214)

Welcome Aboard...

SNL ASC program

Jeff Zaine became the Mechanical Safety & Security Subelement lead for the ASC V&V program at SNL in November of 2022.  Jeff  is currently the manager of Sandia’s Multiphysics Modeling and Simulation department in Livermore, CA and came to the ASC program from the W80-4 program at SNL where he was a weapon system engineer and a product realization team lead.  He has a bachelor’s degree in Mechanical Engineering from the University of California at Santa Barbara and an MBA from Florida State University.  Prior to joining Sandia in 2019, Jeff had an extensive background in product development and insertion at Space Systems/Loral (SS/L).  He spent ten years within the antenna engineering organization designing and qualifying advanced composite reflectors and antenna feeds for satellite applications.  Jeff’s subelement focuses on developing, verifying, and validating science-based physics models and simulation capabilities supporting nuclear safety assessments in relevant mechanical environments, such as manufacturing, stockpile-to-target sequence (STS) and storage, and the development of advanced security systems.  The project has direct programmatic impact on the design and qualification activities of Sandia’s life extension programs and component product realization teams.

Michael Buche started as a postdoc in the Computational Solid Mechanics & Structural Dynamics Department at SNL in June 2021 after completing his PhD in Theoretical and Applied Mechanics at Cornell, where he established a background in constitutive modeling, statistical thermodynamics, applied mathematics, and general mechanics.  Among several other projects, Michael was introduced to an ongoing ASC/PEM project which was developing a composite wedge localization element.  The mathematical formulation was challenging, but after Michael completed his initial development, he worked with the team to implement it in Sierra.  Now Michael is a Senior Member of Technical Staff, is the PI of the project, and is finalizing the development of the composite wedge localization element.  This capability will be used in Sierra combined with FRANC3D to model ductile fracture in engineering materials.

LANL ASC program

Diego Lonardoni is a physicist with a background in theoretical and computational physics.  His experience includes theoretical modeling, code development, and HPC.  He is an expert in quantum Monte Carlo methods applied to the study of nuclear systems, from structure/dynamics calculations of finite nuclei to the study of neutron stars.  His interests range from nuclear physics and astrophysics, to hydrodynamics, plasma physics, continuum mechanics, and software development.  Diego earned a Ph.D. in Physics at the University of Trento, Italy, and moved to Argonne National Laboratory as a postdoc in the Theory Group.  He continued his research career as a fellow/research professor with the Facility for Rare Isotope Beams at Michigan State University while a guest scientist in LANL's Theoretical Division's Nuclear and Particle Physics, Astrophysics and Cosmology group.  He finally joined the XCP-2 Eulerian Codes group at LANL in late January 2022.  His efforts are currently focused on material strength modeling, nuclear reactions in plasmas, inertial confinement fusion applications, and code modernization.

Nathan Vaughn-Kukura earned his undergraduate degrees in physics and applied math from the University of North Carolina (2015) and began at LANL as a student in ASC’s 2017 Computational Physics Student Summer Workshop.  Nathan went on to earn a PhD in Applied & Interdisciplinary Math from the University of Michigan (2020) and continued at LANL as a postdoc in the Lagrangian Codes Group (XCP-1) in 2021.  With interests in the development of algorithms and numerical methods for computational physics applications and their implementation in HPC environments, Nathan joined LANL as a staff scientist in XCP-1 where he is developing a new material contact algorithm for the Lagrangian Applications Project and is working on the Ristra project's new low energy density multiphysics code targeting modern HPC systems. For Ristra, Nathan has been instrumental in the implementation of the single and multi-material hydrodynamics.

LLNL ASC program

Johannes Doerfert joined LLNL in September of 2022 as a member of the Center for Applied Scientific Computing (CASC).  His involvement in the LLVM compiler framework (since 2014) and the OpenMP language standard (since 2018) will be a great benefit to ASC program computational efforts.  Johannes was recently named one of the 2023 Better Scientific Software (BSSw) fellows, a government-funded program providing recognition and funding to leaders and advocates of high-quality scientific software.

Benjamin Isaac joined the Design Physics division at LLNL in 2019 as a computational physicist on LLNL’s ALE3D code.  He is an alumni of the Predictive Science Academic Alliance Program II (PSAAP II) program, which supported his research at the University of Utah, and allowed him to interact with both LANL and LLNL.  He continues development of ALE3D code, focusing on chemistry and radiation, as well as participating in enabling graphics processing unit (GPU) capabilities in the code.

SNL LDRD: Restoring power to the grid.

Researchers create model to improve “black starts.”

SNL scientists have been working on an innovative computer model to help grid operators quickly restore power to the electric grid after a complete disruption, a process called a “black start.” Their model combines a restoration optimization model with a computer model of how grid operators would make decisions when they don’t have complete knowledge of every generator and distribution line.  The model also includes a physics-based understanding of how the individual power generators, distribution substations and power lines would react during the process of restoring power to the grid.  (See the full article in SNL LabNews)

LLNL LDRD: Chemists double down with breakthrough method to study radioactive materials.

A new approach developed at LLNL allows for the study of radioactive and/or precious elements in a much more efficient way, requiring 1,000 times less materials than previous state-of-the-art methods, without compromising the data quality. The method and its application have recently been reported in two successive articles, one in Nature Chemistry and one in Inorganic Chemistry. The research was featured on the front cover of both journals, as well as highlighted in the journal Nature.  (See the full article on the LLNL website)

SNL LDRD: Securing supply chains with quantum computing.

Programming technique could help solve massive optimization problems. The Russo-Ukrainian conflict and the COVID-19 pandemic have shown how vulnerable global supply chains can be.  International events can disrupt manufacturing, delay shipping, induce panic buying and send energy costs soaring.  New research in quantum computing at SNL is moving science closer to being able to overcome supply-chain challenges and restore global security during future periods of unrest. Alicia Magann, a Truman Fellow at SNL, has led the development of a new way to design programs on quantum computers, which she and her team think could be especially useful for solving these kinds of massive optimization problems someday in the future when quantum technology becomes more mature.  Alicia Magann, along with SNL scientists Kenneth Rudinger, Mohan Sarovar, and Matthew Grace developed Feedback-based Algorithm for Quantum Optimization, or FALQON, as a new framework for programming quantum computers, an emerging technology that could become a powerful tool for global security. (See the full article in SNL LabNews)

Questions? Comments? Contact Us.

NA-114 Office Director: Thuc Hoang, 202-586-7050

• Integrated Codes: Jim Peltz, 202-586-7564
• Physics and Engineering Models/LDRD: Anthony Lewis, 202-287-6367
• Verification and Validation: David Etim, 202-586-8081
• Computational Systems and Software Environment: Si Hammond, 202-586-5748
• Facility Operations and User Support: K. Mike Lang, 301-903-0240
• Advanced Technology Development and Mitigation: Thuc Hoang