NNSA


OFFICE OF ADVANCED SIMULATION AND COMPUTING AND INSTITUTIONAL R&D PROGRAMS (NA-114)

Banner for Newsletter
 
 
The Advanced Simulation and Computing (ASC) program delivers leading-edge computer platforms, sophisticated physics and engineering codes, and uniquely qualified staff to support addressing a wide variety of stockpile issues for design, physics certification, engineering qualification, and production. The Laboratory Directed Research and Development (LDRD) and Site-Directed Research and Development (SDRD) programs fund leading-edge research and development central to the U.S. DOE national laboratories’ core missions.

Quarterly Highlights |  Volume 6, Issue 3 | July 2023

Welcome to the third 2023 issue of the NA-114 newsletter - published quarterly to socialize the impactful work being performed by the NNSA laboratories and our other partners.  I’d like to begin this edition with a special recognition of the LLNL team who worked tirelessly to put the El Capitan lease-to-own (LTO) procurement arrangements in place over the last two years (LLNL team shown in the image below).  Other highlights include:

  • SNL’s reentry vibration toolkit being leveraged by the W87-1 and W93 programs for accurate pre-flight-test predictions.
  • LLNL’s demonstration of over 20x wall-time speedup using the LLNL FEusion library in ALE3D (Arbitrary Lagrangian–Eulerian three-dimensional analysis) on the ASC El Capitan early-access platform.
  • LANL’s new capabilities for mesoscale simulations for accurate predictions of a moving shock in high explosives (HE).

Please join me in thanking the professionals who delivered the achievements highlighted in this newsletter and on an ongoing basis, all in support of our national security mission.

Thuc Hoang
NA-114 Office Director

Figure 1: Team from the LLNL ASC Program Office, LLNL Supply Chain Management, LLNL General Counsel, LLNL Office of the Chief Financial Officer, and ASC HQ (from left to right): Scott Futral, Gary Ward, Kevin Camacho, Becky Springmeyer, Rob Neely, Terri Quinn, Jonathan Celniker, Elisa Deusanio, Kim Bosque, Joanne Kim, Becky Ip, Angela Tanner, and Thuc Hoang.

SNL’s reentry vibration toolkit is leveraged by the W87-1 and W93 programs to provide accurate pre-flight-test predictions to help avoid costly overdesign or later redesign of new nuclear deterrent system components.

Figure 2: Depiction of vehicle reentry and vibration states. A reentry vehicle flight path is depicted on the left. The orange boxes depict where the transition from laminar to turbulent boundary layer occurs, while the green boxes show where the boundary layer is fully turbulent. The transitional boundary layer can provide the largest vibrational loading to components, depending on the vehicle release conditions. The turbulent boundary layer state provides the longest sustained loading and is present when the weapon needs to function.

Reentry vehicles experience significant vibration during reentry due to aerodynamic loading (see Figure 2).  This vibration is stressing to ND system components designed by SNL and currently, accurate pre-flight-test predictions are valuable for the construction of preliminary component environment specifications during new system development.  A predictive mod-sim capability for use in the design and qualification of current and future systems is critical to avoid either costly initial overdesign or later redesign.  The reentry vibration toolkit was developed to enable these vibration predictions and is being leveraged by the W87-1 and W93 programs.  There will likely be much fewer flight test opportunities in the future, so accurate mod-sim will be necessary to supplement limited flight tests and to provide vital information for new components in new environments.  (SAND2023-03189A)

 


The LLNL FEusion library capability in ALE3D has demonstrated over 20x wall-time speedup on GPU architectures enabling increased resolution and physical fidelity for explosively fragmented metal simulations on the ASC El Capitan early-access platform.

Figure 3: The Pipe Bomb Fragmentation Test demonstrates FEusion’s capabilities. The model uses a background mesh to model the HE inside a steel cylinder and the surrounding air. The fragmenting steel is simulated in the foreground. This fragmentation is very difficult to simulate using typical finite element methods using only one mesh.

Understanding the behavior of explosively fragmented metals is an important component of many national security challenges.  The Livermore FEusion library capability in the ASC Arbitrary Lagrangian–Eulerian three-dimensional analysis (ALE3D) modeling framework provides an embedded mesh method that greatly facilitates fragmentation simulations.  This method couples two separate, spatially overlapping meshes for fluid-solid and solid-solid hydrodynamic interactions.  The embedded mesh approach can greatly simplify the pre-processing requirements while potentially avoiding many run-time issues related to tangling of an ALE mesh conforming around Lagrange objects.  Furthermore, mesh coupling is allowed between two ALE3D meshes including shell and hexahedral elements, ALE3D/Paradyn coupling, and ALE3D/SPH (Spheral) coupling. 

The FEusion library capability in ALE3D was ported to GPU architectures as part of an L2 Milestone.  The milestone achieved its high-level goals, demonstrating over 20x wall-time speedup, such as for the Pipe bomb fragmentation test in Figure 3.  The enhanced performance of the GPU is allowing the team to increase the resolution of these simulations as well as the physics fidelity.  The successful port to GPU architectures positions ALE3D to successfully utilize El Capitan’s exascale capabilities in this important national security area.  (LLNL-ABS-851464)


LANL is developing capabilities for mesoscale simulations implementing inelastic deformation of granular matter for accurate predictions of a moving shock in HE to strengthen assessments for HE options in the future stockpile.

Unlike solids and fluids, the dynamic response of granular materials has the additional complexity of being dominated by the presence of irregular spaces between the grains.  Under dynamic deformation, a grain squashes from impact, crushes, and squeezes through available pore spaces.

Figure 4: Mesoscale simulations of granular materials under shock compaction using the FLAG hydrocode. Plots show a compaction wave propagating upward, away from the brass piston. Zone pressure distribution enables tracking of the elastic stress wave front along with the compaction wave front. The left pair of images show a simulation incorporating a size distribution of square shaped particles resembling table salt. The right pair of images show a simulation incorporating a size-distribution of circular particles resembling silica beads.

 

Figure 5: Using the FLAG hydrocode, flyer-plate simulations were performed on unconsolidated granular material (figure a) for identifying the effects of various consolidation conditions on the velocities of a compaction wave (figure b). Plots show a pressure wave propagating through a compacted granular matter incorporating a size-distribution of circular grain shapes (figure c).

 

The statistical nature of the complex shock response arising from tens of thousands of grains may result in a different compaction state than what was intended during its performance.  The solution to this problem is at the intersection of geoscience and computational solid mechanics, which strives to develop a physically based, mesoscale-consistent continuum description for granular materials.  LANL’s new mesoscale model seeks to predict pressure-porosity evolution, compaction-front velocity, density, temperature, and pore-distribution.  This capability will improve the assessments for high explosives under consideration for use in the future stockpile.  (LA-UR-23-24339)


SNL’s Library of Advanced Materials for Engineering now supports a modular material softening failure capability providing good experimental agreement in weld analysis; utilized for W88-0 safety analysis.

Figure 6a: On left, calibrations of J2 plasticity without (red) and with (blue) modular material softening failure compared to the experimental data (black)
Figure 6b: On right, further calibration of J2 plasticity model with modular match experimental data.

 

SNL implemented a new modular material softening failure capability in the modular plasticity framework in the Library of Advanced Materials for Engineering (LAMÉ).  Accurate modeling of material softening is often essential for failure predictions (see Figure 6, left graph).  The addition of material softening capabilities to modular plasticity greatly increases the number of material models with failure capabilities available to analysts for predicting failure.  These new failure capabilities have been integrated with Sierra/SM (Solid Mechanics) and tested to provide a robust suite of failure-enabled material models.

The newly added modular material softening failure capability was used in the analysis of a W88-0 joint weld for abnormal mechanical environments.  As shown in Figure 6 (right chart), the introduction of this failure capability also improved the calibration of a plasticity model to better match experimental data.  The inclusion of the failure mechanism better represents the physics of the system response, improves the predictive capability of the model, and yields more accurate results for the customer.  (SAND2023-01609O)


LANL’s new nuclear data evaluation reduces uncertainties in neutron scatter and capture cross sections in tantalum and significantly reduces the criticality overprediction by the U.S. nuclear data library for the new Aliquot plutonium pit casting process.

Figure 7: Monte Carlo simulations of integral experiments using three 181Ta evaluations, ENDF (red points), Japanese Evaluated Nuclear Data Library (JENDL, yellow points), and the new LANL evaluation (dark grey points). Blue points show the experimental results. Five integral experiments are shown with the hardest (fastest) neutron spectrum on the left (1.14 MeV) and the softest (0.26 MeV) on the right.

Tantalum (Ta) is used for casting in the production of plutonium pits at LANL.  The new Aliquot Casting consists of a Pu cylinder surrounded by a tantalum band.  Criticality for this casting is sensitive to inelastic, elastic, and capture cross sections of the tantalum.  A series of integral experiments revealed that the current U.S. nuclear data library, the Evaluated Nuclear Data File ENDF/B-VIII.0, significantly overpredicts criticality.  Even more pronounced discrepancies are seen in other libraries.

A new tantalum evaluation has been developed by combining selected physics measurements with modern nuclear reaction physics into a consistent set of nuclear data.  This evaluation significantly reduces the overprediction by ENDF for three fast integral experiments while preserving good performance for two lower energy experiments.  This work reduces uncertainties on tantalum cross sections and supports criticality assessments.  (LA-UR-23-24339)

 


SNL developed a simulation capability to correct distortions that can occur during metal additive manufacturing (AM) build processes enabling high confidence and quick deployment of AM parts for the W87-1.

Figure 8: Simulated distortion compensation results for “house” build geometry showing distortion magnitudes with initial as-designed geometry (left) and distortion compensated geometry (right).

An integrated team of ASC Verification & Validation (V&V) and ASC Physics and Engineering Models (PEM) researchers at SNL developed a simulation capability to correct metal AM builds for distortions that can occur during the build process.  Metal AM is a relatively new technology that could be used to enable new designs, speed prototyping, and enable improved production capabilities throughout the nuclear deterrence program.  This work is motivated by the W87-1 program and a need to improve the timeline for designing and deploying additively manufactured components.  Using this new tool will allow for quick, tractable analysis of AM builds to enable high confidence and quick deployment of AM parts for the W87-1 and future nuclear deterrence programs (see Figure 8).  Future work will validate this approach to different alloys and additional part geometries.  (SAND2023-03188A)


LANL’s Crosslink mesh generation software recently delivered a high-quality, 64 million-element computational mesh as a demonstration of intuitive, fast, parametrizable, and scalable meshing of 3D multi-material domains enabled by Crosslink's native 3D block topology creation capability.

Figure 9: Two million cell mesh for a 3-D HE shaped charge that was generated with CrossLink.

Mesh generation is one of the first steps in the simulation of complex weapon systems.  It is also one of the hardest steps because the quality and distribution of mesh cells must be carefully controlled at scale (billions to tens of billions of cells) to capture the physics accurately.  LANL's Crosslink software is designed to make meshing of complex 3D multi-material domains intuitive, fast, parametrizable, and scalable.  This will enable the LANL weapons program to perform large-scale, multi-physics simulations to inform production decisions.

Crosslink has recently delivered high-quality, 3D meshes for multi-material simulations in LANL's current and next generation production codes.  These meshes of an axisymmetric shaped charge (Figure 9), ranging from 1 million to 64 million cells, are being used to study scaling of the multiphysics codes on conventional and advanced architectures.  It should be noted that these meshes have better structure along the central axis than 2D meshes that have been spun along an axis of revolution.  This is enabled by Crosslink's native 3D block topology creation capability, a feature missing in older meshing codes. 

(LA-UR-23-24339)


LLNL added new equation-of-state (EOS) models to improve simulations of shock-compressed polymer composites, which require accurate material models for composite materials with embedded particulates, used in many national security applications.

Figure 10: Comparison of pressure (black and red) and relative velocity of the particles relative to the polymer (green) between ARES (solid lines) and ALE3D (dots). In this plot, p1 and w1 indicate pressure and relative velocity at location 1 (halfway through the sample), and p2 indicates pressure at location 2 (back of the sample).

Composite materials with embedded particulates enable many applications and require accurate material models to be properly simulated.  Shockwaves in polymer composites exhibit an interesting asymptotic pressure region, which occurs due to a viscous pressure that is created from particle-polymer interaction within the material.  The ALE3D and ARES hydrocodes have now implemented a model (Herbold, 2022) that adds an additional viscous pressure to the EOS pressure.  This viscous pressure is computed using an estimate of the relative velocity between the particles and polymer, which is a function of the local pressure gradients and particle drag.  The ALE3D implementation further improves the material model through a two-species Mie-Güeneisen EOS that accounts for the local particle mass fraction.

The model implementation was verified by comparing ALE3D and ARES simulation results for a double-shock experiment.  This experiment consists of a flyer with both a low- and high-density material.  Initially, a low-pressure shock travels through the receiving composite material.  After an open reflection on the higher density material, a second stronger shock reaches the composite.  Results show good agreement between the ARES and the ALE3D implementations.  The ARES implementation has additional agreement over experiments with a range of shock velocity and particle fill, providing confidence in the results from both hydrocodes.  (LLNL-ABS-851466)

Reference: Herbold, E., Nonequilibrium equation of state for shock compressed particulate composites. Bulletin of the American Physical Society, 67, (2022).


SNL novel analysis of dislocation effects on helium bubble formation in stainless steels guides improved design of weapon components for long-term tritium storage.

Figure 11: Molecular dynamics code LAMMPS is used to perform large ensemble simulations to study time distribution for individual He atoms to join existing bubbles with and without dislocations. Dislocations significantly reduce time, suggesting that He is attracted to the tensile regions of dislocations, making bubbles more likely to form near dislocations. Hence, modulating dislocation structures can impact material aging behaviors.

Stainless steel reservoirs used for long-term tritium gas containment suffer from aging effects that can lead to vessel cracking.  Tritium-to-helium decay leads to the formation of internal helium (He) bubbles that affect the stainless-steel embrittlement process.  A fundamental understanding of how dislocations impact He bubble formation can therefore directly guide future nuclear weapon component designs.  Modulating dislocation structures can change bubble structures and provide an effective means of improving aging behavior for components.  However, this understanding cannot be easily obtained from traditional experiments. 

SNL researchers recently made a breakthrough discovery through development of a quinary Fe-Ni-Cr-H-He interatomic potential to enable molecular dynamics (MD) simulations of He bubble formation in Fe-Ni-Cr-H systems for the first time.  This allows researchers to perform MD computational experiments and examine how long a diffusing He atom takes to join an existing bubble in the presence of dislocations as shown in Figure 11 (a).  The results of such computational experiments are highly statistical.  SNL has theoretically proved that 1,000 simulations can capture the statistics accurately.  For this work, the team performed two sets of such 1,000 simulations: one set with dislocations and the other set without dislocations.  The time distribution of the He atom to join an existing bubble is summarized in Figure 11 (b).  Notice that dislocations significantly reduce the join time.  Further analysis revealed that this reduction in time occurs because the He atom is trapped in a broad tensile region of dislocations that is closer to the bubble.  This is different from the previous belief that the He atom is trapped only at the dislocation core.  Although actual dislocations are more complex than the straight dislocations simulated (He can still be trapped at junctions of a dislocation network), this discovery guides proper application of the materials in components and informs aging model development for component performance evaluation.  (SAND2023-02935)


LLNL Flux scheduler enables increased throughput of national security workloads on El Capitan Early Access System 3 (EAS-3) hardware.

Workflow throughput on El Capitan EAS-3 hardware has increased 22 times since improvements were made to the Flux job scheduler.  It is projected that large gains in national security workflows will be easily obtained when applied to the incoming El Capitan platform, which is of the same architecture but with the next-generation GPUs.  Many NNSA workflows use two-dimensional (2D) simulations that have not performed well in the past on GPU platforms due to their small size and inability to fully utilize the GPUs.  This has been addressed in the Flux job scheduler by scheduling multiple jobs within a node, effectively over-subscribing the GPUs, and enabling them to run at their full capacity.  This dramatically increases the number of jobs that can run concurrently on machines, making them much more agile in responding to priority needs.

The availability of this increased throughput further motivated the LLNL code teams to port additional numerical kernels from the central processing units (CPUs) to GPUs to better address the needs of 2D simulations.  This resulted in additional feature-complete multi-physics codes, further increasing the ability of El Capitan to respond to computational needs.  The combination of improved multi-physics codes and Flux job scheduler were then able to demonstrate sizeable throughput efficiency gains as shown in Table 1.  (LLNL-ABS-851465)

Nodes, ranks, and runs for each class of machine

Machine

Number of Nodes

Ranks per run

Runs per 24 hours

CTS-1

4

144

14.11

EAS-3

1

4

40.25

EAS-3

1

1

79.33

Table 1:  Comparison of CTS-1 throughput to EAS-3 before (middle row) and after (bottom row) the improvements to the Flux scheduler.  The improved scheduler allows a computational study to run 79.33 simulations in 24 hours on a single node versus 4 nodes of CTS-1 required to simulate only 14.11 runs in 24 hours.  Since 4 nodes of CTS-1 are required versus only 1 of EAS-3, this indicates that a single node of EAS-3 can accomplish roughly 22x the simulation throughput of the single CTS-1 node.


Welcome Aboard...

SNL ASC program

Alden Pack

Alden Pack started supporting the ASC program in 2020 as an electromagnetic (EM) analyst and software developer for the Electromagnetic Theory and Simulation department at SNL.  He is product owner for Gemma, SNL’s next-generation, frequency-domain electromagnetic code with specialized models to evaluate EM radiation (EMR) shielding effectiveness and other normal EM environments in support of system qualification.  Gemma is part of the Radiation Analysis Modeling and Simulation of Electrical Systems (RAMSES) suite of radiation and electrical numerical codes and leverages the Kokkos library for thread-level and inter-node parallelism.  Alden also supports various nuclear deterrence programs as a ModSim lead and ModSim support.

Erin Stelter

Erin Stelter began supporting the ASC program in June 2022 as a business and financial analyst for the V&V subprogram.  Before coming to the ASC program, Erin worked at a bus manufacturer and parts distributor in Louisville, KY, providing purchasing, product, and business development, as well as project management support.  She enjoys the process of working with others and having a combined goal, such as the yearly ASC Implementation Plan, that culminates with a valuable end-product.  In her spare time, Erin enjoys the outdoors, shopping at weekend markets, and taking weekend trips around New Mexico and southern Colorado.

Jason Peters

Jason Peters began supporting the ASC program in September 2021 as a solutions architect working primarily on the schedule and storage systems that support the high-performance computing (HPC) clusters.  Before coming to the ASC program, Jason worked as a systems administrator for the MCS Linux server team at SNL.  Jason brings a wide range of experience with IT technologies including storage systems, backup, virtualization, and networking. 

 


LANL ASC program

Greg Salvesen

Greg Salvesen converted to staff at LANL in 2021.  He is an astrophysicist specializing in systems called X-ray binaries, which consist of a star feeding a disk of gas around a black hole.  His thesis work, post-doc, and projects all had components of V&V/Uncertainty Quantification.  Greg's astrophysics work mainly targets computational models and X-ray observations of black hole accretion disks.

Alex Lovell-Troy

Alex Lovell-Troy is a cloud software architect, formerly with Cray where he designed some of the software LANL uses to manage its largest systems.  He has a background in systems administration and scientific computing at various sites including the University of Illinois and the Large Binocular Telescope Observatory.  Outside of system management, Alex spent time as a consulting data architect and vice president of data consulting for a boutique managed services company.  Alex will be working on open-source efforts to improve the HPC community with a focus on next-generation HPC system management software.  He lives with his family in Northern Virginia.  His children, 9 and 12, are arguably more accomplished than he is, having been featured in local newspapers on two continents and with producer credits on horror movies.


LLNL ASC program

Kristi Belcher

Kristi Belcher joined LLNL in 2020 as a software developer in the Applications, Simulations, and Quality (ASQ) Division.  She primarily focuses on C++ feature development and continuous integration work for both the RAJA/Umpire and MARBL teams in the ASC program.  Kristi has B.S. (Texas State University) and M.S. (University of Oregon) degrees in computer science.  Kristi is involved in many outreach efforts such as recruiting at conferences and other venues, mentoring COMP interns, and volunteering at career days for local high schools.  She has several hobbies outside of work such as paddle boarding, windsurfing, and pickleball.

Bradley Davis

Bradley Davis was designated as Deputy Engineer - Facility Projects and Budget Planning in August 2022 for Livermore Computing (LC), where he leads the LC Conventional Facilities project portfolio.  He assists with the management of the HPC Integrated Project Team in collaboration with Operations & Business (O&B) to execute LC’s HPC infrastructure projects.  Brad was recognized this year with a Secretary’s Honor Achievement Award as part of the leadership team for the Exascale Computing Facility Modernization Project.  Brad is a Project Management Professional (PMP), is EVMS certified, and has a BS Industrial Engineering degree from Cal Poly, San Luis Obispo.  Brad was first introduced to LC through the Cluster Academy summer program and later interned with Anna Maria Bailey and the LC Facilities group prior to joining LLNL.


SNL LDRD: Real-time detection of noble gases could improve monitoring of underground explosions.

Figure 13: Steve Bauer, a recently retired SNL geoscientist, is preparing his sensitive mass spectrometer to detect the gases released by crushing a piece of granite. (Photo by Craig Fritz)

SNL geoscientists have detected specific gases being released from fractured rocks in real time after a series of small chemical explosions using different explosive compounds was set underground at the Energetics Materials Research and Testing Center outside of Socorro, New Mexico.  This fundamental research could one day improve the prediction of earthquakes or detection of underground explosions.  An improved understanding of how and why rocks fracture could also be valuable for enhanced geothermal systems and strain detection in unconventional oil and gas production.  (See the SNL news release)

 


LLNL LDRD: Understanding material degradation in titanium alloys.

Figure 14: Digital representation of a titanium oxide microstructure simulated using phase-fielding modeling. (Image courtesy of LLNL)

Titanium and its alloys are attractive for a wide variety of structural and functional applications due to the metal’s excellent strength, toughness and stiffness, and corrosion resistance.  Specific applications include lightweight structural materials, bioimplants, and energy storage materials.  However, if exposed to hydrogen sources, these alloys are susceptible to hydrogen incorporation and hydride formation, leading to crack initiation and mechanical failure resulting from lattice deformation and stress accumulation.  When it comes to understanding hydrogenation and hydride formation in titanium alloys, many unanswered questions and challenges remain.  (Read more in the full article on the LLNL website)

 


NNSS SDRD: Dale Turley wins “Most Valuable Principal Investigator” award with bubble collapse project.

Figure 15: Comparison of images captured for shocked and detonating NM without (top row) and with butane bubbles (bottom row).

The Special Technologies Laboratory’s Dale Turley is the winner of this year’s Most Valuable Principal Investigator (MVPI) award for his SDRD project “Study of Bubble Collapse in Optically Transparent High Explosive as a Method to Probe the Detonation Process” (22-052).  Dale Turley’s project examined what happens to a butane gas bubble when injected into homogeneous explosive nitromethane (NM) revealing the effects of hot-spot formation on the detonation process.  While previous studies have examined bubble collapse in NM, this work clearly shows evidence that the detonation process occurs near the bubble collapse and what happens when a detonation shock wave overtakes the butane bubble.  The External Advisory Board lauded this project for its “clever experimental setup [and] great visualizations [that] could lead to improved modeling capability.” Congratulations to Dale Turley on his fantastic project and MVPI win.  (Read more here and in the SDRD Annual Report Overview, now available on NNSS.gov).


SNL LDRD: Employing a cyber shuffle to stop hackers from taking over military aircraft.

Figure 16: Part of the SNL team working on a moving target defense that makes computer networks commonly used on space and aircraft less vulnerable to cyberattack stand aboard a C-130 transport aircraft. (Photo by Craig Fritz)

A cybersecurity technique that shuffles network addresses like a blackjack dealer shuffles playing cards could effectively befuddle hackers gambling for control of a military jet, commercial airline, or spacecraft.  Many aircraft, spacecraft, and weapons systems use an onboard computer network, the military standard 1553 (MIL-STD-1553), to let systems like radar, flight controls, and the heads-up display talk to each other.  SNL partnered with researchers at Purdue to show that a moving target defense that utilizes randomness can effectively protect MIL-STD-1553 networks against sophisticated hacking. (Read more here.)

 


Questions? Comments? Contact Us.

NA-114 Office Director: Thuc Hoang, 202-586-7050

  • Integrated Codes: Jim Peltz, 202-586-7564
  • Physics and Engineering Models/LDRD: Anthony Lewis, 202-287-6367
  • Verification and Validation: David Etim, 202-586-8081
  • Computational Systems and Software Environment: Si Hammond, 202-586-5748
  • Facility Operations and User Support: K. Mike Lang, 301-903-0240
  • Advanced Technology Development and Mitigation: Thuc Hoang

Advanced Simulation and Computing