Quarterly Highlights | Volume 6, Issue 3 | July 2023
In This Issue
ASC & LDRD Community—Upcoming Events (at time of publication)
- International Conference on Neuromorphic Systems (ICONS) in Santa Fe, NM (with virtual hybrid option); August 1 – 3
- 39th Conference on Uncertainty in Artificial Intelligence (UAI 2023), University Center at Carnegie Mellon University, Pittsburgh, PA; August 1 – 4
- PSAAP IV Pre-Proposal Meeting at the Houston Airport (IAH) Marriott in Houston, TX; August 8 – 9
- Monterrey Data Conference, Monterrey CA; August 21 – 24*
- DARPA/ERI Summit, Seattle, WA; August 22 – 24*
- HotChips Conference, Virtual/Stanford University, CA; August 27 – 29
- 32nd ASME Annual Conference on Information Storage and Processing Systems at Western Digital - Milpitas in San Jose, CA; August 28 – 30
- Smoky Mountains Conference 2023, Knoxville, TN; August 29 – 31*
- W93 Tech Summit, DOE Forrestal; September 6-7
- Nuclear Explosives Code Development Conference (NECDC), LANL; October 16 – 20
- 2023 DOE Data Days (D3) Workshop at LLNL; October 24 – 26
- SCDS Full Summit at DOE Forrestal (Save-the-Date); October 24 – 26
- Predictive Science Panel (PSP) at LANL; November 7 – 10
Welcome to the third 2023 issue of the NA-114 newsletter - published quarterly to socialize the impactful work being performed by the NNSA laboratories and our other partners. I’d like to begin this edition with a special recognition of the LLNL team who worked tirelessly to put the El Capitan lease-to-own (LTO) procurement arrangements in place over the last two years (LLNL team shown in the image below). Other highlights include:
- SNL’s reentry vibration toolkit being leveraged by the W87-1 and W93 programs for accurate pre-flight-test predictions.
- LLNL’s demonstration of over 20x wall-time speedup using the LLNL FEusion library in ALE3D (Arbitrary Lagrangian–Eulerian three-dimensional analysis) on the ASC El Capitan early-access platform.
- LANL’s new capabilities for mesoscale simulations for accurate predictions of a moving shock in high explosives (HE).
Please join me in thanking the professionals who delivered the achievements highlighted in this newsletter and on an ongoing basis, all in support of our national security mission.
NA-114 Office Director
SNL’s reentry vibration toolkit is leveraged by the W87-1 and W93 programs to provide accurate pre-flight-test predictions to help avoid costly overdesign or later redesign of new nuclear deterrent system components.
Reentry vehicles experience significant vibration during reentry due to aerodynamic loading (see Figure 2). This vibration is stressing to ND system components designed by SNL and currently, accurate pre-flight-test predictions are valuable for the construction of preliminary component environment specifications during new system development. A predictive mod-sim capability for use in the design and qualification of current and future systems is critical to avoid either costly initial overdesign or later redesign. The reentry vibration toolkit was developed to enable these vibration predictions and is being leveraged by the W87-1 and W93 programs. There will likely be much fewer flight test opportunities in the future, so accurate mod-sim will be necessary to supplement limited flight tests and to provide vital information for new components in new environments. (SAND2023-03189A)
The LLNL FEusion library capability in ALE3D has demonstrated over 20x wall-time speedup on GPU architectures enabling increased resolution and physical fidelity for explosively fragmented metal simulations on the ASC El Capitan early-access platform.
Understanding the behavior of explosively fragmented metals is an important component of many national security challenges. The Livermore FEusion library capability in the ASC Arbitrary Lagrangian–Eulerian three-dimensional analysis (ALE3D) modeling framework provides an embedded mesh method that greatly facilitates fragmentation simulations. This method couples two separate, spatially overlapping meshes for fluid-solid and solid-solid hydrodynamic interactions. The embedded mesh approach can greatly simplify the pre-processing requirements while potentially avoiding many run-time issues related to tangling of an ALE mesh conforming around Lagrange objects. Furthermore, mesh coupling is allowed between two ALE3D meshes including shell and hexahedral elements, ALE3D/Paradyn coupling, and ALE3D/SPH (Spheral) coupling.
The FEusion library capability in ALE3D was ported to GPU architectures as part of an L2 Milestone. The milestone achieved its high-level goals, demonstrating over 20x wall-time speedup, such as for the Pipe bomb fragmentation test in Figure 3. The enhanced performance of the GPU is allowing the team to increase the resolution of these simulations as well as the physics fidelity. The successful port to GPU architectures positions ALE3D to successfully utilize El Capitan’s exascale capabilities in this important national security area. (LLNL-ABS-851464)
LANL is developing capabilities for mesoscale simulations implementing inelastic deformation of granular matter for accurate predictions of a moving shock in HE to strengthen assessments for HE options in the future stockpile.
Unlike solids and fluids, the dynamic response of granular materials has the additional complexity of being dominated by the presence of irregular spaces between the grains. Under dynamic deformation, a grain squashes from impact, crushes, and squeezes through available pore spaces.
The statistical nature of the complex shock response arising from tens of thousands of grains may result in a different compaction state than what was intended during its performance. The solution to this problem is at the intersection of geoscience and computational solid mechanics, which strives to develop a physically based, mesoscale-consistent continuum description for granular materials. LANL’s new mesoscale model seeks to predict pressure-porosity evolution, compaction-front velocity, density, temperature, and pore-distribution. This capability will improve the assessments for high explosives under consideration for use in the future stockpile. (LA-UR-23-24339)
SNL’s Library of Advanced Materials for Engineering now supports a modular material softening failure capability providing good experimental agreement in weld analysis; utilized for W88-0 safety analysis.
SNL implemented a new modular material softening failure capability in the modular plasticity framework in the Library of Advanced Materials for Engineering (LAMÉ). Accurate modeling of material softening is often essential for failure predictions (see Figure 6, left graph). The addition of material softening capabilities to modular plasticity greatly increases the number of material models with failure capabilities available to analysts for predicting failure. These new failure capabilities have been integrated with Sierra/SM (Solid Mechanics) and tested to provide a robust suite of failure-enabled material models.
The newly added modular material softening failure capability was used in the analysis of a W88-0 joint weld for abnormal mechanical environments. As shown in Figure 6 (right chart), the introduction of this failure capability also improved the calibration of a plasticity model to better match experimental data. The inclusion of the failure mechanism better represents the physics of the system response, improves the predictive capability of the model, and yields more accurate results for the customer. (SAND2023-01609O)
LANL’s new nuclear data evaluation reduces uncertainties in neutron scatter and capture cross sections in tantalum and significantly reduces the criticality overprediction by the U.S. nuclear data library for the new Aliquot plutonium pit casting process.
Tantalum (Ta) is used for casting in the production of plutonium pits at LANL. The new Aliquot Casting consists of a Pu cylinder surrounded by a tantalum band. Criticality for this casting is sensitive to inelastic, elastic, and capture cross sections of the tantalum. A series of integral experiments revealed that the current U.S. nuclear data library, the Evaluated Nuclear Data File ENDF/B-VIII.0, significantly overpredicts criticality. Even more pronounced discrepancies are seen in other libraries.
A new tantalum evaluation has been developed by combining selected physics measurements with modern nuclear reaction physics into a consistent set of nuclear data. This evaluation significantly reduces the overprediction by ENDF for three fast integral experiments while preserving good performance for two lower energy experiments. This work reduces uncertainties on tantalum cross sections and supports criticality assessments. (LA-UR-23-24339)
SNL developed a simulation capability to correct distortions that can occur during metal additive manufacturing (AM) build processes enabling high confidence and quick deployment of AM parts for the W87-1.
An integrated team of ASC Verification & Validation (V&V) and ASC Physics and Engineering Models (PEM) researchers at SNL developed a simulation capability to correct metal AM builds for distortions that can occur during the build process. Metal AM is a relatively new technology that could be used to enable new designs, speed prototyping, and enable improved production capabilities throughout the nuclear deterrence program. This work is motivated by the W87-1 program and a need to improve the timeline for designing and deploying additively manufactured components. Using this new tool will allow for quick, tractable analysis of AM builds to enable high confidence and quick deployment of AM parts for the W87-1 and future nuclear deterrence programs (see Figure 8). Future work will validate this approach to different alloys and additional part geometries. (SAND2023-03188A)
LANL’s Crosslink mesh generation software recently delivered a high-quality, 64 million-element computational mesh as a demonstration of intuitive, fast, parametrizable, and scalable meshing of 3D multi-material domains enabled by Crosslink's native 3D block topology creation capability.
Mesh generation is one of the first steps in the simulation of complex weapon systems. It is also one of the hardest steps because the quality and distribution of mesh cells must be carefully controlled at scale (billions to tens of billions of cells) to capture the physics accurately. LANL's Crosslink software is designed to make meshing of complex 3D multi-material domains intuitive, fast, parametrizable, and scalable. This will enable the LANL weapons program to perform large-scale, multi-physics simulations to inform production decisions.
Crosslink has recently delivered high-quality, 3D meshes for multi-material simulations in LANL's current and next generation production codes. These meshes of an axisymmetric shaped charge (Figure 9), ranging from 1 million to 64 million cells, are being used to study scaling of the multiphysics codes on conventional and advanced architectures. It should be noted that these meshes have better structure along the central axis than 2D meshes that have been spun along an axis of revolution. This is enabled by Crosslink's native 3D block topology creation capability, a feature missing in older meshing codes.
LLNL added new equation-of-state (EOS) models to improve simulations of shock-compressed polymer composites, which require accurate material models for composite materials with embedded particulates, used in many national security applications.
Composite materials with embedded particulates enable many applications and require accurate material models to be properly simulated. Shockwaves in polymer composites exhibit an interesting asymptotic pressure region, which occurs due to a viscous pressure that is created from particle-polymer interaction within the material. The ALE3D and ARES hydrocodes have now implemented a model (Herbold, 2022) that adds an additional viscous pressure to the EOS pressure. This viscous pressure is computed using an estimate of the relative velocity between the particles and polymer, which is a function of the local pressure gradients and particle drag. The ALE3D implementation further improves the material model through a two-species Mie-Güeneisen EOS that accounts for the local particle mass fraction.
The model implementation was verified by comparing ALE3D and ARES simulation results for a double-shock experiment. This experiment consists of a flyer with both a low- and high-density material. Initially, a low-pressure shock travels through the receiving composite material. After an open reflection on the higher density material, a second stronger shock reaches the composite. Results show good agreement between the ARES and the ALE3D implementations. The ARES implementation has additional agreement over experiments with a range of shock velocity and particle fill, providing confidence in the results from both hydrocodes. (LLNL-ABS-851466)
Reference: Herbold, E., Nonequilibrium equation of state for shock compressed particulate composites. Bulletin of the American Physical Society, 67, (2022).
SNL novel analysis of dislocation effects on helium bubble formation in stainless steels guides improved design of weapon components for long-term tritium storage.
Stainless steel reservoirs used for long-term tritium gas containment suffer from aging effects that can lead to vessel cracking. Tritium-to-helium decay leads to the formation of internal helium (He) bubbles that affect the stainless-steel embrittlement process. A fundamental understanding of how dislocations impact He bubble formation can therefore directly guide future nuclear weapon component designs. Modulating dislocation structures can change bubble structures and provide an effective means of improving aging behavior for components. However, this understanding cannot be easily obtained from traditional experiments.
SNL researchers recently made a breakthrough discovery through development of a quinary Fe-Ni-Cr-H-He interatomic potential to enable molecular dynamics (MD) simulations of He bubble formation in Fe-Ni-Cr-H systems for the first time. This allows researchers to perform MD computational experiments and examine how long a diffusing He atom takes to join an existing bubble in the presence of dislocations as shown in Figure 11 (a). The results of such computational experiments are highly statistical. SNL has theoretically proved that 1,000 simulations can capture the statistics accurately. For this work, the team performed two sets of such 1,000 simulations: one set with dislocations and the other set without dislocations. The time distribution of the He atom to join an existing bubble is summarized in Figure 11 (b). Notice that dislocations significantly reduce the join time. Further analysis revealed that this reduction in time occurs because the He atom is trapped in a broad tensile region of dislocations that is closer to the bubble. This is different from the previous belief that the He atom is trapped only at the dislocation core. Although actual dislocations are more complex than the straight dislocations simulated (He can still be trapped at junctions of a dislocation network), this discovery guides proper application of the materials in components and informs aging model development for component performance evaluation. (SAND2023-02935)
LLNL Flux scheduler enables increased throughput of national security workloads on El Capitan Early Access System 3 (EAS-3) hardware.
Workflow throughput on El Capitan EAS-3 hardware has increased 22 times since improvements were made to the Flux job scheduler. It is projected that large gains in national security workflows will be easily obtained when applied to the incoming El Capitan platform, which is of the same architecture but with the next-generation GPUs. Many NNSA workflows use two-dimensional (2D) simulations that have not performed well in the past on GPU platforms due to their small size and inability to fully utilize the GPUs. This has been addressed in the Flux job scheduler by scheduling multiple jobs within a node, effectively over-subscribing the GPUs, and enabling them to run at their full capacity. This dramatically increases the number of jobs that can run concurrently on machines, making them much more agile in responding to priority needs.
The availability of this increased throughput further motivated the LLNL code teams to port additional numerical kernels from the central processing units (CPUs) to GPUs to better address the needs of 2D simulations. This resulted in additional feature-complete multi-physics codes, further increasing the ability of El Capitan to respond to computational needs. The combination of improved multi-physics codes and Flux job scheduler were then able to demonstrate sizeable throughput efficiency gains as shown in Table 1. (LLNL-ABS-851465)
Number of Nodes
Ranks per run
Runs per 24 hours
Table 1: Comparison of CTS-1 throughput to EAS-3 before (middle row) and after (bottom row) the improvements to the Flux scheduler. The improved scheduler allows a computational study to run 79.33 simulations in 24 hours on a single node versus 4 nodes of CTS-1 required to simulate only 14.11 runs in 24 hours. Since 4 nodes of CTS-1 are required versus only 1 of EAS-3, this indicates that a single node of EAS-3 can accomplish roughly 22x the simulation throughput of the single CTS-1 node.
SNL ASC program
Alden Pack started supporting the ASC program in 2020 as an electromagnetic (EM) analyst and software developer for the Electromagnetic Theory and Simulation department at SNL. He is product owner for Gemma, SNL’s next-generation, frequency-domain electromagnetic code with specialized models to evaluate EM radiation (EMR) shielding effectiveness and other normal EM environments in support of system qualification. Gemma is part of the Radiation Analysis Modeling and Simulation of Electrical Systems (RAMSES) suite of radiation and electrical numerical codes and leverages the Kokkos library for thread-level and inter-node parallelism. Alden also supports various nuclear deterrence programs as a ModSim lead and ModSim support.
Erin Stelter began supporting the ASC program in June 2022 as a business and financial analyst for the V&V subprogram. Before coming to the ASC program, Erin worked at a bus manufacturer and parts distributor in Louisville, KY, providing purchasing, product, and business development, as well as project management support. She enjoys the process of working with others and having a combined goal, such as the yearly ASC Implementation Plan, that culminates with a valuable end-product. In her spare time, Erin enjoys the outdoors, shopping at weekend markets, and taking weekend trips around New Mexico and southern Colorado.
Jason Peters began supporting the ASC program in September 2021 as a solutions architect working primarily on the schedule and storage systems that support the high-performance computing (HPC) clusters. Before coming to the ASC program, Jason worked as a systems administrator for the MCS Linux server team at SNL. Jason brings a wide range of experience with IT technologies including storage systems, backup, virtualization, and networking.
LANL ASC program
Greg Salvesen converted to staff at LANL in 2021. He is an astrophysicist specializing in systems called X-ray binaries, which consist of a star feeding a disk of gas around a black hole. His thesis work, post-doc, and projects all had components of V&V/Uncertainty Quantification. Greg's astrophysics work mainly targets computational models and X-ray observations of black hole accretion disks.
Alex Lovell-Troy is a cloud software architect, formerly with Cray where he designed some of the software LANL uses to manage its largest systems. He has a background in systems administration and scientific computing at various sites including the University of Illinois and the Large Binocular Telescope Observatory. Outside of system management, Alex spent time as a consulting data architect and vice president of data consulting for a boutique managed services company. Alex will be working on open-source efforts to improve the HPC community with a focus on next-generation HPC system management software. He lives with his family in Northern Virginia. His children, 9 and 12, are arguably more accomplished than he is, having been featured in local newspapers on two continents and with producer credits on horror movies.
LLNL ASC program
Kristi Belcher joined LLNL in 2020 as a software developer in the Applications, Simulations, and Quality (ASQ) Division. She primarily focuses on C++ feature development and continuous integration work for both the RAJA/Umpire and MARBL teams in the ASC program. Kristi has B.S. (Texas State University) and M.S. (University of Oregon) degrees in computer science. Kristi is involved in many outreach efforts such as recruiting at conferences and other venues, mentoring COMP interns, and volunteering at career days for local high schools. She has several hobbies outside of work such as paddle boarding, windsurfing, and pickleball.
Bradley Davis was designated as Deputy Engineer - Facility Projects and Budget Planning in August 2022 for Livermore Computing (LC), where he leads the LC Conventional Facilities project portfolio. He assists with the management of the HPC Integrated Project Team in collaboration with Operations & Business (O&B) to execute LC’s HPC infrastructure projects. Brad was recognized this year with a Secretary’s Honor Achievement Award as part of the leadership team for the Exascale Computing Facility Modernization Project. Brad is a Project Management Professional (PMP), is EVMS certified, and has a BS Industrial Engineering degree from Cal Poly, San Luis Obispo. Brad was first introduced to LC through the Cluster Academy summer program and later interned with Anna Maria Bailey and the LC Facilities group prior to joining LLNL.
SNL LDRD: Real-time detection of noble gases could improve monitoring of underground explosions.
SNL geoscientists have detected specific gases being released from fractured rocks in real time after a series of small chemical explosions using different explosive compounds was set underground at the Energetics Materials Research and Testing Center outside of Socorro, New Mexico. This fundamental research could one day improve the prediction of earthquakes or detection of underground explosions. An improved understanding of how and why rocks fracture could also be valuable for enhanced geothermal systems and strain detection in unconventional oil and gas production. (See the SNL news release)
LLNL LDRD: Understanding material degradation in titanium alloys.
Titanium and its alloys are attractive for a wide variety of structural and functional applications due to the metal’s excellent strength, toughness and stiffness, and corrosion resistance. Specific applications include lightweight structural materials, bioimplants, and energy storage materials. However, if exposed to hydrogen sources, these alloys are susceptible to hydrogen incorporation and hydride formation, leading to crack initiation and mechanical failure resulting from lattice deformation and stress accumulation. When it comes to understanding hydrogenation and hydride formation in titanium alloys, many unanswered questions and challenges remain. (Read more in the full article on the LLNL website)
NNSS SDRD: Dale Turley wins “Most Valuable Principal Investigator” award with bubble collapse project.
The Special Technologies Laboratory’s Dale Turley is the winner of this year’s Most Valuable Principal Investigator (MVPI) award for his SDRD project “Study of Bubble Collapse in Optically Transparent High Explosive as a Method to Probe the Detonation Process” (22-052). Dale Turley’s project examined what happens to a butane gas bubble when injected into homogeneous explosive nitromethane (NM) revealing the effects of hot-spot formation on the detonation process. While previous studies have examined bubble collapse in NM, this work clearly shows evidence that the detonation process occurs near the bubble collapse and what happens when a detonation shock wave overtakes the butane bubble. The External Advisory Board lauded this project for its “clever experimental setup [and] great visualizations [that] could lead to improved modeling capability.” Congratulations to Dale Turley on his fantastic project and MVPI win. (Read more here and in the SDRD Annual Report Overview, now available on NNSS.gov).
SNL LDRD: Employing a cyber shuffle to stop hackers from taking over military aircraft.
A cybersecurity technique that shuffles network addresses like a blackjack dealer shuffles playing cards could effectively befuddle hackers gambling for control of a military jet, commercial airline, or spacecraft. Many aircraft, spacecraft, and weapons systems use an onboard computer network, the military standard 1553 (MIL-STD-1553), to let systems like radar, flight controls, and the heads-up display talk to each other. SNL partnered with researchers at Purdue to show that a moving target defense that utilizes randomness can effectively protect MIL-STD-1553 networks against sophisticated hacking. (Read more here.)
Questions? Comments? Contact Us.
NA-114 Office Director: Thuc Hoang, 202-586-7050
- Integrated Codes: Jim Peltz, 202-586-7564
- Physics and Engineering Models/LDRD: Anthony Lewis, 202-287-6367
- Verification and Validation: David Etim, 202-586-8081
- Computational Systems and Software Environment: Si Hammond, 202-586-5748
- Facility Operations and User Support: K. Mike Lang, 301-903-0240
- Advanced Technology Development and Mitigation: Thuc Hoang