Quarterly Highlights | Volume 6, Issue 1 | January 2023
In This Issue
ASC & LDRD Community—Upcoming Events (at time of publication)
- RIKEN-CCS Symposium and DOE-MEXT Collaboration meeting in Kobe, Japan; February 6 -9*
- Stewardship Capability Delivery Schedule Summit, SRNL; February 7-9
- Trilab CCE Meeting at SNL-NM; February 8-9
- JOWOG 34 ACS meeting at LLNL; March 6-9
- 2023 Conference on Data Analysis (CoDA) in Santa Fe, NM; March 7-9
- SOS2023 in Lake Tahoe, CA; week of March 13*
- Predictive Science Panel (PSP) meeting at LLNL; March 21-24
- CORAL-2 Quarterly Review at AMD Austin; March 21-23
- NA-11 SRT&E Budget Summit (virtual); March 27-29
- HPC User Forum at Princeton, NJ; April 18-19
- Salishan Conference, OR; April 24-27*
- ECP Independent Project Review at ORNL; May 2-4
- ASC Principal Investigators (PI) Meeting at Y-12, May 16-18
- International Supercomputing Conference 2023 in Hamburg, Germany; May 21 – 25
Welcome to the first 2023 issue of the NA-114 newsletter - published quarterly to socialize the impactful work being performed by the NNSA laboratories and our other partners. This issue begins with a highlight from Los Alamos National Laboratory (LANL) in which LANL codes, models, new design tools, and the ASC high performance computing (HPC) systems supported the successful completion of the W93 Phase I study. This issue also acknowledges the impressive work performed by newly appointed distinguished members of technical staff, laboratory fellows, senior scientists, and prestigious awardees at LANL, Lawrence Livermore National Laboratory (LLNL), and Sandia National Laboratories (SNL). Other highlights include:
- LLNL modeling of energy storage in metals under extreme deformation conditions significant to plutonium aging.
- SNL’s new UGRID mesh generator accelerating mesh generation for NNSA hypersonic analyses for the Navy’s Hyper Velocity Projectile, supporting aerodynamic analysis and aerothermal analysis.
- Utilization of the Sierra computing platform to support the LLNL B83 Annual Assessment Report with high-resolution 3D simulations.
- SNL’s work bolstering molecular dynamics models of materials in extreme environments through machine learning (ML). A related simulation using the Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) capturing the decomposition of a supercritical fluid is shown in the banner image above.
Please join me in thanking the professionals who delivered the achievements highlighted in this newsletter and on an ongoing basis, all in support of our national security mission.
NA-114 Office Director
ASC capabilities at LANL enabled a successful W93 Phase 1 study, completed in six months using LANL codes, models, Common Modeling Framework, new design tools, and HPC systems.
The W93/Mk7 team successfully completed their Phase 1 study in six months. This study surveyed a set of design concepts. The team performed over 1,000 full system simulations on ASC’s advanced technology systems (ATSs) and commodity technology systems (CTSs). LANL’s Common Modeling Framework (CMF) played a crucial role in employing best modeling practices and organizing and transferring simulation results amongst the team. In addition, new design tools added to CMF were developed just prior to this study that expedited design refinements of these eight concepts. The team employed an array of ASC computational modeling capabilities from fast approximate models for rough assessments to ASC’s most advanced tools during detailed design refinement. Many of these later computational capabilities have undergone decadal improvement in physics fidelity and validation against experiments. In addition, the vast number of simulations were only achievable through an extensive number of performance enhancements implemented within the last few years. Altogether, the technical depth of this six-month study could not have been accomplished without ASC codes and computational platforms. (LA-UR-22-30299)
Distinguished honors across the ASC program...
LLNL ASC program individuals appointed as Distinguished Members of Technical Staff
Four LLNL ASC program associates were recently honored with the Distinguished Members of Technical Staff (DMTS) designation for their extraordinary scientific and technical contributions to the Laboratory and its mission. Appointment to DMTS is the highest technical staff level achievable by a scientist or engineer at LLNL. LLNL Director Kim Budil announced the twenty-six member DMTS class for 2021 and 2022 in an all-hands meeting on September 27th. ASC program affiliates (named below) have made significant and on-going contributions to the ASC Program.
Anna Maria Bailey (Livermore Computing (LC)) is widely known for her skillful management of the LC HPC facilities housing the ASC program computing systems, and her role in engineering the recent Exascale Computing Facility Modernization (ECFM) project.
Todd Gamblin (LC) is the visionary founder of the Spack open-source project which automates highly complex software builds for HPC applications and is widely used by ASC code developers and the global scientific community.
Marty Marinak (Weapons Simulation and Computing (WSC)/ASC – Inertial Confinement Fusion (ICF)) leads the ASC Hydra code development effort which has proven to be an essential tool in computational studies for the ICF program.
Mike Puso (WSC/ASC – Engineering codes) is a noted expert developer of contact mechanics codes with significant contributions over the years to LLNL’s engineering codes Paradyn, Diablo, and Tribol.
Established in 2011, the DMTS is reserved for Laboratory scientists and engineers who have demonstrated a sustained history of high-level achievements in programs of importance to LLNL or distinguished scientific and technical achievements; have become a recognized authority in the field; or have made a fundamental and important discovery that has had sustained, widespread impact. Only a very limited number of the scientists and engineers will be selected for recognition at the DMTS level. (LLNL-ABS-843063)
LANL E.O. Lawrence Awards and Laboratory Fellow Appointments
LANL ASC program associates have been honored as recipients of highly competitive scientific achievement awards, as influential technical leaders in computational sciences impacting both ASC and international scientific communities.
Luis Chacon and Dana Dattelbaum received DOE E.O. Lawrence awards in 2021 and 2020, respectively.
Luis Chacon of Applied Mathematics and Plasma Physics (T-5) was selected for seminal contributions in multiscale algorithms for fluid, kinetic, and hybrid simulation of plasmas, enabling scientific breakthroughs in fast magnetic reconnection and self-organization in magnetic fusion systems, and in reactivity degradation in inertial fusion systems.
Dana Dattelbaum of Dynamic Experiments (M-DO) was selected for pioneering physical insights into shock and detonation physics, innovations in the development of equations of state (EOS) for energetics and polymers, and critical data for hydrodynamic simulations.
Tim Germann, Lin Yin, Hui Li, and Ricardo Lebensohn were four of nine LANL Laboratory Fellows appointed in 2022.
Tim Germann of Physics and Chemistry of Materials (T-1) was selected for his distinguished career at LANL. Germann is an internationally recognized authority for three main bodies of work: computational materials science, computational epidemiology, and computational co-design.
Lin Yin of the Laboratory’s Plasma Theory and Applications group is one of the world’s foremost experts on the physics of laser plasma interactions. She was named a fellow for her important discoveries in the physics of laser scattering and energy coupling in ICF experiments on major laser facilities for the weapons program.
Hui Li of Nuclear and Particle Physics, Astrophysics and Cosmology (T-2) was selected as a fellow for major discoveries in the field of plasma astrophysics. He is recognized as the main person to discover and fully explain Rossby wave instabilities and their role in transporting angular momentum in proto-stellar and protoplanetary disks.
Ricardo Lebensohn of Fluid Dynamics and Solid Mechanics (T-3) was named a fellow for his impact and recognition in the field of microstructure and property relationships of polycrystalline materials. His ideas and computational approaches drive the field and influence virtually everyone performing microstructure-aware computational modeling of polycrystalline materials. (LA-UR-22-33011)
SNL appoints Senior Scientists and Distinguished Members of the Technical Staff
SNL’s ASC program is privileged to recognize several technical staff members who earned special appointments to Senior Scientist (the top 1% of SNL’s workforce) and Distinguished Member of the Technical Staff (DMTS, the top 10%) for their extraordinary scientific and technical accomplishments to the Laboratory, the ASC program, and its mission.
Newly appointed Senior Scientists include Joel Stevenson and Laura Swiler.
Joel Stevenson has been instrumental in supporting SNL science and engineering application code teams and weapon system analysts, with strong engagement in ASC HPC Tri-lab activities including leadership in the Advanced Technology Computing Campaigns (ATCCs), the Common Computing Environment (CCE), and Remote Computing Enablement (RCE).
Laura Swiler is an expert in statistical analysis, uncertainty quantification, verification and validation, and sensitivity analysis. She has served in several national-level leadership roles and multiple journal editor roles and is recognized as an influential program leader and mentor. (SAND2022-10333O)
LLNL ASC team models energy storage in metals under extreme deformation conditions significant to plutonium aging.
The LLNL ASC Physics and Engineering Models (PEM) team investigates microstructural energy storage both due to its influence on temperature evolution and because of interest in energy storage associated with plutonium aging. It has been known for nearly a century that part of the mechanical work imparted on a solid subjected to plastic deformation is stored in the solid itself, while the remainder is dissipated as heat. The amount of energy stored under extreme deformation rates, such as those achievable in high-rate material experiments at the National Ignition Facility (NIF) or on the Z-machine, is difficult, if not impossible, to measure experimentally. This unknown stored energy contributes to uncertainty in the determination of temperature-dependent properties such as mechanical strength. Together with Prof. J. K. Mason at UC Davis, LLNL scientists performed molecular dynamics simulations of high-rate deformation of tantalum metal and measured the energy stored in the material. Their paper published in Acta Materialia reports that energy storage in tantalum is indeed significant and that neglecting it can result in overestimating the material’s temperature by a hundred Kelvin. Using methods of in silico computational microscopy, the authors found that, as expected, much of the stored energy results from the multiplication of dislocations – crystal defects whose motion produces plasticity. However, a surprisingly large fraction of energy is also stored in point defects (and their clusters) generated as debris in the wake of moving dislocations. Whereas energy storage in dislocations reaches its ceiling at a strain around 0.5, energy storage in point defect debris continues to larger strains and may reach as much as half of the energy ultimately stored in dislocations. The paper presents substantial evidence in support of a hypothesis advanced by G. I. Tailor in 1933 that a close relationship between stored energy and mechanical strength can exist. Based on the assumed connection, the authors propose a phenomenological model for predicting energy storage under complex dynamic deformation conditions.
SNL’s UGRID mesh generator accelerates mesh generation for NNSA hypersonic analyses for the Navy’s Hyper Velocity Projectile, aerodynamic analysis of the Philly hypersonic glide vehicle geometry, and aerothermal analysis of the Hot for Hypersonics Flight Campaign.
SPARC’s Multi-Fidelity Toolkit (MFTK) is designed to enhance the analyst’s ability to tailor the level of aerodynamic fidelity to the application’s accuracy and performance needs in an automated fashion. SNL has developed a new automated unstructured mesh generation tool in MFTK, called UGRID, suitable for arbitrarily complex reentry vehicle shapes. UGRID meshes support both mid-fidelity simulations (i.e., inviscid Euler and momentum energy integral technique) and low-fidelity simulations (i.e., modified Newtonian aerodynamics and flat plate correlations). UGRID speeds up generating grids for machine learning (ML) reduced order model (ROM) testing in support of the Navy’s Hyper Velocity Projectile (HVP) program, for aerodynamic analysis of the Philly hypersonic glide vehicle (HGV) geometry, and for the aerothermal analysis of the Hot for Hypersonics (H4H) Flight Campaign. UGRID supports hypersonic applications and general supersonic flow simulations including ascent trajectories. It is designed to maximize versatility of vehicle specification by enabling mesh generation via a computer-aided design (CAD) model, sphere-cone parameters, axisymmetric outer-mold line (OML) coordinates, and legacy analytical axisymmetric shape definitions (e.g., MAGIC/SABRE). It automatically detects and assigns boundary conditions, producing grids that are directly deployable in SPARC simulations (see Figure 3). UGRID approximates the shock shape automatically and allows the user the flexibility to model wake regions. (SAND2022-16489)
The Sierra computing platform supports the LLNL B83 Annual Assessment Report with high-resolution 3D simulations.
The exceptional computational power of the ASC Sierra supercomputer and the hard work of the LLNL WSC team made a strong impact on the FY22 Annual Assessment Report. Kirsten Howley, a Livermore scientist, was able to carry out a series of very high-resolution 3D simulations supporting the B83 assessment. This effort required up to 1,800 nodes of Sierra and used several billion zones per simulation, while requiring only a small amount of wall-clock time. This flexibility is enabling the design team to fully explore multiple regimes of interest in a timely manner. This high level of resolution and fast turn-around time were not possible, prior to the delivery of Sierra. (LLNL-ABS-843492)
SNL bolsters molecular dynamics models of materials in extreme environments through machine learning.
Researchers at SNL, in collaboration with Massachusetts Institute of Technology (MIT), are taking on core inaccuracies of continuum fluid dynamics by deploying ML into classical particle simulations. SNL and ASC Predictive Science Academic Alliance Program-III (PSAAP-III) Center for the Exascale Simulation of Materials in Extreme Environments (CESMIX) developed ML tools (SNAP and Allegro) that are being trained on materials of interest in phase regimes that are challenging to model with any other simulation method. Training of these ML models for uniform accuracy over a wide range of densities and temperatures has been a long-standing challenge in the field. Recent work has drastically reduced the iterative training process, enabling rapid predictions of phase boundaries (see Figure 4 below). Validation of these predictions has been independently achieved using both experiments and first-principles methods. These new ML tools complement and bolster first-principles calculations for EOS models by allowing for dynamic predictions of materials in extreme environments, providing insight into pulsed power experiments and in uncertainties in multiscale modeling workflows. (SAND2022-14286 O)
LANL is resolving a longstanding puzzle about fundamental symmetry of delta-Pu to improve models for stockpile stewardship.
Accurately describing the bonding between atoms in any material is key to accurate modeling of the material’s behavior. Such a description of bonding in delta phase plutonium (Pu) remains problematic despite decades of research. Contributing to the problem is a vexing magnetic behavior that is equally unusual and poorly understood. Pu does not behave like the typical magnetic materials we do understand. Attempts to model the bonding behavior focus on how the magnetic moments of individual Pu atoms are arranged, i.e., how the moments order in relation to one another. These attempts have advanced our understanding, but all the previously used arrangements of magnetic moments break a fundamental symmetry of delta Pu: the individual bonds between Pu atoms are not all equivalent. LANL has derived an arrangement that alleviates this vexation and gives results in better agreement with experimental data. The arrangement retains the proper symmetry, with which LANL calculations no longer start slightly off balance. Focusing on how this arrangement affects the bonds as the atoms move closer and farther away from each other now provides an improved model of the lattice vibrations in delta Pu. Not only does the improved model give better agreement with experimental lattice vibrations than previous descriptions, it also results in the unique negative thermal expansion observed for delta Pu. The new description brings us one step closer to understanding the unusual magnetic behavior of Pu. From an applied point of view, it lays the foundation for calculations that more accurately describe the thermodynamic behavior of “real-world" delta-Pu. The radioactive nature of Pu means the material is full of decay products and lattice defects, which affect the thermodynamic behavior. The correct symmetry of the new arrangement and the better agreement with experiment indicate a path toward better understanding, and hence control, of Pu for national security applications. (LA-UR-22-30299)
Kansas City National Security Campus (KCNSC) developed a simulation capability for thin film deposition, successfully deployed on fixture development for a deposition chamber, reducing the design cycle from three months to one week.
To accelerate fixture development through Virtual Cycles, KCNSC developed a custom ASC code, written in Python, which can simulate physical vapor deposition (PVD). The benefits of this model are twofold; it enables the rapid simulation and development of fixturing and also provides more robust processes capable of producing higher quality parts. The model is supplied geometry as a series of triangular surface meshes, each of which can be assigned unique motion patterns. Then, the main code calculates the deposition rate for each facet at every point in time which is a function of the facet’s current location, exposed area, and the source intensity. The model output is the deposition buildup through time, which can be visualized and used to assess the coating uniformity for a given fixture or shadow mask design. This simulation process has been validated and successfully applied across a number of projects. Most recently, it was deployed to develop a fixture for a new chamber which was able to decrease the coating thickness variance by a factor of three. As a result of simulation, KCNSC was able to evaluate 60+ scenarios with 30+ unique shadow mask designs within a week, whereas a typical development cycle for a single design can take two to three months. Overall, this new model enabled KCNSC to hit a required tolerance twice as tight as the standard operation on the first attempt saving months of development time and cost. (NSC-614-4909 UUR)
LANL/Univ. of Rochester collaborate to improve ray tracing capabilities for 3D physics of ICF and high energy density physics applications.
LANL ASC codes are state-of-the-art tools for modeling ICF and high-energy-density (HED) physics experiments. In turn, experiments at NNSA ICF facilities like NIF and Omega are used to validate codes in the extreme regimes found in many national security applications. Laser raytracing is a key element of high-fidelity modeling of these systems. However, a current limitation is that the LANL ASC laser package only works for 2D simulations, although a 3D extension is under active development. This precludes a complete assessment of the physics of interest in many laser-driven experiments, including the directly driven cylindrical implosion experiments that are used to study instability growth in the HED plasma regime. Cylindrical implosions at NIF show a significant asymmetry arising from the laser drive which was not predicted in 2D simulations.
LANL scientists are collaborating with the Flash Center at the University of Rochester to use the FLASH code, which has 3D laser raytracing, to better understand the origins of this drive asymmetry. FLASH computations reproduce the octagonal pattern observed in the experiment, which arises from differential absorption of the 45- and 50-degree NIF beams used to drive the target. This work demonstrates the importance of continued development of the 3D laser raytracing in xRAGE, the limitations on validating physics models based on NIF experiments with “2D symmetry,” and provides an evaluation of potential errors in interpretation when using 2D calculations if inadvertent 3D effects occur in the experiments. (LA-UR-22-32099)
ASC simulation tools and HPC systems played critical role in December 5th fusion ignition breakthrough.
The HPC publication HPCwire recently featured an article detailing the role of supercomputing in achieving the recent LLNL fusion ignition breakthrough. ASC program computational methods and systems are central to many of the advances in simulation and modeling related to ICF. LLNL computational scientist, Brian Spears, deputy lead modeler for inertial confinement fusion (ICF) at the National Ignition Facility (NIF), provided background for the HPCwire article. Spears credits the ability to simulate experiments using HPC for maximizing the value of ICF laser shots through analysis of results and in planning of next experiments. The ASC program provided leadership-class computers including graphics processing unit (GPU)-enabled Sierra and its unclassified counterpart, Lassen, enabling new cognitive simulation approaches to marry prediction and experiment, and central processing unit (CPU)-based commodity technology systems (CTSs) such as Jade as “workhorses” for the traditional design calculations in the analysis workflow. The ASC exascale supercomputer El Capitan (to be delivered next year) and new experimental ASC systems featuring advanced hardware for optimized artificial intelligence (AI), are expected to contribute greatly to the next generation of research at LLNL to further advance ICF research. (LLNL-ABS-844125, see the full article: Supercomputing's Critical Role in the Fusion Ignition Breakthrough)
SNL’s new machine learning technique for network traffic prediction targets job management efficiency for weapons simulations.
Network congestion can severely disrupt the efficient and effective deployment of weapons simulations on supercomputing platforms by introducing latency into simulation runs. SNL researchers and academic collaborators in the Department of Electrical and Computer Engineering at Queen’s University in Ontario, Canada have developed a ML technique for predicting incoming communication traffic using network interface counters in HPC systems. Based on rolling linear regression, the technique is designed to be deployed on emerging intelligent network interfaces, also known as “SmartNICs,” enabling the intelligent management of network resources to avoid congestion and contention with scientific applications executing on the host CPU. Early results indicate high potential for improved job management that frees-up simulation cycles for the weapons community.
The new ML-based technique is lightweight, executing in linear time and requiring less than 4KB of memory, meaning it is both efficient and leaves the majority of SmartNIC resources available to service other offloaded tasks. Evaluated against nine representative applications drawn from the Exascale Computing Project suite of proxy applications, the technique achieves a normalized root mean squared error (NRMSE) of less than 3.2%; when the sole randomized algorithm (a Monte Carlo method) is removed from consideration, the error drops to less than 0.72% (prediction error for the LAMMPS application is illustrated in Figures 10 and 11). Error can be further reduced by using different regression parameters for predictions of near future events versus those that are more distant, and lower error is desirable because better predictions allow smarter data movement. Intelligent network technology is just emerging into the HPC ecosystem and is expected to play an increasingly important role in future DOE SC and NNSA systems, including through the deployment of ML traffic prediction techniques as described in this highlight. (SAND2022-15844 O)
LLNL ASC capabilities help design vanadium experiments to isolate dynamic strength for the FY23 “Flexible Pit Production” Stewardship Capability Delivery Schedule pegpost.
In experiments that probe conditions of interest on the microsecond time scale it is challenging to isolate specific effects such as material strength. A recent publication in the Journal of Applied Physics details the computationally driven redesign of a gas-gun-based hole-closure experimental platform that probes large-strain material response on the microsecond time scale. The redesign made use of Physics and Engineering Models (PEM)-developed constitutive models and reduces the confounding effects of porosity and anelasticity while retaining desired sensitivity to material strength. This is accomplished by utilizing a two-layer flyer as shown in Figure 12(a), with the layered flyer significantly reducing unwanted tensile excursions in the target. The work required 3D calculations due to the wave interactions, and ASC computing resources enabled the assessment of a wide array of potential design configurations including the materials and thicknesses for the layered flyer. Robust contact mechanics in the ALE3D integrated code were also enabling for the redesign. The final configuration provides better control of the pressure pulse and allows one to evaluate the predictive capabilities of commonly used strength models at strain rates in excess of 105/s and strains up to 150% in the vicinity of the hole (see Figure 12(b)).
Previous work published in Acta Materialia showed the hole-closure platform’s utility in model calibration, and the Journal of Applied Physics paper further highlights the platform’s utility for assessing the predictive capabilities of models. Figure 12(c) contains comparisons of various model parameterizations to experimental data, providing an assessment of model extrapolation outside of the range of conditions typically used for model calibration. The publication highlights that various models calibrated to the same data can produce significantly different predictions when applied to the extreme conditions in the experiments. Knowledge gained from this computational study has been applied to a set of recently fielded experiments on vanadium supporting a FY23 Stewardship Capability Delivery Schedule pegpost (Enabling Efficient and Flexible Pit Production) and has informed experimental design for upcoming experiments on U6Nb that will examine variations with manufacturing. (LLNL-ABS-843476)
NNSA LDRD/SDRD Quarterly Highlights
LANL research solves longstanding physics problem: Anti-butterfly effect enables new benchmarking of quantum computer performance.
LANL’s method of running a quantum system backward, then forward in time, distinguishes information leaks from the desired information scrambling. Bin Yan, a Quantum Theorist at LANL, is corresponding author of the paper on benchmarking information scrambling published in July 2022 in Physical Review Letters. “Our protocol quantifies information scrambling in a quantum system and unambiguously distinguishes it from fake positive signals in the noisy background caused by quantum decoherence.” Using this protocol, LANL researchers can determine the degree to which quantum computers can effectively process information. The LANL team demonstrated the protocol with simulations on IBM cloud-based quantum computers. Information scrambling has proved relevant across a wide range of research areas, including quantum chaos in many-body systems, phase transition, quantum machine learning, and quantum computing. (LA-UR-22-27314, see the full article)
Navigating when GPS goes dark: SNL’s high-tech sensors could guide vehicles without satellites, if they can handle the ride.
Words like “tough” or “rugged” are rarely associated with a quantum inertial sensor. The remarkable scientific instrument can measure motion a thousand times more accurately than the devices that help navigate today’s missiles, aircraft, and drones. But its delicate, table-sized array of components that include a complex laser and vacuum system has largely kept the technology grounded and confined to the controlled settings of a lab. SNL Atomic Physicist, Jongmin Lee, wants to change that. (See the full article in SNL LabNews)
Inspiring the next generation: Exceptional NNSS computing and data scientist given senator’s “Women in STEM” Award.
NNSS' Ajanaé Williams, a scientist on the Computing and Data Science team, received Senator Jacky Rosen’s Nevada Women in Science, Technology, Engineering, and Math (STEM) award in December 2022. As a scientist at NNSS, Ajanaé is part of the experimental process from start to finish, from software development to setting up detector systems and fielding experiments. She inspires girls and young women throughout Nevada to enter STEM fields and break expectations of what “a scientist looks like.” (See the full article on the NNSS website, and more information about the Nevada Women in STEM award)
Questions? Comments? Contact Us.
NA-114 Office Director: Thuc Hoang, 202-586-7050
- Integrated Codes: Jim Peltz, 202-586-7564
- Physics and Engineering Models/LDRD: Anthony Lewis, 202-287-6367
- Verification and Validation: David Etim, 202-586-8081
- Computational Systems and Software Environment: Si Hammond, 202-586-5748
- Facility Operations and User Support: K. Mike Lang, 301-903-0240
- Advanced Technology Development and Mitigation: Thuc Hoang