Igniting Gamma Ray Bursts Inside Supercomputers

Computational Technologies (CT) investigators recently created the first supercomputer simulations to capture the dynamic physics of the fireballs arising from gamma ray bursts—the most powerful and mysterious explosions in the universe. The simulations used the newly mature software framework known as IBEAM, the Interoperability Based Environment for Adaptive Meshes. To model the extremes of the bursts, IBEAM developers combined several groundbreaking features in the software.

There have been more than 100 theories about what causes gamma ray bursts. Today’s leading candidates are neutron stars colliding or old, massive stars dying in hypernova explosions. Whatever their origin, these cosmic blasts propel shockwaves of gas at velocities approaching the speed of light. “We are simulating what happens in that flow, how material that is very rapidly ejected interacts with itself and the surrounding medium,” said Alan Calder, research scientist at the University of Chicago's Center for Astrophysical Thermonuclear Flashes. As first described by Albert Einstein 100 years ago, special relativity governs the fluid-like motions of gas at near-light speeds. Thus, IBEAM includes a relativistic hydrodynamics component to model this aspect of the problem.

Even more challenging than special relativity is simulating the behavior of light, a feat that requires representing millions of individual photons to make realistic comparisons with observations. Not only must IBEAM follow photons as they travel through space and time, but it also must model light beams pointing at multiple angles. Since gamma ray bursts can be as short as 2 seconds, changes need to be calculated using short time steps. Add these factors together, and radiation transport dominates IBEAM simulations, consuming 95 percent of the computing cycles.

"Usually radiation transport is so expensive that you do a hydrodynamics calculation and take a snapshot," said Paul Ricker, assistant professor of astronomy at the University of Illinois at Urbana-Champaign. "You then have a separate code that does radiation at that one point in time." Using IBEAM, Ricker said, "we can self-consistently get a dynamic product of how things evolve, not just a static snapshot."

Such computational demands force the IBEAM group to use two spatial dimensions and assume homogeneity in the third dimension. “Three dimensions requires petaFLOPS, beyond what computers will be able to do for some time,” Calder said. Even running 2-D simulations only becomes tractable because IBEAM includes adaptive mesh refinement (AMR), a technique that focuses the resolution where changes occur. CT researchers at NASA Goddard Space Flight Center developed the PARAMESH software to do AMR with a variety of simulation codes. Harnessing PARAMESH, IBEAM can accommodate the massive ranges in physical quantities that occur in gamma ray bursts. According to Ricker, density and temperature increase by as much as a factor of 1,000, while pressure increases by a factor of 1 million.

With relativistic hydrodynamics, radiation transport, and AMR together in IBEAM, “we are definitely unique,” said Doug Swesty, research assistant professor of physics and astronomy at the State University of New York at Stony Brook. “This is the first time these three things have been coupled in one place.”

Using their one-of-a-kind software framework, the team ran seven gamma ray burst simulations of varying sizes. For the larger cases, IBEAM used the full “Lomax” SGI Origin 3800 system at NASA's Ames Research Center, needing all of its 496 processors and 256 gigabytes of memory. “These simulations were as big as we can fit on the machine,” Swesty said. All of the calculations modeled a box in space that is roughly 10 times the size of our solar system. For the biggest case—a performance run—IBEAM divided the 2-D computational mesh into 256² boxes on each processor. Across Lomax’s 496 processors, the simulation had an effective resolution of 5701².

IBEAM met these resolution challenges while outstripping the CT Project's performance requirements. The minimum performance milestone was 10 percent of peak speed on an entire NASA parallel computer, but IBEAM attained 17 percent of peak, or 208.4 gigaFLOPS, on Lomax. "It has been essential to have access to the resources that NASA provides," Ricker said. To extend performance, the IBEAM group has proposed for and received time on Ames' '"Columbia" SGI Altix 3700 system. This 10,240-processor machine is #2 on the current international TOP500 Supercomputer Sites list. "We should be able to scale up to 2,000 processors," Swesty said. "The interesting thing would be to see what happens when we get to 10,000 processors."

Whether on Columbia or another supercomputer, IBEAM is primed for scientific analysis. “We are going to be doing dozens and dozens of simulations. We plan to run continuously for the next 6 to 8 months,” Swesty said. The focus will be on light curves. When the shock moves into the interstellar medium, it heats up material in its path and radiates photons. Simulations will look at how the outgoing radiation fluctuates over time, which is plotted as a light curve. 

As Ricker explained, “IBEAM directly produces the observable quantity: radiation. We just collect the information from the simulation.” Led by Chryssa Kouveliotou, NASA Marshall Space Flight Center scientists will be comparing IBEAM-generated light curves with those from observations. They will draw on an archive of 2,704 light curves from the Burst And Transient Source Experiment (BATSE), which flew on the Compton Gamma Ray Observatory during the 1990s. Swift, NASA's latest gamma ray observatory, has been producing light curves since late December. “It is a good time to be doing this,” Swesty stressed. “There are a lot of new data coming. They will help constrain our models.” Image to left: This artist rendering depicts NASA's new Swift observatory with a gamma ray burst exploding in the background. Three weeks after a November 20, 2004 launch, Swift captured its first burst within a mere 65 seconds of the event. The IBEAM group will be comparing their simulations with observations from Swift and the predecessor Compton Gamma Ray Observatory (Image credit: © Spectrum Astro and NASA Education and Public Outreach, Sonoma State University, Aurore Simonnet).

Even with radiation transport, IBEAM cannot yet generate a full light curve. After the initial explosion, the light gradually fades away and cools off in an afterglow. “Ultimately we would like to reproduce that behavior and use the simulations to determine what is happening in the afterglow,” Swesty said. “The simulations have not gone long enough for us to see the afterglow decay away. We see the beginning of the afterglow.”

Moving toward greater realism with afterglows, code developers are adding microphysics to IBEAM. In general, the microphysics describes the emission of radiation from hot gasses and the interaction of the photons with the surrounding material. “The calculations we have done so far look at single-group radiation transport, where we consider all photon energies together,” Ricker said. “Our intention is to handle multi-group transport, where we follow the spectrum of the radiation as well as its total intensity. The resulting light curves will depend more on what direction the gas is moving in as well as how hot it gets.” Microphysics solvers will simulate specific radiation frequencies while enabling scientists to trace photon distribution and energy at points along each light curve.

The IBEAM modelers will also be using different geometries for the shocks and density fluctuations in the nearby interstellar medium. Such tinkering will help determine what kinds of shock-fluctuation interactions produce different light curves. “When we match the observed data, they are the conditions that give rise to these observed light curves,” Swesty said. Although it is not a stated goal, the team might very well find the underlying cause, or causes, for gamma ray bursts.

Other potential applications exist for IBEAM, for “radiation transport is important to different physical environments in the universe,” Ricker said. One possibility is modeling Type II supernova explosions, where the radiation is neutrinos rather than photons. Ricker also envisions applying IBEAM to his studies of the large-scale evolution of the universe.

For the broader research community, IBEAM with hydrodynamics is currently available on the investigation Web site. The group expects to have a limited release of the framework with radiation transport capabilities at some point in the future. Preparing for that eventuality, University of Illinois computer scientist Brian Foote is spearheading a “framework transformation of IBEAM,” Ricker said. “It will be cleaner, easier to use, and more sophisticated from the computer science perspective.” Programmers plan to rewrite the Fortran 90 portions of the code using the new, object-oriented Fortran 2003 standard. “Now, we have to recompile the code to use different solvers,” Ricker explained. “With Fortran 2003, we could install IBEAM on the machine and just recompile the solver rather than the whole framework.”

Related Links:

http://ct.gsfc.nasa.gov

http://www.ibeam.org