To meet growing demand for supercomputing by NASA scientists and engineers, the NASA Center for Computational Sciences (NCCS) is adding a second 1,024-processor Scalable Unit to the “Discover” cluster. With this expansion, the Linux Networx Custom Supersystem will have a total of 2,560 processors and offer five times the computing power of the “Halem” HP AlphaServer SC45 system it replaces. NCCS and Linux Networx Inc. staff installed Scalable Unit 2 the week of July 23, and the NCCS expects to complete integration in August.

Discover went into production mode in January and has had usage rates over 90% on some days. Job sizes range from 4 (a single node) to 1,024 processors (256 nodes). To accommodate the augmented Discover’s electricity and cooling needs, Halem was retired on May 1. The 1,392-processor Halem system served as the NCCS capacity computing platform for more than 4 1/2 years, supplying on the order of 50 million processor-hours to the NASA Earth and space sciences community.

NCCS and Software Integration and Visualization Office (SIVO) staff provided substantial assistance to move users from other systems, mostly Halem. “High-usage Halem investigator teams were each assigned a coordinator to ease their transition,” said Sadie Duffy, NCCS User Services Lead (CSC). Early on, all users received a User FAQ in their home directories; the FAQ and a User Guide are also available on the NCCS website. Some users found the move relatively simple, Duffy said, because they already computed on smaller Linux clusters in their departments. Familiarity also stems from Discover having nearly the same user environment as the NCCS “Explore” SGI Altix 3700 computer.

SIVO’s Advanced Software Technology Group (ASTG) offered training for porting software codes to Discover. They also worked closely with scientists, especially those with high-use codes, to benchmark and tune their applications for improved performance and scalability, said Hamid Oloso, ASTG applications support lead (Northrop Grumman IT).

One of those high-use codes is ModelE, the newest atmospheric general circulation model at the Goddard Institute for Space Studies (GISS) in New York City. Results from ModelE simulations, mostly run at 4 by 4 1/2 degrees on Halem, were part of the U.S. contribution to the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (2007).GISS is currently testing and refining ModelE on Discover.

“The model used for our IPCC experiments runs roughly twice as fast on the new cluster compared to Halem without any changes in the code,” said Reto Ruedy, GISS Climate Group Manager (Sigma Space Partners LLC). “Since shared-memory machines are becoming a rarity, we converted the parallelization of our code from OpenMP to MPI. This transition was made much less painful due to the support of Tom Clune and his group and the fact that we could take full advantage of the Earth System Modeling Framework (ESMF). This version of ModelE can simulate 100 years in 5 days, and we now are able to run our long experiments with a finer resolution.”

On Discover, a 2- by 2 1/2-degree, MPI version of ModelE achieved even greater speed-up, as high as 3x per processor compared to Halem, noted ASTG Lead Tom Clune. Many other codes have seen 2x or better performance increases. In general, codes scale as well as on other NCCS computers, Clune said. Performance is particularly good on the Intel “Woodcrest” (2.66 GHz) processors inside the Scalable Units. While having a slower clock rate than the Base Unit’s “Dempsey” (3.2 GHz) processors, they are 30 to 40% faster because of more efficient floating-point operations and speedier memory access, explained NCCS Lead Architect Dan Duffy (CSC). Scalable Unit 1 was formally accepted in April. Scalable Unit 2 will be slightly different in configuration, with 32 of its 256 nodes having 8 gigabytes of memory (versus the usual 4 gigabytes) for jobs that can benefit from it.

Teams whose applications produce large amounts of data often rely on scientific visualization to better understand their results. In a unique approach, the NCCS has fully integrated 16 Visualization Nodes into Discover. They can access all the same file systems as the rest of the cluster. These 4-processor nodes use AMD’s “Opteron” (2.6 GHz) chips because they have superior memory-to-processor bandwidth, a capability particularly helpful for high input/output rates. Additionally, each Visualization Node has 8 gigabytes of memory and an NVIDIA Quadro FX 4500 high-end Graphics Processing Unit (GPU).

Linux Networx Inc. and NASA Center for Computational Sciences staff installed Scalable Unit 2 the week of July 23.

SIVO’s Scientific Visualization Studio (SVS) will be working with users to define Discover’s visualization environment. It will include common visualization software packages (e.g., IDL) and may offer both batch and interactive modes. “A growing area of interest is the use of GPUs for creating visualizations of simulations while they are running,” said SVS programmer Jim Williams (GST, Inc.). “We hope to explore how we can use the Visualization Nodes to allow researchers to visually monitor their codes in real time. We may also be able to produce high-temporal-resolution animations that would require prohibitive amounts of storage were they to be done after the simulation finishes.” Pioneer user access to the Visualization Nodes is available by contacting NCCS User Services at support@nccs.nasa.gov.

http://www.nccs.nasa.gov
http://sivo.gsfc.nasa.gov