NCAR-Wyoming Supercomputing Center (NWSC)

• 153,000 square feet in total
• 12,000 square feet of raised floor for supercomputers
  
  
• Design, construction, and commissioning: approximately $70 million
  
  
• Inaugural computing systems: approximately $25–35 million

Location

8120 Veta Drive, Cheyenne, Wyoming
Driving directions

Opening Date

Official opening: October 15, 2012

Purpose, Users, Remote Access

The NWSC provides advanced computing services to scientists across the nation who study weather, oceanography, air pollution, climate, space weather, energy production, seismology, carbon sequestration, computational science, and other topics related to the Earth system. Its premier data storage and archival facility holds unique historical climate records and a wealth of other scientific data. 

Researchers from the U.S. academic community, including NCAR and the University of Wyoming, use the computing, visualization, and archiving resources at the NWSC.

The majority of users access the facility’s resources remotely via the Internet from desktop or laptop computers.

Yellowstone Supercomputer

The NWSC's inaugural supercomputer, Yellowstone, was installed in 2012. It includes a cluster of high-performance supercomputing processors, a massive data archiving facility, and a special system for visualizing scientific data.

Hardware

• IBM iDataPlex supercomputer system, consisting of
• Intel Sandy Bridge EP processors and
• Mellanox FDR InfiniBand full fat tree.
• 72,576 processor cores

Memory

• 144.6 terabytes

Speed

• Peak computational rate: 1.5 petaflops (1.5 quadrillion calculations per second).
• Relative to the June 2012 list of the world's fastest supercomputers, a 1.5-petaflops system would rank among the world’s 20 fastest supercomputers
• As increasingly powerful supercomputers come online, a 1.5-petaflops system will gradually move down in global speed rankings.

Racks

The system is made up of 100 racks—tall, black, refrigerator-sized cabinets—distributed as follows:

• Yellowstone computing cluster: 74 racks
• 63 iDataPlex compute
• 10 switch
•  1 management
• GLADE (Globally Accessible Data Environment) storage cluster: 20 racks
• 19 data storage
•  1 management/switch
• Data analysis and visualization cluster: 3 racks
• Test systems: 3 racks

Erebus system

Yellowstone's computing power, compared to...

• NCAR was the first customer for the Cray 1A, which supported computational science at the center from 1977 to 1989 Compared to that early supercomputer, the Yellowstone system provides:
•  9.7 million times the computational rate
•  3.4 million times the disk capacity
• 19 million times the central memory size
  
  
• One petaflops is one quadrillion (1,000,000,000,000,000) floating point operations per second, which is approximately 143,000 calculations per second for every man, woman, and child on Earth.
  
  
• At 1.5 petaflops, Yellowstone will be capable of more than 214,500 calculations per second for every person on Earth.
  
  

• To get Yellowstone's power and speed from a set of computers like the ones we use at work or at home, you would need to link 40,000 state-of-the-art laptop or desktop computers together and allow each one to exchange data with any other one at 12 gigabytes per second. That speed would let you read the entire contents of a laptop’s 1 terabyte disk drive in under 1.5 minutes.

Benefits to Society

The research conducted on Yellowstone will lead to improved understanding of many processes, including:

• Tornadoes and other dangerous weather events
• Geomagnetic storms that affect communications systems and power grids
• Water issues, from future snowpack and water availability to drought and flood behavior
• Earthquakes and tsunamis
• Sources and global dispersal patterns of air pollution
• Subsurface water and energy resources
• Potential subsurface reservoirs for carbon sequestration
• Wildfire dynamics
• Effects of wind extremes on wind turbines and of clouds on solar cells

more: 10 ways Yellowstone will make a difference

Ownership and Operation

Staffing

• ~18 permanent staff (software engineers, systems administrators, facility engineers, and administrative staff)
  
  
• ~11 contract staff (security, custodial, grounds keeping, and other services)

Partners

• UCAR/NCAR under NSF sponsorship
• State of Wyoming
• University of Wyoming (UW)
• Wyoming Business Council
• Cheyenne LEADS
• Black Hills Energy (formerly known as Cheyenne Light, Fuel & Power Company)

The facility was designed with a total capacity of 4 to 5 megawatts of electricity, but with Yellowstone now in production, usage is considerably lower. Total power for computing, cooling, office, and support functions has averaged 1.8 to 2.1 MW.

Award Recognition

2013 Green Data Center of the Year

The NWSC received top honors from Datacenter Dynamics in this category, which recognizes those organizations that "have seen increasing pressures of power costs, regulation, community and environmental responsibility as an opportunity rather than a problem and have demonstrated their vision of sustainability as a critical driver in the design and operation of their data facilities."

2013 Green Enterprise IT (GEIT) Award

The facility has been awarded first place in the “Facility Design Implementation” category for its sustainable approach in designing and building the NWSC. The Facility Design Implementation Award recognizes cutting-edge data center projects that demonstrate energy and resource efficiency in a new, operational data center.

LEED Gold status

The NWSC facility has achieved LEED Gold status. LEED, or Leadership in Energy and Environmental Design, is an internationally recognized rating system for green buildings developed by the U.S. Green Building Council for evaluating environmentally sustainable construction.

Partner Contributions

• Funding from NSF to UCAR, NCAR’s managing organization, is $57.6 million. UCAR will operate and maintain the supercomputing center under NSF sponsorship.
• The State of Wyoming, through the Wyoming Business Council, has provided $20 million for construction costs.
• UW will provide $1 million annually for 20 years for ongoing upgrades to computing equipment and data storage.
• Cheyenne LEADS administered $4.5 million of Wyoming Business Council Business Ready Community Funds for site infrastructure. This included development of a fiber optic network, road development to the site, an extension of water and sewer lines, and regional drainage, landscaping, and signage.
• Cheyenne LEADS transferred ownership of the NWSC site, valued at $3 million, to UCAR. 
• Cheyenne Light, Fuel & Power constructed primary and secondary power feeds to the site.

more: A public-private success story

Timeline

• June 2010:  groundbreaking
  
  
• November 2011:  IBM selected to install first supercomputing system
  
  
• Late 2011:  facility construction and commissioning completed
  
  
• October 15, 2012:  grand opening

Contractors and Vendors

Construction:

Saunders Construction, Inc.

Architecture and Engineering:

H+L Architecture

California Data Center Design Group

Integral Group

RMH Group

Martin & Martin Consulting Engineers

Rider Levett Bucknall

Pre-construction Services:

Saunders Construction, Inc.

Design Peer Review:

Reliable Resources

Commissioning:

E Cube, Inc.