Die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Steinbuch Centre for Computing (SCC)A large Multi-Functional HPC Centre
Rudolf Lohner, Frank Schmitz
2 | Steinbuch Centre for Computing | ZKI in Göttingen
University 11 faculties 120 institutes 4,000 employees 18,500 students250 mio € budget
University 11 faculties 120 institutes 4,000 employees 18,500 students250 mio € budget
Forschungszentrum 10 programs (research areas)
21 large institutes 4,000 employees
€ mio 310 budget
Forschungszentrum 10 programs (research areas)
21 large institutes 4,000 employees
310 mio € budget
Karlsruhe Institute of Technology (KIT)
10 km15 min
science
education
structures for the administration
institutes
services
infrastructure
KIT…
3 | Steinbuch Centre for Computing | ZKI in Göttingen
KIT / SCC
Karlsruhe Institute of Technology - two strong partners united
Universität Karlsruhe (TH) One of the most research oriented universities in Germany Nominated to one of three top universities in October 2006 Capitalized with over 100 million € over 5 years
Forschungzentrum Karlsruhe Member of the Helmholtz Association Internationally acknowledged non-university institution
Under the roof of KIT –Steinbuch Centre for Computing
Computing Center of the Universität Karlsruhe (TH) Institute of Scientific Computing of the Forschungszentrum Karlsruhe
4 | Steinbuch Centre for Computing | ZKI in Göttingen
IWR / FZK RZ Uni KA
Virtual Computing Centre since 1996
IT-Services for UKA High Performance Computing Numerical Methods on HPC Integrated Information Mgmt.
IT-Services for FZK Scientific Computing GridKa, CampusGrid, D-Grid Nat. & Intl. Grid Projects
Virtual Computing Centre Karlsruhe since 1996
5 | Steinbuch Centre for Computing | ZKI in Göttingen
SCC-BoardFour Professors and one
Technical Director
Networks andTelecommunication
Data Management and
Information-Services
Systemsand Server
Desktop- Workstation-
and Communi-cations Services
SOA and Virtualisation
Distributed Systems and Grid
Print-, Plot- und
Media Production
I/O, Operational
Services
and Help Desk
HPC -Systems,
-Applications, -Grid
IT-Security and Service-
Management
Project-Managemet and -Coordination
Quality- und Product-Mgmt.
Strategic Communiation & Relationship Mgmt.
Organisation Teaching, Advanced Education
Common Administration and Organisation
10 SCC Departments
6 | Steinbuch Centre for Computing | ZKI in Göttingen
KIT / SCC Roadmap
Karlsruhe Institute of Technology (KIT) Foundation Contract to be signed in 2007 Official start of KIT:
January 1st, 2008
Steinbuch Centre for Computing (SCC) Founded simultaneously with KIT Official start of SCC:
January 1st, 2008
7 | Steinbuch Centre for Computing | ZKI in Göttingen
SCC Portfolio
Infrastructure
Services
Projects
8 | Steinbuch Centre for Computing | ZKI in Göttingen
HPC platforms at SCC: HP XC6000 / XC4000
Q1 04
Q4 04
Q1 05
Q1 06
Q3 06
Phase 2:» Total of 3000 processor cores
» Total of 15.6 TFlop/s peak performance
» Total of 12 TB of main memory
» Total 110 TB of local disk space (scratch)
Phase 0 (Q1 2004)»12 2-way nodes (Itanium 2)»4 file server nodes
- 2 TB shared storage»Single rail Quadrics interconnect
Phase 1 (Q4 2004)»108 2-way nodes
- Intel Madison 6M, 1.5 Ghz, 12 GB»8 file server nodes
- Approx. 11 TB storage system»Single rail Quadrics interconnect
Phase 1 (Q1 2005)»6 16-way nodes
- Intel Madison 6M, 1.6 Ghz- 2 partions, each 8 CPUs, 64 GB
»Single rail Quadrics interconnect
Phase 2 (Q3 2006)»750 4-way nodes (two sockets)
- dual core AMD Opteron 2.6 Ghz- 16 GB main memory
»10 server nodes»10 file server nodes
- Approx. 56 TB storage system»Infiniband DDR interconnect
9 | Steinbuch Centre for Computing | ZKI in Göttingen
Globales Filesystem 56 TB
Infiniband 4x (DDR) Fast Interconnect
Ethernet Administration
Login nodesLustre
. . .
. . .
750 Compute nodes
Service nodes
Local Disksà 72 GB
. . .
Configuration of the HP XC4000
10 | Steinbuch Centre for Computing | ZKI in Göttingen
Parallel File System HP SFS on XC4000
Shared storage for all nodes of XC system
10 file server nodes 2 MDS / Admin 2 OSS for $HOME 6 OSS for $WORK
56 TB file space 8 TB $HOME 48 TB $WORK
Bandwidth Read / write from one node: 340 MB/s / 340 MB/s Total read / write bandwidth of $HOME: 600 MB/s / 360 MB/s Total read / write bandwidth of $WORK: 3600 MB/s / 2200
MB/s
11 | Steinbuch Centre for Computing | ZKI in Göttingen
GridKa, the Tier-1 computer center for LHCLHC=large hadron collider (CERN)
Grid based structure in a worldwide environment for high energy physics
• gLite
• storage resource manager (SRM) based on dCache
• Resource broker for world wide usage of LHC resources
• PBSpro for resource management
• Certification Authority (CA)
12 | Steinbuch Centre for Computing | ZKI in Göttingen
GridKa, usage in Germany
22 Research Centers
44 Working Groups
350 Scientists
Aachen (4)●Bielefeld (2)●Bochum (2)●Bonn (3)●Darmstadt (1)▲Dortmund (1)●Dresden (2)●Erlangen (1)●Frankfurt (1)●Freiburg (2)●Hamburg (1)▲Heidelberg (1)▲(6)●Karlsruhe (2)●Mainz (3)●Mannheim (1)●München (1)●(5)▲Münster (1)●Rostock (1)●Siegen (1)●Wuppertal (2)●
▲
13 | Steinbuch Centre for Computing | ZKI in Göttingen
GridKa, future plans
FZK has signed the WLCG MoU and is able to deliver resources for the LHC experiment up to 2022 as the Tier-1 center
resources for WLCG (Worldwide LHC Computing Grid Projekt
14 | Steinbuch Centre for Computing | ZKI in Göttingen
Motivation for the project CampusGrid
• heterogeneous IT-environment: vector-, SMP-, cluster-, blade- systems, SAN, NAS, Unix, Linux, Windows, Solaris, SuperUX, ….
• global view by the user
• only one user management (ADS from Microsoft)
• one job management
• metacomputing (MPI, ..), “real-time” applications
• access data for visualization at the local workstation
• global accounting
• seamless integration into different projects and middleware concepts like gLite, LCG, D-Grid, Unicore, GT4, ….
15 | Steinbuch Centre for Computing | ZKI in Göttingen
Infrastructure of the project CampusGrid
FSC Celsius
(MetaDataServer)
96 FSC RX 220
IBM eServer e325IBM p630 Power4
IBM xSeries 335
32 Sunfire V20z
Brocade Silkworm 48000
qLogic/Silverstorm IB directorIB-FC gateway
FibreChannel
InfiniBand10/20 Gbit/s
Brocade Switch and IBM Blade JS20/HS20/HS21
EMC Clariion CX700, S-ATA, FC disks
Intel Whitebox
16 Gbit/s
4 Gbit/s4 Gbit/s4 Gbit/s
4 Gbit/s
4 Gbit/s
FSC Celsius
(MetaDataServer)
4 Gbit/s
16 Gbit/s16 Gbit/s 32 Gbit/s
16 Gbit/s
NEC SX-8R environment
16 | Steinbuch Centre for Computing | ZKI in Göttingen
D-Grid (the German Grid initiative)
• starting September 2005 to build a sustainable Grid infrastructure
• six Community Grid projects an the D-Grid Integration Project (DGI)
• funded by BMBF, the Federal Ministry of Education and Research
• astronomy, climate research, high energy physics, engineering research, medical research, humanities
• gLite, GT4, UNICORE
• SRM/dCache for the data access
• heterogeneous CPU access (AIX, Linux, Solaris, SuperUX)
17 | Steinbuch Centre for Computing | ZKI in Göttingen
Shosholoza Project: America’s Cup
Form optimization/Model reduction/Weather forecast
Head of the Scientific Advisory Board
Metras Project: Weather Forecast
Cooperation with the University of Hamburg
Task force SCC (7 employees)
Simulation of Solid Oxid Fuel Cells
Detailed chemistry and 3D
United Airways: Simulation of the respiratory tract
Cooperation with the clinical Center (2 employees)
Numerical HPC
Numerics in HPC at the SCC - Projects
18 | Steinbuch Centre for Computing | ZKI in Göttingen
United Airways Project
19 | Steinbuch Centre for Computing | ZKI in Göttingen
Numerics in HPC at the SCC - Projects
Multicore and coprocessor technology
Cooperation with HP and ClearSpeed Dedicated Research Group
Junior Professor
Numerics on specific architectures Multicore Coprocessors
Visualization Project: on HP SVA
Instationary 3D flow
Software Development
HiFlow-Project: FEM, 3D, HPC (www.hiflow.de)
HiVision-Project: Visualization
OpenLB: Lattice-Boltzmann/HPC (www.openlb.org)
20 | Steinbuch Centre for Computing | ZKI in Göttingen
The influence of a Storage Lake in the Arid Elqui Valley in Chile on the Local Climate
out: Δx = 800 m, Δy = 800m
in(1): Δx = 400 m, Δy = 400 m
in(2): Δx = 200 m, Δy = 200 m
in(3): Δx = 100 m, Δy = 100 m
(out)
(in)
225 x 135 horizontal gridpoints 100 vertical levels
21 | Steinbuch Centre for Computing | ZKI in Göttingen
Thank you!
Questions?