YOUR INFORMATION IS OUR BUSINESS
Dr. Wolfgang Obelöer Technischer Leiter itiso GmbH
Was ist Object Storage und wer braucht’s? Eine Übersicht und die Lösung von Huawei
DLR Storagetechnology 2016 15.-17.06.2016
„Radisson Blu Hotel“ in Rostock
YOUR INFORMATION IS OUR BUSINESS
Agenda • Über itiso • Was ist Object Storage? • Beispiel: Huawei OceanStor 9000 • Einsatzfelder • Zusammenfassung
YOUR INFORMATION IS OUR BUSINESS
Über itiso • Herstellerunabhängiges Systemhaus
• Unser Thema: Datenmanagement
• Gegründet Januar 2005: Hamburg (Zentrale), Hannover, Frankfurt
• Portfolio:
YOUR INFORMATION IS OUR BUSINESS
Was ist Object Storage
YOUR INFORMATION IS OUR BUSINESS
Object Storage • Objekte
– sind wie Files in einem simplen File-System – werden dynamisch erstellt und gelöscht – haben eine variable Länge – beinhalten erweiterbare Attribute – werden durch eine ID angesprochen
• Der Object Storage managt alle Operationen auf den Objekten – für eine 4TB Platte müssen z.B. 1 Milliarde 4KB-Blöcke
verwaltet werden – dies wird durch den Object Storage komplett verborgen
YOUR INFORMATION IS OUR BUSINESS
Object Storage vs. Block vs. File De facto-Standard: S3
YOUR INFORMATION IS OUR BUSINESS
Erasure Coding • - Basiert auf Vandermonde-Reed-Solomon error correction • - Anwendbar für Archive, Backup und Multimedia
YOUR INFORMATION IS OUR BUSINESS
Erasure Coding (einfaches Beispiel)
Coding
Value Split CodeA1=1A2=2A3=3A4=5
X1=1
X2=2A=12
Example
MirrorA1=1A2=1A3=2A4=2
YOUR INFORMATION IS OUR BUSINESS
Vorteile Object Storage • Skalierbarkeit (Verteilter Ansatz möglich) • Schneller Zugriff (Unabhängige Verarbeitung) • Hohe Datensicherheit (Erasure Coding) • S3 als Schnittstelle (Standard in der Cloud) • Niedrige Kosten durch Einsatz von Standard
X86-Hardware • Rechte etc. in Metadaten im Objekt (Selbst-
verwaltung)
YOUR INFORMATION IS OUR BUSINESS
Beispiel: Huawei OceanStor 9000
YOUR INFORMATION IS OUR BUSINESS
Huawei OceanStor 9000
40 PB
Single file system
740 m2 of equipment room
288 OceanStor 9000 C nodes
Large-capacity cluster verification lab (Chengdu Research Center)
YOUR INFORMATION IS OUR BUSINESS
Huawei Storage Portfolio
S2200T/S2600T 18500/18800
Massive Storage
DeviceManager
(Device Management)
eBackup
(Virtualization Backup) (Unified Management)
Stor
age
Solu
tions
So
ftwar
e Pr
oduc
ts
Har
dwar
e Pr
oduc
ts
All-flash array 2 controllers 500 µs latency
Smart Series Hyper Series Info Series
Dorado2100 G2 Dorado5100
(Data Protection Software) (Intelligent Resource Management Software) (Big Data Storage Value-Added Software )
InfoTier InfoEqualizer InfoAllocator InfoExplorer InfoProtector
SmartTier SmartMotion SmartQoS SmartThin SmartPartition SmartVirtualization
HyperCopy HyperSnap HyperClone HyperReplication
2-16 controllers 3 TB cache 3216 disks SPC-1: 1,005,893 IOPS
Unified Storage
Media HPC Analysis Archiving DR & Backup Active-Active File Sharing Virtualization Database Acceleration
Converged Storage Solutions Big Data Storage Solutions Flash Acceleration Solution
2 controllers 16 GB cache 276 disks
eSight
Video Surveillance
5300 V3/5500 V3 6800 V3 5600 V3/5800 V3
2-8 controllers 512 GB cache 750 disks
2-8 controllers 1 TB cache 1250 disks
2-8 controllers 4TB cache 3200 disks
Storage Integration
3-288 nodes 100 PB capacity Mutiple protocol
9000
YOUR INFORMATION IS OUR BUSINESS
OceanStor 9000 - Higlights
•Green design hardware
•4.3PB in single rack
Low TCO
•Erasure code distributed data protection
•E2E integration Check
High Reliability
•Convergence of file & object storage
•Multiple Interfaces
Openness & Convergence
OceanStor 9000 Object Storage System •Full distributed architecture
•100 PB single namespace
Flexible Scalability
YOUR INFORMATION IS OUR BUSINESS
Original Object Storage
Object Storage
File System
Disk Array
Object Gateway
OceanStor 9000
Original object storage: object does
not need to map to file system,
shorter access delay
Concurrent access, compared to
the object gateway and disk array
architecture, full redundancy, no
single point of failure
Scale-out,capacity and
performance increase linearly with
the increase of nodes
OceanStor 9000: Number of Objects: 10 billion Number of buckets: 100 million
### Number of Objects: 100 million Number of buckets: 4 million
Object Storage
YOUR INFORMATION IS OUR BUSINESS
OceanStor 9000: Architektur
Node Node Node Node Node Node Node Node Node Node
Hardware Node Layer
Node Node Node Node Node Node Node Node Node Node
Application layer
Data processing
layer
HPC Video surveillance
Media and entertainment
Object storage
Backup and archiving software
Backup and archiving
Web disk
Data analysis
Precision marketing
Connections to Hadoop
Commercial promotion
FTP NFS CIFS HDFS NDMP
Distributed Storage Layer
Object Storage NAS Storage
Swift/S3
Managem
ent
File storage
Interconnect: 10 GE or Infiniband
YOUR INFORMATION IS OUR BUSINESS
Flexible Skalierbarkeit
YOUR INFORMATION IS OUR BUSINESS
Flexible Skalierbarkeit
YOUR INFORMATION IS OUR BUSINESS
Zuverlässige Datenabsicherung • N+M redundancy, support N+1 to N+3 redundancy,
maximum 3*36 disks failure - business is not interrupted
• Hot spare space replaces hot spare disks. • Data reconstruction replaces disk reconstruction. • Data recovery speed reaches 1 TB/h, 10 times
higher than traditional mode
Traditional RAID5
Traditional RAID6
Distributed Erasure Code
Max. allowed number of faulty disks 1 2 108
Need hot spare disk or not yes yes no
1 TB data recovery time 9.5 hours 10 hours 1 hour
+1 +2 +3
YOUR INFORMATION IS OUR BUSINESS
Data „Aerobics“
Add Checksum
Write diagram Additional Process
System
OBS
local
Data EC
Write to Disk
Rebuild data
Adding checksum to enable detection of corrupted data
Detect errors based on health state of HDD, when reading data or during silent check and process automated error recovery
YOUR INFORMATION IS OUR BUSINESS
Object Deduplizierung
Object level (Object with the same fingerprint would be deduplicated )
Post proceed, calculate fingerprint when upload, backend task compare the fingerprint of all object and delete duplicate data
Global deduplication, Duplicate data comparison is based on all the objects of the entire system, not limited to a single user or a single barrel
Flexible settings, deduplication configuration base on user or bucket, bucket will inherit the policy of it’s user
OceanStor 9000
Account (Dedup enable) Account (Dedup disable)
Bucket
YOUR INFORMATION IS OUR BUSINESS
Performance-Optimierung Cache-Aware Algorithm
Optimized Protocol
File File
Location of current read
Location of next read
Cache miss causes dropped frames and erratic display
Cache hit, smooth video editing and streaming, rapid response
Legend:
Read data Prefetch data
……
One client, one link, one server
One client, multi-link, multiple-server layout breaks through the bottlenecks in the network links and issues with single-server deployments
Mac 10.10 or Redhat 6.3 private client
Location of current read
Location of next read
Single client performance up to 2.5GB/s
Global Cache
SSD-Layer with for caching and autotiering
YOUR INFORMATION IS OUR BUSINESS
Geringe TCO
High-density and energy-saving hardware
Intelligent software and multi-level power conservation
Energy-Saving ATOM CPUs
ATOM Others
50%
Power consumption/processing unit
• 4 U 72-slot enclosure for 3.5-inch hot swappable disks •Single cabinet capable of providing a maximum of 4.3 PB capacity
High-Density nodes
Node control
Normal
Intelligent CPU frequency control
Intelligent fan speed control
Disk start delay
100%
70%
35%
24% 27%
Flow
Hea
vy
Low
Power consumption/node
Intelligent Energy-Saving Design High Efficiency PSU/FAN
•Gold Power PSU: The power
conversion efficiency over 90% •16 level intelligent self-adjusted fan
• Node expansion in 1 min
YOUR INFORMATION IS OUR BUSINESS
Huawei OceanStor 9000
OceanStor 9000
Performance
Flexible Scalability
Open Convergence
• 400 GB/s system throughput, exclusive InfoTurbo acceleration technology achieves up to 2.5 GB/s in bandwidth over a single client.
• Fully symmetric and distributed architecture, accommodation of up to 288 nodes, 100 PB storage space
• All-IP network topology to reduce complexity, and supporting various data types, such as file and object
YOUR INFORMATION IS OUR BUSINESS
Einsatzfelder
YOUR INFORMATION IS OUR BUSINESS
Anwendungs-Szenarien
Data Components
Video
Music
Picture
Data file
Seldom updated
Undefined value
Large capacity and high growth speed
Long storage time
25%
75% (unstructured data)
YOUR INFORMATION IS OUR BUSINESS
Object Storage und HSM (Versity) Right (High)
Performance
High Data Protection
Continuous Data Health Check
Flexible Data Placement
HSM (Versity VSM)
YOUR INFORMATION IS OUR BUSINESS
Zusammenfassung • Object Storage ist sinnvoll einsetzbar für viele
Anwendungsgebiete • Sichere und kostengünstige
Datenspeicherung • Huawei OceanStor 9000 bietet
– Flexible Skalierung – Sehr umfangreiche Softwarefeatures – Alle wichtigen Protokolle
YOUR INFORMATION IS OUR BUSINESS
YOUR INFORMATION IS OUR BUSINESS
itiso GmbH Flughafenstraße 52a 22335 Hamburg http://www.itiso.de
Ihr Ansprechpartner Dr. Wolfgang Obelöer