Ceph Data Recovery
Ceph is an open-source distributed storage system underpinning Red Hat Ceph Storage, IBM Storage Ceph, and countless self-managed deployments behind Proxmox, OpenStack and Kubernetes (via Rook). Because Ceph spreads data across many object storage daemons according to CRUSH placement rules, recovery work typically centres on monitor quorum, placement group health and pool configuration rather than any single disk.
Platform Lineage and Naming
Ceph originated as a research project and became a widely adopted open-source project maintained by the Ceph Foundation. Red Hat acquired Inktank, Ceph's commercial steward, in 2014 and packaged the project as Red Hat Ceph Storage; IBM later folded that lineage into IBM Storage Ceph following its acquisition of Red Hat.
Successive major releases (through names such as Luminous, Nautilus, Octopus, Pacific, Quincy, Reef and Squid) progressively replaced the original FileStore OSD backend with BlueStore, changed default deployment tooling from ceph-deploy to cephadm, and improved erasure-coded pool support for RBD and CephFS. Ceph is also embedded as the default storage layer in Proxmox VE and is commonly deployed through the Rook operator on Kubernetes or as the backing store for OpenStack Cinder, Glance and Swift.
- RADOS — Reliable Autonomic Distributed Object Store, Ceph's core layer
- Red Hat Ceph Storage, IBM Storage Ceph
- Inktank — the company Red Hat acquired to obtain Ceph stewardship
- RBD (RADOS Block Device), CephFS, RGW (RADOS Gateway)
- cephadm, Rook (Kubernetes operator), Proxmox VE integrated Ceph
Generations and Models We Evaluate
| Generation / family | Models |
|---|---|
| Distributions | Community Ceph, Red Hat Ceph Storage, IBM Storage Ceph |
| OSD backends | BlueStore (current default), FileStore (legacy, XFS/ext4-based) |
| Deployment contexts | Bare-metal cephadm clusters, Proxmox VE hyper-converged, Rook-on-Kubernetes, OpenStack-integrated |
| Access layers | RBD block images, CephFS distributed file system, RGW S3/Swift-compatible object |
Architecture and Data Layout
At the core, RADOS distributes objects across Object Storage Daemons (OSDs), each typically owning one storage device, using the CRUSH algorithm to compute placement deterministically rather than relying on a central lookup table. Monitors (MONs) maintain and distribute the cluster map, and manager daemons (MGRs) handle metrics and orchestration; losing monitor quorum can make an otherwise healthy cluster of OSDs unreachable.
Data is grouped into placement groups (PGs), which are the unit CRUSH actually places across OSDs within a pool; each pool is configured for either replication (commonly three-way) or erasure coding. RBD images, CephFS file data, and RGW object data are all ultimately stored as RADOS objects inside pools, meaning corruption at the pool or PG level can affect all three access methods simultaneously.
CephFS additionally relies on one or more Metadata Server (MDS) daemons and a dedicated metadata pool to track the file system's directory hierarchy, separate from the data pool holding file contents — so file system recovery often depends on reconstructing MDS journal and metadata pool state as much as recovering data objects.
Protocols and formats: RBD (block), CephFS (POSIX file, NFS/SMB gateways), RGW S3, RGW Swift, CDMI (older RGW)
- BlueStore stores object data directly on raw block devices with its own internal metadata (RocksDB-backed), avoiding a conventional local file system — legacy FileStore instead stored objects as files on XFS or ext4.
- Pools use either replication (multiple full copies across OSDs per CRUSH rules) or erasure coding (data plus coding chunks split across OSDs), and the choice materially changes how many OSD losses a pool can tolerate.
- CRUSH maps encode the physical failure domains (host, rack, room) that placement decisions respect; a misconfigured or corrupted CRUSH map can leave data technically present but unreachable by normal cluster operation.
Failure Scenarios
Logical failures
- Loss of monitor quorum preventing cluster map access
- Corrupted or accidentally edited CRUSH map
- Placement groups stuck in incomplete, down or inconsistent states
- Accidental pool deletion or pg_num/pool misconfiguration
- CephFS MDS journal corruption or metadata pool damage
- Failed major-version upgrade leaving OSDs unable to start
- RBD image or snapshot deleted at the client or orchestration layer (OpenStack, Proxmox, Kubernetes)
Hardware failures
- Simultaneous OSD failures exceeding the pool's replication or erasure-coding tolerance
- Failure of multiple monitor nodes preventing quorum
- Journal or WAL/DB device failure (BlueStore) affecting dependent OSDs
- Network partition isolating a rack or failure domain during rebalancing
- Underlying drive failures during a large-scale rebalance or recovery operation
Encryption and credentials
Ceph supports OSD-level encryption at rest (dm-crypt) managed through cephadm or Ceph's key management integration. Encryption key material must be preserved alongside affected OSDs and monitor data, as encrypted BlueStore volumes cannot be interpreted without it.
What Not To Do Before an Evaluation
- Do not run rebuilds, reconstructions or re-initialisations against an array that has already lost more drives than its protection level allows.
- Do not recreate pools, aggregates, disk groups, storage pools or clusters — these operations write new metadata over the structures a recovery needs.
- Do not swap drives between slots, and do not reorder shelves. Record the original slot and shelf positions before removing anything.
- Do not run file-system repair tools against production volumes before the underlying storage layer has been evaluated.
- Do not restore a backup or replication set over the affected volumes until the recovery scope has been assessed.
- Do not eradicate deleted volumes or empty recycle/destroyed states on platforms that hold deleted data for a retention window.
Our Evaluation and Recovery Process
- Intake and platform identification — array model, generation, firmware, protection layout and the sequence of events that led to the failure.
- Read-only evaluation of the media and array structures, including assessment of drive health and the extent of any physical damage.
- Forensic imaging of all contributing media, with cleanroom work where drives require it. Originals are preserved unaltered.
- Reconstruction of the storage layer — pools, aggregates, parity groups, chunklets, extent groups or objects — from the images.
- Extraction of the layers above: file systems, virtual machines, databases, mailboxes and shares.
- Verification against a file list and customer-nominated critical data, followed by secure return on encrypted media.
Frequently Asked Questions
Can data be recovered from a single failed OSD?
It depends on the pool's protection scheme. Replicated pools may have intact copies elsewhere in the cluster, while erasure-coded pools require enough surviving chunks to reconstruct an object; a recovery evaluation determines what remains available.
We deleted a pool by mistake — is it recoverable?
Pool deletion in Ceph removes the associated objects and is not reversible through normal cluster operations, though remnants may sometimes be reconstructed depending on how quickly OSDs were reused; an evaluation is needed to assess feasibility.
Does it matter whether we run Proxmox, Rook or bare-metal Ceph?
The underlying RADOS structures are the same, but the orchestration layer affects how configuration and keyring data are stored, which matters for reassembling a working cluster during recovery.