WEKA Data Platform Data Recovery
The WEKA Data Platform (built on WekaFS) is a distributed, high-performance file system designed to run across clusters of NVMe-attached servers, commonly used for AI, machine learning and other latency-sensitive workloads. Recovery cases typically concern the distributed data protection scheme across cluster nodes and the relationship between hot NVMe tiers and tiered object storage, rather than a single drive.
Platform Lineage and Naming
WEKA (WekaIO) developed WekaFS as a purpose-built parallel file system optimised for NVMe flash and low-latency networking, aimed at replacing or complementing traditional HPC and AI storage architectures. It has been positioned as a software-defined platform that can run on-premises across commodity or vendor-qualified NVMe servers, or in public cloud environments.
Over successive releases WEKA added native tiering to object storage (on-premises or cloud), snapshot and snap-to-object capabilities for long-term retention, and broadened protocol access beyond its native client to include NFS, SMB and S3 gateways alongside POSIX semantics.
- WekaIO — the company's earlier name, now generally referred to as WEKA
- WekaFS — the underlying distributed file system technology
- WEKA Data Platform — the current product branding
Generations and Models We Evaluate
| Generation / family | Models |
|---|---|
| Deployment | On-premises NVMe server clusters, cloud-hosted (AWS, Azure, GCP) clusters |
| Protection levels | N+2 and N+4 distributed data protection configurations |
| Storage organisation | Clusters divided into buckets across which data is distributed and protected |
Architecture and Data Layout
WekaFS distributes file system data and metadata across all NVMe drives and nodes participating in a cluster, organised into buckets, which are logical groupings across which the distributed data protection scheme operates. This differs from a per-node file system with local RAID, since a single file's data may be spread across many nodes' NVMe devices.
Distributed data protection is configured as N+2 or N+4, meaning the cluster can tolerate the loss of two or four failure domains (drives or nodes, depending on configuration) respectively without data loss, using an approach that provides RAID-like protection without concentrating rebuild I/O on any single device.
WEKA supports tiering cooler data out to object storage — on-premises or cloud-based — while keeping hot data and metadata on NVMe; snapshots can likewise be exported to object storage (snap-to-object) for retention beyond the live cluster's capacity, meaning a full recovery picture may need to account for both the NVMe cluster and any tiered object storage target.
Protocols and formats: POSIX (native WEKA client), NFS, SMB, S3 (via gateway, both as client access and tiering target)
- N+2 and N+4 protection distribute redundancy across the cluster's drives and nodes according to configured failure domains, allowing multiple simultaneous failures to be tolerated without data loss, up to the configured level.
- Data tiered to object storage is organised differently from the hot NVMe tier, so recovering a fully tiered dataset can require reconstructing state from both the cluster's internal metadata and the external object store.
Failure Scenarios
Logical failures
- Cluster metadata corruption following an interrupted software upgrade or configuration change
- Accidental deletion of files, snapshots, or snap-to-object exports
- Tiering policy misconfiguration leaving data inconsistently placed between NVMe and object tiers
- Bucket rebalancing interrupted mid-operation by node failure
Hardware failures
- Multiple node or drive failures exceeding the configured N+2 or N+4 protection level
- Network fabric failure disrupting low-latency interconnect between cluster nodes
- Object storage tier failures or inconsistency affecting tiered or snap-to-object data
- Simultaneous failures during an active bucket rebalance or protection-level change
What Not To Do Before an Evaluation
- Do not run rebuilds, reconstructions or re-initialisations against an array that has already lost more drives than its protection level allows.
- Do not recreate pools, aggregates, disk groups, storage pools or clusters — these operations write new metadata over the structures a recovery needs.
- Do not swap drives between slots, and do not reorder shelves. Record the original slot and shelf positions before removing anything.
- Do not run file-system repair tools against production volumes before the underlying storage layer has been evaluated.
- Do not restore a backup or replication set over the affected volumes until the recovery scope has been assessed.
- Do not eradicate deleted volumes or empty recycle/destroyed states on platforms that hold deleted data for a retention window.
Our Evaluation and Recovery Process
- Intake and platform identification — array model, generation, firmware, protection layout and the sequence of events that led to the failure.
- Read-only evaluation of the media and array structures, including assessment of drive health and the extent of any physical damage.
- Forensic imaging of all contributing media, with cleanroom work where drives require it. Originals are preserved unaltered.
- Reconstruction of the storage layer — pools, aggregates, parity groups, chunklets, extent groups or objects — from the images.
- Extraction of the layers above: file systems, virtual machines, databases, mailboxes and shares.
- Verification against a file list and customer-nominated critical data, followed by secure return on encrypted media.
Frequently Asked Questions
How many node or drive failures can a WEKA cluster tolerate?
It depends on whether the cluster is configured for N+2 or N+4 protection; either can tolerate multiple simultaneous failures, but exceeding the configured level can affect data availability, which an evaluation would confirm.
If data was tiered to object storage, is it separate from the NVMe cluster for recovery purposes?
Not entirely — WekaFS metadata ties hot and tiered data together, so a full recovery evaluation typically needs to consider both the NVMe cluster state and the object storage tier together.
Can WEKA clusters in the cloud be recovered the same way as on-premises clusters?
The WekaFS data structures are consistent across deployment types, though cloud deployments require working with the specific cloud provider's underlying volumes and snapshots as part of the evaluation.