Open source and free small files storage choices

by DevOpsZen — on  ,  ,  ,  ,  , 

The requirement of storage system for small files are:

  • Efficient
  • Safe
  • Cheap
  • Scalable
  • Simplicity
  • Compatibility
  • High Availability


Ceph provides object storage which compatible with S3 interface, Block storage (RADOS Block Device) which are striped and replicated across multiple machines, and POSIX compatible network file system (Cephfs).

Single Node Ceph Install steps:


GlusterFS is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks.

Facebook haystack

Facebook uses Haystack Object Store to store over 15 billion photos.

OpenStack Swift

Swift is the OpenStack Object Store solution optimized for durability, availability, and concurrency.


TFS is the photos storage project for small files such as photos.


Beansdb is a distributed key-value storage system designed for large scale online system, aiming for high avaliablility and easy management. It took the ideas from Amazon’s Dynamo, then made some simplify to Keep It Simple Stupid (KISS).


The Baidu File System (BFS) is a distributed file system designed to support real-time applications.


FastDFS is an open source high performance distributed file system (DFS). It’s major functions include: file storing, file syncing and file accessing, and design for high capacity and load balance.

Cassandra FS

S3 like file system implemented based on Cassandra

Pithos (based on Cassandra)

pithos is a daemon which provides an S3-compatible frontend for storing files in a Cassandra cluster.


SeaweedFS is a simple and highly scalable distributed file system.