S3 Storage

Module type

storage

S3Storage: A storage module for saving media files to an S3-compatible object storage.

Features

• Uploads media files to an S3 bucket with customizable configurations.

• Supports random_no_duplicate mode to avoid duplicate uploads by checking existing files based on SHA-256 hashes.

• Automatically generates unique paths for files when duplicates are found.

• Configurable endpoint and CDN URL for different S3-compatible providers.

• Supports both private and public file storage, with public files being readable online.

Notes

• Requires S3 credentials (API key and secret) and a bucket name to function.

• The random_no_duplicate option ensures no duplicate uploads by leveraging hash-based folder structures.

• Uses boto3 for interaction with the S3 API.

• Depends on the HashEnricher module for hash calculation.

Configuration Options

YAML

s3_storage:
  path_generator: flat
  filename_generator: static
  bucket:
  region:
  key:
  secret:
  random_no_duplicate: false
  endpoint_url: https://{region}.digitaloceanspaces.com
  cdn_url: https://{bucket}.{region}.cdn.digitaloceanspaces.com/{key}
  private: false

Command Line:

Option	Description	Default	Type
s3_storage.path_generator	Optional. how to store the file in terms of directory structure: ‘flat’ sets to root; ‘url’ creates a directory based on the provided URL; ‘random’ creates a random directory.	flat	string
s3_storage.filename_generator	Optional. how to name stored files: ‘random’ creates a random string; ‘static’ uses a replicable strategy such as a hash.	static	string
s3_storage.bucket	Optional. S3 bucket name	None	string
s3_storage.region	Optional. S3 region name	None	string
s3_storage.key	Optional. S3 API key	None	string
s3_storage.secret	Optional. S3 API secret	None	string
s3_storage.random_no_duplicate	Optional. if set, it will override path_generator, filename_generator and folder. It will check if the file already exists and if so it will not upload it again. Creates a new root folder path no-dups/	False	bool
s3_storage.endpoint_url	Optional. S3 bucket endpoint, {region} are inserted at runtime	https://{region}.digitaloceanspaces.com	string
s3_storage.cdn_url	Optional. S3 CDN url, {bucket}, {region} and {key} are inserted at runtime	https://{bucket}.{region}.cdn.digitaloceanspaces.com/{key}	string
s3_storage.private	Optional. if true S3 files will not be readable online	False	bool

API Reference