S3 Storage

S3 Storage#

Module type

storage

S3Storage: A storage module for saving media files to an S3-compatible object storage.

Features#

  • Uploads media files to an S3 bucket with customizable configurations.

  • Supports random_no_duplicate mode to avoid duplicate uploads by checking existing files based on SHA-256 hashes.

  • Automatically generates unique paths for files when duplicates are found.

  • Configurable endpoint and CDN URL for different S3-compatible providers.

  • Supports both private and public file storage, with public files being readable online.

Notes#

  • Requires S3 credentials (API key and secret) and a bucket name to function.

  • The random_no_duplicate option ensures no duplicate uploads by leveraging hash-based folder structures.

  • Uses boto3 for interaction with the S3 API.

  • Depends on the HashEnricher module for hash calculation.

Configuration Options#

YAML#

# steps configuration
steps:
...
  storages:
  - s3_storage
...

# module configuration
...

s3_storage:
  path_generator: flat
  filename_generator: static
  bucket:
  region:
  key:
  secret:
  random_no_duplicate: false
  endpoint_url: https://{region}.digitaloceanspaces.com
  cdn_url: https://{bucket}.{region}.cdn.digitaloceanspaces.com/{key}
  private: false

Command Line:#

Option

Description

Default

Type

s3_storage.path_generator

Optional. how to store the file in terms of directory structure: ‘flat’ sets to root; ‘url’ creates a directory based on the provided URL; ‘random’ creates a random directory.

flat

string

s3_storage.filename_generator

Optional. how to name stored files: ‘random’ creates a random string; ‘static’ uses a hash, with the settings of the ‘hash_enricher’ module (defaults to SHA256 if not enabled).

static

string

s3_storage.bucket

Optional. S3 bucket name

None

string

s3_storage.region

Optional. S3 region name

None

string

s3_storage.key

Optional. S3 API key

None

string

s3_storage.secret

Optional. S3 API secret

None

string

s3_storage.random_no_duplicate

Optional. if set, it will override path_generator, filename_generator and folder. It will check if the file already exists and if so it will not upload it again. Creates a new root folder path no-dups/

False

bool

s3_storage.endpoint_url

Optional. S3 bucket endpoint, {region} are inserted at runtime

https://{region}.digitaloceanspaces.com

string

s3_storage.cdn_url

Optional. S3 CDN url, {bucket}, {region} and {key} are inserted at runtime

https://{bucket}.{region}.cdn.digitaloceanspaces.com/{key}

string

s3_storage.private

Optional. if true S3 files will not be readable online

False

bool

API Reference

Contents