Generic Extractor

Generic Extractor#

Module type

extractor

This is the generic extractor used by auto-archiver, which uses yt-dlp under the hood.

This module is responsible for downloading and processing media content from platforms supported by yt-dlp, such as YouTube, Facebook, and others. It provides functionality for retrieving videos, subtitles, comments, and other metadata, and it integrates with the broader archiving framework.

For a full list of video platforms supported by yt-dlp, see the official documentation

Features#

  • Supports downloading videos and playlists.

  • Retrieves metadata like titles, descriptions, upload dates, and durations.

  • Downloads subtitles and comments when enabled.

  • Configurable options for handling live streams, proxies, and more.

  • Supports authentication of websites using the ‘authentication’ settings from your orchestration.

Dropins#

  • For websites supported by yt-dlp that also contain posts in addition to videos (e.g. Facebook, Twitter, Bluesky), dropins can be created to extract post data and create metadata objects. Some dropins are included in this generic_archiver by default, but custom dropins can be created to handle additional websites and passed to the archiver via the command line using the --dropins option (TODO!).

You can see all currently implemented dropins in the source code.

Auto-Updates#

The Generic Extractor will also automatically check for updates to yt-dlp (every 5 days by default). This can be configured using the ytdlp_update_interval setting (or disabled by setting it to -1). If you are having issues with the extractor, you can review the version of yt-dlp being used with yt-dlp --version.

Configuration Options#

YAML#

# steps configuration
steps:
...
  extractors:
  - generic_extractor
...

# module configuration
...

generic_extractor:
  subtitles: true
  comments: false
  livestreams: false
  live_from_start: false
  proxy: ''
  proxy_on_failure_only: true
  end_means_success: true
  allow_playlist: false
  max_downloads: inf
  bguils_po_token_method: auto
  extractor_args: {}
  ytdlp_update_interval: 5
  ytdlp_args: ''

Command Line:#

Option

Description

Default

Type

generic_extractor.subtitles

Optional. download subtitles if available

True

bool

generic_extractor.comments

Optional. download all comments if available, may lead to large metadata

False

bool

generic_extractor.livestreams

Optional. if set, will download live streams, otherwise will skip them; see –max-filesize for more control

False

bool

generic_extractor.live_from_start

Optional. if set, will download live streams from their earliest available moment, otherwise starts now.

False

bool

generic_extractor.proxy

Optional. http/https/socks proxy to use for the webdriver, eg https://proxy-user:password@proxy-ip:port

string

generic_extractor.proxy_on_failure_only

Optional. Applies only if a proxy is set. In that case if this setting is True, the extractor will only use the proxy if the initial request fails; if it is False, the extractor will always use the proxy.

True

string

generic_extractor.end_means_success

Optional. if True, any archived content will mean a ‘success’, if False this extractor will not return a ‘success’ stage; this is useful for cases when the yt-dlp will archive a video but ignore other types of content like images or text only pages that the subsequent extractors can retrieve.

True

bool

generic_extractor.allow_playlist

Optional. If True will also download playlists, set to False if the expectation is to download a single video.

False

bool

generic_extractor.max_downloads

Optional. Use to limit the number of videos to download when a channel or long page is being extracted. ‘inf’ means no limit.

inf

string

generic_extractor.bguils_po_token_method

Optional. Set up a Proof of origin token provider. This process has additional requirements. See authentication for more information.

auto

string

generic_extractor.extractor_args

Optional. Additional arguments to pass to the yt-dlp extractor. See yt-dlp/yt-dlp.

{}

json_loader

generic_extractor.ytdlp_update_interval

Optional. How often to check for yt-dlp updates (days). If positive, will check and update yt-dlp every [num] days. Set it to -1 to disable, or 0 to always update on every run.

5

int

generic_extractor.ytdlp_args

Optional. Additional arguments to pass to yt-dlp, e.g. –no-check-certificate or –plugin-dirs.See yt-dlp documentation here for more information: yt-dlp/yt-dlp this is not to be confused with ‘extractor_args’ which are specific to the extractor itself.

string

API Reference

Contents