core.base_module
================

.. py:module:: core.base_module




Module Contents
---------------

.. py:class:: BaseModule

   Bases: :py:obj:`abc.ABC`


   Base module class. All modules should inherit from this class.

   The exact methods a class implements will depend on the type of module it is,
   however modules can have a .setup() method to run any setup code
   (e.g. logging in to a site, spinning up a browser etc.)

   See BaseModule.MODULE_TYPES for the types of modules you can create, noting that
   a subclass can be of multiple types. For example, a module that extracts data from
   a website and stores it in a database would be both an 'extractor' and a 'database' module.

   Each module is a python package, and should have a __manifest__.py file in the
   same directory as the module file. The __manifest__.py specifies the module information
   like name, author, version, dependencies etc. See BaseModule._DEFAULT_MANIFEST for the
   default manifest structure.



   .. py:attribute:: MODULE_TYPES
      :value: ['feeder', 'extractor', 'enricher', 'database', 'storage', 'formatter']



   .. py:attribute:: config
      :type:  Mapping[str, Any]


   .. py:attribute:: authentication
      :type:  Mapping[str, Mapping[str, str]]


   .. py:attribute:: name
      :type:  str


   .. py:attribute:: tmp_dir
      :type:  tempfile.TemporaryDirectory
      :value: None



   .. py:property:: storages
      :type: list



   .. py:method:: config_setup(config: dict)


   .. py:method:: setup()


   .. py:method:: auth_for_site(site: str, extract_cookies=True) -> Mapping[str, Any]

      Returns the authentication information for a given site. This is used to authenticate
      with a site before extracting data. The site should be the domain of the site, e.g. 'twitter.com'

      extract_cookies: bool - whether or not to extract cookies from the given browser and return the
      cookie jar (disabling can speed up) processing if you don't actually need the cookies jar

      Currently, the dict can have keys of the following types:
      - username: str - the username to use for login
      - password: str - the password to use for login
      - api_key: str - the API key to use for login
      - api_secret: str - the API secret to use for login
      - cookie: str - a cookie string to use for login (specific to this site)
      - cookies_jar: YoutubeDLCookieJar | http.cookiejar.MozillaCookieJar - a cookie jar compatible with requests (e.g. `requests.get(cookies=cookie_jar)`)



   .. py:method:: repr()