Skip to content

Configuration

YAML Configuration File

application_name - the name of the application

connections - a list of connections used in the application. You may as many connections as you require.

async_mode - a boolean value that determines if the application should run in async mode. Default is false

Connections are further defined by:

  • name - name of the connection that is used to refer to and identify this connection

  • backend - a python path to connection implementation class

  • credentials - credentials required by the connection that will be passed to the connect method of your connection implementation class.

resources_config - the top level resource allocation of the apuplication, including:

  • lakehouse - the name of connection (specified in connections section above) that will be used for the AMSDAL Lakehouse connection

  • lock - the name of connection that will be used for locking functionality required to keep your data consistent

  • repository - connection that should be used by default as well as for each model

    • default: connection name
    • models:
      • model_name_a: connection name
      • model_name_b: connection name
      • ...

Example

application_name: MyApp
connections:
  - name: lakehouse-connection-1
    backend: postgres-historical
    credentials: 
      configs:
        spark.ui.enabled: false
  - name: data-connection-2
    backend: sqlite
    credentials:
      - db_path: ./default.db
  - name: data-connection-3
    backend: sqlite
    credentials:
      - db_path: ./users.db
  - name: lock-connection
    backend: amsdal_data.lock.implementations.redis_lock.RedisLock
    credentials:
      - host: localhost
      - port: 6379
resources_config:
  lakehouse: lakehouse-connection-1
  lock: lock-connection
  repository: 
    default: data-connection-2
    models:
      user: data-connection-3

application_name: MyApp
async_mode: true
connections:
  - name: lakehouse-connection-1
    backend: postgres-historical-async
    credentials: 
      configs:
        spark.ui.enabled: false
  - name: data-connection-2
    backend: sqlite-async
    credentials:
      - db_path: ./default.db
  - name: data-connection-3
    backend: sqlite-async
    credentials:
      - db_path: ./users.db
  - name: lock-connection
    backend: amsdal_data.lock.implementations.redis_lock.RedisLock
    credentials:
      - host: localhost
      - port: 6379
resources_config:
  lakehouse: lakehouse-connection-1
  lock: lock-connection
  repository: 
    default: data-connection-2
    models:
      user: data-connection-3

Review

{
  "config_path": "./config.yml",
  "http_port": 8080,
  "check_model_exists": true,
  "json_indent": 4
}