Asset properties

Understand Y42 asset properties and types.

Base Asset Properties

All assets have five base properties. These properties are specified in a YAML configuration, except for data transformations which are expressed with a SQL query.

Asset type - models, sources or exposures

Name - Assets in a Y42 space share the same namespace. Each asset must have a unique name.

Materialization

Tests

Tags, Descriptions and Metadata

View all properties

models/stg_customers.yml

stg_customers.sql

version: 2
models: 
  - name: stg_customers
    meta: 
      experts: 
        users: 
          - lea.dang@y42.com 
    description: Staging model 
    config: 
      tags: 
        - customer 
    columns: 
      - name: id 
        data_type: FLOAT 
        description: The id is unique 
        tests: 
          - unique 
      - name: first_name 
        data_type: STRING 
      - name: last_name 
        data_type: STRING

Asset Types

There are three types of assets: source assets, dbt model assets, and exposures. All assets can be defined via UI or in code as showcased below.

Source Assets

Source assets are typically used to ingest or reference data from external sources and persist it in a database. This allows you to easily access and transform the imported data so that it can be used for analysis. A source asset and its tables are specified in the source YAML file. You can also define its schema along with several other options.

sources/shop_source.yml

version: 2
sources:
  - name: shop_source
    meta:
      config:
        type: source-bigquery
        connection: shop_integration
      experts:
        users:
          - lea.dang@y42.com
    tables:
      - name: raw_customers
        config:
          y42_table:
            publish_view: false
            import: raw_customers
            columns:
              - last_name
              - id
              - first_name
            schema:
              $schema: http://json-schema.org/draft-07/schema#
              type: object
              properties:
                last_name:
                  type: string
                id:
                  type: number
                  airbyte_type: integer
                first_name:
                  type: string
            group: sources_tests
            supported_sync_modes:
              - full_refresh
              - incremental
            default_cursor_field: []
            source_defined_cursor: null
            source_defined_primary_key: []
          experts:
            users:
              - lea.dang@y42.com
            teams: []
        columns:
          - name: id
            data_type: FLOAT
            tests: 
              - unique 
          - name: first_name
            data_type: STRING
          - name: last_name
            data_type: STRING
        description: Raw data
        tags:
          - shop
          - customers

Models / DBT Assets

dbt model assets are used to define data transformations that modify the imported data in some way. This could be done by applying filters, aggregations, or other types of manipulations on the source data. The transformed results are then persisted in a database and can be used for further transformation or analysis.

While all other assset configurations are defined in .yml files, modeling logic definitions are exceptions to the rule - they are stored in .sql files.

stg_customers.sql

select id, first_name, last_name 
from {{source('src', 'src_customers')}}

models/stg_customers.yml

version: 2
models: 
  - name: stg_customers
    meta: 
      experts: 
        users: 
          - lea.dang@y42.com 
    description: Staging model
    config: 
      tags: 
        - shop
        - customers
    columns: 
      - name: id 
        data_type: FLOAT 
        description: The id is unique 
        tests: 
          - unique 
      - name: first_name 
        data_type: STRING 
      - name: last_name 
        data_type: STRING

Exposures Assets

Exposures are data assets that reference to a downstream use of your project's data. This could encompass a dashboard, an application, a Python notebook, or a data science project. An exposure groups all relevant upstream assets together to define the data required for external use. By employing exposures, you can:

Group multiple assets together to assess whether all upstream dependencies have been refreshed for a specific exposure and effortlessly refresh the data if needed.

Provide additional context to external data consumers in the exposure data catalog pages.

daily_purchases_update.yml

version: 2
exposures:
  - name: daily_purchases_update
    type: dashboard
    owner:
      name: Lea Dang
      email: lea.dang@y42.com
    depends_on:
      - ref('stg_purchases')
      - source('shop_source','purchases')
    maturity: high
    url: https://bi.tool/purchases-dashboard
    description: Daily updated dashboard of all purchases.
    tags:
      - purchases
    meta:
      experts:
        users:
          - lea.dang@y42.com
        teams: []

Stateful Data Assets Virtual data builds