Reference data from your data warehouse
Reference data from your data warehouse in your Y42 space schema.
Before you can use an existing data in a Y42 space, you must reference it in your space schema. To do so, you can use the Native Reference source or Airbyte BigQuery/ Snowflake source.
|Y42 source asset||Choose this method when...||Limitations|
|Native Reference (BigQuery)||The schema is in the same GCP project as the space schema OR is a shared schema.||The schema must be in the same location/region as the space schema.|
|Native Reference (Snowflake)||The schema is accessible by the Snowflake user.||NA|
|Airbyte BigQuery/ Snowflake||You want to physically replicate the data into your space schema.||Ingestion rows usage limits apply.|
The Native Reference source represents dbt’s Sources and can be defined using dbt Source properties (opens in a new tab). If you use a Native Reference source, Y42 will access the schema you want to import using a GCP or Snowflake service account. Y42 then creates Logical Views (opens in a new tab) in BigQuery or Non-Materilaized Views (opens in a new tab) in Snowflake of your selected tables in the space schema. After which, you can reference these Views just as you would with other assets.
The schema you want to reference must be located in the same hosting location or region (opens in a new tab) as the space schema.
Imports will not generate any logs. Please consult the FAQ for common issues.
Native Reference views do not show any row count.
Views do not behave the same as materialized tables. Please refer to BigQuery's documentation (opens in a new tab) for any limitations on schema changes, data freshness, and access. Depending on your data warehouse, data may be automatically updated without triggering an import, schemas may be out of date unless an import is triggered, etc.:
- BigQuery Logical views automatically reflect the base table’s data regardless of when they have been created.
Views do not automatically propagate parent table changes. To view updated data you have to run an import via a DAG or an ad-hoc import run.
Quoted column names - By default, Snowflake uses UPPER-CASED characters for table and column names, unless specified otherwise. As a result, if you're dealing with tables or columns that have case-sensitive or lower-cased names, it's necessary to use quotes around both the table and column names when making queries.
Y42 will utilize the space's service account credentials to access the tables that you would like reference. As such, you do not need to configure a new service account.
Y42 requires additional permissions if the schema you want to reference does not belong to the same GCP project as the Y42 space.
In the GCP project of the schema that you would like to load, navigate to IAM and follow this guide (opens in a new tab). You will have to grant at least BigQuery Data Owner permissions to the project and/ or schema.
We recommend using Native Reference sources as the default method to reference tables from BigQuery and Snowflake. However, there may be scenarios in which you might prefer to use the Airbyte BigQuery source or Airbyte Snowflake source to replicate a schema into the space schema.