Learn how to turn your branches into data warehouse environments.
With Virtual Data Builds, you can manage your data environment by turning branches into virtual environments. This functionality offers flexibility and control over which branches to publish, and it allows for seamless integration with datasets and schemas in Bigquery and Snowflake.
By default, the
main branch is automatically published. To publish additional branches, follow these steps:
- Open Space Settings.
- Navigate to
Branch environments> Choose a branch >
Publish dataset / schema.
- Enable the publish option.
- Specify the dataset or schema where you'd like to publish the data.
A new dataset (for Bigquery) or schema (for Snowflake) will be created using the following naming convention:
The naming convention for the two types of assets materialized in the dataset/schema is as follows:
The assets will be created based on the state of the branch that you publish. The assets target the correct physical UUID tables, leveraging the internals of Virtual Data Builds.
Y42 creates a view for every asset that has been built successfully, referencing the correct underlying UUID table of the DWH within the dataset/schema of the respective published branch.
- The assets always reference the latest successful build of the respective branch (excluding builds that have failed tests).
- Within the Y42 UI, you can find a redirection link straight to the view as part of the last successful build. This functionality only works if the branch is published as a virtual environment.
- If certain assets shouldn't be published to the virtual branch environment of the respective branch, you can simply do so in the UI by hovering over the link and clicking on “Unpublish.”
- Under the hood, this modifies a property called
publish_view: boolean, which can also be edited directly in code mode.
This feature is particularly useful if certain sources or models, e.g., staging models or raw sources, shouldn't be published for downstream consumption.