Trigger and manage Fivetran Connector syncs.
Y42 allows you to trigger and manage your Fivetran Connectors. Fivetran Source serves as a bridge between your Fivetran Connector and the Fivetran-produced tables within your data warehouse.
When a build is triggered, Y42 initiates a Sync request to Fivetran API, then monitors Sync status updates. Upon a successful sync, Y42 creates views for each updated Fivetran table, and subsequently triggers builds for any downstream assets, including data tests and models.
Before setting up a Fivetran Source, you have to setup a Fivetran Connector (opens in a new tab) and initiate the first Sync. Your Fivetran Connector needs to be run at least once before the Connector can be triggered by Y42.
You will need to provide the following details during the setup process:
- Connector ID: A unique identifier generated by Fivetran for each Connector. This can be found under Connector > Setup.
- API Key: Can be generated under User > API Key.
- API Secret: Can be generated under User > API Key.
- Project / Database: The project / database of the Fivetran generated dataset / schema.
- Dataset / Schema The dataset / schema of the Fivetran generated tables.
You can choose to use dbt packages for Fivetran or set up your source without them.
- Y42 facilitates the use of community-maintained dbt packages for Fivetran (opens in a new tab). To install these packages, open the
packages.ymlfile located in the root directory in code mode. Then, add the following entry to the yml file:
_10 - package: fivetran/package_name_10 version: x.xx.x
Copy the name and version from the latest package description page, like the Google Ads package (opens in a new tab). For instance, add the following to the yml file:
_10 - package: fivetran/google_ads_10 version: 0.10.1
Next, install the packages by pressing CMD + K (or CTRL + K for Windows) and selecting Install package dependencies. This will create a new "dbt_packages" folder in the catalogue, containing a folder for each installed package, along with all its dependencies.
Locate the source folder within a package folder under
dbt_packagefolder in the Code view. The community maintaining Fivetran dbt packages typically names source folders with a "source" suffix (e.g.,
- Finally, click on the desired source and select "External Fivetran." This links the source in the package with your Fivetran credentials, enabling API calls to the Fivetran API for initiating data syncs.
In the asset creation pop up, select "External Fivetran" and enter a unique name for the Fivetran Source.
On the source setup page, on the Authorization panel, select from existing secrets or create a new one.
To create a new secret, enter a unique name and provide Fivetran API key and secret (opens in a new tab).
Open the settings tab and enter:
Fivetran Connector ID. Can be found under Connector > Setup section.
Project / Databaseand
Dataset / Schemaof your Fivetran tables. Those are the properties of your DWH where Fivetran created tables after the initial sync.
Open the schema tab and verify that you are referencing the correct tables.
- An individual Fivetran table build will automatically trigger builds for all tables of the Fivetran Connector. Fivetran does not allow individual table syncs.
- Failed data tests on Fivetran tables will only stop the DAG and not prevent any updates on the tables themselves.