Fivetran
Trigger and manage Fivetran Connector syncs.
Overview
Y42 allows you to trigger and manage your Fivetran Connectors. Fivetran Source serves as a bridge between your Fivetran Connector and the Fivetran-produced tables within your data warehouse.
Fivetran Source
Trigger Fivetran syncs out of Y42.When a build is triggered for one of the source tables, Y42 initiates a Sync request to Fivetran, then monitors Sync status updates. Upon a successful sync, Y42 creates views for each updated Fivetran table, and subsequently triggers builds for any downstream assets, including data tests and models.
Requirements and prerequisites
Before setting up a Fivetran Source, you have to setup a Fivetran Connector (opens in a new tab) and initiate the initial Sync. Your Fivetran Connector needs to be run at least once before the Connector can be triggered by Y42.
You will need to provide the following details:
- Connector ID: A unique identifier for the Connector. This can be found under Connector > Setup.
- API Key: Can be generated under User > API Key.
- API Secret: Can be generated under User > API Key.
- Project / Database: The project / database of the Fivetran generated dataset / schema.
- Dataset / Schema The dataset / schema of the Fivetran generated tables.
Set up guide
Create a Fivetran Source.
In the asset creation pop up, select Source and enter a unique name for the Fivetran Source. Select Fivetran from the Source type dropdown and click on Create.
Create a new connection or use an existing connection.
On the source setup page, open the Authorization panel. Select from any existing connections or create a new connection.
For new connections, enter all the previously gathered details.
Declare the database and schema of your Fivetran tables.
Open the settings tab and enter the Project / Database and Dataset / Schema of your Fivetran tables.
Verify selected tables and columns.
Open the schema tab and verify that you are referencing the correct tables.
If you do not see the correct tables or columns, verify that you have entered the correct database and schema in the previous step.
Commit your changes.
Considerations
- An individual Fivetran table build will automatically trigger builds for all tables of the Fivetran Connector. Fivetran does not allow individual table syncs.
- Failed data tests on Fivetran tables will only stop the DAG and not prevent any updates on the tables themselves.