Sync your data pipelines with third-party systems using Python.
Python Actions allow you to sync your data pipelines with workflows in Asana, JIRA, Make or any tool you prefer. This feature supports a wide range of use cases, such as Reverse ETL data into Salesforce, task creation in Asana, notifications in Slack, data generation in Google Sheets, or data entry into operational databases.
Python Actions provide a Zapier or Make-like functionality, with the added benefits of having all your processing in one place, within one data lineage, and having integrated documentation, within your team's existing workflow.
Press CMD + K or click the
+ button to access the create new asset menu. Find Python actions under the Consume tab within this menu.
To add a Python action asset, you need to configure the following:
- Script: The Python code that triggers the action in a third-party system.
- Dependencies: Define the upstream dependencies.
Within your python code, you can use the
@data_loader decorator to trigger actions in third party systems.
Begin by importing the
data_action, along with any other necessary modules to sync data externally.
Create a function that transforms the upstream data and sends it to a third party system.
If needed, custom secrets can be referenced in Python scripts using the
To access data from sources or models, use the
Dependencies define the upstream dependencies of your Python, helping Y42 understand how the Python asset connects with other assets in the space. It also determines when to trigger the asset and updates the lineage view with the new relationships.
Save your changes by committing them. You can build the asset using dag selectors or via the
Build history tab.
+ selector, when used in front of the asset name in the command, also triggers all upstream dependencies.
- Multiple functions in one script: Multiple functions can be included in a single script with the
@data_loaderdecorator. Each decorated function will be triggered.
- Non-decorated functions: Functions without the
@data_loaderdecorator won't trigger independently. However, they can be invoked within a decorated function.
- Dependencies management: For scripts with multiple decorated functions, the combined set (union) of dependencies from all functions is displayed.
You can reference custom secrets in Python scripts using the following syntax:
_10@data_action_10def send_data(context, assets):_10 # Reference secrets if needed_10 all_secrets = context.secrets.all() # get all secrets saved within this space_10 one_secret = context.secrets.get('<SECRET_NAME>') # get the value of a specific secret saved within this space
Use the logging module (opens in a new tab) to log messages.
You can view the logs by accessing an asset's Build history and selecting the Logs tab.
To enhance the readability of logs, especially when dealing with dataframes, we recommend utilizing the dataframe to HTML function (opens in a new tab). This function converts your dataframe into an HTML table, making your logs more structured and easier to read.
While Python actions primarily trigger external actions, you can preview the data by optionally making the decorated function return a DataFrame.
Preview the data and view logs for each function decorated with
@data_action by clicking on the "Preview" option located above the function name.
Referencing sources and models in your Python script adds them to the Dependencies tab, creating a visible dependency in Lineage mode.
In the lineage view, a link is automatically established between the Python source table and the dbt model.
To install custom modules, list them in a
requirements.txt file. Begin by switching to Code mode, then create or edit the
requirements.txt located in the
requirements.txt file content: