Create an orchestration
Overview
Now that we have built our robust data pipeline and ensured the accuracy of our models, it's time to take advantage of Y42's schedules. By setting up regular runs, we ensure that our data pipeline stays up-to-date with fresh information.
Instructions
Set up an orchestration
-
Open the bottom drawer
-
Navigate to
Build scheduler
-
Click on
Add scheduled DAG
-
Name the schedule:
orders_schedule
-
Enter the following selector statement:
y42 build -s +orders
-
Click on
Visualize DAG
and verify that you see the following DAG:Scheduler DAG.
-
Set up the Schedule frequency unit:
Minute(s)
-
Set Options to
Run every every 15 minutes
-
Click on
Save
Commit your changes
- Click on the
Commit & Push
button at the top - Add a commit name:
adding orders schedule
- Click on Commit
- Wait for the pre-configured checks
Trigger a manual run
- You should now see your schedule as row in the
Build scheduler
tab including the command and the schedule as well as metadata about the last runs - Click on
Trigger Run
to trigger an ad-hoc out of schedule run - You will see a blue bar appearing in the
Last Runs
column
Observe the build
- Navigate back to the
Build
tab - You should see a similar job to the one that you created before except that the DAG Name is
orders_schedule
- Every 15 minutes the same build command will be executed
View the Schedule as code
-
Change to the
Code editor
mode -
Click on the
orchestrations
folder -
Click on
orders_schedule.yml
-
You should see Y42’s YML structure for orchestrations:
_10version: 2_10_10orchestrations:_10- name: orders_schedule_10config:_10enabled: true_10command: y42 build -s +orders_10cron: 0 9 1/1 * *
Congrats! 🎉
Great job on reaching this point! You've just built your first end-to-end data pipeline on Y42! With everything you have learned you are now able to setup your very own data pipeline.
If you would like to continue, checkout our extended tutorial to learn how to:
- Use branching on Y42
- Implement data tests
- and more