11. Create an orchestration

Create an orchestration


Now that we have built our robust data pipeline and ensured the accuracy of our models, it's time to take advantage of Y42's schedules. By setting up regular runs, we ensure that our data pipeline stays up-to-date with fresh information.


Set up an orchestration

  1. Open the bottom drawer

  2. Navigate to Build scheduler

  3. Click on Add scheduled DAG

  4. Name the schedule: orders_schedule

  5. Enter the following selector statement: y42 build -s +orders

  6. Click on Visualize DAG and verify that you see the following DAG:

    Scheduler DAG.Scheduler DAG.
  7. Set up the Schedule frequency unit: Minute(s)

  8. Set Options to Run every every 15 minutes

  9. Click on Save

Commit your changes

  1. Click on the Commit & Push button at the top
  2. Add a commit name: adding orders schedule
  3. Click on Commit
  4. Wait for the pre-configured checks

Trigger a manual run

  1. You should now see your schedule as row in the Build scheduler tab including the command and the schedule as well as metadata about the last runs
  2. Click on Trigger Run to trigger an ad-hoc out of schedule run
  3. You will see a blue bar appearing in the Last Runs column

Observe the build

  1. Navigate back to the Build tab
  2. You should see a similar job to the one that you created before except that the DAG Name is orders_schedule
  3. Every 15 minutes the same build command will be executed

View the Schedule as code

  1. Change to the Code editor mode

  2. Click on the orchestrations folder

  3. Click on orders_schedule.yml

  4. You should see Y42’s YML structure for orchestrations:

    version: 2
    - name: orders_schedule
    enabled: true
    command: y42 build -s +orders
    cron: 0 9 1/1 * *

Congrats! 🎉

Great job on reaching this point! You've just built your first end-to-end data pipeline on Y42! With everything you have learned you are now able to setup your very own data pipeline.

If you would like to continue, checkout our extended tutorial to learn how to:

  • Use branching on Y42
  • Implement data tests
  • and more