Development modes

Explore the three development modes in the integrated data development environment.

Overview

The Y42 integrated data development environment provides a comprehensive suite of tools to supercharge the data pipeline building workflow. It offers three distinct modes to help you get the most out of your projects: the Data Catalog, Code Editor, and Lineage.

Data catalog

Data Catalog mode is a UI-based approach that allows you to rapidly create, read and modify assets. It translates the code and configuration files spread across multiple directories in your project, and renders them as consolidated assets replete with metadata, column-level data and build history - and vice versa! You can also add new assets using the Data Catalog, which will automatically create the corresponding YAML and SQL files.

Data Catalog mode

Code Editor

Code Editor mode provides a familiar developer experience through an enhanced version of Code OSS, the open-source distribution of Visual Studio Code. Powered by WebAssembly, the Code Editor is tailor-made to handle the demands of managing complex data pipelines. It offers productivity essentials such as built-in lineage preview, data previews, and other quality of life improvements such as code auto-suggestions and static code analysis.

Code Editor mode

Lineage

Lineage mode shows relationships between all assets as well as their properties, making it a breeze to understand the flow of data in complex data pipeline architectures. In Lineage mode, you can zoom in from a birds-eye view of your DAG and navigate directly to a detailed view of any asset in your space.

Lineage mode

Automatic sync between modes

All development modes are synced with the state of your space, including staged changes, in real-time. As such, you are free to choose the tool for the job, to leverage the unique workflow benefits that each mode offers. For example, you could define a model's metadata using Data Catalog mode, hop into the Code Editor to the transformation logic in SQL, then trace its dependencies in Lineage mode to understand how the model fits into the overall flow of data.

Synced state between the Data Catalog (left) vs Code Editor (right)

For example, when you create a new model within the Data catalog, the system automatically generates corresponding .sql and .yml files in the Code editor. Similarly, any modifications made to the model definition within the Data catalog will be automatically synchronized and reflected in the Code editor's .sql or .yml files.

In the reverse scenario, if you opt to use the Code editor mode and apply changes to your model's .sql or .yml files, these changes will become visible within the Data Catalog upon your next visit. This automatic update also includes the addition of new .sql files via the Code editor, resulting in their corresponding models appearing in the Data catalog.

Additionally, you can switch development modes and continue right where you left off, at any time. As each mode is synced with your code and configuration files, they always represent the same underlying state. Ultimately, the state of the files committed to the space's Git repository is the single source of truth - the answer to your asset lifecycles, the universe of modeldependencies and everything about your data pipelines.