In data management and ETL workflows, ensuring continuous operation and data integrity in your production environment is crucial. Keboola Development Branches provide a safe, isolated environment where you can experiment and test changes without impacting your live pipeline. Whether you're adjusting a component configuration, modifying transformations, or optimizing data extraction processes, Development Branches offer a secure workspace to thoroughly verify your changes before implementation.
When you create a Development Branch in Keboola, it creates a complete, exact replica of your master branch project. This includes all configurations, transformations, and pipeline components. You can freely modify these elements within the branch, knowing that your production environment remains untouched.
Any data created or processed within the Development Branch is written to separate storage buckets designated specifically for development, ensuring complete isolation from production data.
To start working safely, create a Development Branch by navigating to your Keboola project dropdown menu and selecting the 'new' icon. Name the branch and add a description to keep your work organized. Once created, Keboola duplicates your entire production environment, clearly marking the UI with indicators that you're currently in a Development Branch.
Within the Development Branch, you can make various modifications:
For example, if you have a BigQuery transformation fetching the top 50 results, you can safely change the SQL query to fetch the top 10 results within your Development Branch to test its impact without affecting your production data.
Run your modified components or transformations within the Development Branch. Keboola intelligently uses production data as input while ensuring all output data goes into a development-specific bucket. Verify the results by checking the job logs and inspecting the development bucket data—confirming that your test runs do not affect your production buckets.
For instance, running a modified transformation in a Development Branch ensures that your output data is stored separately, confirming that the production environment remains unaffected.
Once you've thoroughly tested your changes and are satisfied with the results, you can merge them back into your production environment. Keboola gives you the flexibility to:
Before merging, Keboola notifies you of any conflicting changes made in other branches or in production, helping you avoid conflicts and unintended modifications.
After merging, verify the changes in the production environment. Confirm that transformations, configurations, and pipeline workflows have updated correctly. This final step ensures your deployment is successful and stable.
Let's consider a typical Keboola pipeline named "Academy":
You decide to optimize the pipeline by adjusting the transformation to select only the top 10 prices instead. Here's how the process unfolds safely with Development Branches:
Keboola Development Branches empower you to confidently innovate and optimize your data workflows. By providing a secure, isolated environment, Keboola ensures you can continuously improve your pipelines without risking downtime, data loss, or production disruptions. Start leveraging Development Branches today to safely test, iterate, and deploy changes, enhancing your data operations' reliability and efficiency.