Keboola Development Branches

What Are Keboola Development Branches and Why Use Them?

In data management and ETL workflows, ensuring continuous operation and data integrity in your production environment is crucial. Keboola Development Branches provide a safe, isolated environment where you can experiment and test changes without impacting your live pipeline. Whether you're adjusting a component configuration, modifying transformations, or optimizing data extraction processes, Development Branches offer a secure workspace to thoroughly verify your changes before implementation.

Key Benefits of Using Development Branches in Keboola

• Risk-Free Testing: Safely test configuration changes without affecting your production data or workflows.
• Complete Isolation: Any changes made in a Development Branch are isolated from the master branch, preventing unintended impacts.
• Collaborative Workflow: Multiple Development Branches can coexist, empowering multiple team members to work on different features simultaneously.
• Controlled Merging: Choose exactly what to merge into production, ensuring precise control over your deployments.

How Do Development Branches Work in Keboola?

When you create a Development Branch in Keboola, it creates a complete, exact replica of your master branch project. This includes all configurations, transformations, and pipeline components. You can freely modify these elements within the branch, knowing that your production environment remains untouched.

Any data created or processed within the Development Branch is written to separate storage buckets designated specifically for development, ensuring complete isolation from production data.

Step-by-Step Guide to Working with Development Branches

1. Creating a Development Branch

To start working safely, create a Development Branch by navigating to your Keboola project dropdown menu and selecting the 'new' icon. Name the branch and add a description to keep your work organized. Once created, Keboola duplicates your entire production environment, clearly marking the UI with indicators that you're currently in a Development Branch.

2. Modifying Pipeline Configurations

Within the Development Branch, you can make various modifications:

• Component Configurations: Adjust data sources, extractors, and loaders without impacting production.
• Transformations: Change data transformations safely—e.g., modifying SQL queries, adjusting limits, or refining data aggregation methods.
• Pipeline Flows: Test how changes affect the overall data pipeline workflow.

For example, if you have a BigQuery transformation fetching the top 50 results, you can safely change the SQL query to fetch the top 10 results within your Development Branch to test its impact without affecting your production data.

3. Running and Testing Changes

Run your modified components or transformations within the Development Branch. Keboola intelligently uses production data as input while ensuring all output data goes into a development-specific bucket. Verify the results by checking the job logs and inspecting the development bucket data—confirming that your test runs do not affect your production buckets.

For instance, running a modified transformation in a Development Branch ensures that your output data is stored separately, confirming that the production environment remains unaffected.

4. Merge Changes Safely Back to Production

Once you've thoroughly tested your changes and are satisfied with the results, you can merge them back into your production environment. Keboola gives you the flexibility to:

• Merge All Changes: Incorporate all modifications made in the Development Branch.
• Partial Merging: Selectively choose specific changes to merge, ideal for granular control.
• Retain or Remove Branches: Decide whether to keep the Development Branch for future use or delete it to keep your workspace clean.

Before merging, Keboola notifies you of any conflicting changes made in other branches or in production, helping you avoid conflicts and unintended modifications.

5. Verify Production Updates

After merging, verify the changes in the production environment. Confirm that transformations, configurations, and pipeline workflows have updated correctly. This final step ensures your deployment is successful and stable.

Comprehensive Example: Optimizing a Pipeline with Development Branches

Let's consider a typical Keboola pipeline named "Academy":

1. An HTTP Data Source downloads a CSV file regularly.
2. A BigQuery transformation selects the top 50 prices from the extracted table.
3. The resulting table is stored in the "top_prices" production bucket.

You decide to optimize the pipeline by adjusting the transformation to select only the top 10 prices instead. Here's how the process unfolds safely with Development Branches:

• Create a Development Branch from the master branch.
• Edit the BigQuery transformation in this branch, changing the SQL query's LIMIT from 50 to 10.
• Run the transformation within the Development Branch, verifying that results are written only to the development bucket.
• Confirm production data remains unchanged by inspecting the original "top_prices" table in the master branch.
• Once satisfied, merge these changes back into production and verify that the transformation in production now correctly selects the top 10 prices.

Best Practices and Recommendations

• Limit Simultaneous Branches: For simplicity and clarity, use one Development Branch at a time, or carefully coordinate multiple branches to avoid conflicts.
• Regularly Review Changes: Always review your changes thoroughly before merging to production to ensure data integrity and pipeline stability.
• End-to-End Testing: Test the entire pipeline within the Development Branch to identify and resolve potential issues early.
• Clear Documentation: Always document your changes with clear commit messages, providing context and clarity for future reference.

Conclusion

Keboola Development Branches empower you to confidently innovate and optimize your data workflows. By providing a secure, isolated environment, Keboola ensures you can continuously improve your pipelines without risking downtime, data loss, or production disruptions. Start leveraging Development Branches today to safely test, iterate, and deploy changes, enhancing your data operations' reliability and efficiency.