Building data pipelines has become even easier with Keboola’s Flow Builder

How To

May 24, 2022

Flow Builder - the collaborative new feature that speeds up data pipeline deployment

Keboola is proud to announce the launch of its new feature: the Flow builder.

What is Flow Builder?

The Flow Builder is a new feature that allows you to visually build end-to-end data pipelines from a single drag-and-drop user interface.

Building data pipelines can be extremely complicated - from juggling the challenges of multiple data sources and formats to requiring advanced programming knowledge for transformations and understanding the intricacies of how to change your loading process to tailor it to every data destination’s specific conditions.

But Keboola already simplified each step of the process with its reusable ready-made components. Simply select the right component to either extract, transform, or load your data.

Now Keboola takes it a step further by joining all the components into a single Flow Builder. You can create data flows by dragging and dropping the components together on a single screen, and even schedule and automate your data pipelines with a button click.

This simplicity brings you five main benefits.

The 5 Advantages of Visual Flow Builder

The Flow Builder offers five benefits to Keboola’s users:

Holistic overview. View your multi-step data pipeline in a single browser window. This is especially advantageous when your data pipeline is complex because it gives you a simplistic big picture view of everything that is happening.

User-friendliness. The Flow Builder is built as a drag-and-drop tool. This allows non-technical people to jump on board and start building their pipelines without any need for data engineering skills.

Collaboration. Teams can view and work on the same data flows together. It also allows users to set up the data pipeline as a team and then dig deeper into various components individually. With its in-built version control, your entire team can see how the collaborative efforts changed the data pipelines through time.

Multi-flow. The Visual Flow Builder allows you to build multiple data flows in a single view. Each flow is its own data pipeline.

Reusability. Simply copy-and-paste an existing data flow to a new flow to reuse the work done across different data pipelines.

Let’s now build a new data flow to see how to unlock the benefits in practice.

How to set up your data pipelines with Flow Builder - A step-by-step walk-through

We will build an end-to-end data flow using Flow Builder in 3 steps.

Step 1: Access the new Flow Builder feature

Log into your Keboola account to get started.

Don’t have a Keboola account yet? No worries! Keboola offers an always-free (no credit card required) tier that allows you to follow along. Create an account here.

Once in Keboola, select from the top menu Flows > Flows > New Flow:

Give your new flow a name and an easy to understand description. Then click create flow.

This will open up the Flow Builder view, where you can view and build your data flow.

Step 2: Select one or multiple ready-made components

Start by selecting the components to collect your data from your data sources. The general process is simple:

Select the right component to extract/transform/load data from/to your specified data source.
Authorize Keboola to access data in that component.
Extract/transform/load data.

We will show you an end-to-end example of setting up ready-made components by walking you through reading data files from Google Drive.

We start by clicking on Select Components > Google Drive to collect a file from our Google Drive folder.

We proceed by clicking on the Google Drive component> show details cog > Select or Create Configuration to add new configuration. The configuration grants access to Keboola so it can read the data files on your Google Drive:

Name the configuration (e.g., “Google Drive Config”) and click on Create (shown below) and the Configure (not shown):

The wizard guides you through Google’s authorization to give Keboola access to our Google Drive (not shown here, but super easy to follow).

Step 3: Build out the entire flow

Continue adding other ready-made components and organize them in a logical flow or select the ones you want to run in parallel using drag & drop function.

For each flow:

Select one or multiple ready-made components.
Organize them into a logical flow using the drag and drop functionality.
Configure each component by providing the credentials and instructions for what or where to extract or write or for what code to execute in a transformation.
Save the changes

Once you’ve built your flow end-to-end simply click on Run flow, to set the data pipeline into action.

Step 4: Schedule and automate your flow

Within the Flow Builder, click on Set schedule under Automate to set up the orchestration that will automatically run your job at your preferred recurring time.