Automate with Keboola Flows

What Are Keboola Flows and Why You Need Them

Keboola Flows are powerful automation tools designed to streamline and simplify your data management processes. They allow you to integrate various data components, such as extractors, transformations, and writers, into automated workflows. By scheduling these workflows to run regularly, you eliminate manual tasks, minimize human errors, and accelerate data-driven decision-making across your organization.

Flows are particularly useful for:

• Automating regular data extraction from multiple sources.
• Ensuring timely transformation and cleansing of raw data.
• Automatically loading data into databases or BI tools for immediate analysis.
• Reducing operational overhead and manual maintenance.
• Enhancing data reliability and consistency.

Creating Your First Flow: Step-by-Step Guide

Let's walk through creating an automated workflow in Keboola. We'll call our example flow "Daily Run." Here's how easy it is to set up:

1. Create a new Flow: Navigate to the Flows section within Keboola’s intuitive user interface and start by creating a new flow named "Daily Run."
2. Chain your components: Begin by selecting the necessary components from your project. Typically, you'll start with an extractor (for example, the Snowflake extractor), followed by a transformation component, and finally a data writer.
3. Configure sequential execution: Tasks are executed sequentially based on the flow you've defined. Tasks within the same step, however, run in parallel. This flexibility allows you to optimize execution times, especially when dealing with multiple data sources or transformations.
4. Save and schedule: After configuring your components, save your settings and set your desired execution schedule.

Understanding Flow Components

Flows within Keboola consist of several core components:

• Extractors: These components gather raw data from various sources. Common examples include databases like Snowflake, PostgreSQL, MySQL or external sources such as Google Drive.
• Transformations: These allow you to modify, enrich, and clean your data. Transformations can be based on SQL or other data transformation languages supported by Keboola.
• Writers: After transformation, writers load the data into target databases, analytics tools, or cloud storage, ensuring your data reaches the desired destination.

Scheduling Flows Effectively

Scheduling is crucial for automating your data processes. Keboola provides flexible scheduling options:

• Preset schedules: Choose from popular presets such as every 15 minutes, hourly, daily, or weekly.
• Custom schedules: Define your precise timing. For example, you could run flows at specific intervals throughout the day, such as midnight, 6:00 AM, 12:00 PM, and 6:00 PM.
• Time zone considerations: Keboola defaults to UTC, but we highly recommend adjusting scheduling to your local time zone. This is particularly important when daylight saving changes occur, ensuring reports and data refreshes align with your organizational needs.

Notifications and Monitoring

Once your flow is set and running, there’s no need for constant manual monitoring. Keboola’s notification system ensures you stay informed about your data pipeline status:

• Success notifications: Get alerted when your flow completes successfully.
• Warning notifications: Receive notifications if tasks complete with warnings, enabling proactive troubleshooting.
• Error notifications: Immediate alerts for failures and errors help you quickly diagnose and resolve issues.
• Prolonged execution notifications: Alerts when a job takes longer than expected, helping you identify bottlenecks or performance issues.

You can easily set these notifications to be delivered via email to any user involved in the project or even external stakeholders.

Advanced Error Handling: The "Continue on Failure" Feature

A common challenge in data workflows is handling occasional failures without disrupting the entire process. Keboola provides a powerful solution with the "continue on failure" feature:

• Avoid workflow disruption: If one extractor or task within a step encounters an error, the flow can continue running other tasks without interruption.
• Proactive troubleshooting: By setting up notifications for warnings, you'll be informed immediately about any issues.
• Enhanced reliability: This feature is especially useful when dealing with unstable APIs or multiple data sources, where occasional errors might occur.

Real-Time Monitoring and Manual Execution

Keboola provides complete transparency into your data pipeline:

• Real-time status updates: Track the execution of individual components and the overall flow in real-time from the "Jobs" section.
• Manual flow execution: Run your flow manually anytime, providing flexibility for ad-hoc data needs.
• Detailed logs and insights: Access comprehensive logs to quickly identify and resolve issues.

Real-World Example

Consider a marketing analytics use-case:

1. Extraction: You regularly pull customer interaction data from Snowflake, sales data from Google Drive, and campaign data from Facebook Ads.
2. Transformation: Keboola transforms raw data into actionable insights by aggregating spend, clicks, and conversions.
3. Load: The resulting insights are loaded into your BI tool, ready for your marketing team to analyze every morning.
4. Automation: Keboola Flows schedules this entire process daily at 5 AM, ensuring fresh data is always available for your team's morning meeting.

Conclusion: Keboola Flows empower your team to automate, optimize, and scale data processes effortlessly. With robust scheduling, notifications, and error-handling capabilities, your data pipeline becomes reliable, efficient, and easy to manage.