Analysts and data scientists use SQL queries to pull data from the data storage underbelly of an enterprise. They mold the data, reshape it, and analyze it, so it can offer revenue-generating business insights to the company.
But analytics is only as good as the material it works with.
That is, if the underlying data is missing, compromised, incomplete, or wrong, so will the data analysis and inferences derived from it.
To understand the end result - data - we need to understand the ETL process that brings the data to life. Or at least to the analysts’ hands.
An ETL pipeline is a software architecture design that describes a specific type of data pipeline.
The ETL pipeline flows data from source systems to target systems. In between, data is filtered, cleansed, aggregated, enriched, and transformed to be ready for BI tools and analytics.
The ETL pipeline can be broken down into three processes: extract, transform, and load.
The extract phase extracts or collects data from raw data sources, such as APIs, sensors, marketing tools, and other sources of data.
The raw data can come in many structured and unstructured formats, such as JSON, XML, Excel, or CSV files, as well as entries from transaction databases.
Once data is extracted, it is moved into the transform phase.
The transform stage cleanses, enriches, and alters data into a format and shape that is ready for consumption by users and (business intelligence) applications.
Data transformation includes:
The output destination of the transformation stage is the staging database, where the cleansed, filtered, aggregated, and enriched data is pushed, before being loaded in the target destination.
Load is the third and final stage of the ETL pipeline. Data is loaded into the data storage of your choice. Be that a database (MySQL, PostgreSQL, etc.), data mart, data warehouse (Amazon Redshift, Snowflake, and Google BigQuery), or data lake (Amazon S3, Apache Hadoop, Microsoft Azure Data Lake Storage (ADLS), …).
We treat loading data separately, because the implementation details (the data engineering) of the loading phase depends heavily on the target data storage - from schema changes to typing coercion, a lot of engineering efforts go into loading data properly.
The ETL process might seem straightforward, but it hides many pesky challenges:
ETL pipelines are usually built using devoted ETL tools. ETL tools are preferred over custom scripting because of many advantages:
But tooling does not solve all of the problems.
The main engineering tradeoff you need to decide upon is whether your ETL process needs to support real-time data or can be updated in batches.
Real-time data needs specialized technology (like Apache Kafka streams) or tooling (ETL tools supporting data streaming), while batch processing can extract, transform, and load data in batches at pre-defined (scheduled) time intervals.
Real-time data offers you immediate business insights, but it also comes at the cost of increased engineering overhead and more challenging tooling or solutions.
Ultimately, the decision will depend on your business model - whether you will need real-time decision-making or not.
Keboola is a data integration platform that automates data operations end-to-end.
Use it to build and deploy your ETL pipelines:
Try it out. Keboola offers a no-questions-asked, always-free tier, so you can play around and build your pipelines with a couple of clicks.