Automation brings incredible value to data teams and hyper-growing companies. It removes hours of sweating over repetitive and mundane tasks, which frees up time for data engineers, analysts, and scientists to get on with more engaging and revenue-generating work.
Now, imagine that you can extend the benefits of automation beyond the selected repetitive tasks… to the entire data pipeline.
Enter Keboola’s scaffolds: a new way of getting data workflows to production in a one-click deployment.
Scaffolds are like recipes for data products. You can think of Keboola Connection as a smart, futuristic kitchen that makes some of the best meals from these recipes.
Imagine walking up to a shiny machine displaying pictures of tasty dishes. Click on the picture of the food you want to eat and voila! There’s a meal ready for you.
In the background, Keboola Connection reads the scaffold recipe. It selects the ingredients it needs from the cupboards then washes them, chops them, and mixes them up. It sautés and slowly simmers them until the meal is ready for your enjoyment. All according to the recipe that you selected.
Need an analysis of digital marketing? A scaffold can quickly set up Keboola to connect to your advertising APIs and consolidate all expected metrics in an easy to understand dashboard. Curious about your subscription business? Use a scaffold to calculate all of the metrics that you require.
Feeling the creative juices flowing? You can also take the scaffold recipe into your own hands and adjust it to your taste.
There are several advantages to implementing scaffolding within your organization:
Scaffolds, like recipes, come in many different varieties, and you can mix and match. Just to name a few:
As you can see, scaffolds can accelerate deployment of any project - from simple data extractions and transformations to advanced modeling, analysis, and machine learning applications. What’s even better, any project or workflow can be turned into a scaffold. A single click deploys the scaffold and a wizard makes sure all the required information (such as data source credentials) is accounted for. A few seconds later a completely new workflow appears in the project - not as a black box, but as open-code transformations and component configurations that can be modified and built upon. The scaffolds can even tag and describe tables and buckets for quick sharing into the Data Catalog, or for other scaffolds to recognize the patterns and assemble themselves into more complex workflows and use cases.
The sky’s the limit when it comes to scaffolds, and Keboola adds new ones regularly to keep your options fresh:
The scaffolds currently featured in Keboola.
Let’s take a look at how you can use scaffolds within Keboola Connection. Once logged in, navigate to Components > Scaffolds.
Start your scaffolding journey from the home screen.
You will see a list of the scaffolds that Keboola already offers. As new scaffolds are being added to the list, you can use the search bar to find the one that you want to use. (Pro tip: start typing the name of the app from which you want to analyze/extract data)
Let’s say that you want to track and analyze customer reviews (did someone say natural language processing with machine learning?). Natural language processing can be quite complex and laborious, but with Keboola it’s just a matter of clicking a few buttons (e.g. “+ USE THIS”).
A drill-down look at the components of the ReviewTrackers Reviews to Snowflake scaffold.
Once we succumb to the appeal of the blue button, we can see that the scaffold is already doing everything for us (✅for the win). All we need to do is connect our ReviewTrackers account.
A green checkmark next to every procedure in the scaffold shows us which procedures are already automated. All we need to do is log into ReviewTrackers.
Press “→ USE SCAFFOLD” to see the magic unfold. In a matter of seconds, the entire pipeline has been run from the end (reviews) to end (analysis and warehousing). You can even inspect the particular procedures that have been deployed at each step - just click on the link on the right-hand side.
Breakdown of the scaffold. Clicking a blue link on the right allows us to explore that particular procedure.
For example, we can see that the scaffold has created 7 tables for us:
If you want to visualize the results, you can now connect your Looker account, or use the already processed data for further machine learning.
Voila! Your meal is served. Bon appétit.
Try scaffolding for yourself with Keboola’s 14-day no-questions-asked trial. Go to try.keboola.com and use the code guide-mode to activate your free trial account.