Customer Stories
September 27, 2023
How Apify saved 3.5 engineering days per month while DOUBLING upsells
From “scraping by” to top-notch analytics with Keboola
Apify is a platform that helps you build, deploy, and monitor web scrapers and browser automation tools.
Guided by the principle “Whatever you can do in a browser, Apify can automate for you”, the platform helps customers by automating a range of tasks, such as monitoring competitors' product prices, scraping texts, and even developing advanced AI language models.
To gain a deeper insight into Apify's data-driven journey and how they leveraged Keboola to overcome challenges, we spoke to Matej Šesták, the company’s Data Team Lead. Currently heading a team of four data specialists, Matej's journey with Apify began three years ago, when he was single-handedly tackling data challenges.
Challenges Faced By Apify Before Keboola
Apify used to track a variety of metrics but lacked an efficient system for handling analytical processes.
Without a specific analytical stack in place, the company’s data operations relied heavily on its software infrastructure. Matej would write AWS Lambda functions to transform raw data in the production Mongo database, then pipe it into Redash to visualize metrics.
This setup brought several issues:
- Inaccessible data: Querying some data would break production jobs, so it was off limits to analytics.
- Production delays: As MongoDB was used for both production and analytics, running high-load queries slowed down production and impacted Apify's user experience.
- Difficult to onboard new people: While MongoDB is great for software developers, its MongoDB Query Language (MQL) and internal logic made it difficult to onboard new analytics team members without an in-depth knowledge of the backend architecture.
- Expensive to add new data sources: If Matej wanted to analyze new raw data or enrich datasets before driving insights, he had to spend hours developing custom data extractors to load data into MongoDB.
- Scaling challenges: The above challenges made it hard to expand analytics and develop new data use cases due to the constraints of their current system.
Matej recognized the need for streamlined data and reporting systems that met industry standards. He opted for Snowflake as a dedicated analytical database and, to further support the company's growth, he chose Keboola to manage the ETL processes and orchestrate Apify’s extensive data tasks.
Why Choose Keboola?
“The thing that impressed us the most is how many things you can get done with Keboola. You can do any type of data engineering with this one tool. From extracting data, transforming it, to orchestrating jobs, everything is super simple and can be done in a single UI.”
When looking for the right data platform to streamline Apify’s data engineering jobs, Matej evaluated many candidates, including Airflow and Fivetran. But in the end, he chose Keboola. Here’s why:
- Better pricing: Keboola was much more financially accessible than other tools.
- Strong recommendations: After speaking to numerous consulting companies, many of them vouched for Keboola.
- Existing user base: A vast pool of people have existing knowledge of Keboola, making future hiring a lot less challenging.
- Comprehensive solution: Keboola matched all of Apify’s data engineering needs for autonomous data extraction and orchestration of transformation jobs.
- User-friendly: The platform’s intuitive design simplifies the onboarding process for new team members.
- Efficiency: Keboola's plug-and-play system, complemented by a range of pre-configured solutions, enabled Apify's data team to hit the ground running without depending solely on software engineers.
“Keboola is a time saver. Especially for smaller data teams. You don’t waste time figuring out how different tools work together. Keboola puts them all in one place for you.”
So, what has the Apify data team achieved with Keboola?
The Results After Using Keboola
Keboola was instrumental in automating financial reporting, determining profits on a per-customer basis, and using data to drive better sales processes.
Let’s take a look at the use cases in more detail.
Use case #1: Automate financial reporting to save 3.5 engineering days every month
Apify uses financial reporting to monitor internal processes (sales team), control (finance team and external accounting firm), and liaise with their investors.
The challenges before Keboola
Apify had many challenges surrounding the computation of revenue per customer:
- Different subscription contracts (e.g. monthly, quarterly, yearly, or custom time frame) had to be translated into a consistent monthly reporting format.
- The "pay-as-you-go" model had complexities such as applying discounts, managing canceled plans, and other unique contract stipulations.
- Apify’s system relied on a 1000+ line script in their backend to join and analyze all these nuances into one financial report.
The current solution not only made onboarding difficult, but it also posed challenges when discrepancies arose between internal and external accounting analyses.
How did Keboola help?
Matej's team turned to Keboola to refine their financial reporting. They did this by:
- Automating data collection from different data sources (including internal invoicing data and external accounting reports for comparison).
- Building transformations as independent units of business logic.
- Orchestrating all separate jobs into a single data pipeline that produces a financial report and automatically compares it with external accounting data.
“Without Keboola, we wouldn't be able to put all the different data sources together, compare them, and have the data and metrics always available for everyone at the company all in one place.”
The benefits after Keboola
Using Keboola unlocked many advantages for Apify:
- Transparent financial reporting: Keboola's modular architecture makes financial reporting transparent. It allows the Apify team to quickly inspect and verify the accuracy of transformations.
- Increased trust: With greater transparency and the ability to compare with external accounting reports, Apify’s team members can fully trust their data and perform many financial controlling operations in-house.
- Improved metrics: Matej and his team added additional metrics to help the company track its financial success - MRR by customer segments and its direction (growing, falling, stable) is just one of them. Improving the metrics was easy thanks to Keboola’s modular design.
- Time savings. On average, Matej and his team save 3.5 days of engineering time every month by avoiding manual code inspections and automating their pipelines with Keboola.
- Increased data quality. Automation isn’t just about saving time. Keboola’s orchestrations also allow the team to automatically generate reports and avoid the human errors that were present in the past.
“The report became easier to understand. It wasn’t one huge script hidden somewhere in the code base. You could open the data pipeline, inspect the components, and quickly understand what it does.”
Use case #2: Understand gross margin to drive better profits while lowering costs for customers
The challenges before Keboola
With a monthly profit report already set up, Apify always had visibility of blended gross margin across the company.
But understanding which customers and products drive profits (and which ones eat into the bottom line) was trickier:
- Costs per customer or product had to be computed from AWS usage costs. Translating between the different consumption prices of AWS services per region and their pricing matrix was challenging because the prices vary dynamically.
- Additionally, it was difficult to automatically incorporate the engineering costs of developing bespoke solutions per customer.
How did Keboola help?
Keboola helped the data team at Apify to:
- Quickly and automatically collect billing data from AWS with Keboola’s AWS Cost and Usage reports Extractor.
- Keep the data fresh and valid with its orchestration engine.
- Analyze data per customer and product using transformations.
- Make the data available to the sales team by using the Salesforce Writer.
The benefits after Keboola
The main advantages appeared at the intersection of sales enablement and product delivery. The Apify team can now:
- Identify the most and least profitable products and customers. This wasn’t possible before Keboola.
- Understand internal pricing flexibility. The sales team is empowered with the upper and lower limits of the pricing they can offer, enabling them to better tailor their packages to customers.
- Spot worrying trends. When a customer’s costs start to drastically increase, the Apify team can intervene and help the customer avoid unwanted costs.
- Build more price-performant products. When the development team identified the highest costs, they were able to develop new products that were cheaper for both Apify and the customer - a win-win. For example, the system can automatically suggest a different region for a data loading bucket to lower transfer costs.
Use case #3: Use reverse ETL to improve sales processes and double upsell sales
The challenges before Keboola
Before they chose Keboola, the communication between Apify employees and customers took place across many channels and tools:
- Sales used Salesforce for their CRM and emailing.
- Customer Support used Intercom for communicating with customers in need of help.
- Event analytics was captured using Mixpannel.
For each data source, a backend engineer had to manually integrate the data from that source’s API into the production MongoDB database. This made cross-departmental analytics slow and heavily reliant on the software development team.
How did Keboola help?
As with other use cases, Keboola was used as a single source of truth for all customer-centered analyses:
- Automated data collection with Keboola’s out-of-the-box extractors and loaders.
- Keboola’s transformations join, de-duplicate, and integrate data into a cohesive table.
- The data is served back to business users with Keboola’s reverse ETL features.
The benefits after Keboola
Using Keboola to collect, clean, align, and serve data helped the Apify team to:
- Speed up new data source integrations. Keboola’s extractors and writers are pre-built, which is a huge time-saver. This also allows the software engineering team to spend more time on building superb products (instead of writing API integrations for the analytics team).
- Improved marketing and sales initiatives. By building custom metrics and integrating them into Hubspot and Salesforce, the growth teams at Apify can do a better job. For example, the sales team can view the customers whose usage is growing and quickly upsell them to a higher plan. Alternatively, marketing can identify the segment of customers whose usage is falling and send them a promotional campaign before they churn.
- Align customer communication across departments. After syncing all communication tools (Salesforce, Intercom, etc.), Apify stopped over-emailing customers and burning bridges with unaligned communication strategies.
The result?
“With the improved upsells and targeting, we were able to increase the conversion of customers to higher-paid plans by 2x.”
What’s next for Apify?
Matej and his team plan to use Keboola to:
- Launch product analytics to better understand user activations and conversions with product changes, and how to improve them.
- Improve data robustness with new features like native data types.
- Onboard new teams onto Keboola - especially their engineering teams - to empower more people at Apify to self-serve their data needs.
As Matej said:
“Keboola is a simple data platform that solves all your data needs right out of the box without much configuration.”
Curious about the benefits that your company could unlock with Keboola? Try Keboola for free (no credit card required).
Did you enjoy this content?
Have our newsletter delivered to your inbox.
