Recently I watched a webinar organised by Snowflake, it was called “Data Warehouse Automation, Ingestion and Industry Leading Analytics with Snowflake and Qlik“. In the past I could never find time for webinars, and it’s amazing how life can give you opportunities to try something new. It was nice to see a live demonstration of how the enterprise can build a cloud-native BI solution, from data ingestion to analysis, which is also partially automated. (You can watch it on demand as well.)

Snowflake is a cloud data platform, and Qlik is one of the leading analytics and dashboarding platform. But if it feels like something is missing from the end-to-end data warehouse, it’s all good, because the trick of course is that Qlik has recently acquired Attunity, therefore they have enabled the bundle of data integration products to treat your data all the way from source systems to shiny dashboards.

Two products that came with Attunity are going to cover the need in CDC and ETL tools. They are Qlik Replicate and Qlik Compose (both were formerly known under Attunity brand). OptimalBI has been both an Attunity and Qlik partner for many years, therefore there is nothing new for our team. You can find that we have blogged a lot about both of the products under the previous name. Surely, the biggest benefit for customers is the ability to stay with just one vendor for a wide range of data solutions. Usually it guarantees that all the products will seamlessly work together, but I believe in this case they did it well before the rebranding already. But enterprises will definitely appreciate the benefit of contacting only one support one sales team for all data-related questions.

Let’s dive into details, how does the end-to-end BI solution looks like with all the Qlik products and the Snowflake? Here is a perfect picture from the Qlik website explaining where their new Data Integration platform is positioned (in turquoise).

Qlik Replicate can access almost any modern or archaic data source you can imagine to enable CDC (change data capture) streaming, which means that any change to the source data will be captured in real-time. Replicate doesn’t allow much data transformation, but that is the next step.

Qlik Compose is a data warehouse automation tool. Here is a disappointing part: data warehouse automation doesn’t mean a magic button that is going to create a data warehouse for you, but this is a tool that can save a lot of data engineer’s time by helping with ETL scripts generation and visual source to target fields mapping. Compose allow easy data loads orchestration, and it works perfectly with the CDC results of the Qlik Replicate. Compose can help with generation of both Data Vault and Star schema structures and load scripts. Unfortunately, the transformation toolset is not powerful enough, so some complex logic has to be manually scripted in SQL.

All of these copies of data should be stored somewhere, and that’s where the Snowflake could really help, the violet rectangle on the picture above. Snowflake allow saving costs on both storage and compute. I have already reviewed Snowflake features in one of my previous blogs.

Two purple squares on the right side of the picture are Qlik Data Catalyst, another recent Qlik acquisition, and the best known Qlik product Qlik Sense or, as generously mentioned, actually any visualisation or analytics tool. Any of them can easily connect to Snowflake.

I would agree with the webinar presenters that this stack of world-renowned tools is probably the best end-to-end BI solution of what you can get these days.

Kate
Data masseuse