Snowflake Data Apps
Data Eng Weekly - Ep 25
After the Snowflake summit last week, Snowflake announced the release of their "Native App Framework," which is equivalent to an app store for data warehouses.
Before diving into some examples, let's provide some context:
Snowflake's vision is to become the app store for analytics, offering a secure platform similar to iOS for smartphones.
Users have access to a marketplace of apps that can be installed securely with just one click.
They ensure that the apps available in the marketplace are safe and that data remains within the user's data warehouse.
This addresses the challenge of selling data applications to corporate clients, as the underlying data remains within the customer's environment.
There is no need to deploy the app in the client's IT infrastructure, eliminating a significant hurdle in the sales process.
In my previous company, we had to package and send an application to a developer on the client side, the process was quite painful.
There were various challenges involved, such as ensuring compatibility with the client's IT infrastructure, handling deployment and installation complexities, and addressing any security concerns.
This often resulted in a time-consuming and cumbersome process.
With the release of their "native app framework," Snowflake claims to solve that problem.
Let’s try an app
One of the available apps that caught my attention is a free app developed by sundeck.io.
This app enables users to monitor their credit consumption in the Snowflake warehouse more effectively.
What makes it even more interesting is that the source code for this app is open source and can be found on GitHub.
Onboarding for the app from the Snowflake marketplace was a straightforward process.
I simply opened the app in the marketplace, agreed to the terms and conditions, and within 10 minutes, I received confirmation by mail from Snowflake that the app was ready to use.
Upon onboarding the OpsCenter app from the Snowflake marketplace, a new database was created within my Snowflake account.
I could easily access it through the Apps section in Snowsight.
From there, I could open the OpsCenter app, and it launched a Streamlit webpage.
The onboarding experience for deploying apps in the Snowflake marketplace was good. The process was straightforward, requiring just a few clicks to install the app in the warehouse. This eliminated the need for app hosting hurdles and compliance concerns.
How to build an app ?
I won't go through the tutorial provided by Snowflake, but instead, I will answer some key questions that have been on my mind.
1- How to install an app in dev mode?
Before creating an application your need to create an Application package which is simply a Snowflake database with some extended functionalities:
At its core, an application package is a Snowflake database that is extended to include additional information about an application. In that sense, it is a container for an application that includes:
Shared data content
Application files
You can then simply upload your files to an external stage (S3 bucket) and install the app using the following query:
CREATE APPLICATION HELLO_SNOWFLAKE_APP
FROM APPLICATION PACKAGE HELLO_SNOWFLAKE_PACKAGE
USING '@hello_snowflake_package.stage_content.hello_snowflake_stage';
How to add Python code to the app ?
Python code is injected into the app through User Defined functions (UDF):
These UDFs are created by a setup SQL script run when the app is installed.
How to use Python third-party libraries?
Snowflake provides a predefined list of third-party packages that you can use, which can be found at the following link.
However, this list can be quite limiting, as it may not include all the packages you may need. In such cases, the alternative is to import the package code yourself using an external stage.
How to add a Streamlit application?
Streamlit source code has to be placed in a separate folder. The app itself is created by the SQL initialization script with the following statement:
From within Streamlit, you will be able to query any database as long as you grant the required permissions to the app.
Limitations
The app framework still has some limitations to be aware of:
Not all Streamlit features are currently available, including custom components, media elements, internet access, file uploader, and pyplot figures (??).
It is currently only available on AWS.
Tables created using AUTOINCREMENT are not supported.
There may be some other minor limitations that you may come across during your usage.
Even with its limitations, the Snowflake app framework represents an exciting direction for Snowflake.
It will be interesting to see the types of apps that will be deployed to the marketplace and how corporate clients will embrace this new way of consuming external software.
Thanks for reading,
-Ju
I would be grateful if you could help me to improve this newsletter. Don’t hesitate to share with me what you liked/disliked and the topic you would like to be tackled.
P.S. you can reply to this email; it will get to me.