Snowflake’s Stream

Stream

Stream is a CHANGE DATA CAPTURE methodology in Snowflake; it records the DML changes made to tables, including (Insert/Update/delete). When a stream is created for a table, it will create a pair of hidden columns to track the metadata.

create or replace stream s_emp on table emp append_only=false;

I have two tables, emp and emp_hist. Emp is my source table, and emp_hist will be my target.

Now, I will insert a new row in my source table to capture the data in my stream.

Let’s see our stream result.

In the same way, I’m going to delete and update my source table.

I deleted one record and made an update in a row, but here in the stream, we could see two deleted actions.

1. The first delete action was for the row that I deleted, and the second one is for the row that I updated.
2. If the row is deleted from the source, the stream will capture the METADATA$ACTION as DELETE and METADATA@ISUPDATE as FALSE.
3. If the row is updated in the source, the stream will capture both the delete and insert actions, so it will capture the old row as delete and the updated row as insert.

Create a Merge Query to Store the Stream Data into the Final Table

I’m using the below merge query to capture the newly insert and updated record (SCD1) into my final table.

merge into emp_hist t1

using (select * from s_emp where not(METADATA$ACTION=’DELETE’ and METADATA$ISUPDATE=’TRUE’) ) t2

on t1.emp_id=t2.emp_id

when matched and t2.METADATA$ACTION=’DELETE’ and METADATA$ISUPDATE=’FALSE’ then delete

when matched and t2.METADATA$ACTION=’INSERT’ and METADATA$ISUPDATE=’TRUE’

then update set t1.emp_name=t2.emp_name, t1.location=t2.location

when not matched then

insert (emp_id,emp_name,location) values(t2.emp_id,t2.emp_name,t2.location);

Query for SCD2

BEGIN;

update empl_hist t1

set t1.emp_name=t2.emp_name , t1.location=t2.location,t1.end_date=current_timestamp :: timestamp_ntz

from (select emp_id,emp_name,location from s_empl where METADATA$ACTION=’DELETE’) t2

where t1.emp_id=t2.emp_id;

insert into empl_hist  select t2.emp_id,t2.emp_name,t2.location,current_timestamp,NULL

from s_empl t2 where t2.METADATA$ACTION=’INSERT’;

commit;

 Tasks

Tasks use user-defined functions to automate and schedule business processes. A single task can perform a simple to complex function in your data pipeline.

I have created a task for the above-mentioned merge query. Instead of running this query manually every time, we can create a task. Here, I have added a condition system$stream_has_data(’emp_s’) in my task creation. So, if data is available in the stream, then the task will run and load it to the target table, or else it will be skipped.

create task mytask warehouse=compute_wh

schedule=’1 minute’ when

system$stream_has_data(’emp_s’)

as merge into emp_hist t1

using (select * from emp_s where not(METADATA$ACTION=’DELETE’ and METADATA$ISUPDATE=’TRUE’) ) t2

on t1.emp_id=t2.emp_id

when matched and t2.METADATA$ACTION=’DELETE’ and METADATA$ISUPDATE=’FALSE’ then delete

when matched and t2.METADATA$ACTION=’INSERT’ and METADATA$ISUPDATE=’TRUE’

then update set t1.emp_name=t2.emp_name, t1.location=t2.location

when not matched then

insert (emp_id,emp_name,location) values(t2.emp_id,t2.emp_name,t2.location);

Recently, the news broke that Optimizely acquired Netspring, a warehouse-native analytics platform.

I’ll admit, I hadn’t heard of Netspring before, but after taking a closer look at their website and capabilities, it became clear why Optimizely made this strategic move.

Simplifying Omnichannel Analytics for Real Digital Impact

Netspring is not just another analytics platform. It is focused on making warehouse-native analytics accessible to organizations of all sizes. As businesses gather more data than ever before from multiple sources – CRM, ERP, commerce, marketing automation, offline/retail – managing and analyzing that data in a cohesive way is a major challenge. Netspring simplifies this by enabling businesses to conduct meaningful analytics directly from their data warehouse, eliminating data duplication and ensuring a single source of truth.

By bringing Netspring into the fold, Optimizely has future-proofed its ability to leverage big data for experimentation, personalization, and analytics reporting across the entire Optimizely One platform.

Why Optimizely Acquired Netspring

Netspring brings significant capabilities that make it a best-in-class tool for warehouse-native analytics.

With Netspring, businesses can:

• Run Product Analytics: Understand how users engage with specific products.
• Analyze Customer Journeys: Dive deep into the entire customer journey, across all touchpoints.
• Access Business Intelligence: Easily query key business metrics without needing advanced technical expertise or risking data inconsistency.

This acquisition means that data teams can now query and analyze information directly in the data warehouse, ensuring there’s no need for data duplication or exporting data to third-party platforms. This is especially valuable for large organizations that require data consistency and accuracy.

Ready to capitalize on these new features? Contact Perficient for a complimentary assessment!

The Growing Importance of Omnichannel Analytics

It’s no secret that businesses today are moving away from single analytics platforms. Instead, they are combining data from a wide range of sources to get a holistic view of their performance. It’s not uncommon to see businesses using a combination of tools like Snowflake, Google BigQuery, Salesforce, Microsoft Dynamics, Qualtrics, Google Analytics, and Adobe Analytics.
How?

These tools allow organizations to consolidate and analyze performance metrics across their entire omnichannel ecosystem. The need to clearly measure customer journeys, marketing campaigns, and sales outcomes across both online and offline channels has never been greater. This is where warehouse-native analytics, like Netspring, come into play.

Why You Need an Omnichannel Approach to Analytics & Reporting

Today’s businesses are increasingly reliant on omnichannel analytics to drive insights. Some common tools and approaches include:

• Customer Data Platforms (CDPs): These platforms collect and unify customer data from multiple sources, providing businesses with a comprehensive view of customer interactions across all touchpoints.
• Marketing Analytics Tools: These tools help companies measure the effectiveness of their marketing campaigns across digital, social, and offline channels. They ensure you have a real-time view of campaign performance, enabling better decision-making.
• ETL Tools (Extract, Transform, Load): ETL tools are critical for moving data from various systems into a data warehouse, where it can be analyzed as a single, cohesive dataset.

The combination of these tools allows businesses to pull all relevant data into a central location, giving marketing and data teams a 360-degree view of customer behavior. This not only maximizes the return on investment (ROI) of marketing efforts but also provides greater insights for decision-making.

Navigating the Challenges of Omnichannel Analytics

While access to vast amounts of data is a powerful asset, it can be overwhelming. Too much data can lead to confusion, inconsistency, and difficulties in deriving actionable insights. This is where Netspring shines – its ability to work within an organization’s existing data warehouse provides a clear, simplified way for teams to view and analyze data in one place, without needing to be data experts. By centralizing data, businesses can more easily comply with data governance policies, security standards, and privacy regulations, ensuring they meet internal and external data handling requirements.

AI’s Role in Omnichannel Analytics

Artificial intelligence (AI) plays a pivotal role in this vision. AI can help uncover trends, patterns, and customer segmentation opportunities that might otherwise go unnoticed. By understanding omnichannel analytics across websites, mobile apps, sales teams, customer service interactions, and even offline retail stores, AI offers deeper insights into customer behavior and preferences.

This level of advanced reporting enables organizations to accurately measure the impact of their marketing, sales, and product development efforts without relying on complex SQL queries or data teams. It simplifies the process, making data-driven decisions more accessible.

Additionally, we’re looking forward to learning how Optimizely plans to leverage Opal, their smart AI assistant, in conjunction with the Netspring integration. With Opal’s capabilities, there’s potential to further enhance data analysis, providing even more powerful insights across the entire Optimizely platform.

What’s Next for Netspring and Optimizely?

Right now, Netspring’s analytics and reporting capabilities are primarily available for Optimizely’s experimentation and personalization tools. However, it’s easy to envision these features expanding to include content analytics, commerce insights, and deeper customer segmentation capabilities. As these tools evolve, companies will have even more ways to leverage the power of big data.

A Very Smart Move by Optimizely

Incorporating Netspring into the Optimizely One platform is a clear signal that Optimizely is committed to building a future-proof analytics and optimization platform. With this acquisition, they are well-positioned to help companies leverage omnichannel analytics to drive business results.

At Perficient, an Optimizely Premier Platinum Partner, we’re already working with many organizations to develop these types of advanced analytics strategies. We specialize in big data analytics, data science, business intelligence, and artificial intelligence (AI), and we see firsthand the value that comprehensive data solutions provide. Netspring’s capabilities align perfectly with the needs of organizations looking to drive growth and gain deeper insights through a single source of truth.

Ready to leverage omnichannel analytics with Optimizely?

Start with a complimentary assessment to receive tailored insights from our experienced professionals.

Connect with a Perficient expert today!
Contact Us

Joining the Crew

Let’s face it.  Data projects are laced with problems. I have been in data my entire career and I love these challenges. 15 years in the industry has exposed me to first-hand digital transformation and cloud migrations.  At Perficient, we face challenges like this often and we embrace them with a strong approach to delivery at the executive level.

What drew me to Perficient was its commitment to innovation, strong track record of success, and its collaborative and supportive work environment. Frankly, everyone I met seemed happy and wanted to help each other.  Despite this being my first impression, I was intrigued.

Why Perficient?

Since joining Perficient, I have had the opportunity to engulf myself in a one-of-a-kind culture. I have become familiar with Perficient’s unique entrepreneurial culture and how it fosters a collaborative and supportive network of colleagues on a global scale. Not to mention the countless similarities between Snowflake and Perficient that I discover with each day leading the team.

The answer to the million-dollar question, “Why Perficient?” is simple. Choose Perficient because:

• It has a standout Entrepreneurial Culture
• The Collaborative and Supportive Work Environment
• Opportunity for Global Growth
• Working Better Together

Entrepreneurial Culture

Perficient constantly pushes the boundaries of what is possible for its clients and always looks for new, innovative ways to help clients solve their business challenges. This entrepreneurial culture of innovation is what drew me in. You cannot attend any technology conference without hearing about the platforms new AI offerings.  The difference is we are drinking our own champagne with these platforms.  We are leading several projects around AI, and in one instance, we illustrated what can be achieved by using our own consulting data. The team is always willing to try new things to help our customers.  I am excited to lead a team of talented professionals passionate about making a difference.

Collaborative and Supportive Work Environment

At Perficient, everyone is encouraged to share their ideas and contribute to the success of our global clients. This sense of community at Perficient is what immediately struck me. Everyone is willing to help each other out, and there is a genuine sense of camaraderie.  When a customer relayed concern regarding assembly details, our internal teams immediately began collaborating to explore an AI-driven product for a leading manufacturer.  I am encouraged to have a collaborative and supportive team willing to dive into tangible solutions for the benefit of our customers.

Opportunity for Global Growth

Perficient offers opportunities for growth and development with its strong global presence. Often, we work on newer, best-of-breed technologies, where learning quickly is the norm and having the flexibility to deliver according to the customers’ needs is essential. As the Snowflake Practice Leader at Perficient, I am driving many Snowflake-related opportunities that are global in delivery. Our team is focused on running projects that scale to the global needs of our customers.

Better Together

Having spent time in two astounding organizations, I continue to uncover similarities between the two.  Both Snowflake and Perficient are well regarded once engaged.  Snowflake’s recent NPS (Net Promoter Score) as of Q2-FY25 is 67. NPS is a measure commonly used to associate word of mouth in marketing or said more practically as the likelihood someone will refer Snowflake to a new customer.  The industry is typically around 30, and 70 or above is considered great.  At Perficient, nearly 95% of our customers are choosing to continue to work with us again. It’s a statistic that was earned by building trust through consistent execution. With Snowflake, consumption pricing allows consumers to benefit from the flexibility of their data needs.  You pay for what you consume.  This bodes well for the Perficient delivery model flexibility, as we thrive on being agnostic on the cloud (like Snowflake) with a strong global workforce, often moving with customer needs.

The Answer to the Million-dollar Question:

Leaders in the Data Cloud space should choose Perficient because of our commitment to push boundaries for our clients through a culture that enables our teams to think freely and outside the box to create one-of-a-kind solutions. The entrepreneurial mindset at Perficient gives way to a collaborative environment where everyone has a willingness to contribute to client success with original ideas. We apply these qualities to a global scale to create a unique recipe for success from a Snowflake + Perficient, better together perspective.

Looking to the future: What is next for the Snowflake Practice at Perficient?

Looking ahead, I am honored to be leading the global Snowflake Perficient team. The momentum of AI and cloud data modernization has Perficient well positioned to not only drive business outcomes but deliver with our flexible global delivery.  It is a challenge that is worth our best. One that we are leveraging with our own consulting practice.  At Perficient, we do not try to do everything like our other global consultancy peers, but what we do-> we do with purpose and excellence. To that end, I am confident our team will deliver world-class projects and am excited about the opportunity to make a difference in the lives of our clients and to help Perficient continue to grow and succeed.  Come join us on the journey here.

Learn more about the Perficient, Snowflake practice here.