Posts Tagged ‘ETL’

On Operational projects that involves heavy data volume load on a daily basis, there’s a need to monitor the DB Disk Space availability. Over a period of time, the size grows occupying the disk space. While there are best practices to handle the size by adopting strategies of Purge for outdated data and add buffer/temp/data/log […]

Often there’s a need to monitor the OS Disk Drive Space availability with the Drive holding ETL operational files (log, cache, temp, bad files etc.). Over a period of time, the # of files grows occupying the disk space. While there are best practices to limit the # of operational files and clear them from […]

ETL testing is a type of testing technique that requires human participation in order to test the extraction, transformation, and loading of data as it is transferred from source to target according to the given business requirements. Take a look at the block below, where an ETL tool is being used to transfer data from […]

In the last blog PART1, we discussed Full load with the help of an example in the SSIS (SQL Server Integration Service). In this blog, we will discuss the concept of Incremental load with the help of the Talend Open Studio ETL Tool. Incremental Load: The ETL Incremental Loading technique is a fractional loading method. […]

what is ETL? ETL is a process that extracts the data from different source systems, then transforms the data (like applying calculations, concatenations, etc.), and finally loads the data into the Data Warehouse system. The full form of ETL is Extract, Transform, and Load. What is a data warehouse (DW)? A Data Warehouse (DW) is […]

What is a Slowly Changing Dimension? Slowly Changing Dimension (SCD) is a dimension that allows us to store and manage both current and previous data over time in a data warehouse. It is considered and implemented as one of the most critical ETL tasks in tracking the history of dimension records. There are three types […]

What is a Slowly Changing Dimension? A Slowly Changing Dimension (SCD) is a dimension that stores and manages both current and historical data over time in a data warehouse. It is considered and implemented as one of the most critical ETL tasks in tracking the history of dimension records. Type 1 SCDs – Overwriting In […]

Quite often, while building the Data Integration Pipeline, Performance is a critical factor. The factors below are vital for following the guidelines while working on ETL processing with Informatica PowerCenter. The following items are to be considered during ETL DEV: Pre-Requisite Checks and Analysis Basic Tuning Guidelines Additional Tuning Practices Tuning Approach Pre-Requisite Checks/Analysis  Before […]

Data is the lifeline of any modern organization. At any point, every day, you work on molding data points into information to derive profits. Therefore, having the right building blocks is a crucial part of running a good business. This is where the dynamic duo of ETL and SQL comes into play. While you may […]

What is SSRS? SSRS stands for SQL Server Reporting Services. It is a reporting tool developed by Microsoft that comes free with the SQL Server. It produces formatted reports with the tables of data, graph, and reports. Reports are hosted on a server and configured to run using parameters supplied by users. When we run the […]

Denodo: A  data virtualization platform Data virtualization is a core technology that enables modern data integration and data management solutions. Factors of data virtualization: Connect, introspect, and govern any data source with zero data replication: Quickly connect disparate structured and unstructured sources. Catalog your entire data ecosystem. Data stays in the sources and it is […]

Azure Data Factory is a strong ETL tool, with the capacity of creating ETL pipelines using low code/no code approach. This can be achieved with using “Activities”. Activities are the tasks that are conducted on data within a pipeline. In this post I demonstrate an ETL process which copies data from one source to another, […]