Posts Tagged ‘ETL’

Several weeks ago, I was watching a cable network news channel and the news anchor was discussing what it will take to win the next presidential election. As he discussed it in more detail, he commented about how there will need to be much better analysis of data to pinpoint the right segment of the […]

Logical deletes are a common practice when loading dimensions in a data warehouse. If a dimensional record is deleted in the source database, it may need to continue to exist in your data warehouse. In my recent project, this was one of the requirements set during the design phase. Since ODI was being used as […]

I often use the SQL Server (MSDB) Database to store and execute SSIS packages.  However, I found it cumbersome to migrate multiple packages back and forth from network locations to the MSDB or from the various environments (DEV, QA, PROD).  So, I created a simple script to generate the dtutil commands necessary to quickly migrate […]

In continuation with my previous post on Big Data Trends, I would like to reference couple of Stream processing solutions that are able to process huge volumes of data with low latency. These solutions address the shortcomings of the traditional ETL delivery mechanism. ETL shortcomings ETL and message queue systems address offline reporting and batch […]

We’re constantly looking for ways to streamline the BI development process.  Data mapping (source to target) can be a complex and cumbersome process due to: Ongoing model changes – the target moves! Poor (or “loose”) communication between those mapping and the development team. Lack of consistency in collecting and documenting mappings (Excel, I’m looking at […]

To build a custom plugin in groovy, do this.  I know this seems redundant, but it wasn’t as clear as it should have been. 1. Create a project directory.  Just a plain ol’ directory. Wherever you like. 2. Add the source file subdirectories: src\main\groovy\…  (with the package path you’d like to use.  We have src\main\groovy\com\perficient\gradle) […]

Quick tip of the day: Use Gradle plugins to package up functionality for easy reuse.  For example, we developed a simple “database” plugin to handle the common tasks associated with building and upgrading a database.  Things like: build – create a new database from scratch. Runs the CREATE DATABASE and sqlcmd’s the full DDL upgrade […]

While the Gradle manual is extensive, since we’re not building an executable or the like from source code, many of the concepts are a little opaque.  Here’s a quick primer on Gradle for BI: Gradle is “project” based, with each project containing a set of inter-dependent tasks which in turn contain actions: The whole thing […]

We’ve chosen Gradle as our build system for our iterative BI environment.  It’s a powerful tool, but there’s a bunch of awesomeness in there. Gradle uses Groovy as its scripting language.  Groovy is just plain great.  You get the power of the Java platform in a scripting language and can do things like this: def […]

Having been in consulting for over 18 years, I am often asked by business owners or senior management how they can deliver solutions that are faster and better. They also want world-class products that captivate the customer, in addition to better predictability in the solutions delivered. I have found a way to have repeatable success […]

Written by Andrew Holowaty – National Microsoft BI Practice