Posts Tagged ‘ETL’

In continuation with my previous post on Big Data Trends, I would like to reference couple of Stream processing solutions that are able to process huge volumes of data with low latency. These solutions address the shortcomings of the traditional ETL delivery mechanism. ETL shortcomings ETL and message queue systems address offline reporting and batch […]

We’re constantly looking for ways to streamline the BI development process.  Data mapping (source to target) can be a complex and cumbersome process due to: Ongoing model changes – the target moves! Poor (or “loose”) communication between those mapping and the development team. Lack of consistency in collecting and documenting mappings (Excel, I’m looking at […]

To build a custom plugin in groovy, do this.  I know this seems redundant, but it wasn’t as clear as it should have been. 1. Create a project directory.  Just a plain ol’ directory. Wherever you like. 2. Add the source file subdirectories: src\main\groovy\…  (with the package path you’d like to use.  We have src\main\groovy\com\perficient\gradle) […]

Quick tip of the day: Use Gradle plugins to package up functionality for easy reuse.  For example, we developed a simple “database” plugin to handle the common tasks associated with building and upgrading a database.  Things like: build – create a new database from scratch. Runs the CREATE DATABASE and sqlcmd’s the full DDL upgrade […]

While the Gradle manual is extensive, since we’re not building an executable or the like from source code, many of the concepts are a little opaque.  Here’s a quick primer on Gradle for BI: Gradle is “project” based, with each project containing a set of inter-dependent tasks which in turn contain actions: The whole thing […]

We’ve chosen Gradle as our build system for our iterative BI environment.  It’s a powerful tool, but there’s a bunch of awesomeness in there. Gradle uses Groovy as its scripting language.  Groovy is just plain great.  You get the power of the Java platform in a scripting language and can do things like this: def […]

Having been in consulting for over 18 years, I am often asked by business owners or senior management how they can deliver solutions that are faster and better. They also want world-class products that captivate the customer, in addition to better predictability in the solutions delivered. I have found a way to have repeatable success […]

Written by Andrew Holowaty – National Microsoft BI Practice