It is amazing to see the technology terms we come up with to explain new technology or trend. The consulting thought leadership coins the words to group a set of technology, trend to make it easier for people to have a context. However the success and adoption of the technology/trend defines the term’s reputation. For example Data warehouse was an in-thing only to be shunned when it did not deliver on its promises. Industry quickly realized the mistake and called it Business Intelligence and hid Data Warehouse behind BI until things settled. Now no one questions value of DW or EDW or perceive that as a risky project.
Some terms are really great and they are here to stay for a long time. Some withers away, some change and take a different meaning. One such term which got my attention is IoT – Internet of Things – what is this? It sounds like ‘Those things’ but really what is this trend or technology?
Wikipedia gives you this definition:
“The Internet of Things (IoT) is the interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure. Typically, IoT is expected to offer advanced connectivity of devices, systems, and services that goes beyond machine-to-machine communications (M2M) and covers a variety of protocols, domains, and applications. The interconnection of these embedded devices (including smart objects), is expected to usher in automation in nearly all fields, while also enabling advanced applications like a Smart Grid.”
That is a lot of stuff. Looks like pretty much everything we do with Internet. I am sure this term will change and take shape. But let’s look how this relates to Enterprise Data Management. So from an enterprise data perspective, Let us consider a subset of IoT – machine generated internet data and consolidation of data from the systems operating on the cloud. What we end up with is a whole lot of data which is new, and also not in the traditional Enterprise Data framework. The impact and exposure are real, and much of the IoT data may live outside the firewalls.
In essence, the Enterprise Data Management need to deal with the added dimension of Architecture, Technology, and Governance of IoT. Considering IoT Data as out of scope for Enterprise Data Management will lead to more issues than it can solve, especially if you are generating or depend on the IoT data.
It is almost always advantageous to be able to “make sure” your Cognos TM1 environment is “ready for use” after a server restart. For example, you may want to:
Hopefully, you know what a TM1 chore is (“a chore is a set of tasks that can be executed in sequence that are typically TurboIntegrator processes) and understand that, as an administrator, you could login to TM1 and manually execute a chore or process, but there is a better way.
Let TM1 Server do it!
To have the TM1 server execute a chore immediately after (every time) it starts up, you can leverage a TM1 configuration file parameter to designate a chore as a “startup chore”. This is similar to w MS Windows service that is set to “automatic” (most likely like your machines TM1 servers):
To indicate that your chore should be run when the server starts up, you go into the (TM1s.cfg) configuration file and add the parameter: StartupChores.
You simply list your chore (or chores separated by a colon, for example:
Don’t worry too much about adding this to the configuration; if this parameter is not specified, then no Chores will be run, and if the chore name specified does not match an existing Chore then an error is written to the server log, and TM1 tries to execution the next chore indicated (if no valid chores are found, the server will simply start/become available as normal).
These chores will run before the server starts up (technically, the server is “up”, just not “available” to any user yet):
Startup chores run before user logins are accepted and before any other chores begin processing.
Here is my example:
Once I restarted my server, I checked my server log and verified that the chore (Backup TM1) did in fact execute:
Since Startup chores are run before any logins are allowed, you’ll have trouble trying to monitor the Startup chores with tools like TM1Top or even Operations Console – and therefore there is no way to cancel a Startup chore with the exception of killing the server process.
While there are several BI technologies and more coming into the foray every day, SSRS has remained a key player in this area for quite some time now. One of the biggest advantages of SSRS reporting is that it involves the participation of the end user and that is very intuitive to use.
Let’s go back few years when excel was the go to tool for dash boarding. Every time a director or VP wanted a report, he would go to his developers to extract information from the database to help him make dashboards for his meetings. The end user had to rely on the developers to extract information and had to spend several minutes if not hours to make a dashboard. This all works ok when the meeting is scheduled for a specific day of the week or month. We all know this is a myth and most meetings happen impromptu. In such cases, there is not enough time to extract data and to extrapolate that information into graphs.
Here is why SSRS came in as a key player. With a strong foundation of Microsoft, SSRS brought in some of the best features and much needed features:
While these features may not look ground breaking in the first look, these features actually bring in a lot of value. These features save a lot of time and that time in business directly translates into revenue. The developers can design dashboards once and deploy them to a server. The VP or director can press a button to get these reports on his machine. Furthermore, the reports can be exported in several formats. What I really like about the reports though is the look and feel. Microsoft retained the aesthetics of MS excel reports and by that I mean that you can have a pie chart in excel and in SSRS look exactly same. This is a great feature especially for the audience since it most people do not like to see the look of the reports change over time. Another great feature is that SSRS has fantastic security options and one can implement a role based reporting.
In summary, SSRS is a power packed tool and you should reap benefits of the great features that come with it.
Years of work went into building the elusive single version of truth. Despite all the attempts from IT and business, Excel reporting and Access databases were impossible to eliminate. Excel is the number one BI tool in the industry and for the following good reasons : accessibility to the tool, speed and familiarity. Almost all the BI tools export data to Excel for those reasons. Business will produce the insight they need as soon as the data is available, manual or otherwise. It is time to come to terms with the fact change is imminent and there is no such thing as Perfect Data but only what is good enough to business. As the saying goes:
‘Perfect is the enemy of Good!’
So waiting for all the business rules and perfect data to produce the report or analytics, is too late for the business. Speed is of essence, when the data is available, business wants it; stale data is as good as not having it.
In the changing paradigm of Data Management, agile ideas and tools are in play. Waiting for Months, weeks or even a day to analyze the data from Data warehouse is a problem. Data Discovery through Agile BI tools which doubles as ETL, offers significant reduction in data availability. Data Virtualization provides access to data in real-time for quicker insights along with metadata. In-Memory data appliances produce analytics in fraction of the time compared to traditional Data warehouse/ BI.
Currently there are 4 options for rounding numbers in Cognos TM1. They are:
The most popular method to apply rounding in TM1 is in reporting. Cognos TM1 leverages MS Excel for reporting and supports all of the formatting and calculations available within Excel. Typically, “templates” are created that apply the organizations (or individuals) desired formatting and/or rounding in a consistent way. In addition, Excel workbooks can be published to TM1Web for viewing by wider audiences (other reporting options, such as Cognos BI, also support formatting/rounding in report presentation).
The following is a simple illustration of using Excel formatting on TM1 data:
Another popular method for rounding numbers in TM1 is to round as data is being loaded (into TM1). This allows information to always be presented in the expected format (or precision) throughout the TM1 application. Based upon specific requirements, it is also common to model a TM1 application with reporting cubes to isolate the calculation and transactional processing from the reporting and presentation (of specific information). In this case, data may be transferred from a source cube to various reporting cubes and during that transfer process logic can be applied to round to the desired precision. Simply put, you may have a summary cube specifically for reporting that holds dollars rounded up to the nearest thousand.
Cognos TM1 cube Viewer does support some formatting options for viewing data in TM1 cubes. Although this method is somewhat elementary, some precision can be set which will invoke some level of rounding for display.
The format dialog in Cognos TM1:
Finally, Cognos TM1 supports the ability to create cube rules that apply business logic to data in TM1. This business logic can include rounding. Generally speaking, as a user navigates through a cube, TM1 executes the rule (in real time) applying the logic to certain data intersection points within the cube. The result of the rule can be based upon just about any algorithm. Below is an example of very simple rounding logic.
The user-entered value is “Valueof” and there are 2 TM1 cube rules applied:
So what do I recommend? A best practice recommendation would be to evaluate your requirements and take an approach that best serves the model’s needs. The options that may best serve from an enterprise perspective would be to maximize flexibility by storing the raw numbers in TM1 and then either:
To be clear, it is important to understand that programmatically introducing rounding (either via during a load/transfer of data or by TM1 cube rule) can introduce material differences in some consolidation situations (shown in the cube view image above as “All Locations”).
Big Data is on everyone’s mind these days. Creating an analytical environment involving Big Data technologies is exciting and complex. New technology, new ways of looking at the data which is otherwise remained dark or not available. The exciting part of implementing the Big Data solution is to make it a production ready solution.
Once the enterprise comes to rely on the solution, dealing with typical production issues is a must. Expanding the data lakes and creating multiple applications accessing, changing and deploying new statistical learning solutions can hit the overall platform performance. In the end-user experience and trust will become an issue if the environment is not managed properly. Models which used to run in minutes may turn into hours and days based on the data changes and algorithm changes deployed. Having the right DevOps process framework is important to the success of Big Data solutions.
In many organizations the Data Scientist reports to the business and not to IT. Knowing the business and technological requirements and setting up the DevOps process is key to make the solutions production ready.
Key DevOps Measures for Big Data environment:
One of the top key issue is Big Data security. How secured is the data and who has the access and the oversight of the data? Putting together a governance framework to manage the data is vital for the overall health and compliance of the Big Data solutions. Big Data is just getting the traction and much of best practices for Big Data DevOps scenarios yet to mature.
The speed in which we receive information from multiple devices and the ever-changing customer interactions providing new ways of customer experience, creates DATA! Any company that knows how to harness the data and produce actionable information is going to make a big difference to their bottom line. So Why Virtualization? The simple answer is Business Agility.
As we build the new information infrastructure and the tools for the modern Enterprise Information Management, one has to adapt and change. In the last 15 years, the Enterprise Data Warehouse has matured to a point with proper ETL framework and Dimension models.
With the new ‘Internet of Things’ (IoT) a lot more data is created and consumed from external sources. Cloud applications create data which may not be readily available for analysis. Not having the data for analysis will greatly change the critical insights outcome.
Major Benefits of Virtualization
In conclusion, having the Virtualization tool in the Enterprise Data Management portfolio of products will add more agility in Data Management. However, use Virtualization appropriately to solve the right kind problem and not as a replacement to traditional ETL.
Data Virtualization offers a unique opportunity for IT and Business to leverage this technology to cut down the development time for adding new sources of data. The providers of this technology is the top software vendors like IBM, Microsoft etc. (see Forrester wave) … with the new entrant Cisco (bought Composite recently). This is not a complete list. There are other players in this market.
Many BI tools offer connectivity to different types of data sources as part of their interface (think ODBC) – but it falls more in the ETL side of the offering. Virtualization provides a way to hide the physical names and provides a common model / canonical models for the business user’s consumption.
ETL development for adding new data sources to Enterprise Data Warehouse (EDW) takes a long time simply because of the rigor needed for loading and validation of the data. Business users want these new data for analysis or even just for cross checking as soon as possible. Adding new data sources in a reasonably shorter turnaround time like in days as opposed to weeks and months is possible by using Data Virtualization tools.
IT can leverage Data virtualization for providing quick access to the needed Data to power users without compromising the control. After establishing the trustworthiness of the data, bigger roll out can follow suit. Putting the proper processes for access and letting IT manage the meta-data (Logical) layer will be a good way to have an oversight on the usage. These processes will give the needed control to IT in managing the Data sources to avoid operational nightmares.
Cloud BI comes in different forms and shapes, ranging from just visualization to full-blown EDW combined with visualization and Predictive Analytics. The truth of the matter is every niche product vendor offers some unique feature which other product suite does not offer. In most case you almost always need more than one suite of BI to meet all the needs of the Enterprise.
De-centralization definitely helps the business in achieving agility and respond to the market challenges quickly. At the same token that is how companies may end up with silos of information across the enterprise.
Let us look at some scenarios where a cloud BI solution is very attractive to Departmental use.
Getting the business case built and approved for big CapEx projects is a time-consuming proposition. Wait times for HW/SW and IT involvement means lot longer delays in scheduling the project. Not to mention the push back to use the existing reports or wait for the next release which is allegedly around the corner forever.
Business users have immediate need for analysis and decision-making. Typical turnaround for IT to get new sources of data takes anywhere between 90 days to 180 days. This is absolutely the killer for the business which wants the data now for analysis. Spreadsheets are still the top BI tool just for this reason. With Cloud BI (not just the tool) Business users get not only the visualization and other product features but also the data which is not otherwise available. Customer analytics with social media analysis are available as a third-party BI solution. In the case of value-added analytics there is business reason to go for these solutions.
Power users need ways to slice and dice the data, need integration of other non traditional sources (Excel, departmental cloud applications) to produce a combined analysis. Many BI tools comes with light weight integration (mostly push integration) to make this a reality without too much of IT bottleneck.
So if we can add new capability, without much delay and within departmental budget where is the rub?
The issue is not looking at the Enterprise Information in a holistic way. Though speed is critical, it is equally important to engage Governance and IT to secure the information and share appropriately to integrate into the Enterprise Data Asset.
As we move into the future of Cloud based solutions, we will be able to solve many of the bottlenecks, but we will also have to deal with security, compliance and risk mitigation management of leaving the data in the cloud. Forging a strategy to meet various BI demands of the enterprise with proper Governance will yield the optimum use of resources and /solution mix.
How do you create a recoverable backup for a TM1 server instance (TM1 service)? What is best practice? Here is some advice.
Note: as with any guideline or recommendation, there will be circumstances that support deviating from accepted best practice. In these instances, it is recommended that all key stakeholders involved agree that:
“In information technology, a backup, or the process of backing up, refers to the copying and archiving of computer data so it may be used to restore the original after a data loss event. The verb form is to back up in two words, whereas the noun is backup” (http://en.wikipedia.org/wiki/Backup).
To be clear, what I mean to refer to here is the creation of an archived copy or image of a specified Cognos TM1 server instance at a specified moment in time that can be used to completely restore that TM1 server to the state it was in when the archive was created.
The following outlines the steps recommended for creating a valid backup:
Certainty some of the above steps could be eliminated in the process of creating a backup, however in an enterprise environment where business processes depend upon availability and correctness , it is highly recommended that the outlined steps become standard operating procedure for creating your Cognos TM1 backups.
Common sense, right? Let’s hope so.