Andrew Wiles Olap Warehousing Blog: Olap Warehouse Toolkit

SQLBits III Report

Sun, 14 Sep 2008 13:57:35 GMT

SQLBits 2008 is over. It is the first time that I have attended this event and I have to say that I was very impressed with the quality of the speakers and the content. I was particularly pleased that there was a strong, well attended BI track. A huge amount of kudos and credit should go to the organisers who cooked up this top notch event in their spare time and for no financial gain. If you have not been to a SQLBits I can definitely recommend it as a superb resource for understanding what is going on in the world of SQL Server.

I was privileged to provide one of the seminars at this year’s event. My slot was scheduled for early afternoon and I have to say that once I had attended the first 2 sessions of the day I was more than a little concerned about matching the standard of the other speakers. The session seemed to go well and I enjoyed the interaction with what was clearly a very knowledgeable audience. I would like to thanks those who got engaged for your questions and comments.

For those reading this blog who were not there the presentation was entitled “Does your cube support your users reporting needs” and was based on the warehouse toolkit work which caused me to kick off this blog in the first place. Preparing the presentation has also given me the impetus to update the databases for the first time in quite a while. If you follow the File Downloads links on this page you will see a new SQL 2008 relational and Olap database.

Over the next few days I will try to get my PowerPoint deck uploaded and as the whole of SQLBits was filmed by Microsoft I will also see if I can get hold of a copy of the video so that those who didn’t attend can see what I was talking about. This of course subject to the provision that it is not too embarrassing!

Cakes, Layers and Slices

Fri, 01 Feb 2008 14:51:49 GMT

The following diagram shows the way in which I like to organise data when building a financial warehouse. I refer to this model as "Cakes, Layers and Slices" which provides a simple way to explain the component tableas and views to a client in a form that they can grasp.

The fact table is the "cake" containing all the data required to build a cube. The cake is split into "layers" representing each of the "reporting cycles" in the model. Actual data is a special case of a reporting cycle in which the data accumulates over time. Each layer of the cake (especially actual data) can also be split into "slices" representing a span of time.

The layers and slices may be implemented as physical tables or views with the fact table being a union of all the layers. In an ideal world in which the data volume is not too large the entire model would be built as views from the imported warehouse data. As the data volumes grow the individual elements in the model can be built as tables with appropriate ETL to move data from one part of the model to another.

The schema associated with each element (cake, layer and slice) is kept identical so that all transformations are carried out in the ETL or views that contribute.

The result is easy to manage and elements can be swapped in and out as the content grows and the reporting cycles progress through the year.

Xmla is dead?

Mon, 02 Jul 2007 15:07:00 GMT

I have for the last couple of weeks, on and off, been attempting to connect to SAP BW using the Microsoft ADOMD.Net provider. In theory this should be possible as Microsoft and SAP both claim to be Xmla 1.1 compliant as defined by the Xmla council specifications. My efforts to find technical information regarding the SAP implementation of Xmla have had emails bouncing between The USA and Germany with varying levels of surpise and incredulity that I should attempt such a feat.

This would be quite sad if I had not experienced pretty much the same problem 2 years ago with Hyperion Essbase, an excercise that I eventually gave up on.

Xmla was heralded as the way forward for Olap about 5 years ago. Microsoft, Hyperion, SAP and SAS Institute all put resources into the Xmla council along a number of other members most notably Simba. One of my old colleagues at SAS represented them on the council for a year or so and was full of excitement at the opportunity of getting involved in the delivery of a genuine cross-companystandard.

In truth the council has failed.

Microsoft, SAS, SAP and Hyperion (Essbase) have all implemented an Xmla based solution but, with the possible exception of the Simba O2X driver that can access Microsoft Analysis Services and SAP BW (technology from which was used to deliver Microsoft's SQL Server to SAP capability), I have seen no evidence of cross platform interoperability.

The Xmla council site www.xmla.org has been taken over by Simba, some time after the last working threads on the site died out about 2 years ago. The industry is left with 4 (or maybe 5 if you count Mondrian) independent and incompatible implementations of Xmla.

Is Xmla is dead? Does anyone else care?

Windows Vista

Thu, 12 Apr 2007 09:04:54 GMT

About a month ago I upgraded to Windows Vista. Anyone reading the myriad of blog entries devoted to problems with Vista upgrade might think this is the act of a madman but as a software developer I need to be prepared for the future.

I have to say that my experience has been OK (Intentionally neutral)

Before upgrading I did some research and bought appropriate hardware; something that a lot of the complainants out there seem to have ommitted. My configuraion is now a DELL XPS M1710 with 4Gb RAM running the x64 version of Vista Business.

I initially had a problem with the performance of Windows Explorer which was taking 15-20 seconds to load. It turns out that this was related to a problem with DNS on my network (Explorer tries to obtain information from Active Directory to enumerate the users full name) so I suppose that in some ways this did me a favour. Having fixed my DNS explorer performance is now fine. The general performance of applications is good but this will have more to do with the machine specification than the OS.

There are several applications that I use, most notably Visual Studio and anything that comes from Adobe/Macromedia, that need to be run with administrator rights. This is slightly irritating (i.e. an extra mouse click each time the app is launched) but certainly not a show stopper. As a developer I do wonder why Adobe felt they needed to write so much stuff to protected part of the OS in the first place but I'm sure that when I get around to upgrading these apps they will have sorted out the problems. I have had no problems with printers, wireless networking etc.

On the plus side Vista seems to boot a lot faster than Windows XP ever did on my previous laptop (INSPIRON 8600 with 1Gb RAM). It is particularly noticable that the applications invoked on startup e.g. SQL Server load very quickly. The sleep mode also appears to be stable which was never the case with XP and means that I switch the machine off far less often. I have experienced no instability in the form of unexpected OS exceptions which again I used to have occasionally on XP. The new start menu with its built in search is quite nice once you get used to it and I even use the rolodex window search because it works rather than just because it looks cool.

Like all transitions there were a number of occasions where I cursed Microsoft for changing the interface and moving/renaming features that I use frequently, but having found the new functionality I am now working as productively as I was before the change.

My overall feeling about Vista is very neutral. It does what I expect an operating system to do in much the same way that Windows XP did. It looks a bit nicer if you can afford to switch the Aero interface on but this is not as important as the changes that Microsoft have made to Office 2007. It seems to me that Microsoft have directed Vista firmly at the consumer market where it is bound to do well if only because every new PC will ship with Vista pre-insalled. As a business user I would struggle to come up with a compelling business case for upgrade particularly if I had thousands of PC's specified with <1Gb RAM to manage. In this respect Microsoft have created for themselves a huge challenge if they want to make Vista the operating system of chioce before the end of the decade...

TM1 V9.1

Wed, 14 Mar 2007 10:32:17 GMT

TM1 version 9.1 has been made available to partners within the last week as a beta so I been having a play. There are changes to integrate the recently aquired Temtec Executive Viewer and to improve the configuration of the TM1Web interface. Of more interest to me however are changes that improve the granularity of locking for data and metadata update and the new rules editor.

Data entry is IMHO TM1's real USP. The effectiveness of this capability has previously been somewhat hampered by the need to lock large sections of the database during most forms of update. The decision to improve the granularity of locking should make a big difference for applications in which multi-user collaboration is required.

The new rules provides colorisation, outlining and improved tools for working with the TM1 function library. The old rules editor was a real weakness in TM1 being little better than notepad. The editor still lacks features such as intellisense which one would expect from a state of the art development tool and I would like to see some better tools for syntax checking and highlighting but overall it is a big step in the right direction.

First Results

Mon, 05 Mar 2007 12:10:25 GMT

The following table shows my inititial findings using the published financial model for comparison. Creating sample data for my high volume model is causing some problems at the moment so I have had to put this to one side in the short term.

These results show an interesting split in performance. TM1 shows a significant advantage in query performance with a 27% advantage over Analysis services. The metatdata performance of the TM1 model is however substantially worse that that for Analysis Services. The biggest issue here being the time taken to open a connection.

	SSAS
	X50			X1
	Start	End	Total	Ave
Connection.Open	24.1149	25.9775	1.8626	0.0373
adSchemaDimensions	0.7719	1.2426	0.4707	0.0094
adSchemaHierarchies	18.1821	18.2321	0.0500	0.0010
adSchemaMembers	32.4954	32.9260	0.4306	0.0086
adSchemaLevels	0.0152	0.0652	0.0500	0.0010
Cellset.Open	19.7400	21.2822	1.5422	0.0308

	TM1				Variance
	X50			X1
	Start	End	Total	Ave
Connection.Open	43.2477	60.5626	17.3149	17.3149	46380%
adSchemaDimensions	49.2379	49.5183	0.2804	0.0056	-40%
adSchemaHierarchies	21.3191	21.5194	0.2003	0.0040	301%
adSchemaMembers	11.2474	28.7325	17.4851	0.3497	3961%
adSchemaLevels	40.9002	41.0705	0.1703	0.0034	241%
Cellset.Open	8.9033	10.0349	1.1316	0.0226	-27%

* Note: The TM1 provider did not allow the repeated opening and closing of a connection but threw an exception. The result for Connection.Open is therefore based on a single call.

Environment:

Machine: DELL Inspiron 8600, 1Gb RAM

Analysis Services: SQL Server 2005 build 9.00.2047

TM1: Version 9.0 SP2

All comparisons carried out using the ODBO provider for the application. For SSAS this is the MSOLAP.3 provider, for TM1 this is the TM1OLAP.1 provider.

Each set of results is based on making 50 calls to the same routine and then averaging the resonse time.

Wherever large variances occur results have been repeated to ensure that external factor have not influenced the result.
Queries run one time through to pre-load any cache.

Getting back to business

Tue, 27 Feb 2007 16:20:22 GMT

After several weeks of relative inactivity on the Olap warehouse project I am making a concerted effort this week to progress toward my first set of benchmarking. I have been further refining the financial model and have created a comprehensive set of data for testing. This model whilst relatively small is calculation intensive and therefore should be representative of models used for budgeting and financial analysis. I am also looking into building a larger sales model which can be loaded with a very large amount of data.

Where possible I am building in some of the "best practice" feedback that other people have contributed so far. In doing this I am trying not to let the Tm1 and SSAS models drift apart in functionality. By this I mean that I want the models to demontstrate the same dimensionality and hierarchical structures so that from an end-user perspective they are directly equivalent.

The initial set of results will focus on objective measurements such as data load and query performance. It is also my intention to make some subjective observations on the functionality and ease of use of the tools.

Target is this friday for the first set of performance observations...

Source Data Management

Fri, 12 Jan 2007 13:07:42 GMT

Segmentation and management of source data can be a challenge in financial applications. Most accountanting applications have the notion of a month end close after which data from a specified period should not be updated. For some time I have wanted to support this bahaviour in a warehouse environment. This week I have pushed some changes to the relational database for the toolkit that supports this. The design is developing along the folloeing lines.

Data segmented by reporting cycle

I have created separate tables for each reporting period for actual data. Budgets are also stored in separate tables but in this case the table will contain data for all reporting periods associated with the budget cycle. It is intended that this segmentation will ensure that updates to monthly results are physically separated from historical information to provide a high degree of safety for processes that carry out bulk updates. An additional layer of safety is provided by triggers tied to the Current Reporting Period and Current Year which will prevent data in historical periods from being altered once the period is closed.

Layering of facts

The fact table for the model is built by combining the data in the segmented tables into a combined view. This is carried out via 2 paths or "Layers". The periodic actuals are combined by union through into the first "Layer" named [layer].[Actual] adding period values in the process. The budget data is similarly combined via a separate path into [layer].[Budgets]. These 2 layers are further combined by union into the view [fact].[Financials].

Model Management

The definition of the layer views will need to be dynamically created based on the control data in the model. The next step in this part of the process will be to write some stored procedures that re-build the "layers" based on changes to the control data. For example there will be a requirement to create a set of new actuals tables each time a year is added to the HorizonEnd date. Tables will be added or dropped from the views based on their active status. This approach should allow the management of the model over a significant period of time without risking the integrity of the data associated with any "reporting cycle"

Prior, Current and Future...

Thu, 14 Dec 2006 15:01:03 GMT

Since my last post I have been very busy doing my "real job" and I have therefore not had as much time as I would like to spend on the warehouse toolkit. I have however been able to make a little progress.

I wanted to deliver a flexible way of defining how aggregations should occur for prior, current and future years. The best way to describe this is through an example. The current reporting year is 2006 and the reporting period is October. We have 1 year of prior data (2005) and are creating the budget for 2007. If we create a report that contains actual results, shown against the 2005, 2006 and 2007 budgets we might expect to see a report similar to that shown below.

		2005
	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
Actual	10	10	10	10	10	10	10	10	10	10	10	10
Budget 2005	10	10	10	10	10	10	10	10	10	10	10	10
		2006
	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
Actual	10	10	10	10	10	10	10	10	10			10
Budget 2006	10	10	10	10	10	10	10	10	10	10	10	10
		2007
	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
Actual
Budget 2007	10	10	10	10	10	10	10	10	10	10	10	10

The main behaviours here are that Actuals in the prior year span all periods, in the current year display for current and prior periods and in future years do not display at all. The budgets display all periods for all years. Remember that the period may have been set to a date in the past and data may exist for some of the periods that are displayed as blank.

I have made changes to the model and will be pushing these shortly as release 1.2

Minor revisions

Wed, 29 Nov 2006 09:34:59 GMT

This morning I have pushed some minor revisions to the database files and some additional documentation.

TM1 database

The TM1 model has been updated for some of the suggestions made by Steve Rowe and I have completed the rules so that this model is now functionaly equivalent to the Analysis Services model. The main changes here are to the logic for presenting cumulative totals over time on flow accounts.

Analysis Services database

The only change here is to the documentation headers. I realised that I had left a bunch of headers in the model proclaiming copyright for iT-Workplace Limited. These have been removed and replaced with new headers that contain an explicit statement that the code is available for public use plus revision information.

SQL Server relational database

Same issue as the AS database

Documentation

I have created an Issue Log/FAQ document to keep track of input provided by visitors. I have also added a set of release notes so that anyone wanting to upgrade their databases can keep track of the changes made.

Initial databases available

Tue, 28 Nov 2006 14:46:24 GMT

Today I have pushed a copy of my work-in-progress databases for anyone who would like to take a look. At present these databases contain a simple P&L and Balance sheet with some sample data. I have not done much work on data validation yet other than to confirm that the balance sheet balances!

In the short term I have been concentrating on delivering functionality rather than worrying about performance. Once I am happy that the basic functionality is settling down I will try to create a larger dataset to see how the differnt models perform. At the moment the TM1 model is lagging slightly behind SSAS. This is largely due to where I am on the TM1 rules learner curve. My initial impressions of working with TM1 are:

The TM1 language set used to create rules is rather sparse when compared to the very rich Mdx language implemented by SSAS.
It is very nice to have everything recalculate immediately on making a rules change rather than having to re-process.

Over the next few days I will be validating the arithmetic within these initial models and will also try to create some basic documentation that describes the functional requirments that I am working with.

BTW: Many thanks to Chris Webb http://cwebbbi.spaces.live.com for the mention on his blog.

Opening Comments

Mon, 27 Nov 2006 13:28:34 GMT

Welcome to my Olap Warehousing Blog. The purpose of this blog is to provide me with a reason to carry out some research that has been simmering on the back-burner for some time. I wish to create some standardised warehouse components that will make the implementation of financial Olap solutions a lot simpler. For want of a better title I have called this the "Olap warehouse toolkit" project. This project is aimed at satisfying my curiosity on a number of issues relating to Olap warehousing. The objective is to build an Olap model that demonstrates the following functional capabilities:

Correct Credit/Debit handling for accounting data including accounting and normal balance views.
Flow, Opening and Closing balance and Average calculation over time
Scaling of values to Thousands, Millions etc
Ability to view periodic or cumulative
Ability to view historical reports (i.e. filtering of more recently added data)
Ability to model slowly changing dimensions – i.e. un-restated history
Scenario and version management.
Currency conversion
Built-in inter-company elimination logic

To make life a little more interesting I shall be working with Applix TM1 and Microsoft Analysis Services in parallel and will be trying to deliver directly equivalent functionality with both technologies.

I hope that a few others may find the journey interesting enough to follow and maybe even contribute to. If you would like to find out a bit more about my goals and objectives please take a look at the "Project Charter" document under File Downloads. Over the course of the next week I will be publishing some database files which contain the state of my work to-date.