Data validation: The small difference between a smooth process and an annoying, ineffective task.

Who does not know it? Your four year old kid comes to you, with bright eyes, ready to explore the world and starts asking you: “Mommy, daddy — when you do an IT project that includes transformation of data, how do you validate that data? How do you prove to yourselves as well as anyone else that the data you got in the system is what it should be? How do you prove that you did not forget half of the data and messed up the other half?”

Ahhh… little kids and their silly, yet sweet questions!

So, every parent has already experienced that, right?

Ok, admitted, it’s a ridiculous story. Not only do four-year old’s not care about data validation, in fact quite a few people in IT projects don’t give a sh**. Why is that? Well, probably there is a good explanation: it’s a topic that may appear as not very rewarding. Validating gigabytes or even terabytes of data sounds like checking tons of Excel files (or similar) in a repetitive, exhausting, time consuming and inefficient process. And this is even a pretty accurate description of how this is handled in many projects.

Most people would agree that data validation is important. But it’s considered in many projects as difficult topic. You will always have only the chance to do a small spot check of the data (not true if done right). It will be extremely time consuming (not true if done right). It will be an error prone process, in which real issues will stay undiscovered while many findings will show which are in the end no issues (not true if done right). In total: it’s a process which adds only very little value.

Not true, not true, not true — if you do it right.

What if I told you that the data validation of a multi-terabyte SAP S/4 brownfield conversion could be done in not more than six hours? In a fully automated process that is audit proof and could be repeated with the click of a button as often as necessary?

And this is not a fancy vision for the year 2077, this is what is done already today.

If you are involved in an SAP transformation project where you have the feeling that data validation is not done properly, where too much effort is spent, where the added value is just minimal — stay tuned.

I’d like to share my experience from data transformation projects done in +10 years and how data validation was done. The good, the bad and the ugly.