Modern post client-server organisations are thick with data but where is the best data our enterprise has to offer? Where is the most appropriate, the most up to date and the most accurate data? How do we create a representation of the business world that our users would find recognisable?
I was prompted to ask questions like this when I was first involved with data migration nearly ten years ago. We had gone through the painful process of building a migration tool, stocking it with extracts taken from the most legitimate source available (a rather cranky old database created for regulatory purposes), only to find it all rejected out of hand by our user population.
It turned out that each of set of users had, out of exasperation or necessity, created a separate store of the data items they needed to do their work. Each store, of course, was built to a different format and standard. Some were in Excel, some were bastardised versions of a data-gathering Access database that had been left behind from a previous migration exercise. Some were in non-standard database formats using tools officially superseded a couple of desktop versions ago.
To the technologists these stores represented the worst possible nightmare of design and build aberration. To the users they were the best fit to their need and being closer to the coalface of day to day activity they were also the best matches to external reality.
We were faced with the classic migration dilemma – stick to the official policy and use the discredited corporate data store, or listen to our users and use the best of what they had to offer.
We (fortunately) choose the latter course.
Of course this entailed considerable additional effort. Each candidate store had to be examined for quality and re-jigged to fit the new data structure. Extracts had to be hand coded – sometimes even re-keyed.
However, once the project was allowed be led by the business it became a shared problem. The terms of the question were no longer how are we (the IT professionals), going to load the data we needed into the new system? It became how are we (the business) going to preserve the best of our data and maybe even add value to it by enhancing its quality, as we load it into the new tool?
It was from this that I learned the most important rule of successful data migrations. To guarantee success data migration has to become a business problem, not a technical one. This principal is more important than your choice of ETL tool, more important than the modelling convention you adopt, more important than adherence to strict project management principals.
Recently, at a wedding, I met one of my chief user side contacts from that time. As we reminisced about the characters we had worked with – the awkward, the eccentric, the party animals and the straight-laced – I was struck, once again, by how close our network across what was a geographically dispersed company had been. Of course, what was then a new system is now, in its turn, legacy. My friend had come into the earlier project as an engineering expert, but having got a taste for project life was now project managing the migration from our system to its successor. She was doing all the things we learned – re-establishing the virtual network and scouring the desk tops for those rogue data sources that will maximise the value of the migration exercise. I know that her migration efforts will be a success.