Data Cleansing for Fun & Profit

It was during SQLBits 10 in London that I first uttered the phrase “data cleansing for fun & profit”, as a spoof presentation title during Buck Woody’s “Moves Presenting Like Jagger” presentation.  I hadn’t really thought about it much since then, until just recently, I was thinking about the sort of support calls that come through to me.  And they (almost entirely) amount to dodgy unclean data.

For example: a field named “TelephoneNumber” should contain phone numbers, right?  Not email addresses, nicknames, first line of addresses etc.  The big problem is, though, how do you keep this clean?  Particularly when the data is stored in two separate tables in two separate databases, and is regularly fed from another unclean source over which you have no control?

edit: I seem to have got the presentations confused.  I offer no excuses beyond the usual lack of caffeine… Props to Jonathan “FatherJack” Allen.  Should have checked my facts at http://www.sqlbits.com/Speakers/Buck_Woody

 

Advertisements
This entry was posted in SQLServerPedia Syndication, Uncategorized and tagged . Bookmark the permalink.

2 Responses to Data Cleansing for Fun & Profit

  1. You get the application developers to parse the data on entry, you have constraints on the columns in the database to catch what passes the UI parsing and you have regular reporting / analysis of the data by an MI team that push back to users where data is incorrect*.

    Where data is put in an inappropriate field because there is nowhere else to put it the UI needs development.

    After all of this you still get data that is wrong in one way or another so you still have EXACTLY the same dilema, simply with fewer rows affected!

    Hope I helped!

    * – this generally means it unusable too.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s