Here is a good article with many links and debates. I have used the “accepted” answer and gotten 95% accurate results – so why not 100%? I have not solved that one. I am trying the 2nd answer which in the efficiency graph gives better results.

http://stackoverflow.com/questions/18932/how-can-i-remove-duplicate-rows