Deduplication and You

I was doing some backups and realized my photos, which I keep in Dropbox (sue me) were taking up about 50G. I knew there had to be a lot of duplicates but searching for them through 50G of files would be tedious. This seemed like a task that should be easily solvable with a simple one-line but my shell-foo was simply not up to the task. Python to the rescue. I meant for this to work for my pictures, but really it should work for anything.

If you use this make sure to try it once with the --dry-run option to see what duplicates it finds. Then you can at least spot check and make sure it's identifying them correctly.

tl;dr - I wrote a quick and dirty script to deduplicate my pictures.

$ dedupe_files.py --src ~/Dropbox/Camera\ Uploads --dest ~/Dropbox_Camera\ Uploads.dupes
/home/mcaudill/Dropbox/Camera Uploads/Archive/Photo Booth Library/Pictures/Photo on 1-11-15 at 4.45 PM.jpg is a duplicate of /home/mcaudill/Dropbox/Camera Uploads/Archive/Photo on 1-11-15 at 4.45 PM.jpg (117fa9d33b38d19300bdac7921b7412b)
Moving /home/mcaudill/Dropbox/Camera Uploads/Archive/Photo Booth Library/Pictures/Photo on 1-11-15 at 4.45 PM.jpg to /home/mcaudill/Dropbox_Camera Uploads.dupes
/home/mcaudill/Dropbox/Camera Uploads/Archive/Photo Booth Library/Pictures/Photo on 1-11-15 at 4.47 PM.jpg is a duplicate of /home/mcaudill/Dropbox/Camera Uploads/Archive/Photo on 1-11-15 at 4.47 PM.jpg (fb0bd6014aaea38fa8f73074aad26d0)
Moving /home/mcaudill/Dropbox/Camera Uploads/Archive/Photo Booth Library/Pictures/Photo on 1-11-15 at 4.47 PM.jpg to /home/mcaudill/Dropbox_Camera Uploads.dupes
/home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140727_092314.jpg is a duplicate of /home/mcaudill/Dropbox/Camera Uploads/2014-07-27 09.23.14.jpg (1d3f48a2b596d47c9d16d5e8bdf86778)
Moving /home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140727_092314.jpg to /home/mcaudill/Dropbox_Camera Uploads.dupes
/home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140716_192546.jpg is a duplicate of /home/mcaudill/Dropbox/Camera Uploads/2014-07-16 19.25.46.jpg (5c6829d37ae5ceddef2b986d5634af61)
Moving /home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140716_192546.jpg to /home/mcaudill/Dropbox_Camera Uploads.dupes
/home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140723_101455.jpg is a duplicate of /home/mcaudill/Dropbox/Camera Uploads/2014-07-23 10.14.56.jpg (4a13b37f7fa57249fdb2cadd51b81768)
Moving /home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140723_101455.jpg to /home/mcaudill/Dropbox_Camera Uploads.dupes
/home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140706_205524.jpg is a duplicate of /home/mcaudill/Dropbox/Camera Uploads/2014-07-06 20.55.24.jpg (40d7217492ca9c79073a61706d977130)
Moving /home/mcaudill/Dropbox/Camera Uploads/Archive/Camera Uploads/IMG_20140706_205524.jpg to /home/mcaudill/Dropbox_Camera Uploads.dupes
...