Deduplicate files on an external drive attached to Mac?

Tech Talk MacOS
1

I have an 8TB external drive attached to my Mac Mini. Will dedupe programs work on that? Or can they just examine internal drives?

Thanks.

2

Most tools claim to remove duplicates from any storage connected to the Mac.

To manage duplicate files on external storage connected to a Mac, several tools and methods are available. For a user-friendly approach, applications like Cisdem Duplicate Finder can identify duplicate filesā€”including photos, videos, audio, documents, and archivesā€”on external hard drives, SSDs, or SD cards, and allow for their removal. The software supports scanning single or multiple locations and offers features like previewing duplicates side-by-side, automatic selection of duplicates for deletion, and the ability to exclude specific folders or file types from the scan to improve efficiency.

For users seeking a more advanced solution that leverages macOSā€™s APFS (Apple File System) cloning feature, diskDedupe is a dedicated tool designed specifically for APFS volumes. It uses SHA-256 hashing to detect identical files and replaces duplicates with APFS clones, which share the same underlying data blocks while preserving all file metadata such as timestamps, permissions, and extended attributes. This method saves disk space without deleting files, allowing the same file to exist in multiple locations without consuming additional space. The app offers a simple interface with scan and deduplicate functions, and it intelligently detects existing APFS clones to avoid redundant processing.

Alternatively, command-line tools like jdupes (a fork of fdupes) can be used to find and replace duplicate files with APFS clones on APFS volumes, preserving metadata when using recent versions (1.19.0 and later). This approach requires technical knowledge but provides a free, open-source option for deduplication. Other open-source scripts and utilities, such as apfs-compactor or clonefile-dedup, are also available but may be in early development stages.

While macOSā€™s APFS does not natively support automatic data deduplication, it does use clone files to minimize data storage, which can be leveraged by third-party tools like diskDedupe to achieve space savings. It is important to avoid ejecting the external drive during scanning or deletion to prevent data loss.

3

Thanks for the information.

I researched a variety of software titles on YouTube. I think that the videos were secretly devised as marketing tools for the most popular titles. The narrators would rate titles in order of their choice using slick animation ms. And each narrator came up with the ā€œbestā€ choice.

I guess itā€™s going to come down to trial and error with the freeware versions of 3 or 4 titles. Then Iā€™ll choose what I like best. Iā€™m starting with Nektony Duplicate File Finder.

4

Donā€™t forget to make a backup first! :wink:

5

Absolutely. .

I have BackBlaze and Time Machine running on my Mac Mini.

Also, most of the content of my external drive is now ā€˜mirroredā€™ on my Google Drive. For a variety of reasons, over the years Iā€™ve ended up with many duplicate files across different cloud services. I decided to move to just one cloud service to save money and be a bit better organized. Being a bit paranoid (fear of losing data I guess), I copied everything from the other cloud service to their own folders within my Google Drive. And synced all of that (4.6 TB) to my external hard drive. Then I turned off synching on my Google Drive so the 2 sets of data were independent from each other.

So, Iā€™m starting off by finding and removing duplicates on the external drive. Then Iā€™ll figure out how to do the same in my Google Drive.

1 Like