minus-squaredoeknius_gloek@discuss.tchncs.detoLinux@lemmy.ml•Massive data backup question: What Linux software do you folks recommend for helping sort out and organize terabytes of files and remove duplicates?linkfedilinkarrow-up20·edit-22 days agoI’ve had great success with restic. It will handle your 4TB just fine, here’s some stats of mine: Total File Count: 78374 Total Size: 13.324 TiB and another one, not as large but with lots of files Total File Count: 1295210 Total Size: 2.717 TiB Restic will automatically deduplicate your data so your duplicates won’t waste storage at your backup location. I’ve recently learned about backrest which can serve as a restic UI if you’re not comfortable with the cli, but I haven’t used it myself. To clean your duplicates at the source I would look into Czkawka as another lemming already suggested. linkfedilink
I’ve had great success with restic. It will handle your 4TB just fine, here’s some stats of mine:
Total File Count: 78374 Total Size: 13.324 TiBand another one, not as large but with lots of files
Total File Count: 1295210 Total Size: 2.717 TiBRestic will automatically deduplicate your data so your duplicates won’t waste storage at your backup location.
I’ve recently learned about backrest which can serve as a restic UI if you’re not comfortable with the cli, but I haven’t used it myself.
To clean your duplicates at the source I would look into Czkawka as another lemming already suggested.