I would love to hear specifics on how others deal with storage and back up of large amounts of data (mostly photos and videos).
I have accumulated many terabytes of data over the years. I have 5 hard drives that are connected to my computer but the ones I use for photos are full. I bought a 2 disk NAS drive several years ago and quickly outgrew that (I should have better estimated how much space I’d need in the future, oops). I fairly recently bought an external 8TB drive to back everything up (again) and try and clean a lot of it up. Everything is also backed up onto Code42 Crashplan.
So now my problems is, how do I move forward? I would love to hear the input from others.
Where do you store your primary/original data? Internal, External?
Do you us a HD array? Which one? How Big?
How do you backup locally (to what device)? (assuming 3 copies, 2 local, 1 off site)
Do you use RAID? Which level?
Do you consider RAID (4,5,6) parity to be your local backup?
I have a QNAP 4 bay NAS with 4x4TB drives in a RAID 5 array. All data from my PC is regularly backed up to the NAS (daily at the moment, I used to have it sent to hourly, but that was overkill, considering I only used the PC at home for an hour or 2 most days).
On the PC itself, the data is on SSD and is synced to a spinning rust drive and from the spinning rust to NAS.
Additionally, the data is backed up to HiDrive and OneDrive (I’m currently moving away from OneDrive and the MS cloud, because of GDPR issues) and Carbonite.
That’s some considerable growth, you must take a lot of photos/videos!
I’m a big fan of Synology’s product line, I have a DS1515 with the DX expansion unit for a total of 10 disks. This might be a good option for you. You could start out with a single NAS and have the option of adding an expansion unit later on if you need the extra space.
Anything I’m actively working on is stored on an internal NVMe disk, once I’m finished with it I’ll shift it over to the NAS.
I use Backblaze for my offsite backup, and also OneDrive for a small amount of personal data which I access regularly from multiple devices. I don’t really have a local backup aside from snapshots configured on my Synology NAS.
A RAID really isn’t a good backup solution in most cases. It chiefly protects against a single (or depending on your configuration, multiple) disk failure. An ideal backup solution would protect the entire disk volume, and mitigate against a single point of failure. For example - if a power surge fries your system, a RAID won’t do any good as all those disks will be cooked.
I would recommend you consider a NAS that supports starting with as few as 2 disks and upgrading later to as many more as you think you could use. I am a fan of Synology, and am a fan of over-provisioning, so I bought a 2415+. It was expensive to purchase ($2400Cdn) but even more expensive to populate with disks. (12x$350Cdn for a total of $4200.) I use a lot of virtual machines, and I don’t store ANY significant data local on any machine… it all goes to the NAS. These days there is a newer model with most of the same characteristics: https://www.synology.com/en-us/products/DS2419+ With Synology the first two digits are the MAXimum number of disks (12 + a single expansion of 12 more for the 2419+) and the last two digits are the year of release. (The 2419 is a 2019 model.)
I need MORE. But Raid 5 is the way for me. But there are a few external drives and SSDs connected for “working.” Sadly, 16TB ain’t enough and I’m due to get more storage. Yikes.
(don’t forget offsite and cloud back up. Backblaze has mixed reviews, but it has saved my butt a few times without question)
3.5TB of backups. Duplicati to back up to a network NAS raid 1. The NAS is a very old and slow Buffalo Linkstation.
Everything is also backed up to Crashplan
My computer auto turns on at 5:00am to run all backup tasks. Important tasks run throughout the day at set periods.
I used to use HDDs in raid 0 for photo/video to speed everything up, which worked really well. Everything was backed up so raid 0 was not an issue. Now SSDs are much more affordable so I just use them. The NAS still has HDD’s.
Agree on this… it has always looked interesting, but that’s as far as I ever got. I tried something somewhat similar, but free (in theory) instead, called Proxmox Virtual Environment. Proxmox is based on Xen and Linux drive management. It’s not as focused on RAID as it is on the VM functionality, and if you pay for a license, on the multi-node redundancy capabilities. In the end, Proxmox was fun but the idea that I have to pay an annual fee to get what would otherwise be free community Linux OS patches was too much for me.
I tested Unraid and I liked it. I have so many old HDDs it seemed like a great solution and it was. I added a SSD for the cache, which made it really fast.
Unfortunately I had terrible corruption issues after lots of investigation it turned out to be my old motherboard controllers failing.
When I upgrade my current desktop or my NAS dies I will setup a Unraid device again. Well that’s the current plan.
I found this Youtube channel helpful and interesting on using Unraid.