I want to create a copy of my NAS hosted media folder that is about 30TB. I have a bunch of 4-8TB local (most USB3, some SATA) disks, and I would like to copy these files to the various destinations maximizing space used and time required. Since I have a 10GBE network, I can read data far faster than I can write to any of the destinations, so multiple simultaneous file copies is required at the same time to maximize this activity. doing this manually is painful. trying to select the maximum number of files that can fit (but not go over) each destination is a pain. Any thoughts on a script or an app I can use to assist here is appreciated. I want to leave the files in their native format, so I am looking for a file copy, not block-based backup etc.
If you’re only looking to copy every file to a new server, cut everything to a local directory that you know will fit. So rather than copying one and going to the next, split the mass into local directories first and check each one for size requirements. Once you know exactly which ones to copy and when, you could transfer them all at once. And one of the best ways to accomplish it is to copy your directory trees. Make sure you are only copying top directories though since they’re easiest. The directories that take up too much stance and have lots of folders are best to be put in one place. If you do it correctly, once you’ve finished, you can just merge all the folders back into place.