When Your Backup Fails

You know it was only a matter of time before data loss happens, or, well at least the drive case or drive fails.

Lightening does strike sometimes.We had a few power hits this week due to storms. Not big ones, just enough to flicker the lights once or twice. Hmmm, backup array isn’t powered up. Weird. Maybe that explains emails I’ve gottten from Chronosync that its had errors. I usually follow up these but haven’t on this particular set. Lets take a look.  First I found out the battery in my UPS is not holding charge. It may show charged, but put a load on it and even a 10w led bulb crashes it. Replace battery with larger external one. Test the UPS and it works. How ever array powers down again, but everything else stayed up. Oh look, I plugged the power supply into the filtered power outlet, no the one actually on the UPS ! DUH !

As for the PS for the drive dock in this case  its soft failing.  The case lghtst up, but the drives weren’t getting enough juice to spin. Ok, no big deal. I order a new drive case with space for another drive as I’d been getting tight on space. I find amazon deal of the day with 10TB WD RED’s on sale for $129 off, buy 2. That will double my space as the current JBOD array is 4tb + 6 TB + 4tb + 6 tb. just how I bought drives and started with 2 I think originally, then added on as I needed and had drives. The 4tb’s are matching HGST’s, one WD blue 6tb and one red NAS vers.

Skip many details but I’m checking Neofinder to see what was on the big backup volume. Its mostly Folder = Volume backups and a couple odd files. All the original drives are online, and have backups onto second tier of drives.  However several folders show empty. Thats not good but I guess Neofinder was running when the volumes outright quit. No problem, let me just pull a backup db file from time machine. Hmmm, server only has one from an hr ago, that won’t do since it looks like the backup raid failed a couple days ago. Checking across other machines TM only laptop has TM going back normal length of time. Restore Neofinder db file.

Why isn’t TM backing up correctly ? no good reason, TM has always been flaky for no reason. TM network volume has plenty of space, no errors from TM complaining its corrupted and needs to create new backup for many months. In fact the last time I got this error it was from the machine with deep set of backups.  I’ll investigate later.

Things I did right :I indexed everything with Neofinder with automatic updates 3 times a week. I  have copies of all neofinder db files on all machines. They get updated by an automatic Chronsync batch backup that runs twice a week that cross syncs all machines. While I could run from central server, I opted to go with local copies for performance and when you travel, you need the local copy. I’m not getting into serving the files via the internet if I don’t have to as some of them are several hundred megs each. No fun on a hotel internet connection.

New idea. Put the original drives from the raid into another case. Another thing done right : don’t use a case’s hardware raid controller, use the OS for just such emergencies because when a case fails, you don’t have to worry about finding another exact match years after you bought it to get things working again.

First try : raid shows up, then unmounts, every 30 secs. Disk utility complains things aren’t right. The driver are in the case as 4tb 4tb 6tb blue 6 tb red. I mess around a but thinking maybe OS X is running background repair, looking at log files. Nothing good after 30 minutes.I hope maybe OS X corrects the problem and mounts the drives. Nope.

Crazy idea : what if the order of the drives in the case matters ? It shouldn’t, but its one thing I’ve never tested in this confg. Randomly, I swap the middle two drives so that bottom to top its 4tb 6tb blue 4 tb 6tb red. I know the bottom drive is the 1st volume of the original set. It works ! who would of thought that in a JBOD the order of the drives makes any difference ? well at least for OS X it does. New thing learned.

I run disk first aid, everything is good. Everything is good. So now I wait for drives and new case to show up where I’ll decide how to set up the new backup array, and then copy back everything. At least I have no work hanging over my head to make managing this a pressure situation.

More learning : apple JBOD with HFS+ isn’t a full implementation. The way its supposed to work is that each drive is complete with dir and files, just as a giant volume. Apple apparently is keeping the directory all on the first volume of the set. Worse is that if a data volume drops off, the entire raid goes down. By spec, you should only loose whats on that one drive and everything is ok. Thats why I picked this setup. With HFS+ it doesn’t work this way. Since I’m starting over, I’m going to take a more detailed look at raid options in APFS since its actually using containers. I’ll do some testing and see if apple has improved their JBOD formatting to act and perform more like the spec, or not. I’m going to do some active volume failure tests before committing to a copy of 15tb of data.

No data lost it looks like after checking everything. Only time and money which is the best case for significant failure. I’d planned for this to happen and even tested a few failures. Mostly things worked because I had enough copies meaning at least 3, and in some cases 5 so that in the case of TM, I still had one machine with good backup I could reach into.

Living on the edge of the digital abyss –

S

Working From Home ?
Need To Put A New Mac Editing Machine Together ?
Here’s How To Build Your Own iMac Based NLE or DAW and The Ideal Configuration and Perhipherals

IF you now have to work from home and a laptop just isn’t cutting it, literally, here is a look at building an iMac based system. I built this system with my own time and money, I spent many days doing research on the right things to buy which actually work. I even managed to send one thunderbolt 3 drive case back and I’ll name names here. There are plenty of ins and outs with building an iMac based system and I’ll get you to what matters and save your money doing it.

Installing 10.15.2 Onto A Mac Pro 2010 and a AMD GPU

Being stuck on 10.13.6 on my Mac tower 2010 was not happiness. Sure not the fastest system, but with CPUs updated to 3.46ghz xeons it was more than good enough. It was as fast as a 2013 machine at least, which with updated GPU life was good. Now keeping a 9 year old machine running sounds crazy, but if it serves your needs more than good enough, then who cares ? Sure those new 2019 Mac Pro’s are insanely fast, but my machine does what I need more than fast enough with all the latest hardware and software updates. The speed difference isn’t worth $12K or more to swap out machines.

Now a b ig part of the equation has been the GPU. The NVIDIA Titan X I’ve been running for a couple years is still a fast beast for 1080 and holds up ok for 4K. However apple wasn’t letting me continue with OS updates with the NVIDIA driver mess. I’d gotten tired of waiting for driver updates to appear after OS updates and hope it would all work. With the apparent demise of all NVIDIA support now, I had to try to figure out what I was going to do.

The cheapest and simplest fix was an updated GPU to let a newer OS run. While 10.14 was supported, and I could limp that along for at least another year there was really nothing preventing 10.15 form running on my machine except apple being bad. Enter Catalina Script Patcher. Follow the directions, its not really much more complicated than an OS install, and you’ll have 10.15.2 running on your Mac tower as fast as the downloads and installers can run. Maybe an hour or so with a good internet connection and SSD. Thats the easy part. Also please consider sending the script author a donation for the fine bit of work he is doing, its well worth it to support.

The short answer is that while the AMD 580 is supported, its not exactly a performance monster by my NVIDIA GPU’s standards or others. Sure if its your only choice its ok but after a LOT of research I came to the conclusion that the 2 better choices where the Vega64 or 5700XT as options. The Vega64 is a couple years old now but its still a compute power house for Resolve and Adobe apps. Its still holding its own next to 1080Ti’s and the newer 2000 series cards. The fastest card out there next to a 2080 TI is the AMD RX Vii ( or 7 ). Its out of production but there is still stock out there, but its also as expensive as the 2080 Ti which I can’t run even if I wanted to. So the compromise was to get the new 5700XT. The drivers for it showed up in 10.15.1 beta and are in release in 10.15.2. Technically, its not supported as apple hasn’t let any 5700 XT cards out for the new Mac Pro, but it works fine anyway. My card is using standard PC rooms and I actually seem to have a boot screen – at least the gray screen plus progress bar, it goes black and then the finder loads up 10 secs later.

As for powering the card it uses a 8 and 6 pin connection which is the same as the NVIDIA cards I’ve been running. So life was simple here but there was still one problem. The card I bought is supposed to be a standard dual slot card. There are others which are triple slot sized which I passed on. In this case the card is slightly wider than a dual slot card and covers about 50% of the next PCIe slot. I pulled my BMD 4K extreme card unhappily to fit the Sapphire card in. The other 2 slots are filled with a USB 3 X 4port card and a eSATA X4 port card. All of them are in use and I doubt there is a combo card to be had that will work in the Mac after apple pulled driver support for a lot of chips in the 10.12 era. Bad apple.

For now I’m good. I can’t see buying a Mac Pro. Maybe apple will produce a Mac Regular with a couple PCIe slots and a i9 processor but I doubt it. Hackintosh ? not sure. iMac regular ? maybe since the iMac Pro’s doesn’t really offer enough complete power or GPU power to make them worthwhile. Its a very hard time to stay on Mac when apple seems intent in pricing everyone off and away to PC or Linux land.