A timely reminder

[Suff]

For future reference this setis better than a chisel. But in no way as satisfying in conveying your irritation on a piece of cheap Chinese trash......

[Suff]

For those who are in the slightest bit interested, I thought I'd share a bit of the pain I'm going through. If you are not interested, please switch off now.....

First my data was on a VMWare ESXi server. Simply put, this is a free, very capable and totally inscrutable server which has very little in terms of support tooling when something goes wrong.

I finally got my drives from the UPS depot, but it took me nearly 3 hours to get there and back by metro and train.

My 4xSATA docking station, on the other hand is still stuck in the DPD depot and I won't get it till tomorrow evening...

Because of this, I decided to go to MediaMarkt and get another one. This is for the morning, so after the next bit.

So, anyway, as a test, I thought I'd see if I could first recover my small volume. It sits on one drive and was not damaged in any way. Using one of my nice shiny new 4TB drives, I started the process of getting the data off. First I set up a new virtual machine on my desktop and tried to link the drive. Had I known then what I know now, I would have finished in about 2 hours. However it was good training.

The virtual ESX server did indeed see the drive and the file system. But it would not mount it. No way, no how. I tried connecting it in every way possible. So I wondered if it was the drive type. So I decided to clone it and make it virtual Then the server would think it was internal like the original.

Cloning is something I do a lot of. But this time it was much harder. First I had to image the drive. I use Acronis Trueimage home for this because it is a very capable piece of software. As we shall see.

Once cloned I need to make an image out of it. However I can't afford to make any changes so I have to use tools. TrueImage provides an imaging tool that creates a .VHD file (Virtual Hard Disk Microsoft format). This is where I hit the first hurdle. TrueImage had made a 66gb backup image of the drive because I chose to compress it. But, in order to make the VHD, it needs to re-create the entire 1tb volume plus a bit for the virtual control. This, of course, is bigger than 1tb and I had to use one of the new 4TB drives. This is a problem because I need that space to get all the data off the server drives once they are up and running again.

So, enough for the night, I have the backup but I don't have my Netflix because the XP machine on that drive runs it and my backup is in France and copying very slowly over the internet.

Next day I head into town to Mediamarkt. The bike is still broken so I really can't go to any of the outlying one's. So I just have to pay the price. I have a list in my head, USB3.0 docking station for 2 or more drives and a 5TB LaCie drive the website tells me is in stock. Probably around €300 more than I've already paid.

When I get there, I find that there are only single drive USB3.0 drive bays (I don't have enough ports for that). Looking around I find a storage server (NAS appliance), which takes 4 drives, sitting in the offers bin with €100 off it. I opened it up and checked everything which seemed fine. Then I spent the next hour walking backwards and forwards looking for options. Because that NAS server was €267. In the end I found that the only alternative was 4 external drive bays plus a USB3.0 hub. Total cost €250. The NAS is quite small and neat and connects to the network so all my systems can share it and for €17 it was a no brainer. Had it still been full price, another calculation.

LaCie drive? Not a chance. Gone. There were 8TB drives for €450 (I had already bought 2*4TB for €255 delivered), or I could get a nasty looking Seagate 5TB drive for €219. In the end that is what I did. With hindsight I should also have bought another USB3.0 drive bay and the USB3.0 hub. But I hadn't done the maths by then.

So I got back armed with my storage server, loaded the 5TB drive into it, set it up and started on the next step.

To get to a VMWare drive image from a Microsoft one, Starwind produce the excellent V2V tool, which converts between the two. So, in the end, I wind up pushing the new image to the Seagate drive. Which is so slow (the new drives are working locally at around 140mbyres a second sustained), 35mbytes/s that I know why it's so cheap...

By this time I've done the math. I need a drive larger than the one I'm copying, at either end, to complete the process. That means I'm going to need >1tb on either side just for the 1TB drive plus I'm going to need >3TB on either side for the 3TB drives and I'm going to be pushing a >3tb drive image to the NAS after each drive is done. As you can see, the things going to rack up fast as I have 4 x 3TB drives to recover. I realise I need to start reorganising my mobile and portable drives so I have enough space to do this.

But I'm getting ahead of myself. So after a lot of waiting and copying (and tidying up), I finally get my 1TB virtual drive and I add it to the virtual server. Boot it up and....

Nothing. It still sees the drive, it still know's it's VMFS5 (VMWare filesystem 5), but it won't mount it.

Back to google. 2 hours of obscure surfing later and I find out that I need to "Resignature" the volume before VMWare will read it and mount it. Simply it has to read it, catalog it and then it will mount it. Otherwise it uses the proverbial barge pole and won't touch it.

More surfing. Need to use VSPhere tools. Nope, can't use then as I need the server up and running properly and a week of time to get that installed. It says that you can use the server command line but every example is with VSphere tools.

In the end I get my awkward hat on and look through the options of all the ESX tools on the server itself. Until, hey presto, I find the option resignature in a totally different place with a totally different command.

Getting somewhere now. I resignature the drive and it mounts, immediately. There it is, virtual volume with virtual machines. After a bit of faffing, I take the VM off the drive image and put it in two places, one for safety on the NAS and one on my local spare SSD for running. I get it running with a bit of fiddling (network is very different on my laptop) and I have TV again....

Magic.

Now I have a problem. I need a single drive with >3tb on my laptop as my 5TB and my 2* 4TB and one of my 1TB drives are all in the NAS. All 14TB of it to deal with the 12TB of drives. That leaves me with my D drive on the laptop (1TB), The spare 1TB Drive out of the ESXi server, my second 1TB drive I have not yet got around to putting inside the laptop. Another 1TB drive which is in a USB2 enclosure. I dropped it last year and it lost some data so I'm deeply suspicious of it and don't trust it. On the back lid of the Laptop is another 1TB drive which is basically full. However I have a plethora of 500mb and 320mb drives I have outgrown which can also take data.

So as you can see, I have a lot of data around, but a lot of space too. So Saturday night was data reorganisation round. With compression and file moves and a clone of my D drive to get the thing onto another smaller USB drive and working right, I finally got the whole thing rebooted and up and running this afternoon.

Next step. Gather the disparate drives into a single 4TB volume and I'm home free. Did I really say that. Was this not ridiculous optimism? Windows does not allow USB drives to be dynamic. Since XP it has not allowed non dynamic drives in a single volume.....

This is beginning to feel familiar. So, back to google. Where I see 1,000 entires of "why do you want to do that" and about 1 entry of help. In the end, again, I found the answer by digging into my own knowledge of Windows servers and how they control storage.

I found out that Windows 8 onwards comes with what they call "Storage Pools". Something which is not administered in the storage area but in the system and security area.... Go Figure.....

So I added my 4 drives, 3 x USB and 1 internal, to the storage pool and, hey presto, one 4TB drive. Now, one might ask oneself, why, if Windows can do this, did they not put the damned technology in the SAME DAMNED PLACE as you go to create VOLUMES....

Yes, Yes, I know I'm shouting.... I have to stop that.

So, here I am, I have my VPN machine running and backed up ready for the new server build, when I get the motherboards which are on their way from france (by snail carrier as far as I can tell but then it was free so what can you say), I'm in the process of virtualising my server drives and, once that is done (about by Thursday by the progress so far), I'll be able to get all the files off onto my 4tb store and what is left of my NAS and I'll be able to build the new server.

So here I am, this is what has happened to my mundane life and mundane apartment.....

In this picture you can see my laptop in the process of pulling a backup of the first 3tb drive (the one which failed naturally), you can count 8 drives littered around it, only 5 of which are connected and active. You can't see the two drives inside or the one on the lid for a count of 11 drives around my Laptop.



Here is the server waiting with the pending 3 drives to be imaged. Each one will take the better part of a day.



I hadn't intended for this to be so long. But it's been a long weekend with a LOT of frustration. I wasn't joking about if you are not interested switch off.

What price the data I have nearly lost? The total, so far, is going to be around €1000 to fix the mess. However, when it's done, I will have a reasonably competent NAS for backups and the kernel of a comprehensive storage sync solution for all my servers and data.

Oh and a whole lot of one time knowledge which I probably won't have much use for in the next 3 years...

[Suff]

20 days later and I have nearly three copies of my data. One copy on the original drives. I have the server up and running again with the new motherboard. Well the second one actyally, MB1 was DOA and has to go back.

I'm in a quandary now. I really want two complete copies of my data and the ability to rebuild my machine with the newer faster drives two of which are 1TB larger than the originals.

The problem is that even though I've got some 35TB of space kicking around here, I'm 1TB short of my nirvana of two copies plus enough space to build the new server. I'm thinking about just going out and buying another 5TB external drive, but the cheap Seagate one has pitiful performance (20% of my new drives) and I've already spent close to €1k on getting this done.

Copying has been a real pain. Literally I'm driving my laptop and my USB3 docking station to the limit. I've had constant hardware resets due to the heat and I've had to use sync tools over and again to get my copies done. Now my server is up and running again it's better, but it's very warm here and even the NAS box is crashing in the heat.

At least another two weeks to get all the data synched around.... I've had to upgrade the virtual W7 machine to W10 as it has become corrupted (perhaps the shoddy sync software I downloaded).

Not a journey I want to repeat any time.... Ever, not soon.... At least I got the VPN and Netflix back up first. Helps to while the time away when copies take up to 48 hours.

At least I have the new job starting on the 24th. It's a 30% pay rise so I can replace what I spent on this fairly quickly....

[Suff]

Postscript.

If you look at that picture of my laptop with all the drives around it and zoom in on the drive standing up on the desk to the left in the docking station, you will see at the top right cornerof the label that I have scribbled circles on it in pen.

That’s because that is the drive which kept going offline and kicked off all this mess in the first place.

Last week my NAS device (14TB of drives), in which I had put 3 of the 4 original drives from my server, had stopped working. Just flashing green lights and no way of talking to it.

In the end I took all the drives out and it rebooted to emergency standard. Told me it was uninitialised and asked for me to put a drive in. It then told me it would trash that drive. Lovely, nice failure resistant NAS.

So I reinitialised the NAS and then it asked me to create a new volume. I plugged the other 3 drives back in and waited for them to be recognised. And waited. And waited. Until, eventually, I pulled out drive 2 which was not showing. I turned it over and lo and behold.

It has a scribbled circle in the top right corner.

It is also completely and utterly dead, won’t respond anywhere. Now, being an old and newly cautious IT person, the only data on those drives was duplicated 2 more times elsewhere. However of the two additional copies, half of one of them was on the drive I needed to replace the dead one with.

Today my backups are copying to my NAS (third), backup. Tonight I’ll backup the rest of the data and then tomorrow I’ll sync all the changes since I created the first backup.

Granted I should not have used the drive, but, in my defence I thought it was the motherboard which caused the problem. Everything checked out and seemed to be working fine. I have lost nothing because I no longer trust anything. Although I’m going to have to buy yet another drive to replace my second backup solution.

The moral of the story is always the same. When it comes to your data, trust nothing. Only defence in depth will save you. Fortunately it’s not an iPhone with 14 years worth of contacts on it………..