RAID fun.

[iain187]

when my boss goes abroad and leaves me on my jack jones, i always say by the time you get back something will have gone wrong, just you watch.

this weeks disaster was 2 disks in the exchange servers raid array going tits. :dead:

after a couple of long days, peoples emails are back and my phone has finally stopped ringing :approve: but the one thing that puzzles me is that when the server crashed, the error was hdd2 failed, I thought easy enough to put a hot spare in.

but when i did that it didn't work - only when i got to the RAID bios did I see that hdd0 had also failed. (4 disk raid 5 array, guess what that meant....)

There was no sign of this anywhere else, we do a daily check of the logs etc, and i looked on some random log in the server startup process (ctrl + e or s i think ? dell poweredge anyone??) I could see logs for disk 2 but nothing anywhere for disk 0


Is it possible for some kind of sensor to break in the raid controller hence no logs? I just dont fancy a disk going without us knowing and then a 2nd one killing the server again
I am not an expert here

on the other hand the server is 6 years old and all the senior managers didnt have email access for 2 days, I think we can squeeze some $$$ out of them

 

[azrael]

Once a disk goes the rest are working that much harder, its feasible for another to go. RAID 6

Have had issues before with a poweredge and a faulty hot plug chasis

 

[regi james]

thats unlucky I guess they were a mirrored pair that failed? I guess the corrupted data from the 1st failing disk caused an error for the mirrored one?
sounds unlikely they would both fail.
is there any SMART data that can be read?
just to add im no expert on servers but know a fair bit about computers in general

 

[iain187]

I will have to find more info tomorrow once all the users have settled down.

Man o Man am I glad for backup tapes

 

[Tony Hunter]

I would just set alight to myself and out the door screaming.

 

[Cookie]

I would just set alight to myself and out the door screaming.

Wrong sort of raid

 

[Tony Hunter]

lol,no I'm sh1t with computers end of.

 

[regi james]

Wrong sort of raid

fpmsl

 

[iain187]

lol i wondered... wtf

 

[Silent_Scone]

not uncommon especially if the disks are are fairly old. Just one of those things

It's when you're trying to fix something else and then that happens that you feel like setting yourself on fire lol

 

[dk]

thats unlucky I guess they were a mirrored pair that failed? I guess the corrupted data from the 1st failing disk caused an error for the mirrored one?
sounds unlikely they would both fail.
is there any SMART data that can be read?
just to add im no expert on servers but know a fair bit about computers in general

He said, it's raid 5........

 

[Cookie]

thats unlucky I guess they were a mirrored pair that failed? I guess the corrupted data from the 1st failing disk caused an error for the mirrored one?
sounds unlikely they would both fail.
is there any SMART data that can be read?
just to add im no expert on servers but know a fair bit about computers in general

Reading not your strongpoint, though

 

[PinkoCommie]

Ahh, reminds me of the time that I spent 12 hours trying to recover an Exchange server only to eventually realise that the latest backup that I was trying to restore from was actually taken in the middle of the message store corrupting itself :-(

On the raid side, hot spare FTW

 

[dk]

Reading not your strongpoint, though

With punctuation not being yours....

 

[dk]

Ahh, reminds me of the time that I spent 12 hours trying to recover an Exchange server only to eventually realise that the latest backup that I was trying to restore from was actually taken in the middle of the message store corrupting itself :-(

On the raid side, hot spare FTW

Exchange on a proper San or DAGs ftw!

 

[Longy]

I had two HDD's fail on a Dell Powerfault on a RAID5! was only a couple of months old - nightmare.

RAID6 for me from now on.

 

[Cookie]

With punctuation not being yours....

One too many dots in your ellipsis, you massive spacker.

 

[Mot!]

It's this time of year that hard drives die.

Especially if your company are tight with the air-con. Much like the council I was with - servers smoking and dying they're so hot? LOL

Same happened to me last year. I gave it the cocky walk down to the server room with a HDD ready to drop it in and let the array rebuild it's self. Then realised 2 discs were gone - FML

 

[welshname]

are you running hardware or software raid? I ran software raid for a while but it just kept dying a death. the initial outlay for hardware raid is worth it imho. i know some people still like software though.

 

[Longy]

are you running hardware or software raid? I ran software raid for a while but it just kept dying a death. the initial outlay for hardware raid is worth it imho. i know some people still like software though.

I've had couple of hardware RAID cards fail - which is a massive pain the arse too.

 

[Adam.]

I've had couple of hardware RAID cards fail - which is a massive pain the arse too.

One of the RAID card's in our Xserve's died a few weeks ago. Absolute nightmare!

 

[iain187]

are you running hardware or software raid? I ran software raid for a while but it just kept dying a death. the initial outlay for hardware raid is worth it imho. i know some people still like software though.

hardware...

air con is fine, although one time the air con unit failed in the cctv dvr room and it killed one of our DVRs easy to fix them things though.


i have about 200 mailboxes sat in a recovery store starting to restore - most users are going 'omg my calendars are gone' - at first i thought the restore had just restored their inbox and somehow forgot their diaries, then i realised people just didnt give a s**t that their inboxes were empty seeing as they could email people. haha thilly users

 

[Cookie]

I've had both sets of AC units in a server room die overnight before.. came in in the morning, 45c of heat greets me!

Amazingly, we only lost 3 or 4 disks, most servers powered themselves down to stop overheating

 

[iain187]

apparently my boss is going to get a boot up the arse for getting enterprise exchange instead of limiting users mailboxes

because our exchange restore was 160gb for 250-300 odd users.



big boss coming up next week to 'debrief' my boss on what happened while he was away. i am going to try not to laugh :approve:

 

[dk]

apparently my boss is going to get a boot up the arse for getting enterprise exchange instead of limiting users mailboxes

because our exchange restore was 160gb for 250-300 odd users.



big boss coming up next week to 'debrief' my boss on what happened while he was away. i am going to try not to laugh :approve:

You think that's big?

Our exchange is 950gb for 300 users.....

 

[iain187]

big boss thinks that's big :rasp:

mostly because the company we are owned by are militant on archiving and give you 10mb mailbox / 40mb if you are exec

but they have a ba-jillion users.

however all our policies have to be 'aligned' with parent companies so we shall see.

 

[dk]

Yeah, were in the process of archiving ours, we did start with enterprise vault, but are now doing it with mimecast.

Once that's done we are moving to exchange 2010.

We have a new San I'm going to be implementing our new virtualisation and vdi platform on, a snip at £2m

 

[Darren S]

apparently my boss is going to get a boot up the arse for getting enterprise exchange instead of limiting users mailboxes

because our exchange restore was 160gb for 250-300 odd users.



big boss coming up next week to 'debrief' my boss on what happened while he was away. i am going to try not to laugh :approve:

Lol. Seriously? We're still on Exchange 2003 Standard, so it detaches itself quite often when it reaches it's relatively small limit (around 65GB iirc?). Biggest issue for us is the amount of reports & attachments people need/don't need/get sent. I bet 80% of users never bloody read them!

Exchange 2010 for us in a month or two - once the renewal goes through.

D.

 

[Cookie]

At least you cnuts have Exchange, I have to support Domino :/

 

[iain187]

Lol. Seriously? We're still on Exchange 2003 Standard, so it detaches itself quite often when it reaches it's relatively small limit (around 65GB iirc?). Biggest issue for us is the amount of reports & attachments people need/don't need/get sent. I bet 80% of users never bloody read them!

Exchange 2010 for us in a month or two - once the renewal goes through.

D.

it's either 65 or 75gb - we kept hitting it too.

iirc, at that point my boss ran round panicing trying to get people to archive their mail - particually the senior managers with the 2gb+ mailboxes - which he managed to do.
then he realised he needed to defrag the infomation store whilst it was offline or it just stayed at the limit and kept detaching.

he told me that he tried it but it was going to take longer than a weekend which was too long to have the info store offline, so presumabley he thought fook it let's get a purchase order for exchange 2003 enterprise signed off.

so since then users have just been allowed to let their mailboxes grow as they please.


Do you guys impose limits on your users mailboxes and force them to archive? what is best practice for this

 

[Adam.]

1GB limit on mailboxes for me. we have just under 300 users on our mail server.

If they get too big they stop getting mail and they get a notification. The majority of our users are good though as they have learned the hard way lol.

Any mail that cant get to a full inbox just sits in the delivery que until the users clears out some shite.