Home
Join

138 Replies

  • Words fail me

    Total up the cost of everything in the server room, add in any 3rd party costs to rebuild it and finally throw in the cost of a week or so total downtime to the company 

    Send it on to them and suggest thy review

    If no go you have done what you can, highlight and quantify the risk and presented the best options

    After that it is on them to accept the risk

    Spice (111) flagReport
    Was this post helpful? thumb_up thumb_down
  • Price what it costs to replace EVERYTHING in the server room, switches, servers, storage, UTM/Firewall/router, AND lost productivity while it's all down.   That is what it will cost if the sprinkler goes off, and they do go off accidentally some times.  Fire suppression systems start looking pretty cost effective after that.

    edit.. Damn, Toby beat me to it.

    Spice (51) flagReport
    Was this post helpful? thumb_up thumb_down
  • Write up the cost should your entire server rack need to be replaced, and the downtime it would cause. Then write up the likelihood of the sprinklers going off. Either from an incident or an accidental activation.

    Seems like high risk, and high cost. Money talks, and the cost of removing the sprinklers is no where near full replacement.

    If all else, get some documentation that you told them this was a problem. That way if the worst happens, you are in the clear.

    Edit: What Toby said.

    Spice (23) flagReport
    Was this post helpful? thumb_up thumb_down
  • Or just turn the sprinklers on tomorrow and say "told you so"

    Spice (118) flagReport
    Was this post helpful? thumb_up thumb_down
  • toby wells wrote:

    Or just turn the sprinklers on tomorrow and say "told you so"

    Easy for you to say when you're not the one that will have to rebuild a network from scratch.  

    As other's have said, write up a full cost report on what it would take to replace what's in there and add in time it would take for setup.  Pad the time some to account for conflicts and issues that would probably come up.  You could also add in any research on how often sprinkler systems could activate on a false trigger.  

    Then show a price for what it would take to replace the system with something that is electronics friendly.  Show the risks of what's involved with the current system and new system.  If they choose not to act, get it in writing so if something happens, you can pull the document out that it was decided to accept the provided risk.

    Spice (9) flagReport
    Was this post helpful? thumb_up thumb_down
  • The posts above have it down pat, but you should also be open to negotiate an alternative to a water sprinkler sysyem.
    http://serverfault.com/questions/25319/whats-the-best-fire-suppression-for-a-server-room

    Spice (8) flagReport
    Was this post helpful? thumb_up thumb_down
  • Also consider possible fire damage (from water, I know!) as well as replacing all cabling under the floor.

    Is there anything under the room that water would go to? consider that as well.

    Remember that with a chemical system, the fire goes away and only what was burnt is unrecoverable.

    With a water system, the fire goes away and everything wet is next to unrecoverable.

    Spice (9) flagReport
    Was this post helpful? thumb_up thumb_down
  • Water damage is a very serious and expensive situation.

    Spice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • Point out to them the downtime, loss of productivity, cost of the equipment to replace, and your overtime to get everything back up and running should those sprinklers go off.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • Ever heard of gas suppression systems?

    Spice (4) flagReport
    Was this post helpful? thumb_up thumb_down
  • HALON!

     :D

    Spice (10) flagReport
    Was this post helpful? thumb_up thumb_down
  • Where's a good place to find what your local fire codes are (besides Google)?  I'm just curious, but I couldn't find any useful information searching for Texas Building Codes or Texas Fire Codes, and I don't know if there are additional municipal codes.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • TheSteward wrote:

    Where's a good place to find what your local fire codes are (besides Google)?  I'm just curious, but I couldn't find any useful information searching for Texas Building Codes or Texas Fire Codes, and I don't know if there are additional municipal codes.

    If you get really stumped call your local fire department's business line. I'm sure they'd rather you call then have to come out and flood your server room.

    Spice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • Our choice was sprinkler or nothing, so we went the nothing route.

    Spice (6) flagReport
    Was this post helpful? thumb_up thumb_down
  • We use a forced air system in our UPS Battery room

    We have an EPO (Emergency Power Off) system in the Data Center but you need to make sure that the racks are covered so that if the sprinklers do go off, no water can access the equipment in the racks.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • Josh7176 wrote:

    Ever heard of gas suppression systems?

    As far as my wife is concerned that would be putting me outside.
    Spice (47) flagReport
    Was this post helpful? thumb_up thumb_down
  • Halon isn't typically used anymore, because it's hazardous.  Seems like FM200 is the preference.  When getting fire suppression quotes make sure to get the full cost (system, maintenance, sealing the room and doors, etc).

    Another option is removing the sprinklers from the room.  You won't have fire suppression, but you won't have waterlogged equipment either.  

    Spice (4) flagReport
    Was this post helpful? thumb_up thumb_down
  • alex32165 wrote:

    So this is something that's been bugging me since my very first day.  Our server room that is about 15X25 feet has 2 sprinkler heads in it.  I've brought it up a couple times and it just gets pushed away because they dont want to spend the money. 

    Step 1) Purchase a zippo lighter
    Step 2) Walk into server room
    Step 3) Ignite zippo lighter underneath a sprinkler
    Step 4) ???
    Step 5) Profit
    Spice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • Jeremy_B wrote:

    Our choice was sprinkler or nothing, so we went the nothing route.

    Because of UL regulations my racks are available to everyone, 24/7 shift work environment.  We post extinguishers and have them inspected yearly.  Sprinklers would produce some severely hazardous downtime that neither our customers or we can afford.  Being as I know first hand how often fire suppression systems false, not taking that chance. 

    Was this post helpful? thumb_up thumb_down
  • I saw a server room once that had a sprinkler system... and during winter and spring a leaky roof. It was solved with tarping the server racks and a kiddy pool on top where the leaks were.

    Spice (7) flagReport
    Was this post helpful? thumb_up thumb_down
  • I've got server rooms with sprinklers and server rooms with dry agent (Halon 1301 is long gone).

    You can't fight the fire code. And a dry chem system is way more expensive than a sprinkler head, so you can't fight the economics of it either. Definitely make your case, but bear these facts in mind.

    1. Sprinklers go off when the temperature of the air they are in exceeds some "wow, that's too hot" value. Not all sprinkler bulbs are the same. Temps available range from 100F to 600+F. Make sure you have the bulb that is appropriate for your hazard classification. Is the room occupied? Is the construction inherently fireproof? Do you keep it free of cardboard boxes and other crap?

    2. When one sprinkler goes off, they don't all go off (like in the movies). Only the ones that are in the burning area go off.

    Translation: If your server room sprinkler went off, your servers were on fire. Sure, they're wet, but they were probably charred anyway.

    3. When was the last time you heard about sprinklers going off in a server room?

    4. That's what you have insurance for. You do have hazard insurance, right?

    The primary purpose of a clean agent, such as FM200, is to cool the fire so it no longer supports combustion. It has the beneficial characteristic of leaving no residue to clean up and not damaging typical electronics. ABC dry-chem, in contrast, is corrosive to electronics and should never be used on anything you want to keep.

    Everything dries out - nothing unburns.

    Spice (40) flagReport
    Was this post helpful? thumb_up thumb_down
  • Robert5205 wrote:

    3. When was the last time you heard about sprinklers going off in a server room?
    I understand what you are saying Robert but on the other hand, I must disagree. Just because it never happened before, does not mean it will remain that way ie: shit happens.
    Spice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • Easy-peasy.

    Tally up the total cost of EVERY SINGLE PIECE OF TECH IN THAT ROOM, and present it to the higher ups:

    "This is how much we'll lose if these darned sprinklers go off."

    Voilà.

    Spice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • Perhaps plastic bags tied air tight around the sprinkler heads would solve the issue. This way your company doesn't have to pay to get them removed/replaced and your gear wouldnt get wet if they go off. 

    Spice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • Robert5205 wrote:

    4. That's what you have insurance for. You do have hazard insurance, right?
    what happens while you are sitting around waiting to have said hardware replaced by the insurance company?
    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • lancelotrussell wrote:

    Perhaps plastic bags tied air tight around the sprinkler heads would solve the issue. This way your company doesn't have to pay to get them removed/replaced and your gear wouldnt get wet if they go off. 

    If there truly is a fire, then say goodbye to the plastic bags.
    Spice (4) flagReport
    Was this post helpful? thumb_up thumb_down
  • Same boat here. Roof leaked about a month ago and they all thought I was nucking futs with the way I reacted... Just imagine if the sprinklers went off.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • Robert5205 wrote:

    [...] Everything dries out - nothing unburns.

    This is a really good point: for the most part, electronics can get wet as long as they aren't getting electricity at the same time.  It's not good for them, but there's a decent enough chance they'll survive if they power off first.

    Is there a way to have the racks power down before the sprinklers go off?  Would that be an economical compromise?  Would that even be realistic or safer at all?

    Spice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • While I agree that the best case scenario is to use a dry chemical based fire prevention system, I think it's important to debunk some sprinkler myths:

    1. When the fire alarm goes off, every sprinkler activates

    FALSE - Most sprinkler heads are controlled by a glass trigger filled with a glycerin based liquid that expands based on temperature.  If a fire starts in another part of the office, your server room will be unaffected.

    2. Your only choice is between water damage and fire damage

    FALSE - While water and electronics certainly do not mix, a modern sprinkler uses only 1/6 as much water as a fire hose.  If anyone's dropped their cellphone in water and it lived to tell the tale, your server equipment may actually survive.

    3.  Between having a sprinkler and having nothing, it's safer to choose nothing

    FALSE - Sprinklers typically only activate between 135 and 165 degrees Fahrenheit.  If you've ever seen a time lapse of how quickly modern office furniture burns, you'll realize a little water damage is the least of your worries in the event of a real fire: https://www.youtube.com/watch?v=G6lLbDQcJyA

    In Summary

    I absolutely advocate for a dry chemical solution if one is available, but when faced with nothing vs a water based fire prevention system, I'll choose prevention any day of the week.

    Add in a solid disaster recovery plan with offsite backup, the fact that any server equipment should already be insured and there is a human cost to allowing a server room fire to go unchecked with the potential of burning down the office and injuring / killing staff.

    Spice (15) flagReport
    Was this post helpful? thumb_up thumb_down
  • lancelotrussell wrote:

    Perhaps plastic bags tied air tight around the sprinkler heads would solve the issue. This way your company doesn't have to pay to get them removed/replaced and your gear wouldnt get wet if they go off. 

    If you're going to go that route, just get them capped.  (EDIT: just in case anyone gets the wrong idea, I absolutely am not recommending that)  That wouldn't cost that much, and the cap would actually hold against the pressure those things can put out (a plastic bag would rupture in seconds at best if the sprinkler goes off, and wouldn't block heat to prevent the bulb from breaking).

    That being said, I have a feeling just capping them would be a fire hazard, so you'll probably want to check your local fire code.

    Spice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • We are in the same situation, we have a sprinkler head right above the server rack. It makes me sick to think of the day that thing goes off.

    We priced it out and a Halon system is ~30k, about the same cost as everything in the room. I have expressed this concern before with no reaction. The system costs money up front but if the thing never goes off then there is no cost. 

    I'll bring it up again in our meeting this week just for good measure.

    Spice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • rolandisberto wrote:

    [...] the fact that any server equipment should already be insured and there is a human cost to allowing a server room fire to go unchecked with the potential of burning down the office and injuring / killing staff.

    This.

    No amount of equipment is worth more than people's lives.

    Spice (7) flagReport
    Was this post helpful? thumb_up thumb_down
  • Forget it! You've done your part. Really you have. This is no longer an IT issue.
    You do have this documented in writing, right?
    Viewing your document does not require having any server or other devices running in the server room, right? (Hard copy).

    Spice (5) flagReport
    Was this post helpful? thumb_up thumb_down
  • My question would be business continuity - do you have offsite backups?

    If you don't, and your entire infrastructure is suddenly singing in the rain, are you out of business?

    Spice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • Most modern sprinkler systems (at least in the last 3 major new local govt buildings we've built) do not even pressurize the overhead pipes until the fire detectors in that particular part of the building trigger the fire control system to open valves that feed those pipes. This is in addition to each sprinkler head having its own thermal fuse to open it up. This is by design for two reasons: (1) to minimize the water damage to only the part of the building that needs water sprayed and (2) overhead sprinkler pipes that stay statically pressurized with water that never flows anywhere tend to rust out and spring leaks. 

    Still, when we built our new PubSafety 911 dispatch center, we demanded that the overhead sprinklers be deleted from the design of the computer/radio system room specifically because even if the room was on fire, the servers, network gear, and 800MHz trunked radio equipment inside it needs to keep operating to the last second it possibly can until the fire kills it, even if the building has long since been evacuated.

    Spice (5) flagReport
    Was this post helpful? thumb_up thumb_down
  • Ctrl Alt Oops wrote:

    Robert5205 wrote:

    3. When was the last time you heard about sprinklers going off in a server room?
    I understand what you are saying Robert but on the other hand, I must disagree. Just because it never happened before, does not mean it will remain that way ie: shit happens.

    I must disagree with this as well. The danger is not necessarily the sprinklers going off. If there are sprinkler heads in the server room, then there is/are pipe(s) feeding those heads, and chances are the pipe(s) contain water under high pressure. Pipes can and do break and leak.

    But Robert is right about one thing: sometimes you can't fight fire code. Our server room has sprinklers in it too, and it's because it's mandated by local building code. A ground floor room not encased in at least 6" of concrete on all sides and greater than 10 sq ft? Sprinkle-pipe that sh*t. There is nothing we can do about it except plan and setup our server racks and switches as far away from the pipes as possible.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • Ctrl Alt Oops wrote:

    Robert5205 wrote:

    3. When was the last time you heard about sprinklers going off in a server room?
    I understand what you are saying Robert but on the other hand, I must disagree. Just because it never happened before, does not mean it will remain that way ie: shit happens.

    So, do you invest equally in planning for aliens landing on the roof? Armed hacker terrorists storming the facility to get your backup tapes? A plane crashing into your building? Probably not. Why? Because of the risk.

    ExposedRisk = (probability of the event) * (damage it can cause) * (ease/cost of recovery/mitigation) * (ease/cost of prevention)

    If it took twice as long and cost four times as much to hot-swap a drive as it did to restore an array from a backup, we'd have fewer hot-swap arrays. Except for businesses where the downtime was expensive enough to justify the cost of avoiding the risk. But, if hard drives failed only once every 1,000 years, we'd have backups and not RAID arrays.

    Spice (4) flagReport
    Was this post helpful? thumb_up thumb_down
  • Robert5205 wrote:

    So, do you invest equally in planning for aliens landing on the roof? Armed hacker terrorists storming the facility to get your backup tapes?


    You don't???? *places hand on sidearm*
    Spice (5) flagReport
    Was this post helpful? thumb_up thumb_down
  • Even top tier datacenters have sprinklers as a last resort.

    The DC I worked at had FE-25(HFC-125) as the primary, bust had a dry-pipe system with a deluge valve as the backup.

    Once the alarm went off, the gas system would count down to 30, then deploy, at that time, the deluge valve would dump and the pipes would have water pressure. If the fire didnt go out, the heat would melt a sprinkler fuse and the sprinkler would activate.

    The system required at least 2 smoke heads to trigger before activating.

    Is your system a dry-pipe system?

    Spice (4) flagReport
    Was this post helpful? thumb_up thumb_down
  • toby wells wrote:

    Words fail me

    Total up the cost of everything in the server room, add in any 3rd party costs to rebuild it and finally throw in the cost of a week or so total downtime to the company 

    Send it on to them and suggest thy review

    If no go you have done what you can, highlight and quantify the risk and presented the best options

    After that it is on them to accept the risk

    It may be cheaper to just replace the servers. The cost of retrofitting a room for a fire-suppression system ain't cheap.  

    Spice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • ITSlave wrote:

    It may be cheaper to just replace the servers. The cost of retrofitting a room for a fire-suppression system ain't cheap.  


    When we built our server room, fire suppression cost us $13k for a dry system.
    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • I wouldn't block the sprinklers in any way as has been suggested.  You need to replace the system with something that doesn't use water.  If you can't do that, or your company refuses to do that, then I don't think blocking them (capping, etc.) is a realistic alternative option and here is why:

    If you block the sprinkler intentionally, and a fire DOES happen, and someone dies or is injured in the fire, and a fire inspector can see that you blocked the sprinklers, I can only assume that you would personally be facing criminal charges.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • We had the the same issue until just recently. You have several options.  As others have said, highlight the cost of a waterhead going off or leaking and include loss of productivity plus time to get back to fully operational. Remember while you are down/wet, your company may grind to a stand still especially if you have voip phones. 

    Option 1. Remove water heads in the server room. Note: fire code/building code may prevent this. Our fire code said that as soon as one sprinkler head was touched, the entire system needed to be overhauled due to the age. 

    Option 2. Place 4"-6" PVC over waterhead to route water away from server in the event they go off. Probably illegal and against fire code. Also if there is a fire in the server room and insurance discovers you did this, they may not cover. 

    Option 3. http://turtleshellindustries.com/ an umbrella for your servers. Please keep the guffaws to a minimum. 

    Option 4. Relocate the server room. This may not be feasible, but in our case we were able to. 

    

    Was this post helpful? thumb_up thumb_down
  • Ctrl Alt Oops wrote:

    Robert5205 wrote:

    4. That's what you have insurance for. You do have hazard insurance, right?
    what happens while you are sitting around waiting to have said hardware replaced by the insurance company?

    Well, I never said it was an instant solution. What are you going to do while you are waiting for the new gear to replace the burned stuff?

    I have the luxury of having 4 different campuses that I can fail over to. I have a spare DL360G8 server that sits in a box ready to be deployed in case of a catastrophic failure of an entire server room. I can reload VMs on it and restore some operations in limp-along mode. I have cold spares for all switches, firewalls, and other stuff - not stored in the server rooms or IDFs.

    If it all burns, I'll rebuild it on the ashes.

    Spice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • Tally up the total cost of every thing in the room. Tally up the total cost of lost revenue if the servers where down a  whole week.

    Take that number and hand it the boss and tell him that is how much of his money he he betting that drunk guy with the 5th grade education that built that $12 pluymbers part built it to such supreme tolerances that it will never fail AND that is the bet that under no circumstances with there ever be a false fire alarm happening in the building because we all know that never happens. Pipes never leak. Things like sprinklers never fail or go off wrongly. That is just movie mumbo jumbo.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • Robert5205 is right on.  If there is a fire in the computer room your servers are toast long before the sprinkler opens.  I would gladly have a business interruption for the servers than funerals for our workers. 

    Spice (6) flagReport
    Was this post helpful? thumb_up thumb_down
  • At my last company we had sprinkler heads in the server room too.  Management didn't want to spend the money on a proper solution.  I believe our last IT Director gained some ground on it, but they ended up letting him go and soon after that the project was scrapped.  Go figure.  I dealt with it for about 6 years and my boss over 10 years...  If it's not important or make any sense to the right people, changes will never happen.  If they value IT, you might have a shot.

    Spice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • 1) Document everything

    2) Backup everything

    3) Get 3 quotes on a suitable suppression system ( ex: FM-200 )

    4) Present it to management

    5) Wait for the sky to fall

    Spice (4) flagReport
    Was this post helpful? thumb_up thumb_down
  • For us in NZ, before our system was hosted, it was 1 week downtime to relocation to a hosted solution, or 1 month downtime to effectively source, install, and setup new servers.  We are fortunate that our data-center is now hosted and replicated, housed in a purpose built server farm building with a gas suppression system, but previously I even had trouble getting a server refresh plan approved ...

    Was this post helpful? thumb_up thumb_down
  • I have always wondered why mechanical engineers always do this - oh yeah their not IT. We are building a new building and I have been involved from initial concept (yes rare) and while going over plans with engineers they had designed datacenter to have sprinklers and I told them no way. Of course push back from management as budget was discussed but I did as SQLRage said:

    SQLRage wrote:

    Write up the cost should your entire server rack need to be replaced, and the downtime it would cause. Then write up the likelihood of the sprinklers going off. Either from an incident or an accidental activation.

    Documented all system costs and replacement time, let alone the down time for a 911 dispatch center and the value really out weighed the cost.  Can't wait till I have a server room with not sprinkler heads.
    

    Was this post helpful? thumb_up thumb_down

Read these next...