Servers Maintenance Should be automated?

Question

Would you rather do things manually like me, or get everything in scripts?

do health checks, free up storage/space, update packages, check the firewall, send reports, check log files, backup databases , delete old databases etc.

I have all of these in scripts but i find it useless and meaningless although it saves hell a ton of time.

I don't trust scripts even though I'm the one who wrote em.

Yes it's on a large scale.

What do you think?

Bulma · Answer

My predecessor at my current position refused to script or automate anything (probably a member of the "look busy" club). It resulted in them burning out and taking a less stressful lower paying job as they were literally working 18 hours a day running backups and import jobs manually late PM and early AM in addition to maintaining a crumbling infrastructure (8 year old servers and 100mb switches crumbling).

The best part was when this person tried to tell me I should follow in their footsteps, nope.

Not a case of working smarter for sure.

Spice (4) flagReport

1 found this helpful thumb_up thumb_down

John3367 · Answer

If you are doing the same function over and over... it is good to use a script... BUT I like to look at watch everything. So at some point there is going to be an issue the script isnt working on because something isnt thought of...

Be careful with scripts..

Spice (2) flagReport

2 found this helpful thumb_up thumb_down

Robert5205 · Answer

I make a distinction between a "script" that automates the entry of repetitive commands and a script that makes decisions for you. The former are fine. The later are usually not.

A script is only as good as the person who writes it. And, based on my decades of experience, the vast majority of script writers do a poor job of design and execution. It takes a lot of discipline and time to write code that accounts for every possibility. And it takes more time to test each of those possibilities in combinations. In most cases, the primary goal of script writing is to save time. You can probably see the the type of person whose primary motivation is to save time is unlikely to invest time into ensuring their script is 100-percent accurate. They're like the person on a bad date who keeps looking at their watch.

You write a script to copy a file each night. Did it work or do you just assume it did?

You have it send you an email if it succeeds? So do all your other scripts. That's a lot of emails every day. Eventually, you stop looking at them and just delete them. Maybe you write a script to filter out everything that worked and just leave the failure notices?

Did your script not send you a failure email because it didn't fail or because it failed to send an email?

There's a balance between doing your job and writing a script that helps you do your job. If a script could do your job, they wouldn't need you.

Spice (19) flagReport

6 found this helpful thumb_up thumb_down

John5152 · Answer

Robert5205 has pretty much nailed it!Routine and repetitive jobs - automate, then manually check everything's working on a regular basis.No way would I ever rely on Manual backups for a database of any type, that definitely needs to be automated and scheduled and it couldn't be simpler to do this! What happens if you're on holiday or off sick? Does that mean nothing gets backed up until you return?I always set aside several hours on a Monday which I spend manually restoring my backups to check that they are good!

WCRORLANDO · Answer

Automate where you can, but inspect all results.  This job is as much an art as it is a science.

Supaplex · Answer

That is where you should start fixing things. Add more logging and feedback or some kind of reporting to the script that will make you more "sure" that script worked as expected. This approach saves a lot of time and instead of doing some manual stuff on a weekly basis you will have to check things manually maybe once half a year or so. IT is fun when you have time to improve and research stuff instead of messing with routine tasks. Automate as much as you can. In order to simplify multiple scripts you probably already have you may try to create regular functions that can be used across various scripts you have.https://www.hyper-v.io/creating-function-windows-powershell-saving-module/This saves a lot of time while updating scripts according to changes in your infrastructure. Add some simple choose/select menu on top of the scripts library and convert it to the EXE file so you don't have to start each script manually each time. https://gallery.technet.microsoft.com/PS1-To-EXE-Generator-d39e8be2As a final option, you can create a GUI for your scripts library so you can just click the needed button. It helps me a lot when I have to RDP from my phone to fix things. https://adamtheautomator.com/build-powershell-gui/

Phill7895 · Answer

Automation for the win, back it up with monitoring systems, flesh out your scripts as you find new blips, do the odd manual check on a schedule to ensure all is working and monitoring isn't missing any bugs. But yes, automate all the way.

adrian_ych · Answer

When you say servers.....hosts (hyper-v hosts or ESXi hosts) or physical machines or VMs ? Then what are the servers running or doing ??

Then what type of database ?? Why manual backup DB ? Why delete DB ??

Reports & logs etc do you have a notification or log management set up ??

Spice (1) flagReport

1 found this helpful thumb_up thumb_down

GerardBeekmans · Answer

Robert made a key point regarding all those email alerts. Too many emails just means too much noise and not enough signal. Not enough emails and it becomes a game of "did the email server fail or was there no output". Can't win, there.I use custom scripts with Zabbix to mitigate some of this. No email means all is well but Zabbix does receive the period "all is still ok" updates. If there hasn't been an update received of any kind in a while (good or bad) then that is flagged as an error condition to send an alert about. It likely means the script stopped running correctly for whatever reason and so investigate that issue. Could still be an email server so I have Zabbix dashboard open in a browser full-time and just glance at it throughout the day as I do other work a few tabs over.You also settle into a routine. I expect to receive a certain number of daily emails from Veeam as it runs its jobs. I just made a habit of looking for X emails in the Veeam folder when I come to work and just glance to make sure they all say "[Succes]". If there are fewer than X emails in the folder, investigate if the backup job is just running late or something else happened. Those are rare events and don't take much time.

The ones with warning or failure are coloured differently in the email client so they stand out. Those get actioned. I do this for a few critical parts that I must be assured ran each day. It takes less than a minute. Just part of the routine sweeps.Automation is key, though. If you have a few servers you can afford to do it manually. Or, if you get paid by the hour. If I were to patch each server I manage manually, it would become my full time job. I have too many to worry about. I can't be stuck in Update Mode 5 days a week and evenings. By the time I'm caught up, it'll be "next month" and start it all over again. I'd never get any other work done.I used to be against the whole idea of letting servers update automatically but it came to a point where it wasn't a feasible option anymore. I can't hire a person whose single job is to run updates all day long, year in and year out, so something had to give on that front. It does mean things can break and stay broken for longer before it's noticed. You just add your monitoring tools to help spot it sooner. Sometimes it's an unwinnable situation. We all are forced to do more work with fewer people so we have to work smarter and we're only going to work as smart as the bugs in our tools and script allow us to.Edit: clean up and fix some typos.

TB33T · Answer

I think of it similar to they way I handle backups. They're on a schedule, but I still follow up on a regular basis to make sure it completed correctly.

pigdog · Answer

Neither.  I usually just wait until something breaks.

Sauret · Answer

Best answer is Robert5205​'s.

Jim Schuuz · Answer

Exactly.  If you have a local backup system (not a managed solution) and don't have regularly-scheduled time to verify the backups, you're basically skating on thin ice.  If you have to create a calendar reminder to check and verify (mount, boot, review logs, etc.) then make one.  Too often, people don't discover a hole or problem with their backups until they need them.