Monit is a server monitoring program that can check the processes running on your system to ensure that they're always online and operating within reasonable CPU and memory limits. It also comes with a web interface to monitor your system.
What Does Monit Do?
Essentially, Monit will check in on a given process every two minutes (by default) to see how it's doing. It keeps a log of resource usage, and can take action if an error occurs. This includes relaunching crashed processes, and restarting processes using too many resources.
Monit defines its behavior entirely through config files. Here's the default config for MySQL as an example:
check process mysql
with pidfile /var/run/mysqld/mysqld.pid
start program = "/usr/sbin/service mysql start" with timeout 60 seconds
stop program = "/usr/sbin/service mysql stop" with timeout 60 seconds
if totalmem > 400 MB for 5 cycles then alert
if totalmem > 600 MB for 5 cycles then restart
if cpu > 50% for 5 cycles then alert
if cpu > 90% for 5 cycles then restart
if 3 restarts within 5 cycles then timeout
Monit is configured to monitor the running MySQL process. If it goes down, it can start it again using the start program = config. If MySQL starts using too many resources, it can be restarted automatically, though you'll be warned by email first.
It can also be configured to monitor your system as a whole, and can alert you if your server is experiencing unusual load.
check system wp01
if loadavg(5min) > 1 then alert
if memory usage > 90% for 5 cycles then alert
if cpu usage (user) > 90% for 5 cycles then alert
Everything Monit monitors is viewable from the web interface, which runs over port 2812 by default.
Additionally, you can view data for multiple hosts all in one place with the M/Monit utility, which does require a license but is free to try.
Installing and Configuring Monit
Install Monit from your distro's package manager; for Debian based systems like Ubuntu, that would be:
sudo apt-get install monit
Monit should come with default config, usually located at ~/.monitrc. If it didn't (as was the case with a macOS installation) you can find the default config files here.
Monit's web interface is configured to run on port 2812. This is fine, but it's entirely open by default with a default password. We'll want to lock it down to an authorized IP address and
set httpd port 2812
allow your_ip
allow admin:password
Change this password, of course. There's a few more configuration options for the web interface, but this is good for now.
Monit can then be configured to monitor individual processes. Usually, this is done with the processes PID file (often found in /var/run/) which stores the current PID of the running instance of that process, since the PID will change whenever the process restarts.
check process nginx with pidfile /var/run/nginx.pid
start program = "/etc/init.d/nginx start"
stop program = "/etc/init.d/nginx stop"
You'll need to pass Monit a command to start and stop your process. It should automatically restart if the process goes down, with no additional config, but if you want to restart based on resource usage, you'll have to specify that yourself with a few if statements:
check process nginx with pidfile /var/run/nginx.pid
start program = "/etc/init.d/nginx start"
stop program = "/etc/init.d/nginx stop"
if totalmem > 600 MB for 5 cycles then restart
if cpu > 90% for 5 cycles then restart
if 3 restarts within 5 cycles then timeout
Monit will send out alerts whenever something restarts, so you'll be notified when something is wrong.
Once you're done with the config, reload the monit daemon with:
monit reload
And start monitoring with:
monit start all
This should start the web interface as well, which you can use to verify that Monit is working properly, usually running on port 2812.
This is the basic config to get Monit running out of the box, but Monit has a lot more syntax. You can read the full manual on everything Monit can do to learn more.
Configuring Alerts
Monit can be configured to send email alerts whenever major issues occur, or when processes are restarted. You'll need to add configuration for your mail server:
set mailserver smtp.gmail.com
port 587
username "yourserver@gmail.com"
password "secret"
with timeout 60 seconds
set eventqueue
basedir /var/lib/monit/events
slots 100
The first block defines the mail server to use. The easiest method would be to just use smtp.gmail.com with a Gmail account, though you could install Postfix on your server to run locally, or use an enterprise email service.
The second line tells Monit to queue alerts, if the mail server isn't reachable for whatever reason.
Obviously, you'll also need to set the email address it sends to:
set alert youremail@gmail.com
Additionally, since Monit has the ability to run scripts with the exec command, you can set up custom alerts however you'd like. For example, you could set up a script to send you Slack notifications, store it as /usr/local/bin/slack-webhook.sh, and run it whenever nginx changes PID or is restarted by Monit:
check process nginx with pidfile /var/run/nginx.pid
start program = "/etc/init.d/nginx start"
stop program = "/etc/init.d/nginx stop"
if changed pid then exec "/bin/bash -c 'PROCESS=Nginx /usr/local/bin/slack-webhook.sh'"
if 1 restart within 1 cycle then exec "/bin/bash -c 'PROCESS=Nginx /usr/local/bin/slack-webhook.sh'"
Note that all paths must be fully qualified, including /bin/bash and paths to executables in your local bin. And after you're done modifying the config files, you'll need to reload Monit again.