Winston sat in the darkened room. All around him lights blinked and flashed.
Regardless of what it did, Big Brother knew. There was nothing he could do,
without Big Brother being aware. There was also the chance that should Winston
do something that Big Brother did not like, alarms would sound and people would
come storming into the room. Thank goodness , Winston just sat quietly.
Although this sounds like it comes right out of George Orwell's 1984, we're
talking about a different Winston and different Big Brother. In fact, Winston
can go by many names. However, regardless of what name is used, it is under the
ever present watch of a real-life Big Brother, who watches and waits for one
Fortunately, this Big Brother is truly your friend. However, like
its Orwellian namesake, it is constantly on the look-out for things it doesn't
like, waiting to sound an alarm. What we are talking about is a systems
monitoring tool developed by Sean MacGuire of The MacLawran Group
In essence, there is not much that Big Brother
does, which you cannot find in commercial monitoring tools. Although many
commercial products are available with more features, Big Brother has a number
of advantages which makes it ideal for many environments. One of the most
compelling reasons to use Big Brother is its simplicity. It is composed of just
a handful of scripts and programs, which the clients use to gather information
and report it to a central server where the information is collected and
displayed in an extremely accessible format (see figure 1).
configuration and scripts themselves are very easy to change which allows you to
fit Big Brother to your current needs, while at the same time giving you room to
grow, without putting a strain on your budget. Although it is not covered
directly by the General Public License, you can download for free from the
MacLawran Group's web site. It is covered by a "fair use" license however to
redistribute it, you need written permission from the MacLawran Group. See their
web site for more details.
What Big Brother is Made of
Big brother is a
made up of five key components. There is a central monitoring station or display
server, which receives incoming messages from the clients, processes them and
makes them available in the form of Web pages. This means that although the
display server currently run only on UNIX
or Linux machines, you can monitor
from anywhere. (Shell scripts refer to this server using the BBDISPLAY
Network connectivity is monitored from the server using a shell
script (bb-network.sh), which tries to contact the various clients using a
simple ping. In addition, you can configure Big Brother to check for
connectivity to specific services on the client,
such as HTTP,
FTP, and SMTP.
The machine that checks connectivity does not necessarily need to be the display
server and is referred to in the scripts as BBNET. In some cases, it might make
sense to have the BBNET machine sit on a machine acting as a router
the information it gathers to a server inside a firewall, for example, which
would allow you to check connectivity to the Internet, as well.
cases, checking connectivity to the machine is sufficient. If so, you do not
need to do any further configuration as all the work is done by the server.
However, if the machine needs to report information to the display server (such
as disk space, running processes, etc.), you need to configure the client.
This is accomplished by the "Local System Monitor" component which is embodied in the
shell script bb-local.sh.
Big Brother can also be configured to send pager
messages to the system administrator,
based on any monitored event.
When one of
those events occur , the machine where the event
occurred uses the bb-page.sh
shell script to send a message to the BBPAGER server, which then uses Kermit to
contact the pager via modem.
Like the BBNETserver, the BBPAGER server does not
need to be the same machine as the display server. Note that Kermit is not
included as part of the big brother package. Finally, there are a handful of
support programs, such as the Big Brother daemon
(bbd) which sits on the various
servers and clients program (bb), which sends the appropriate messages to the
display and pager servers.
Keep in mind that the exact same event
different machines can have different meanings depending on your
For example, a database server, which stores all of the data in a single file
may take up an entire filesystem
and from the operating system's standpoint, the
size never changes. Therefore, the filesystem
might be constantly at 99% full.
As a result, you probably won't want report it. However, a file server that
reaches 80% full might generate a warning and then a "panic" when the file
system reaches 95% full.
Another important aspect is that monitoring is not
just limited to UNIX
and Linux machines. Instead clients are available for
Windows NT and NetWare. Note that only the binaries are provided for Windows NT
and NetWare clients.
The primary configuration file is etc/bb-hosts, which
sits under the directory where you install Big Brother. By default, this is
/usr/local/bb, but can be changed when you compile
the source for your specific
system. The bb-hosts file has a similar structure to the /etc/hosts, but also
include information about what whether or not connectivity should be checked,
and if so, specifically what network
services should be looked at.
of the pages are prepared using HTML,
your BBDISPLAY server needs to have a an
HTTP server running on it in order to server the pages to the clients. Big
Brother updates the page index.html in regular intervals so the information is
always current. However, this does not mean you have to keep pressing the
refresh or update button to see the current information, as each page is
automatically updated every 60 seconds.
At the top of the page in the figure below is
a legend, describing what each of the different colors mean, along with the time
the page was last updated and links to various sources of information. Clicking
on the picture of Big Brother (actually Sean MacGuire), brings you to the Big
Brother home page.
Big Brother also has the ability to group machines, so
that they are displayed in a separate table. Each column represents one specific
aspect being monitored, with the rows being the particular system. If something
is not monitored on a particular system, there will be a dash in that cell of
the table. Otherwise, there will be one of the colored balls listed in the
legend. If you had it up in a browser, you would see that the yellow and red
balls are blinking, which gives them an additional visual effect.
the column headings (what is being monitored) are links to a help page. If you
click on them, you are brought to the exact spot for that particular aspect,
which gives you details about what is being monitored, as well as what script is
used to do the monitoring and in some cases specific information about how the
monitoring is being done.
Setting up Display Server
The first step is to
get Big Brother and extract the files. Configure it for your system by changing
into the ./doc directory and running ./bbconfig. , where is the name of
the your OS. Leave off the OS name for a list of supported systems. However, you
need to make sure you get this right or Big Brother will not compile
Next, change to the ./src under the Big Brother root. Run "make" to compile
all of the components, then "make install" to install them in their default
location. If you need to change the location, you can change the location in the
After the binaries are installed, edit runbb.sh in the Big Brother
root directory and set the BBHOME variable
to the directory where you installed
Big Brother. This is extremely important as the entire functionality centers
around this variable.
Next,change to ./etc and edit the bb-hosts file and
determines what aspects f the system will be monitored. It has a structure
similar to a tradition hosts file, but is broken into three parts:
IP-ADDR HOSTNAME # DIRECTIVES
If you have turned on fully qualified domain
variable), then the HOSTNAME **must** also contain the domain.
DIRECTIVES is essentially a list of what needs to be monitored on the remote
site. If this machine is going to be one of the servers, then you should set the
appropriate variable(s) in the list of directives (BBDISPLAY, BBPAGER or BBNET
depending of what kind of server this is. Table 1 shows you a list of the more
common directives. For some examples, check the bb-hosts file provided for you.
(You will to have to change it anyway). Finally, run ./bbchkcfg.sh and
./bbchkhosts.sh to check for errors in the configuration files.
that the Big Brother information is displayed by the HTTPD server. Therefore you
need to tell your web server where to get the Big Brother pages. This is
normally done by making a symbolic link
between the Big Brother root directory
and somewhere underneath your DocumentRoot for your web server. Commonly this is
just the DocumentRoot/bb, so all you have to do is enter and you're in. Note
that you must make sure that the server is configure to follow the symbolic
When you think you have this configured correctly, move back to the
Big Brother root directory and start Big Brother by running the script
./runbb.sh. Like many script which starts services, you can use the arguments
start, stop and restart. Note that by default, Big Brother is not configured to
run when the system boots, so you probably need to add something to your
/etc/rc.d directory (or where the start-up scripts are on your machine). If BB
fails to start, check the file BBOUT for any errors. At this point, the server
display server should be ready to go.
Setting up the clients
clients is very easy if they have the same operating system
hardware as the server. If you have different operating systems, then you have
seen that the same program will deliver a different output, sometimes with the
same options. Therefore, you need to have each program deliver the information
in a format that Big Brother can understand. That's why you need to configure
Big Brother at the very beginning.
Even if your machines are different, you
should configure your bb-hosts file to list all of the clients. Although you
**could ** have a different bb-hosts file on each machine, I find it is easier
to use the same file on each machine, although it may require the bb-hosts file
to be edited if you want to monitor different things on each client.
the bb-hosts file is configure correctly, you create a "tarball" for the client
using the script doc/bbclient . This tarball contains the necessary programs and
files. Copy the tarball into the BBHOME directory on the client
and unpack it.
(Note you could have different BBHOMES on the clients, but I think that just
makes things harder to manage.)
If you have different platforms, you will
need to install a client
on one machine for each different platform and then
create a tarball, which is then copied to the other clients of that type. Make
sure to copy your master bb-hosts file (it has a common format across platforms)
and check your configuration using bbchkcfg.sh and bbchkhosts.cfg.
the steps are fairy simple, configuring a Windows NT clients is completely
different that for UNIX.
As of this writing, the Big Brother client
available in binary
form as a zip archive, which you need to unpack on a local
drive. Both Alpha (axp) and Intel (x86) versions are available and you need to
rename the appropriate one for your system. For example, on an Intel machine you
ren bbnt-x86.exe bbnt.exe
You next run bbnt.exe to install
bbnt [-y] -install BBDISPLAY FQDN
Here BBDISPLAY is
of the Big Brother display server. FQDN
is either Y or N
depending on whether or not Big Brother should return the Fully qualified domain
name of the machine. IPPORT is the port used for communication between the local
Big Brother client
and the Big Brother server. It is important that you use the
exact same port as the server has, otherwise the server will not get updated.
Note that the -y option simply tells Big Brother to install without prompting
Note that that Big Brother runs as a system service under
windows NT. Once installed it can be managed from either the Control Panel ->
Services, Server Manager->Services, or the command line
using the net command
(e.g. net stop, net start, and so on). Because it is running as a service,
there is a potential security
problem should the bbnt.exe program get replaced.
Therefore, you need to make sure that it is readable, executable and writable
by an administrative account
The NT client
also has two addition
options to either upgrade (-upgrade) or remove Big Brother completely (-remove).
In order to avoid any problems it is a good idea to stop the service before you
try to remove or upgrade it.
Configuring Big Brother for your site
NT clients has a very comfortable GUI
configuration interface (bbcgf.exe). In
general, the available configuration options are self-explanatory. However,
detailed configuration instructions are available in the included README file.
I have managed several machines where the default configuration on UNIX
machines is sufficient. However, Big Brother has a number of different
parameters which you can use. The primary configuration file is etc/bbdef.sh
(the BB definitions file), which bbrun.sh reads when it starts up. Here you
define not only basic parameters (like whether or not to display the
fully-qualified names), but also specific behavior such as how full the disk
needs to be before it is reported.
By default, Big Brother monitors your
filesystems and reports when they get too full (90% warning, 95% panic). These
levels can be changed globally by setting the DFWARN and DFPANIC variables to
different values, depending on your needs. One useful aspect is the ability to
define these levels on a per filesystem
basis. For example, many sites have
databases which take up entire partitions. Although the information in the
database changes, how much space is taken up doesn't. If DFPANIC is set to 95%,
for example, but the filesystem
is **always** at 99%, you will get a lot of
unnecessary messages. The trick is to copy the file etc/bb-dftab INFO to
etc/bb-dftab and edit to suit your needs.
Similarly, the CPUWARN and
CPUPANIC are used to report on CPU
activity. These are based on the load average
as reported by the uptime command (and then multiplied by 100). By default,
these are set to 150 and 300 respectively, and can also be changed in the
You can also monitor specific processes to see if they are
running on the client.
The PROCS variable
defines which processes to monitor and
report as a warning if they are not running, whereas the PAGEPROC defines which
are defined as a panic.
The intent here is that should the specified process not
be running, Big Brother will page the administrator.
The MSGS and PAGEMSGS
are used to monitor log files. By default, MSGS is set to "NOTICE WARNING".
This means that only messages containing NOTICE or WARNING are looked at.
PAGEMSG is set to "NOTICE", which means that should such a message appear, it
generates a red condition (panic) and would page an administrator
if the system
is so configured. There is also the IGNMSGS variable,
which tells Big Brother
which messages to specifically ignore.
Keep in mind that the runbb.sh script
reads the configuration information from bbdef.sh file when it is started.
Therefore, unlike other Linux daemons, changes are not recognized until Big
Brother is restarted. Therefore, after each change, you will have to restart Big
Although you probably will not need to change it, another important
file is etc/bbsys.sh. This contains information about your specific operating
system, such as the location of programs on your system, and which arguments to
use. My suggestion is that you do not mess with this file unless you know what
you are doing. It is generated for you when you install Big Brother and there is
normally no need to change it.
Another important variable
is BBTMP which
defines the location of the directory, which Big Brother users to store
temporary files. By default this is $BBHOME/tmp and there is generally no reason
to change it. However, if you do want it somewhere else, you need to make sure
that normal users do not have write access as it represents a potential security
Configuration of Big Brother goes beyond just setting variables.
Obviously, because you have the source code and it is mostly scripts, you can
define the behavior anyway you want. The problem with changing the scripts and
programs is that you will probably have to re-do a lot of work, when Big
Brother gets updated. Instead, Big Brother provides a mechanism whereby you can
define your own tests. That is, you can configure Big Brother to run addition or
"external" tests. These are located in the $BBHOME/ext directory and a template
is provided to show you the general syntax.
The good news
**and** the bad news is that's not all. It's good for system administrators
because there is a lot more you can do with Big Brother than we discussed here.
It's bad for me because we don't have the space here to go into all of the
details. You already have the basics to install Big Brother and be able to
monitor the key aspects of your system.
One thing we did not cover much is
the notification features. As I mentioned, Big Brother can be configured to send
email, pages (including the Short Message Service - SMS), and other kinds of
notification to administrator
when certain events occur. Unfortunately, being
as powerful as it is, discussing this topic requires more space that we have.
My suggested is that once you have it running, you should investigate the
scripts and configuration files. Once you have Big Brother running (maybe even before
you install it), I would suggest you subscribe to the Big Brother mailing list.
(send email to email@example.com with the **body** of the message
"subscribe bb"). This provides a great forum for asking your questions, as well
as getting some neat tips and tricks from more advanced users. Both the scripts
and the mailing list provide a wealth of information about what Big Brother can
already done and what more you can do with it.
Table 1 - Common bb-host
BBDISPLAY Central server where information is displayed.
BBPAGER Server which sends pages to administrators.
BBNET Host which check connectivity to the network
http://www_path Host and path to check
for http connections. Multiple paths can be specified by separating using a pipe
ftp Check the ftp service
smtp Check the smtp server
pop3 Check the pop3 server
telnet Check the telnet service
ssh Check the ssh server
nntp Check the nntp server
dns Checks for name resolution server
noping Don't do ping test for this host
dialup If host is down then display clear button
NOTE: The service name must be written exactly the was it does in
/etc/services. On some system the POP3 server is written ''pop-3''.