Welcome to Linux Knowledge Base and Tutorial
"The place where you learn linux"
Karen Lilly Creations

 Create an AccountHome | Submit News | Your Account  

Tutorial Menu
Linux Tutorial Home
Table of Contents
Up to --> Linux Tutorial

· System Monitoring
· Finding Out About Your System
· What the System Is Doing Now
· Big Brother

Glossary
MoreInfo
Man Pages
Linux Topics
Test Your Knowledge

Site Menu
Site Map
FAQ
Copyright Info
Terms of Use
Privacy Info
Disclaimer
WorkBoard
Thanks
Donations
Advertising
Masthead / Impressum
Your Account

Communication
Feedback
Forums
Private Messages
Recommend Us
Surveys

Features
HOWTOs
News
News Archive
Submit News
Topics
User Articles
Web Links

Google
Google


The Web
linux-tutorial.info

Who's Online
There are currently, 151 guest(s) and 1 member(s) that are online.

You are an Anonymous user. You can register for free by clicking here

  
Linux Tutorial - System Monitoring - Big Brother
  Checking Other Things ---- Solving Problems  


Big Brother

Winston sat in the darkened room. All around him lights blinked and flashed. Regardless of what it did, Big Brother knew. There was nothing he could do, without Big Brother being aware. There was also the chance that should Winston do something that Big Brother did not like, alarms would sound and people would come storming into the room. Thank goodness , Winston just sat quietly.

Although this sounds like it comes right out of George Orwell's 1984, we're talking about a different Winston and different Big Brother. In fact, Winston can go by many names. However, regardless of what name is used, it is under the ever present watch of a real-life Big Brother, who watches and waits for one false move.

Fortunately, this Big Brother is truly your friend. However, like its Orwellian namesake, it is constantly on the look-out for things it doesn't like, waiting to sound an alarm. What we are talking about is a systems monitoring tool developed by Sean MacGuire of The MacLawran Group (www.maclawran.ca//bb-dnld/).

In essence, there is not much that Big Brother does, which you cannot find in commercial monitoring tools. Although many commercial products are available with more features, Big Brother has a number of advantages which makes it ideal for many environments. One of the most compelling reasons to use Big Brother is its simplicity. It is composed of just a handful of scripts and programs, which the clients use to gather information and report it to a central server where the information is collected and displayed in an extremely accessible format (see figure 1).

The configuration and scripts themselves are very easy to change which allows you to fit Big Brother to your current needs, while at the same time giving you room to grow, without putting a strain on your budget. Although it is not covered directly by the General Public License, you can download for free from the MacLawran Group's web site. It is covered by a "fair use" license however to redistribute it, you need written permission from the MacLawran Group. See their web site for more details.

What Big Brother is Made of

Big brother is a made up of five key components. There is a central monitoring station or display server, which receives incoming messages from the clients, processes them and makes them available in the form of Web pages. This means that although the display server currently run only on UNIX or Linux machines, you can monitor from anywhere. (Shell scripts refer to this server using the BBDISPLAY variable.)

Network connectivity is monitored from the server using a shell script (bb-network.sh), which tries to contact the various clients using a simple ping. In addition, you can configure Big Brother to check for connectivity to specific services on the client, such as HTTP, FTP, and SMTP. The machine that checks connectivity does not necessarily need to be the display server and is referred to in the scripts as BBNET. In some cases, it might make sense to have the BBNET machine sit on a machine acting as a router and direct the information it gathers to a server inside a firewall, for example, which would allow you to check connectivity to the Internet, as well.

It some cases, checking connectivity to the machine is sufficient. If so, you do not need to do any further configuration as all the work is done by the server. However, if the machine needs to report information to the display server (such as disk space, running processes, etc.), you need to configure the client. This is accomplished by the "Local System Monitor" component which is embodied in the shell script bb-local.sh.

Big Brother can also be configured to send pager messages to the system administrator, based on any monitored event. When one of those events occur , the machine where the event occurred uses the bb-page.sh shell script to send a message to the BBPAGER server, which then uses Kermit to contact the pager via modem. Like the BBNETserver, the BBPAGER server does not need to be the same machine as the display server. Note that Kermit is not included as part of the big brother package. Finally, there are a handful of support programs, such as the Big Brother daemon (bbd) which sits on the various servers and clients program (bb), which sends the appropriate messages to the display and pager servers.

Keep in mind that the exact same event on two different machines can have different meanings depending on your environment. For example, a database server, which stores all of the data in a single file may take up an entire filesystem and from the operating system's standpoint, the size never changes. Therefore, the filesystem might be constantly at 99% full. As a result, you probably won't want report it. However, a file server that reaches 80% full might generate a warning and then a "panic" when the file system reaches 95% full.

Another important aspect is that monitoring is not just limited to UNIX and Linux machines. Instead clients are available for Windows NT and NetWare. Note that only the binaries are provided for Windows NT and NetWare clients.

The primary configuration file is etc/bb-hosts, which sits under the directory where you install Big Brother. By default, this is /usr/local/bb, but can be changed when you compile the source for your specific system. The bb-hosts file has a similar structure to the /etc/hosts, but also include information about what whether or not connectivity should be checked, and if so, specifically what network services should be looked at.

Since all of the pages are prepared using HTML, your BBDISPLAY server needs to have a an HTTP server running on it in order to server the pages to the clients. Big Brother updates the page index.html in regular intervals so the information is always current. However, this does not mean you have to keep pressing the refresh or update button to see the current information, as each page is automatically updated every 60 seconds.

At the top of the page in the figure below is a legend, describing what each of the different colors mean, along with the time the page was last updated and links to various sources of information. Clicking on the picture of Big Brother (actually Sean MacGuire), brings you to the Big Brother home page.

Big Brother also has the ability to group machines, so that they are displayed in a separate table. Each column represents one specific aspect being monitored, with the rows being the particular system. If something is not monitored on a particular system, there will be a dash in that cell of the table. Otherwise, there will be one of the colored balls listed in the legend. If you had it up in a browser, you would see that the yellow and red balls are blinking, which gives them an additional visual effect.

Note that the column headings (what is being monitored) are links to a help page. If you click on them, you are brought to the exact spot for that particular aspect, which gives you details about what is being monitored, as well as what script is used to do the monitoring and in some cases specific information about how the monitoring is being done.

Setting up Display Server

The first step is to get Big Brother and extract the files. Configure it for your system by changing into the ./doc directory and running ./bbconfig. , where is the name of the your OS. Leave off the OS name for a list of supported systems. However, you need to make sure you get this right or Big Brother will not compile correctly.

Next, change to the ./src under the Big Brother root. Run "make" to compile all of the components, then "make install" to install them in their default location. If you need to change the location, you can change the location in the Makefile.

After the binaries are installed, edit runbb.sh in the Big Brother root directory and set the BBHOME variable to the directory where you installed Big Brother. This is extremely important as the entire functionality centers around this variable. Next,change to ./etc and edit the bb-hosts file and determines what aspects f the system will be monitored. It has a structure similar to a tradition hosts file, but is broken into three parts:

IP-ADDR HOSTNAME # DIRECTIVES

If you have turned on fully qualified domain names (the FQDN variable), then the HOSTNAME **must** also contain the domain. DIRECTIVES is essentially a list of what needs to be monitored on the remote site. If this machine is going to be one of the servers, then you should set the appropriate variable(s) in the list of directives (BBDISPLAY, BBPAGER or BBNET depending of what kind of server this is. Table 1 shows you a list of the more common directives. For some examples, check the bb-hosts file provided for you. (You will to have to change it anyway). Finally, run ./bbchkcfg.sh and ./bbchkhosts.sh to check for errors in the configuration files.

Remember that the Big Brother information is displayed by the HTTPD server. Therefore you need to tell your web server where to get the Big Brother pages. This is normally done by making a symbolic link between the Big Brother root directory and somewhere underneath your DocumentRoot for your web server. Commonly this is just the DocumentRoot/bb, so all you have to do is enter and you're in. Note that you must make sure that the server is configure to follow the symbolic links.

When you think you have this configured correctly, move back to the Big Brother root directory and start Big Brother by running the script ./runbb.sh. Like many script which starts services, you can use the arguments start, stop and restart. Note that by default, Big Brother is not configured to run when the system boots, so you probably need to add something to your /etc/rc.d directory (or where the start-up scripts are on your machine). If BB fails to start, check the file BBOUT for any errors. At this point, the server display server should be ready to go.

Setting up the clients

Configuring the UNIX clients is very easy if they have the same operating system and hardware as the server. If you have different operating systems, then you have seen that the same program will deliver a different output, sometimes with the same options. Therefore, you need to have each program deliver the information in a format that Big Brother can understand. That's why you need to configure Big Brother at the very beginning.

Even if your machines are different, you should configure your bb-hosts file to list all of the clients. Although you **could ** have a different bb-hosts file on each machine, I find it is easier to use the same file on each machine, although it may require the bb-hosts file to be edited if you want to monitor different things on each client.

Once the bb-hosts file is configure correctly, you create a "tarball" for the client using the script doc/bbclient . This tarball contains the necessary programs and files. Copy the tarball into the BBHOME directory on the client and unpack it. (Note you could have different BBHOMES on the clients, but I think that just makes things harder to manage.)

If you have different platforms, you will need to install a client on one machine for each different platform and then create a tarball, which is then copied to the other clients of that type. Make sure to copy your master bb-hosts file (it has a common format across platforms) and check your configuration using bbchkcfg.sh and bbchkhosts.cfg.

Although the steps are fairy simple, configuring a Windows NT clients is completely different that for UNIX. As of this writing, the Big Brother client is only available in binary form as a zip archive, which you need to unpack on a local drive. Both Alpha (axp) and Intel (x86) versions are available and you need to rename the appropriate one for your system. For example, on an Intel machine you would run:

ren bbnt-x86.exe bbnt.exe

You next run bbnt.exe to install the program.

bbnt [-y] -install BBDISPLAY FQDN IPPORT

Here BBDISPLAY is the IP address of the Big Brother display server. FQDN is either Y or N depending on whether or not Big Brother should return the Fully qualified domain name of the machine. IPPORT is the port used for communication between the local Big Brother client and the Big Brother server. It is important that you use the exact same port as the server has, otherwise the server will not get updated. Note that the -y option simply tells Big Brother to install without prompting for confirmation.

Note that that Big Brother runs as a system service under windows NT. Once installed it can be managed from either the Control Panel -> Services, Server Manager->Services, or the command line using the net command (e.g. net stop, net start, and so on). Because it is running as a service, there is a potential security problem should the bbnt.exe program get replaced. Therefore, you need to make sure that it is readable, executable and writable by an administrative account **only**.

The NT client also has two addition options to either upgrade (-upgrade) or remove Big Brother completely (-remove). In order to avoid any problems it is a good idea to stop the service before you try to remove or upgrade it.

Configuring Big Brother for your site

The NT clients has a very comfortable GUI configuration interface (bbcgf.exe). In general, the available configuration options are self-explanatory. However, detailed configuration instructions are available in the included README file.

I have managed several machines where the default configuration on UNIX machines is sufficient. However, Big Brother has a number of different parameters which you can use. The primary configuration file is etc/bbdef.sh (the BB definitions file), which bbrun.sh reads when it starts up. Here you define not only basic parameters (like whether or not to display the fully-qualified names), but also specific behavior such as how full the disk needs to be before it is reported.

By default, Big Brother monitors your filesystems and reports when they get too full (90% warning, 95% panic). These levels can be changed globally by setting the DFWARN and DFPANIC variables to different values, depending on your needs. One useful aspect is the ability to define these levels on a per filesystem basis. For example, many sites have databases which take up entire partitions. Although the information in the database changes, how much space is taken up doesn't. If DFPANIC is set to 95%, for example, but the filesystem is **always** at 99%, you will get a lot of unnecessary messages. The trick is to copy the file etc/bb-dftab INFO to etc/bb-dftab and edit to suit your needs.

Similarly, the CPUWARN and CPUPANIC are used to report on CPU activity. These are based on the load average as reported by the uptime command (and then multiplied by 100). By default, these are set to 150 and 300 respectively, and can also be changed in the bbdef.sh file.

You can also monitor specific processes to see if they are running on the client. The PROCS variable defines which processes to monitor and report as a warning if they are not running, whereas the PAGEPROC defines which are defined as a panic. The intent here is that should the specified process not be running, Big Brother will page the administrator.

The MSGS and PAGEMSGS are used to monitor log files. By default, MSGS is set to "NOTICE WARNING". This means that only messages containing NOTICE or WARNING are looked at. PAGEMSG is set to "NOTICE", which means that should such a message appear, it generates a red condition (panic) and would page an administrator if the system is so configured. There is also the IGNMSGS variable, which tells Big Brother which messages to specifically ignore.

Keep in mind that the runbb.sh script reads the configuration information from bbdef.sh file when it is started. Therefore, unlike other Linux daemons, changes are not recognized until Big Brother is restarted. Therefore, after each change, you will have to restart Big Brother.

Although you probably will not need to change it, another important file is etc/bbsys.sh. This contains information about your specific operating system, such as the location of programs on your system, and which arguments to use. My suggestion is that you do not mess with this file unless you know what you are doing. It is generated for you when you install Big Brother and there is normally no need to change it.

Another important variable is BBTMP which defines the location of the directory, which Big Brother users to store temporary files. By default this is $BBHOME/tmp and there is generally no reason to change it. However, if you do want it somewhere else, you need to make sure that normal users do not have write access as it represents a potential security hole.

Configuration of Big Brother goes beyond just setting variables. Obviously, because you have the source code and it is mostly scripts, you can define the behavior anyway you want. The problem with changing the scripts and programs is that you will probably have to re-do a lot of work, when Big Brother gets updated. Instead, Big Brother provides a mechanism whereby you can define your own tests. That is, you can configure Big Brother to run addition or "external" tests. These are located in the $BBHOME/ext directory and a template is provided to show you the general syntax.

Next Steps

The good news **and** the bad news is that's not all. It's good for system administrators because there is a lot more you can do with Big Brother than we discussed here. It's bad for me because we don't have the space here to go into all of the details. You already have the basics to install Big Brother and be able to monitor the key aspects of your system.

One thing we did not cover much is the notification features. As I mentioned, Big Brother can be configured to send email, pages (including the Short Message Service - SMS), and other kinds of notification to administrator when certain events occur. Unfortunately, being as powerful as it is, discussing this topic requires more space that we have.

My suggested is that once you have it running, you should investigate the scripts and configuration files. Once you have Big Brother running (maybe even before you install it), I would suggest you subscribe to the Big Brother mailing list. (send email to majordomo@taex001.tamu.edu with the **body** of the message "subscribe bb"). This provides a great forum for asking your questions, as well as getting some neat tips and tricks from more advanced users. Both the scripts and the mailing list provide a wealth of information about what Big Brother can already done and what more you can do with it.

Table 1 - Common bb-host directives

BBDISPLAY Central server where information is displayed.
BBPAGER Server which sends pages to administrators.
BBNET Host which check connectivity to the network services.
http://www_path Host and path to check
for http connections. Multiple paths can be specified by separating using a pipe ('|).
ftp Check the ftp service
smtp Check the smtp server
pop3 Check the pop3 server
telnet Check the telnet service
ssh Check the ssh server
nntp Check the nntp server
dns Checks for name resolution server
noping Don't do ping test for this host
dialup If host is down then display clear button

NOTE: The service name must be written exactly the was it does in /etc/services. On some system the POP3 server is written ''pop-3''.

 Previous Page
Checking Other Things
  Back to Top
Table of Contents
Next Page 
Solving Problems


MoreInfo

Test Your Knowledge

User Comments:


Posted by scooter on June 29, 2004 03:42pm:

You should also take a look at Nagios.


You can only add comments if you are logged in.

Copyright 2002-2009 by James Mohr. Licensed under modified GNU Free Documentation License (Portions of this material originally published by Prentice Hall, Pearson Education, Inc). See here for details. All rights reserved.
  
Help us cut cost by not downloading the whole site!
Use of automated download sofware ("harvesters") such as wget, httrack, etc. causes the site to quickly exceed its bandwidth limitation and therefore is expressedly prohibited. For more details on this, take a look here

Login
Nickname

Password

Security Code
Security Code
Type Security Code


Don't have an account yet? You can create one. As a registered user you have some advantages like theme manager, comments configuration and post comments with your name.

Help if you can!


Amazon Wish List

Did You Know?
You can get all the latest Site and Linux news by checking out our news page.


Friends



Tell a Friend About Us

Bookmark and Share



Web site powered by PHP-Nuke

Is this information useful? At the very least you can help by spreading the word to your favorite newsgroups, mailing lists and forums.
All logos and trademarks in this site are property of their respective owner. The comments are property of their posters. Articles are the property of their respective owners. Unless otherwise stated in the body of the article, article content (C) 1994-2013 by James Mohr. All rights reserved. The stylized page/paper, as well as the terms "The Linux Tutorial", "The Linux Server Tutorial", "The Linux Knowledge Base and Tutorial" and "The place where you learn Linux" are service marks of James Mohr. All rights reserved.
The Linux Knowledge Base and Tutorial may contain links to sites on the Internet, which are owned and operated by third parties. The Linux Tutorial is not responsible for the content of any such third-party site. By viewing/utilizing this web site, you have agreed to our disclaimer, terms of use and privacy policy. Use of automated download software ("harvesters") such as wget, httrack, etc. causes the site to quickly exceed its bandwidth limitation and are therefore expressly prohibited. For more details on this, take a look here

PHP-Nuke Copyright © 2004 by Francisco Burzi. This is free software, and you may redistribute it under the GPL. PHP-Nuke comes with absolutely no warranty, for details, see the license.
Page Generation: 0.13 Seconds