Device Drivers

One of the purposes of an operating system is to hide the peculiarities of the system's hardware devices from its users. For example the Virtual File System presents a uniform view of the mounted filesystems irrespective of the underlying physical devices. This chapter describes how the Linux kernel manages the physical devices in the system.

The CPU is not the only intelligent device in the system, every physical device has its own hardware controller. The keyboard, mouse and serial ports are controlled by a SuperIO chip, the IDE disks by an IDE controller, SCSI disks by a SCSI controller and so on. Each hardware controller has its own control and status registers (CSRs) and these differ between devices. The CSRs for an Adaptec 2940 SCSI controller are completely different from those of an NCR 810 SCSI controller. The CSRs are used to start and stop the device, to initialize it and to diagnose any problems with it. Instead of putting code to manage the hardware controllers into every application, the code is kept in the Linux kernel. The software that handles or manages a hardware controller is known as a device driver. The Linux kernel device drivers are, essentially, a shared library of privileged, memory resident, low level hardware handling routines. It is Linux's device drivers that handle the peculiarities of the devices they are managing.

Linux supports three types of hardware devices: character, block and network. Character devices are read and written directly without buffering, for example the system's serial ports /dev/cua0 and /dev/cua1. Block devices can only be written to and read from in multiples of the block size, typically 512 or 1024 bytes. Block devices are accessed via the buffer cache and may be randomly accessed, that is to say, any block can be read or written no matter where it is on the device. Block devices can be accessed via their device special file but more commonly they are accessed via the file system. Only a block device can support a mounted file system. Network devices are accessed via the BSD socket interface and the networking subsytems described in the section on networking.

There are many different device drivers in the Linux kernel (that is one of Linux's strengths) but they all share some common attributes:

Kernel code

Device drivers are part of the kernel and, like other code within the kernel, if they go wrong they can seriously damage the system. A badly written driver may even crash the system, possibly corrupting file systems and losing data.

Kernel interfaces

Device drivers provide a standard interface to the Linux kernel or to the appropriate subsystem. For example, the terminal driver provides a file I/O interface to the Linux kernel and a SCSI device driver provides a SCSI device interface to the SCSI subsystem which, in turn, provides both file I/O and buffer cache interfaces to the kernel.

Kernel mechanisms and services

Device drivers make use of standard kernel services such as memory allocation, interrupt delivery and wait queues to operate.

Loadable

Most of the Linux device drivers can be loaded on demand as kernel modules when they are needed and unloaded when they are no longer being used. This makes the kernel very adaptable and efficient with the system's resources.

Configurable

Linux device drivers can be built into the kernel. Which devices are built is configurable when the kernel is compiled.

Dynamic

As the system boots and each device driver is initialized it looks for the hardware devices that it will control. If those devices do not exist (maybe are just not found), the device driver is simply redundant and causes no harm apart from occupying a little of the system's memory.

Memory

Linux provides kernel memory allocation and deallocation routines and it is these that the device drivers use. Kernel memory is allocated in chunks that are powers of 2. For example 128 or 512 bytes, even if the device driver asks for less. The number of bytes that the device driver requests is rounded up to the next block size boundary. This makes kernel memory deallocation easier as the smaller free blocks can be recombined into bigger blocks.

It may be that Linux needs to do quite a lot of extra work when the kernel memory is requested. If the amount of free memory is low, physical pages may need to be discarded or written to the swap device. Normally, Linux would suspend the requestor, putting the process onto a wait queue until there is enough physical memory. Not all device drivers (or indeed Linux kernel code) may want this to happen and so the kernel memory allocation routines can be requested to fail if they cannot immediately allocate memory.

The device driver can also specify that it wants to DMA to and from the memory it allocates. It is the Linux kernel, not the device driver, however, that takes care of the details. This way it is the Linux kernel rather than the device driver decides what constitutes DMA'able memory for the system.

Interfacing Device Drivers with the Kernel

Linux is very dynamic, every time a Linux kernel boots it may encounter different physical devices and thus need different device drivers. Linux allows you to include device drivers at kernel build time via its configuration scripts. When these drivers are initialized at boot time they may not discover any hardware to control. Other drivers can be loaded as kernel modules when they are needed. To cope with this dynamic nature of device drivers, device drivers register themselves with the kernel as they are initialized. Linux maintains tables of registered device drivers as part of its interfaces with them. These tables include pointers to routines and information that supports the interface with the device class.

Character Devices

Figure: Character Devices

Character devices, the simplest of Linux's devices, are accessed as files, applications use standard system calls to open them, read from them, write to them and close them exactly as if the device were a file. This is true even if the device is a modem being used by the PPP daemon to connect a Linux system onto a network. As a character device is initialized its device driver registers itself with the Linux kernel by adding an entry into the chrdevs vector of device_struct data structures. The device's major device identifier (for example 4 for the tty device) is used as an index into this vector. The major device identifier for a device is fixed.

Each entry in the chrdevs vector, a device_struct data structure contains two elements: a pointer to the name of the registered device driver and a pointer to a block of file operations. This block of file operations is itself the addresses of routines within the character device driver, each of which handles specific file operations such as open, read, write and close. The contents of /proc/devices for character devices is taken from the chrdevs vector.

When a character special file representing a character device (for example /dev/cua0) is opened, the kernel must set things up so that the correct character device driver's file operation routines will be called. Just like an ordinairy file or directory, each device special file is represented by a VFS inode . The VFS inode for a character special file, indeed for all device special files, contains both the major and minor identifiers for the device. This VFS inode was created by the underlying filesystem, for example EXT2, from information in the real filesystem when the device special file's name was looked up.

Each VFS inode has associated with it a set of file operations and these are different depending on the filesystem object that the inode represents. Whenever a VFS inode representing a character special file is created, its file operations are set to the default character device operations.

This has only one file operation, the open file operation. When the character special file is opened by an application the generic open file operation uses the device's major identifier as an index into the chrdevs vector to retrieve the file operations block for this particular device. It also sets up the file data structure describing this character special file, making its file operations pointer point to those of the device driver. Thereafter all of the applications file operations will be mapped to calls to the character devices set of file operations.

Block Devices

Every block device driver must provide an interface to the buffer cache as well as the normal file operations interface. Each block device driver fills in its entry in the blk_dev vector of blk_dev_struct data structures. The index into this vector is, again, the device's major number. The blk_dev_struct data structure consists of the address of a request routine and a pointer to a list of request data structures, each one representing a request from the buffer cache for the driver to read or write a block of data.

Figure: Buffer Cache Block Device Requests

Each time the buffer cache wishes to read or write a block of data to or from a registered device it adds a request data structure onto its blk_dev_struct. The figure above shows that each request has a pointer to one or more buffer_head data structures, each one a request to read or write a block of data. The buffer_head structures are locked (by the buffer cache) and there may be a process waiting on the block operation to this buffer to complete. Each request structure is allocated from a static list, the all_requests list. If the request is being added to an empty request list, the driver's request function is called to start processing the request queue. Otherwise the driver will simply process every request on the request list.

Once the device driver has completed a request it must remove each of the buffer_head structures from the request structure, mark them as up to date and unlock them. This unlocking of the buffer_head will wake up any process that has been sleeping waiting for the block operation to complete. An example of this would be where a file name is being resolved and the EXT2 filesystem must read the block of data that contains the next EXT2 directory entry from the block device that holds the filesystem. The process sleeps on the buffer_head that will contain the directory entry until the device driver wakes it up. The request data structure is marked as free so that it can be used in another block request.

Hard Disks

During initialization Linux maps the topology of the hard disks in the system.It finds out how many hard disks there are and of what type. Additionally, Linux discovers how the individual disks have been partitioned. This is all represented by a list of gendisk data structures pointed at by the gendisk_head list pointer. As each disk subsystem, for example IDE, is initialized it generates gendisk data structures representing the disks that it finds. It does this at the same time as it registers its file operations and adds its entry into the blk_dev data structure. Each gendisk data structure has a unique major device number and these match the major numbers of the block special devices. For example, the SCSI disk subsystem creates a single gendisk entry (``sd'') with a major number of 8, the major number of all SCSI disk devices. Figure  8.3 shows two gendisk entries, the first one for the SCSI disk subsystem and the second for an IDE disk controller. This is ide0, the primary IDE controller.

Although the disk subsystems build the gendisk entries during their initialization they are only used by Linux during partition checking. Instead, each disk subsystem maintains its own data structures which allow it to map device special major and minor device numbers to partitions within physical disks. Whenever a block device is read from or written to, either via the buffer cache or file operations, the kernel directs the operation to the appropriate device using the major device number found in its block special device file (for example /dev/sda2). It is the individual device driver or subsystem that maps the minor device number to the real physical device.

Details of the physical charactistics of hard disks can be found here.

IDE Disks

The most common disks used in Linux systems today are Integrated Disk Electronic or IDE disks. IDE is a disk interface rather than an I/O bus like SCSI. Each IDE controller can support up to two disks, one the master disk and the other the slave disk. The master and slave functions are usually set by jumpers on the disk. The first IDE controller in the system is known as the primary IDE controller, the next the secondary controller and so on. IDE can manage about 3.3 Mbytes per second of data transfer to or from the disk and the maximum IDE disk size is 538Mbytes. Extended IDE, or EIDE, has raised the disk size to a maximum of 8.6 Gbytes and the data transfer rate up to 16.6 Mbytes per second. IDE and EIDE disks are cheaper than SCSI disks and most modern PCs contain one or more on board IDE controllers.

Linux names IDE disks in the order in which it finds their controllers. The master disk on the primary controller is /dev/hda and the slave disk is /dev/hdb. /dev/hdc is the master disk on the secondary IDE controller. The IDE subsystem registers IDE controllers and not disks with the Linux kernel. The major identifier for the primary IDE controller is 3 and is 22 for the secondary IDE controller. This means that if a system has two IDE controllers there will be entries for the IDE subsystem at indices at 3 and 22 in the blk_dev and blkdevs vectors. The block special files for IDE disks reflect this numbering, disks /dev/hda and /dev/hdb, both connected to the primary IDE controller, have a major identifier of 3. Any file or buffer cache operations for the IDE subsystem operations on these block special files will be directed to the IDE subsystem as the kernel uses the major identifier as an index. When the request is made, it is up to the IDE subsystem to work out which IDE disk the request is for. To do this the IDE subsystem uses the minor device number from the device special identifier, this contains information that allows it to direct the request to the correct partition of the correct disk. The device identifier for /dev/hdb, the slave IDE drive on the primary IDE controller is (3,64). The device identifier for the first partition of that disk (/dev/hdb1) is (3,65).

More details of the physical charactistics of IDE drives can be found here.

Initializing the IDE Subsystem

IDE disks have been around for much of the IBM PC's history. Throughout this time the interface to these devices has changed. This makes the initialization of the IDE subsystem more complex than it might at first appear.

The maximum number of IDE controllers that Linux can support is 4. Each controller is represented by an ide_hwif_t data structure in the ide_hwifs vector. Each ide_hwif_t data structure contains two ide_drive_t data structures, one per possible supported master and slave IDE drive. During the initializing of the IDE subsystem, Linux first looks to see if there is information about the disks present in the system's CMOS memory. This is battery backed memory that does not lose its contents when the PC is powered off. This CMOS memory is actually in the system's real time clock device which always runs no matter if your PC is on or off. The CMOS memory locations are set up by the system's BIOS and tell Linux what IDE controllers and drives have been found. Linux retrieves the found disk's geometry from BIOS and uses the information to set up the ide_hwif_t data structure for this drive. More modern PCs use PCI chipsets such as Intel's 82430 VX chipset which includes a PCI EIDE controller. The IDE subsystem uses PCI BIOS callbacks to locate the PCI (E)IDE controllers in the system. It then calls PCI specific interrogation routines for those chipsets that are present.

Once each IDE interface or controller has been discovered, its ide_hwif_t is set up to reflect the controllers and attached disks. During operation the IDE driver writes commands to IDE command registers that exist in the I/O memory space. The default I/O address for the primary IDE controller's control and status registers is 0x1F0 - 0x1F7. These addresses were set by convention in the early days of the IBM PC. The IDE driver registers each controller with the Linux block buffer cache and VFS, adding it to the blk_dev and blkdevs vectors respectively. The IDE drive will also request control of the appropriate interrupt. Again these interrupts are set by convention to be 14 for the primary IDE controller and 15 for the secondary IDE controller. However, they like all IDE details, can be overridden by command line options to the kernel. The IDE driver also adds a gendisk entry into the list of gendisk's discovered during boot for each IDE controller found. This list will later be used to discover the partition tables of all of the hard disks found at boot time. The partition checking code understands that IDE controllers may each control two IDE disks.

SCSI Disks

The SCSI (Small Computer System Interface) bus is an efficient peer-to-peer data bus that supports up to eight devices per bus, including one or more hosts. Each device has to have a unique identifier and this is usually set by jumpers on the disks. Data can be transfered synchronously or asynchronously between any two devices on the bus and with 32 bit wide data transfers up to 40 Mbytes per second are possible. The SCSI bus transfers both data and state information between devices, and a single transaction between an initiator and a target can involve up to eight distinct phases. You can tell the current phase of a SCSI bus from five signals from the bus. The eight phases are:

BUS FREE

No device has control of the bus and there are no transactions currently happening,

ARBITRATION

A SCSI device has attempted to get control of the SCSI bus, it does this by asserting its SCSI identifer onto the address pins. The highest number SCSI identifier wins.

SELECTION

When a device has succeeded in getting control of the SCSI bus through arbitration it must now signal the target of this SCSI request that it wants to send a command to it. It does this by asserting the SCSI identifier of the target on the address pins.

RESELECTION

SCSI devices may disconnect during the processing of a request. The target may then reselect the initiator. Not all SCSI devices support this phase.

COMMAND

6,10 or 12 bytes of command can be transfered from the initiator to the target,

DATA IN, DATA OUT

During these phases data is transfered between the initiator and the target,

STATUS

This phase is entered after completion of all commands and allows the target to send a status byte indicating success or failure to the initiator,

MESSAGE IN, MESSAGE OUT

Additional information is transfered between the initiator and the target.

The Linux SCSI subsystem is made up of two basic elements, each of which is represented by data structures:

host

A SCSI host is a physical piece of hardware, a SCSI controller. The NCR810 PCI SCSI controller is an example of a SCSI host. If a Linux system has more than one SCSI controller of the same type, each instance will be represented by a separate SCSI host. This means that a SCSI device driver may control more than one instance of its controller. SCSI hosts are almost always the initiator of SCSI commands.

Device

The most common set of SCSI device is a SCSI disk but the SCSI standard supports several more types; tape, CD-ROM and also a generic SCSI device. SCSI devices are almost always the targets of SCSI commands. These devices must be treated differently, for example with removable media such as CD-ROMs or tapes, Linux needs to detect if the media was removed. The different disk types have different major device numbers, allowing Linux to direct block device requests to the appropriate SCSI type.

Details of the physical charactistics of SCSI drives can be found here.

Initializing the SCSI Subsystem

Initializing the SCSI subsystem is quite complex, reflecting the dynamic nature of SCSI buses and their devices. Linux initializes the SCSI subsystem at boot time; it finds the SCSI controllers (known as SCSI hosts) in the system and then probes each of their SCSI buses finding all of their devices. It then initializes those devices and makes them available to the rest of the Linux kernel via the normal file and buffer cache block device operations. This initialization is done in four phases:

First, Linux finds out which of the SCSI host adapters, or controllers, that were built into the kernel at kernel build time have hardware to control. Each built in SCSI host has a Scsi_Host_Template entry in the builtin_scsi_hosts vector The Scsi_Host_Template data structure contains pointers to routines that carry out SCSI host specific actions such as detecting what SCSI devices are attached to this SCSI host. These routines are called by the SCSI subsystem as it configures itself and they are part of the SCSI device driver supporting this host type. Each detected SCSI host, those for which there are real SCSI devices attached, has its Scsi_Host_Template data structure added to the scsi_hosts list of active SCSI hosts. Each instance of a detected host type is represented by a Scsi_Host data structure held in the scsi_hostlist list. For example a system with two NCR810 PCI SCSI controllers would have two Scsi_Host entries in the list, one per controller. Each Scsi_Host points at the Scsi_Host_Template representing its device driver.

Figure: SCSI Data Structures >

Now that every SCSI host has been discovered, the SCSI subsystem must find out what SCSI devices are attached to each host's bus. SCSI devices are numbered between 0 and 7 inclusively, each device's number or SCSI identifier being unique on the SCSI bus to which it is attached. SCSI identifiers are usually set by jumpers on the device. The SCSI initialization code finds each SCSI device on a SCSI bus by sending it a TEST_UNIT_READY command. When a device responds, its identification is read by sending it an ENQUIRY command. This gives Linux the vendor's name and the device's model and revision names. SCSI commands are represented by a Scsi_Cmnd data structure and these are passed to the device driver for this SCSI host by calling the device driver routines within its Scsi_Host_Template data structure. Every SCSI device that is found is represented by a Scsi_Device data structure, each of which points to its parent Scsi_Host. All of the Scsi_Device data structures are added to the scsi_devices list. The figure above shows how the main data structures relate to one another.

There are four SCSI device types: disk, tape, CD and generic. Each of these SCSI types are individually registered with the kernel as different major block device types. However they will only register themselves if one or more of a given SCSI device type has been found. Each SCSI type, for example SCSI disk, maintains its own tables of devices. It uses these tables to direct kernel block operations (file or buffer cache) to the correct device driver or SCSI host. Each SCSI type is represented by a Scsi_Device_Template data structure. This contains information about this type of SCSI device and the addresses of routines to perform various tasks. The SCSI subsystem uses these templates to call the SCSI type routines for each type of SCSI device.In other words, if the SCSI subsystem wishes to attach a SCSI disk device it will call the SCSI disk type attach routine. The Scsi_Type_Template data structures are added to the scsi_devicelist list if one or more SCSI devices of that type have been detected.

The final phase of the SCSI subsystem initialization is to call the finish functions for each registered Scsi_Device_Template. For the SCSI disk type this spins up all of the SCSI disks that were found and then records their disk geometry. It also adds the gendisk data structure representing all SCSI disks to the linked list of disks shown in the figure above.

Delivering Block Device Requests

Each request to read or write a block of data to or from a SCSI disk partition results in a new request structure being added to the SCSI disks current_request list in the blk_dev vector. If the request list is being processed, the buffer cache need not do anything else; otherwise it must nudge the SCSI disk subsystem to go and process its request queue. Each SCSI disk in the system is represented by a Scsi_Disk data structure. These are kept in the rscsi_disks vector that is indexed using part of the SCSI disk partition's minor device number. For exmaple, /dev/sdb1 has a major number of 8 and a minor number of 17; this generates an index of 1. Each Scsi_Disk data structure contains a pointer to the Scsi_Device data structure representing this device. That in turn points at the Scsi_Host data structure which ``owns'' it. The request data structures from the buffer cache are translated into Scsi_Cmd structures describing the SCSI command that needs to be sent to the SCSI device and this is queued onto the Scsi_Host structure representing this device. These will be processed by the individual SCSI device driver once the appropriate data blocks have been read or written.

Network Devices

A network device is, so far as Linux's network subsystem is concerned, an entity that sends and receives packets of data. This is normally a physical device such as an ethernet card. Some network devices though are software only such as the loopback device which is used for sending data to yourself. Each network device is represented by a device data structure. Network device drivers register the devices that they control with Linux during network initialization at kernel boot time. The device data structure contains information about the device and the addresses of functions that allow the various supported network protocols to use the device's services. These functions are mostly concerned with transmitting data using the network device. The device uses standard networking support mechanisms to pass received data up to the appropriate protocol layer. All network data (packets) transmitted and received are represented by sk_buff data structures, these are flexible data structures that allow network protocol headers to be easily added and removed. How the network protocol layers use the network devices, how they pass data back and forth using sk_buff data structures is described in detail in the Networks chapter (Chapter  networks-chapter). This chapter concentrates on the device data structure and on how network devices are discovered and initialized.

The device data structure contains information about the network device:

Name

Unlike block and character devices which have their device special files created using the mknod command, network device special files appear spontaniously as the system's network devices are discovered and initialized. Their names are standard, each name representing the type of device that it is. Multiple devices of the same type are numbered upwards from 0. Thus the ethernet devices are known as /dev/eth0,/dev/eth1,/dev/eth2 and so on. Some common network devices are:

/dev/ethN	Ethernet devices
/dev/slN	SLIP devices
/dev/pppN	PPP devices
/dev/lo	Loopback devices

Bus Information

This is information that the device driver needs in order to control the device. The irq number is the interrupt that this device is using. The base address is the address of any of the device's control and status registers in I/O memory. The DMA channel is the DMA channel number that this network device is using. All of this information is set at boot time as the device is initialized.

Interface Flags

These describe the characteristics and abilities of the network device:

IFF_UP	Interface is up and running,
IFF_BROADCAST	Broadcast address in device is valid
IFF_DEBUG	Device debugging turned on
IFF_LOOPBACK	This is a loopback device
IFF_POINTTOPOINT	This is point to point link (SLIP and PPP)
IFF_NOTRAILERS	No network trailers
IFF_RUNNING	Resources allocated
IFF_NOARP	Does not support ARP protocol
IFF_PROMISC	Device in promiscuous receive mode, it will receive
	all packets no matter who they are addressed to
IFF_ALLMULTI	Receive all IP multicast frames
IFF_MULTICAST	Can receive IP multicast frames

Protocol Information

Each device describes how it may be used by the network protocool layers:

mtu

The size of the largest packet that this network can transmit not including any link layer headers that it needs to add. This maximum is used by the protocol layers, for example IP, to select suitable packet sizes to send.

Family

The family indicates the protocol family that the device can support. The family for all Linux network devices is AF_INET, the Internet address family.

Type

The hardware interface type describes the media that this network device is attached to. There are many different types of media that Linux network devices support. These include Ethernet, X.25, Token Ring, Slip, PPP and Apple Localtalk.

Addresses

The device data structure holds a number of addresses that are relevant to this network device, including its IP addresses.

Packet Queue

This is the queue of sk_buff packets queued waiting to be transmitted on this network device,

Support Functions

Each device provides a standard set of routines that protocol layers call as part of their interface to this device's link layer. These include setup and frame transmit routines as well as routines to add standard frame headers and collect statistics. These statistics can be seen using the ifconfig command.

Initializing Network Devices

Network device drivers can, like other Linux device drivers, be built into the Linux kernel. Each potential network device is represented by a device data structure within the network device list pointed at by dev_base list pointer. The network layers call one of a number of network device service routines whose addresses are held in the device data structure if they need device specific work performing. Initially though, each device data structure holds only the address of an initialization or probe routine.

There are two problems to be solved for network device drivers. Firstly, not all of the network device drivers built into the Linux kernel will have devices to control. Secondly, the ethernet devices in the system are always called /dev/eth0, /dev/eth1 and so on, no matter what their underlying device drivers are. The problem of ``missing'' network devices is easily solved. As the initialization routine for each network device is called, it returns a status indicating whether or not it located an instance of the controller that it is driving. If the driver could not find any devices, its entry in the device list pointed at by dev_base is removed. If the driver could find a device it fills out the rest of the device data structure with information about the device and the addresses of the support functions within the network device driver.

The second problem, that of dynamically assigning ethernet devices to the standard /dev/ethN device special files is solved more elegantly. There are eight standard entries in the devices list; one for eth0, eth1 and so on to eth7. The initialization routine is the same for all of them, it tries each ethernet device driver built into the kernel in turn until one finds a device. When the driver finds its ethernet device it fills out the ethN device data structure, which it now owns. It is also at this time that the network device driver initializes the physical hardware that it is controlling and works out which IRQ it is using, which DMA channel (if any) and so on. A driver may find several instances of the network device that it is controlling and, in this case, it will take over several of the /dev/ethN device data structures. Once all eight standard /dev/ethN have been allocated, no more ethernet devices will be probed for.

User Comments: