|
Contents
This is the fourth article in a series about the SCSI DISK FMA project:
Overview
To easily forge an error condition, the SCSI DISK (SD) driver provides a mechanism for fault injection. This way, programmers can inject error conditions arbitrarily, which is a requirement during testing of code modifications. This article describes the fault injection mechanism from the point of view of kernel programmers.
Data Structures and Interfaces
Fault injection is a kind of data interception. Fault injection falsifies the data that is sent back from the HBA driver, and finally causes walking into a different code path in the SD driver.
The following diagram shows the relationship between the SD driver and sd fault injection.
SD driver
| ^
v | <----fault injection
SCSA layer
| ^
v |
HBA driver
|
To a kernel programmer, fault injection includes corresponding data structures and ioctl commands. Using these, the programmer can modify the key kernel data structures inside the SD driver during a SCSI command execution.
Fault Injection Data Structures
Major data structures include the following.
/*
* replicates the variables that are exposed to
* struct scsi_pkt.
*/
struct sd_fi_pkt;
/*
* replicates the variables that are exposed to struct sd_xbuf.
*/
struct sd_fi_xb;
/*
* replicates the variables that are exposed to struct sd_lun.
*/
struct sd_fi_un;
/*
* replicates the variables that are exposed for Auto-Request-Sense.
*/
struct sd_fi_arq;
|
You can see details of the structures in usr/src/uts/common/sys/scsi/targets/sddef.h.
Fault Injection Interfaces
The kernel programmer can issue the fault injection command through the sdioctl interface. There are several fault injection commands provided for different purposes.
Table 1 shows a brief introduction to each command.
 |
SDIOCSTART
|
Start fault injection, do the initialization |
SDIOCSTOP
|
Stop fault injection, clean up the resources |
SDIOCINSERTPKT
|
Inject one struct sd_fi_pkt |
SDIOCINSERTXB
|
Inject one struct sd_fi_xb |
SDIOCINSERTUN
|
Inject one struct sd_fi_un |
SDIOCINSERTARQ
|
Inject one struct sd_fi_arq |
SDIOCPUSH
|
Push injected fault onto FIFO |
SDIOCRETRIEVE
|
Return buffer of log from injection session |
|
Table 2 shows the available arguments for each command.
|
SDIOCSTART
|
None |
SDIOCSTOP
|
None |
SDIOCINSERTPKT
|
struct sd_fi_pkt *, can't be NULL |
SDIOCINSERTXB
|
struct sd_fi_xb *, can't be NULL |
SDIOCINSERTUN
|
struct sd_fi_un *, can't be NULL |
SDIOCINSERTARQ
|
struct sd_fi_arq *, can't be NULL |
SDIOCPUSH
|
uint_t *, can be NULL |
SDIOCRETRIEVE
|
char *, can't be NULL |
|
Example: SD Fault Injection
A typical fault injection sequence would be:
SDIOCSTART
|
v
SDIOCINSERTPKT(SDIOCINSERTXB, SDIOCINSERTUN, SDIOCINSERTARQ)
|
v
SDIOCPUSH
|
v
Send out SCSI Command
|
The SDIOCPUSH command moves the index of next fault forward as specified in the argument. If the argument is NULL, it moves forward one step.
The following is a short example that shows how to use sd fault injection.
"sd_fi_test.c"
#include <stdio.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <strings.h>
#include <sys/sysmacros.h>
#include <sys/types.h>
#include <sys/scsi/scsi_types.h>
#include <sys/dditypes.h>
#include <sys/cred.h>
#include <sys/scsi/scsi_pkt.h>
#define SDIOC ('T'<<8)
#define SDIOCSTART (SDIOC|1)
#define SDIOCSTOP (SDIOC|2)
#define SDIOCINSERTPKT (SDIOC|3)
#define SDIOCINSERTXB (SDIOC|4)
#define SDIOCINSERTUN (SDIOC|5)
#define SDIOCINSERTARQ (SDIOC|6)
#define SDIOCPUSH (SDIOC|7)
#define SDIOCRETRIEVE (SDIOC|8)
#define SDIOCRUN (SDIOC|9)
/*
* struct sd_fi_pkt definition.
*/
struct sd_fi_pkt {
uint_t pkt_flags; /* flags */
uchar_t pkt_scbp; /* pointer to status block */
uchar_t pkt_cdbp; /* pointer to command block */
uint_t pkt_state; /* state of command */
uint_t pkt_statistics; /* statistics */
uchar_t pkt_reason; /* reason completion called */
};
/*
* Define a transport error condition.
*/
struct sd_fi_pkt t_pkt_tran_err = {
0, /* pkt_flags */
0, /* pkt_scbp */
0xff, /* pkt_cdbp, 0xff will keep the original value unchanged. */
STATE_GOT_BUS, /* pkt_state */
STAT_TIMEOUT, /* pkt_statistics */
CMD_TRAN_ERR /* pkt_reason */
};
int main(int argc, char **argv) {
int fd = 0;
unsigned char buf[512];
if (argc < 2) {
printf("./test_fault_injection <devpath>\n");
return 0;
}
/*
* Start fault injection.
*/
if ((fd = open(argv[1], O_RDONLY|O_NDELAY)) == -1)
perror("open");
if (ioctl(fd, SDIOCSTART, NULL) == -1)
perror("ioctl SDIOCSTART");
/*
* Inject a transport error.
*/
if (ioctl(fd, SDIOCINSERTPKT, &t_pkt_tran_err) == -1)
perror("ioctl SDIOCINSERTPKT");
if (ioctl(fd, SDIOCPUSH, NULL) == -1)
perror("ioctl SDIOCPUSH");
/*
* Send out a SCSI command (READ).
*/
if (ioctl(fd, SDIOCRUN, NULL) == -1)
perror("ioctl SDIOCRUN");
if (read(fd, buf, sizeof (buf)) < 0)
perror("read");
/*
* Stop fault injection.
*/
if (ioctl(fd, SDIOCSTOP, NULL) == -1)
perror("ioctl SDIOCSTOP");
return 0;
}
|
You can copy the following code and compile it.
X86:
/usr/sfw/bin/gcc -o sd_fi_test sd_fi_test.c
X64:
/usr/sfw/bin/gcc -m64 -o sd_fi_test sd_fi_test.c
SPARC:
cc -xarch=generic64 -o sd_fi_test sd_fi_test.c
|
Run the following command.
./sd_fi_test /dev/rdsk/cxtxdxsx
|
You might then get ereports like the following.
# fmdump -ev
Oct 13 16:20:34.6110 ereport.io.scsi.cmd.disk.tran 0xd2fd358799400801
Oct 13 16:20:34.6110 ereport.io.scsi.cmd.disk.recovered 0xd2fd358799400801
|
Note: For the following conditions the original value remains unchanged.
sd_fi_pkt::pkt_cdbp == 0xff
sd_fi_xb::xb_retry_count == 0
|
For More Information
About the Authors
Xiao Li is a Sun Software Engineer. He has an M.S. in Computer Science from the Graduate University of the Chinese Academy of Sciences.
|
|
Print-friendly Version
