In System V Release 4 (SVR4), the interface between device drivers and the rest of the UNIX kernel has been standardized and documented in section 9 of the Solaris man pages. The reference manual documents driver entry points, driver callable functions and kernel data structures used by device drivers. These interfaces, known collectively as the Device Driver Interface/Driver-Kernel Interface (DDI/DKI) consist of several sections:

DDI/DKI Architecture Independent: These interfaces are supported on all implementations of SVR 4 and will be supported in future releases of System V.
DKI-only: These interfaces are part of SVR4 and may not be supported in future releases of System V. There are only two interfaces in this class, segmap(9E) and hat_getkpfnum(9F).
Solaris DDI: These interfaces specific to Solaris and will be supported in future releases of Solaris 2.x.
Solaris DDI for SPARC based systems: These interfaces are specific to the SPARC processor and may not be available on other processors supported by Solaris.
Solaris DDI for x86 based systems: These interfaces are specific to the x86 processor and may not be available on other processors supported by Solaris.

For more information about the DDI/DKI, see Intro(9) in the Solaris man pages.

The Solaris 2.x DDI/DKI, like its SVR4 counterpart, is intended to standardize and document all interfaces between device drivers and the SunOS kernel. In addition, the Solaris 2.x DDI/DKI is designed to allow source compatibility for drivers on any SunOS 5.x-based machine, regardless of the machine architecture. It is also intended to provide binary compatibility for drivers running on the same processor that supports the Solaris 2.x DDI/DKI, regardless of the specific machine architecture. Drivers that only use kernel facilities that are part of the Solaris 2.x DDI/DKI are known as "Solaris 2.x DDI/DKI-compliant device drivers" or "Solaris DDI-compliant" device drivers.

Note: Suns implementation of the Solaris 2.x DDI/DKI was designed to provide binary compatibility for third party device drivers across currently supported hardware platforms across minor releases of the operating system. Currently there is no validation process for supported hardware platforms.

However, unforeseen technical issues may force changes to the binary interface of the Solaris 2.x DDI/DKI. Sun cannot therefore promise or in any way assure that Solaris 2.x DDI/DKI-compliant device drivers will continue to operate correctly on future releases of Solaris Operating Environment.

Furthermore, future releases may contain additions to the Solaris 2.x DDI/DKI to support future platforms. At that time, device drivers may require these additions to operate across the new set of supported platforms.

2.2 What does it mean if a driver is not Solaris DDI-compliant?

It simply means that your driver may not work (or compile) in later releases. Functions and structures not documented in the DDI/DKI are subject to change/removal at any time.

2.3 How can I tell if a driver is not DDI compliant?

You can tell if your driver is compliant by running the ddict tool or by comparing what the driver uses with section 9 of the Solaris man pages. If the driver is using an interface not in the section 9 man pages, it is non-compliant.

3. Structures, Variables, Files, and Operating System Information

3.1 What are the definitions of and the relationships between instances, device numbers, minor nodes, major numbers, and minor numbers?

"Instance numbers" are part of the "shorthand" name for "dev_info" nodes owned by a particular driver. These nodes are usually associated one-to-one with a particular hardware instance: the two 85C30 chips in a SPARCstation each have a dev_info node called instance zero or "zs" and instance one. Instance numbers are assigned (and owned) by the system, and returned to the driver by ddi_get_instance(9F). Instance numbers and "shorthand names" are not normally visible outside the kernel.

host% ls -l /devices/pseudo/mm:zero crw-rw-rw- 1 root 13, 12 Oct 11 02:20 /devices/pseudo/mm:zero

"Minor nodes" consist of all the information that is held about a user-accessible device that is the minor number (exported by the special file), part of the name of the special file (the 'zero' in "mm:zero", minor number 12 of the "mm" driver), and whether it is block or character. See ddi_create_minor_node(9F) for how to build one.

"Device numbers" are contained in an opaque type "dev_t". They consist of a major number and a minor number.

The "major number" is an internal magic number used by the system to bind special files (such as /devices/pseudo/mm:zero) to device drivers.

In the example above, the "major number" is 13. There is usually no reason for the driver to care what its major number is as they are assigned by the system.

The "minor number" is a component of the device number. It's meaning is entirely up to the device driver, and it is associated with a special file by calling ddi_create_minor_node(9F). In the above example, the minor number is 12. Minor names are associated with minor numbers, as the part following the colon in the name of the special file ('zero' in this case).

3.2 Should I acquire a mutex around any manipulations of a state structure element?

It depends. If it is an element you only initialize in attach then read everywhere else, then there is no need. If it is an element of the data structure that's shared-writable, then yes.

In general though, you should probably start by protecting everything, and relax the protection as you get more confident and a better understanding of your driver and the multithreaded world.

3.3 Can and should I only have one mutex for each instance?

You can, and it often makes sense, but you do not have to. However if you wish to support DR on the Enterprise platforms you will need to have a mutex for each instance.

3.4 What structures can my driver access?

Only those defined in section 9S of the manual pages. Note that these structures may change size, so you should never declare one. For example, if you need a buf(9S) structure, declare a pointer and dynamically allocate one with getrbuf(9F).

3.5 Can I access the user or proc structure?

No.

3.6 I need to prevent user memory from being paged out, so my driver can do DMA to it. Can I call as_fault()?

No, as_fault() is not part of the DDI/DKI. What it does and how it does it may change more or less arbitrarily from minor release to minor release. The only DDI-compliant way to lock down memory is with physio(9F)

3.7 Why is there no man page for printf() or panic() ?

Drivers should not normally print messages. However, though printf() exists in the kernel, if messages need to be printed to the console (and/or the kernel message buffer) drivers should use cmn_err(9F).

There is no equivalent to the SunOS 4.x uprintf() routine.

cmn_err() also allows the driver to panic the system. It is unlikely that drivers will need to do so, however.

3.8 What ddi_iblock_cookie_t should I pass mutex_init(9F) for a mutex used with a timeout(9F)?

NULL is fine for mutexes that are not used in interrupt routines.

3.9 I'm using the ddi_soft_state routines to allocate per-instance state structures. Using the global pointer they require, in this case 'statep', how can I access unit n's state structure in kadb?

Use:
*(*(*statep)+4n)/mX

For, instance #3 (starting at #0), longs #0 through #3 (starting at #0):
in this case, '4n' is 4 times 3 = 0xc in hex.

So to look at the first four longs of instance #3's soft state structure, use:
*(*(*statep)+c)/4X

To just get the address, use:
*(*(*statep)+c)=X

For SunOS 5.0 and 5.1, you have to add x10 to the base address, like this:
*(*(*statep+10)+4n)/mX

3.10 Periodically, in situations where only one "xx" card is in a machine, /dev/xx1 is created instead of /dev/xx0. boot -r does not help. At one point there were two "xx" cards in this machine, and one was removed. Why isn't the remaining card now /dev/xx0?

When you move a card from one slot to another, or remove a card, Solaris remembers the old card. This is what makes instance numbers and dev_t's stick and solves the sliding controller number problems that have always existed.

The device Solaris probed first (ever) should be instance #0. Look in /etc/path_to_inst and you should see which device path name is instance #0.

3.11 What is a 'watchdog reset'?

A watchdog reset occurs when a synchronous trap condition occurs with traps disabled. Since traps are disabled, the processor can't take a trap but it can't continue executing either. The chip then enters error mode and logically raises an error_mode signal. This signal is logically wired back to the watchdog_reset_in pin.

One common way that a new driver can watchdog the system is by using too much stack space (commonly, too many large local variables).

Since the kernel stack doesn't grow, but has red-zones, and the first access into the red-zone could be with traps off from the trap handlers, that could cause a watchdog.

Note: Some chips don't have external error_mode signal, nor external watchdog_reset_in pins and this process all occurs inside the chip.

3.12 Is there a naming convention for I/O controls?

The convention is to pick a letter (say 'M'), and left-shift its ASCII value 8 bits. Then, number your I/O controls in the lower eight bits.

For example:

#define MYIOC ('M' << 8)
#define MYIOCTL_1 (MYIOC | 1)
#define MYIOCTL_2 (MYIOC | 2)

There are two issues to be aware of:

source (namespace pollution): Choose a descriptive prefix that does not conflict with any system macros. For example, TIOC is a particularly dangerous one to use, as the identifier is already defined in <sys/termios.h, and there are already a number of TIO* ioctl values. TIO* is sort of "reserved" for terminal stuff via historical precedent.
Binary: In principle, USL is supposed to keep a registry of ioctl numbers so that a driver can easily flag when it is being given an inappropriate ioctl. In practice, this is very broken as there are not enough letters in the alphabet to deal with all the possible values.

Note: An ioctl command number can be any 32-bit value - the ioctl command numbers do not have to be crunched into 16 bits. Unfortunately an ioctl number registry doesn't really work unless everyone uses it.

3.13 What are the different address spaces that a PCI driver can access?

The PCI address domain consists of three distinct address spaces, as we know, the memory space, the IO space and the configuration space. A driver can access all three address spaces, by setting up appropriate mappings and using the memory, I/O or configuration space access routines provided in the DDI/DKI framework. As a common example, during the driver attach routine the driver could read the configuration space by setting up a mapping via the DDI call pci_config_setup(9f), read the configuration registers using the pci_config_get* routines and write to the configuration registers using pci_config_put * routines.

3.14 I am trying to access my PCI device IO registers through the driver using DDI calls. The driver is mapping the device IO space. I read all zeros, but I was able to read the configuration space and read the contents. Any clues why IO map is not working?

First, check that IO_ENABLE bit is set in the device command register. This will permit access to the device IO space. Although the PCI nexus driver sets these modes by default, the device driver must agree to these settings and if not, it should set proper operation modes.
If the command register shows proper settings, you could be doing something wrong in the mapping operation. For example, perhaps a wrong argument like rnumber is being passed to the mapping routine ddi_regs_map_setup(9f). Check all your arguments. In addition, it is good programming practice not to map the device memory while the PCI configuration space is mapped for modifications. For example, If you called pci_config_setup(9f) to map and modify the configuration space, make sure pci_teardown(9f) has been called before mapping the device memory.

3.15 I can reliably crash the system if I write zero to the command register on my PCI adapter. The same code doesn't crash the x86 machine running Solaris 2.5 . The panic message is as follows:
panic[cpu0]/thread=0x503f6c60: CPU0 Privileged Timeout Error:

On any system, writing a zero to the command register disables access to I/O and memory space, and the device fails to respond to accesses.
How a failure is treated is dependent on the platform. On current SPARC platforms, it causes a trap. On current x86 platforms, writes are dropped and reads return garbage.
The firmware and the framework do write the command register with the desired value, although a driver is free to map and write it as well. (This is not advisable unless the driver disagrees and wants to write a different value.) At no point should the driver disable access to the address space in which it is operating.

3.16 Is it possible to map-in and use both the Base Address Register (BAR) as well as the expansion ROM from the PCI configuration space?

According to section 6.2.5.2 of the PCI 2.1 specifications, there are interactions between the command register MEMORY ENABLE bit and the expansion ROM register EXPANSION ROM ENABLE bit. First, the device will not respond to any memory accesses, whether to registers or to its ROM, unless the MEMORY ENABLE bit is set. Second, the device is allowed to share an address decoder between register accesses and expansion ROM accesses. These result in the following interactions:

MEMORY ENABLE	ROM ENABLE	EFFECT
0	0	All accesses disabled.
0	1	All accesses disabled.
1	0	Register accesses enabled.
1	1	ROM accesses enabled. [*]

[*] The status of register accesses is device-specific when both the MEMORY ENABLE and ROM ENABLE bits are set.

On a device that shares an address decoder, register accesses are not allowed. On a device with independent decoders for register and ROM accesses, register accesses are allowed.

3.17 When I tried to load my driver using add_drv(1M) it fails with the following message "driver loaded successfully but failed to attach". In look through the debug messages, I don't see my attach routine in the driver called at all!

The failure could be occurring in probe(9e) routine. Typically a PCI device driver will not have a probe routine, in which case this explanation is applicable. During the driver load operation, the operating system has to know if the device is present in the system. When add_drv(1M) is called, it checks the "name" property of each device in the device tree, that matches the specified driver name. If there is no match, then the "compatible" property is checked. If that match also fails, the driver loading mechanism fails, because the system could not find a device to attach the driver to. So if the "name" or "compatible" property does not match your driver name, the system has no way of knowing which device in the system the driver is used controls. In order to specify the device in such cases, you must use add_drv(1M) with the -i (alias) option, which would specifics the device ID and vendor ID combination string "pciVVVV,DDDD" for your device. For example, If your driver name is "xxx" for a PCI device with a Vendor ID 1234 and Device ID 56, then you would use add_drv -i '"pci1234,56"' xxx. This will result in driver xxx being loaded on your device. Please note that if the Subsystem ID field in the configuration registers for this device is non-zero, VVVV,DDDD should be the Subsystem Vendor ID and the Subsystem ID respectively. Please refer to the PCI Bus bindings (IEEE-1275) document for further information.

3.18 I have three Base Address Registers (BAR) allocated on my PCI device. How do I know what index to use, to map into each of these registers using ddi_regs_map_setup(9f) in my device driver?

Determine which register index in "reg" property (IEEE 1275) corresponds to your BAR. Use this index value as 'i number' to ddi_regs_map_setup(9f). You must be familiar with the "reg" property format in order to do this. It is explained in the IEEEE-1275 PCI bus bindings document. The rules defined by this document for a non-FCode device state that rnumber=0 refers to the device's configuration space, rnumber=1 refers to the first "active" BAR, if any, and rnumber=2 refers to the second "active" BAR, if any. The expansion ROM and any "hard decode" entries come after the active BARS; see the binding document for details. If the device has FCode, consult the device manual for correlation between the "reg" entries and device registers. You would also need the size of the register for ddi_regs_map_setup(9f) which can be retrieved using ddi_dev_regsize(9f)

3.19 How can a simple character device driver be developed for the Solaris OS? How can a new device driver for the Solaris OS be installed and uninstalled?

Many sample drivers are available in the Solaris Driver Development area of the Sun Developer site: see code samples.

However, these are full-blown drivers and cannot be easily stripped down to a basic driver. To this end, we provide here a bare-bones skeleton driver (called as skelton.c below) that demonstrates the basic structure of a device driver. It can be run as is, and it is intended for quick testing and also as a framework that you can extend. Refer to this program and follow the comments in it.

/************************************************************

        Character Driver Skeleton Program (skelton.c)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        o how to create a test driver (skelton)
                % cc -c -D_KERNEL -xarch=v9 skelton.c
                % ld -r -o skelton skelton.o
        o how to configure
                # cp skelton /tmp
                # cat skelton.conf
                name="skelton" parent="pseudo" instance=0;
                # cp skelton.conf /usr/kernel/drv
                # ln -s /tmp/skelton /usr/kernel/drv/sparcv9/skelton
        o how to load the test driver
                # add_drv skelton
        o how to confirm if the driver is loaded/unloaded or not
                # modinfo
                or
                # modinfo | grep skelton
        o how to unload the test driver
                # rem_drv skelton

        If you insert cmn_err(9F) into this skelton, you will
        be able to see how each entry point is called by
        watching the console output.

        This sample test driver was tested under Solaris 8 and 9.

 ************************************************************/

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#include 
#include 
#define FALSE 0
#define TRUE (!FALSE)

#define SKELTON_NAME            "skelton"

#define SKELTON_DIDNODES        0x04
#define SKELTON_DIDZALLOC       0x08

static int skel_attach(dev_info_t *dip, ddi_attach_cmd_t cmd);
static int skel_detach(dev_info_t *dip, ddi_detach_cmd_t cmd);
static int skel_open(dev_t *devp, int flag, int otyp, cred_t *cred);
static int skel_close(dev_t dev, int flag, int otyp, cred_t *cred);
static int skel_read(dev_t dev, struct uio *uiop, cred_t *credp);
static int skel_write(dev_t dev, struct uio *uiop, cred_t *credp);
static int skel_getinfo(dev_info_t *dip, ddi_info_cmd_t cmd, void *arg,
        void **resultp);
static int skel_ioctl(dev_t dev, int cmd, intptr_t arg, int mode,
        cred_t *credp, int *rvalp);

/* ----- Device Access cb_ops Structure [start] -----

        The data structure (cb_ops(9S)) MUST be provided and initialized
        correctly.

*/
static struct cb_ops skel_cb_ops = {
        skel_open,
        skel_close,
        nodev,
        nodev,
        nodev,
        skel_read,
        skel_write,
        skel_ioctl,
        nodev,
        nodev,
        nodev,
        nochpoll,
        ddi_prop_op,
        NULL,
        D_NEW | D_MP
};
/* ----- Device Access cb_ops Structure [end] ----- */

/* ----- Autoconfiguration Structure [start] -----

        The data structures (modlinkage(9S),modldrv(9S),mod_driverops
and
        dev_ops(9S)) MUST be provided and initialized correctly.

*/
static struct dev_ops skel_ops = {
        DEVO_REV,
        0,                      /* reference count */
        skel_getinfo,
        nulldev,
        nulldev,
        skel_attach,
        skel_detach,
        nodev,
        &skel_cb_ops,
        (struct bus_ops *)NULL
};

extern struct mod_ops mod_driverops;

static struct modldrv md = {
        &mod_driverops, /* Type of module. This is a driver */
        "skel driver",  /* Name of the module */
        &skel_ops,
};

static struct modlinkage ml = {
        MODREV_1,
        &md,
        NULL
};
/* ----- Autoconfiguration Structure [end] ----- */

/* ----- State Pointer [start] -----

        The driver MUST provide a state pointer, which is used by the
soft
        state system to create the list of memory items.
*/

struct skel_state {
        int instance;           /* instance number */
        dev_info_t *dip;        /* dev_info structure */
};

static void *statep;

/* ----- State Pointer [end] ----- */

/* ----- Loadable Module Routines [start] -----

        All drivers MUST implement the _init(9E),_fini(9E) and _info(9E)
        entry points to load,unload and report information about the
        driver module.
*/

static int
_init(void)
{
        int     error;

        error = ddi_soft_state_init(&statep, sizeof(struct skel_state),
1);
        if (error)
                return error;

        error = mod_install(&ml);

        if (error) {
                ddi_soft_state_fini(&statep);
                return error;
        }
        return 0;
}

static int
_info(struct modinfo *modinfop)
{
        return mod_info(&ml, modinfop);
}

static int
_fini(void)
{
        int     error;

        error = mod_remove(&ml);
        if (error)
                return error;

        ddi_soft_state_fini(&statep);
        return 0;
}
/* ----- Loadable Module Routines [end] ----- */

/* ----- Autoconfiguration Entry Point [start] ----- */
static int
skel_attach(dev_info_t *dip, ddi_attach_cmd_t cmd)
{
        int     instance;
        struct skel_state *bs;
        int             flags;

        switch(cmd) {
        case DDI_ATTACH:
                instance = ddi_get_instance(dip);

                if (ddi_soft_state_zalloc(statep, instance) !=
DDI_SUCCESS) {
                        return DDI_FAILURE;
                }

                flags = 0;

                bs = ddi_get_soft_state(statep, instance);
                if (bs == NULL) {
                        goto out;
                }

                flags |= SKELTON_DIDZALLOC;

                if (ddi_create_minor_node(dip, "0", S_IFCHR, instance, 
                        DDI_PSEUDO,0) != DDI_SUCCESS) {
                        goto out;
                }

                flags |= SKELTON_DIDNODES;

                ddi_set_driver_private(dip, (caddr_t)flags);
                ddi_report_dev(dip);
                return DDI_SUCCESS;
        default:
                return DDI_FAILURE;
        }
out:    /* Undo */
        ddi_set_driver_private(dip, (caddr_t)flags);
        skel_detach(dip, DDI_DETACH);
        return DDI_FAILURE;
}

static int
skel_detach(dev_info_t *dip, ddi_detach_cmd_t cmd)
{
        int             instance, flags;
        struct skel_state *bs;

        switch(cmd) {
        case DDI_DETACH:
                instance = ddi_get_instance(dip);
                bs = ddi_get_soft_state(statep, instance);
                if (bs == NULL) {
                        return DDI_FAILURE;
                }

                flags = (int)ddi_get_driver_private(dip);

                if (flags & SKELTON_DIDNODES)
                        ddi_remove_minor_node(dip, NULL);
                if (flags & SKELTON_DIDZALLOC)
                ddi_soft_state_free(statep, instance);

                return DDI_SUCCESS;
        default:
                return DDI_FAILURE;
        }
}

static int
skel_getinfo(dev_info_t *dip, ddi_info_cmd_t cmd, void *arg, void
**resultp)
{
        dev_t   dev;
        int             instance, rval = DDI_FAILURE;
        struct skel_state *bs;

        switch(cmd) {
        case DDI_INFO_DEVT2DEVINFO:     /* seek device's dev_info
pointer */
                dev = (dev_t)arg;
                instance = getminor(dev);
                bs = ddi_get_soft_state(statep, instance);
                if (bs != NULL) {
                        *resultp = bs->dip;
                        rval = DDI_SUCCESS;
                } else {
                        *resultp = NULL;
                }
                break;
        case DDI_INFO_DEVT2INSTANCE:    /* seek device's instance number
*/
                dev = (dev_t)arg;
                instance = getminor(dev);
                *resultp = (void *)instance;
                rval = DDI_SUCCESS;
                break;
        default:
                break;
        }
        return rval;
}
/* ----- Autoconfiguration Entry Point [end] ----- */

/* ----- Character Driver Entry Point [start] ----- */
static int
skel_open(dev_t *devp, int flag, int otyp, cred_t *cred)
{
        return 0;
}

static int
skel_close(dev_t dev, int flag, int otyp, cred_t *cred)
{
        return 0;
}

static int
skel_read(dev_t dev, struct uio *uiop, cred_t *credp)
{
        return 0;
}

static int
skel_write(dev_t dev, struct uio *uiop, cred_t *credp)
{
        return 0;
}

static int
skel_ioctl(dev_t dev, int cmd, intptr_t arg, int mode,
        cred_t *credp, int *rvalp)
{
        return 0;
}
/* ----- Character Driver Entry Point [end] ----- */

3.20 We are having problems loading our driver in the Solaris 2.9 OS, SPARC Platform Edition.

Description:

When we use add_drv, the error message is:

devfsadm: driver failed to attach: ccmux
exit status = 11
Warning: Driver (ccmux) successfully added to system but failed to attach.
Driver (ccmux) installed.

Our driver is trying to push itself between TCP and IP stream layers, and register a /dev/ccmux character device (in that order). It doesn't seem to get to the _init portion of our code.

The OS is in 64-bit mode, running IPv4.

Solution:

Your error is typical of trying to load a 32-bit device driver into a 64-bit OS. A 64-bit kernel can run 32- and 64-bit binaries but can load only 64-bit kernel modules.

In short, you cannot run a 32-bit driver with a 64-bit OS. Since the Solaris 7 release, Solaris has been able to run as either 32-bit mode or 64-bit mode. A 32-bit driver cannot run with a 64-bit mode Solaris release. Ideally, you should compile both 32- and 64-bit versions of the driver, so the OS can pick the right driver depending on the version of the OS that is running. The size of a pointer is different in the two operating systems. The distinction has to occur at compile time. isainfo -kv tells which kernel modules you can load.

# isainfo -kv
64-bit sparcv9 kernel modules

3.21 I wrote a new driver for the SCSI HBA. I have not defined a bus_ops function, nor have I referred to the driver after loading, but I cannot unload the driver.

The following is the dev_ops structure of the driver:

static struct dev_ops star_ops = {
        DEVO_REV,                       /* rev, */
        0,                              /* refcnt */
        star_info,                      /* getinfo */
        star_identify,                  /* identify */
        nulldev,                        /* probe */
        star_attach,                    /* attach */
        star_detach,                    /* detach */
        nodev,                          /* reset */
        &star_cb_ops,                   /* char/block ops */
        NULL                            /* bus ops */
};

The driver is not unloadable if its reference count is more than zero or if it is a nexus driver. The driver is considered a nexus driver if it has a bus_ops structure defined in the dev_ops structure.

Verify the structure's current value using adb as follows:

# adb -k /dev/ksyms /dev/mem
physmem 3e42
star_ops$<dev_ops
star_ops:
star_ops:       rev             refcnt          getinfo
                3               0               star_info
star_ops+0xc:   identify        probe           attach
                star_identify   nulldev         star_attach
star_ops+0x18:  detach          reset           cb_ops
                star_detach     nodev           star_cb_ops
star_ops+0x24:  bus_ops         power
                scsi_hba_busops 0

The preceding structure shows that refcnt is 0, so that is not why the driver cannot be unloaded. The structure values, however, show that bus_ops is not 0; it has some function, making the driver a nexus driver. Therefore, it is not unloadable.

Though bus_ops is not defined in the dev_ops structure, the Solaris OS treats SCSI HBA drivers as nexus drivers. The _init entry point of this driver (SCSI HBA) has the scsi_hba_init function, which internally changes the dev_ops->bus_ops value and makes it a nexus driver.

3.22 Is there any way to keep scheduled events generated by timeout(9F) going during a panic dump?

When the system is panicking, the calls of functions scheduled by timeout(9F) and ddi_trigger_softintr(9F) will not occur.

3.23 If not, is there a way for our HBA driver to find out if a panic dump has been initiated?

Yes, there is a DDI named as ddi_in_panic(9F), which can be used in a driver to check for the panic state. It returns 1 if the system is in panic, and 0 if it is not in panic.

3.24 How can I access the physical device once the new driver is loaded?

Do a reconfigure reboot. This should create the entries in the /devices directory and the symbolic links in the /dev directory.

3.25 Is it possible for two different drivers to create variables that they can share? In other words, can we define a global variable, something similar to kernel variables maxphys or scsi_options?

You can achieve this functionality by having a misc module export this global variable and other modules referencing the same (the misc/scsi module defines scsi_options). Other driver modules can reference this variable by specifying the dependencies in the driver through the -N option of link-editor ld(1M):

ld -N misc/scsi -o xx xx.o

Refer to the Declarations and Data Structures section of Sun Product Documentation for more information.

3.26 What are the different address spaces a PCI driver can access?

A driver can access all three address spaces by setting up appropriate mappings and using the memory, I/O, or configuration space access routines provided in the DDI/DKI framework.

A common example is during the driver attach routine: the driver could read the configuration space by setting up a mapping via the DDI call pci_config_setup(9f), and read/write to the configuration registers via the pci_config_get*/pci_config_put* routines.

3.27 A developer puts a new developed device driver under /usr/kernel, and it causes a system crash while the system boots. How can one avoid loading this buggy driver in the next boot?

The solution is to use boot -a in the OK prompt. boot -a will boot the system interactively. This allows the developer to specify the default directory for modules to be loaded. At the prompt Enter default directory for modules:[/platform/SUNW,Ultra-1 /kernel /usr/kernel], the developer could specify the module directory without /usr/kernel. So the system will not load the drivers under /usr/kernel/drv. It therefore avoids loading the the buggy device driver and causing a system crash again.

3.28 Under certain conditions on the Solaris OS, a device driver cannot be unloaded. What are those conditions?

If a driver has no detach routine (nodev is specified in the driver's dev_ops(9S) for its detach routine).
If EBUSY is returned from the driver's _fini() routine.
If the driver has a positive number of references to it.
If the driver has a bus_ops structure specified in its dev_ops structure, and it has children attached to it (therefore it is a nexus driver).
If the driver has no dev_ops at all!

3.29 In drivers we get the instance number in attach/detach routines, and we get the device (*dev) in open routine. But there is no DDI/DKI call to get the instance number for the given device number in the open routine.

Please use the following Solaris DDI driver entry point in the open routine.

getinfo(9E):
int prefixgetinfo(dev_info_t *dip, ddi_info_cmd_t cmd, void *arg, void **resultp);

When cmd is set to DDI_INFO_DEVT2INSTANCE, getinfo() returns the instance number associated with the dev_t arg. The instance number will be returned in the field pointed to by resultp.

3.30 I have just written a new driver and am ready to test it. How do you create the new device links so the driver can be loaded and tested?

Method #1:

At the boot menu, enter b -r, which is the same as the boot -r you would use on a SPARC system.

Method #2:

As root:

    # touch /reconfigure
    # init 6

You can also use the reboot command in place of init 6.

Method #3:

As root:

    # _INIT_RECONFIG=1 ; export _INIT_RECONFIG
    # /etc/init.d/drvconfig
    # /etc/init.d/devlinks

Method #1 and #2 are the preferred ones, but #3 can be used in case you forgot when rebooting the system after adding new hardware and you don't want to bring it down again.

If you have removed hardware since your last reconfiguration, it is best to use one of the methods that involves a reboot of the system. This is to ensure that everything is closed down properly and the system does not access invalid device nodes that represent missing hardware. In this case, use either method #1 or #2.

3.31 How can I copy the data from the kernel memory space to the user memory space?

Use ddi_copyout(9F), which enables us to copy some data in the kernel memory space to our application program.

For example, when we want to obtain the configuration data within the device driver, ddi_copyout(9F) is useful.

The sample program is shown below. This is the extention of the skelton driver -- refer to 3.19 for Character Driver Skelton Program (skelton.c).

1) The user application program opens the device and call ioctl(2).

  #include 
  #include 
  #include 

  #include "skelton.h"

  main() {

    int fd;
    struct my_skelton_info skelton_info_copy;

    fd = open ("/devices/pseudo/skelton@0:0", O_RDONLY);

    if (fd == -1) {
      perror ("open");
      exit (errno);
    }

    if (ioctl(fd,SKELTON_DUMMY,NULL) == -1) {
      perror("ioctl");
      exit(errno);
    }

    if (ioctl(fd,SKELTON_GETINFO,&skelton_info_copy) == -1) {
      perror("ioctl");
      exit(errno);
    }

    printf("my_skelton_info venid = %d\n",skelton_info_copy.venid);
    printf("my_skelton_info devid = %d\n",skelton_info_copy.devid);
    printf("my_skelton_info revid = %d\n",skelton_info_copy.revid);
 
  }

2) The following sample driver will respond to the application.

o header file (skelton.h)

#ifndef SKELTON_H
#define SKELTON_H

#define SKELTON_DUMMY           _IO('m',1)
#define SKELTON_GETINFO         _IO('m',2)

struct my_skelton_info {
        int     venid;   /* Let's suppose this is the vendor id. */
        int     devid;   /* Let's suppose this is the device id. */
        short   revid;   /* Let's suppose this is the revision id. */
};

#endif SKELTON_H

o extended skelton driver (skelton.c)

        :
        :
#include "skelton.h"
        :
        :
struct skel_state {
    int instance;                   /* instance number */
    dev_info_t *dip;                /* dev_info structure */
    struct my_skelton_info info;    /* additional info */
};
        :
        :
static int
skel_attach(dev_info_t *dip, ddi_attach_cmd_t cmd)
{
    int instance;
    struct skel_state *bs;
    int     flags;

    switch(cmd) {
    case DDI_ATTACH:
        :
        :
        bs = ddi_get_soft_state(statep, instance);
        :
        :
        /* SET VALUE IN THE KERNEL SPACE */
        bs->info.venid = 1234; /* Let's suppose this is the vendor id
value. */
        bs->info.devid = 5678; /* Let's suppose this is the device id
value. */
        bs->info.revid = 99;   /* Let's suppose this is the revision id
value. */

        return DDI_SUCCESS;
    default:
        return DDI_FAILURE;
    }
        :
        :
}

static int
skel_ioctl(dev_t dev, int cmd, intptr_t arg, int mode,
    cred_t *credp, int *rvalp)
{
    int instance;
    struct skel_state *rs;

    instance = getminor(dev);
    rs = ddi_get_soft_state(statep, instance);
    if(rs == NULL) {
        return ENXIO;
    };

    switch (cmd) {
        case SKELTON_GETINFO:
            /* COPY DATA FROM KERNEL SPACE TO USER SPACE */
            if(ddi_copyout((void *)&rs->info, (void *)arg,
                sizeof(struct my_skelton_info), mode) != 0) {
                    return EFAULT;
            }
            return 0;
        case SKELTON_DUMMY:
            cmn_err(CE_NOTE, "_ioctl called (SKELTON_TEST)\n");
            return 0;
        default:
            return ENOTTY;
    }
}

3) The result will be as follows.

You will see the following output. This is the result: values are copied from the kernel memory space to the user memory space.

    # ./skelApp
    my_skelton_info venid = 1234
    my_skelton_info devid = 5678
    my_skelton_info revid = 99

3.32 I want to manage multiple instances in my Solaris device driver. My device needs four-channel communication. I call ddi_create_minor_node() four times from the attach() routine. But inside the ioctl() routine, I cannot get the minor number. How can I achieve this?

The trick commonly used is to combine the instance number with a driver-specific number (for example, the slice number) to create a minor number. Then split it back up into two, when you need it.

A code example follows at the end of this note. Look at how MINOR_NUM macro achieves this.

For more information, visit the Sun Developer site and navigate to the Solaris Driver Development area. There are a number of sample drivers with sources. In particular, the 'bst' driver uses this technique.

/*
 * Create the minor number by combining the instance number
 * with the slice number.
*/
     #define MINOR_NUM(i, s)       ((i) << 4 | (s))

     int
     xkdiskattach(dev_info_t *dip, ddi_attach_cmd_t cmd)
     {
         int instance, slice;
         char name[8];

          /* other stuff in attach... */

         instance = ddi_get_instance(dip);
         for (slice = 0; slice < V_NUMPAR; slice++) {
             
             sprintf(name, "%c", slice + 'a');
             ddi_create_minor_node(dip, name, S_IFBLK,
                 MINOR_NUM(instance, slice), DDI_NT_BLOCK_CHAN, 0);
             /* Other stuff ... */

         }
     }

4. About Drivers

4.1 What is a nexus driver and how do I write one?

Nexus drivers handle bus-specific operations.

You cannot currently write one. The nexus interfaces are in a continual state of change and so they are not documented.

4.2 Can I write a driver for a SCSI Host Bus Adapter? That's a nexus driver, isn't it?

Starting in Solaris 2.4, you can write DDI-compliant SCSI Host Bus Adapter (HBA) drivers.

Yes, they used to be implemented as nexus drivers, but a set of interfaces were added to the DDI/DKI in Solaris 2.4 to hide many of the changing details of nexus drivers from HBA drivers.

The man pages for the interfaces are in section 9 of the man pages.

4.3 Is there a way to send signals to user threads from a device driver?

Starting in Solaris 2.4, there is. You can use proc_signal(9F).

Note: To send signals from interrupt handlers, you have to get a reliable handle on the process that can be used even if the process exits before the handler sends the signal, which is one reason proc_ref(9F),proc_unref(9F), and proc_signal(9F) were introduced in Solaris 2.4.

You can also look at the chpoll(9E) entry point. This is a documented, DDI-compliant way for non-STREAMS drivers to implement polling. In many cases, this is what you really want. Applications can express an interest in a set of file descriptors and be notified by the driver that something has changed. See the man pages for chpoll(9E) and poll(2). poll(2) replaces the SunOS 4.x select(2) system call.

4.4 What is a layered driver?

A layered driver is a driver that calls another driver's routines.

An example might be a pseudo device driver that provides a pseudo disk consisting of several real disks. This driver translates strategy(9E) requests into calls to the real disk driver strategy(9E) routines.

There is no way to write a Solaris DDI-compliant driver that performs this layering. However, drivers must be aware that they may be used by a layered driver such as in open(9E), close(9E), andioctl(9E).

4.5 What does self-identifying means?

Self-identifying devices devices are those that are recognized by the PROMs/BIOS when the system boots. This is specifically for PCI devices. For devices that do not identify themselves to PROM/BIOS, the OS has no information about them and hence has to be specified in conf file. For example, "driver.conf" is a must for a pseudo device.

4.6 What properties for devices that are not self-identifying must be described?

For non-self-identifying devices, you must describe basic properties for devices that are not self-identifying, such as disks, tapes, older ISA devices, and VME devices, in the driver.conf file.

4.7 What are the contents of the xx.conf file?

This file contains any expected SCSI ID or combination of reg, interrupts, and other basic properties. See the driver.conf(4) man page for more information.

4.8 Can you name some self-identifying devices?

Examples of self-identifying devices are all PCI devices, SBus devices, and PnP ISA devices.

4.9 My driver panics the system in ddi_create_minor_node(9F). Is this a known bug in the kernel?

No, it's probably a bug in the driver. Check your getinfo(9E) routine and be sure that it is doing the right thing. See the getinfo(9E) man page and the Writing Device Drivers book in the documentation section of this site for the proper form of using this interface.

4.10 Do self-identifying devices require a configuration file?

No, however if you need to supply driver private properties, they are specified in the configuration file. See driver.conf(4) and pci(4) for more information.

In order for the hwconf file to augment the properties of a hardware dev_info node, it must match the "name" property, the "parent" property, and the first "reg" property. If they don't match, you'll get two dev_info nodes - one from the hardware (PROM) and one from the driver.conf file.

4.11 I want to have two drivers in one module. It doesn't seem to work by putting multiple entries in the modlinkage structure.

Including two drivers in a single loadable module is not supported.

However, it can be done in a non-compliant way by making three loadable modules.

Put all the driver code in one common module and write two small wrapper modules for each driver. The small wrappers need _init, _fini, _info, dev_ops/cb_ops structures for the driver and a line so the two small driver wrappers depend on the main module and get linked with its symbols.

Each mini-driver stub should include:

char _depends_on[] = "misc/foo";

Where "misc/foo" is the name of the common code module.

The common module would be a "misc." module, with it's own _init, _fini and _info functions. The module ops would be of the "misc" type using a modlmisc structure (look this up in <sys/modctl.h>).

The underlying driver just includes the common shared code.

The mini-driver stubs each have their own separate dev_ops structures and can be made unloadable. The misc module with the common driver code will be automatically held via the driver stub modules depending on them.

Note that this method, and using modlmisc, are not DDI-compliant, and so may not work in future releases of Solaris.

4.12 How does a driver retrieve the hostid?

There is no DDI-compliant way to retrieve it, though applications may retrieve it with the sysinfo(SI_HW_SERIAL) system call.

4.13 How do I exercise my driver's aread(9E) and awrite(9E) routines?

Drivers which provide aread(9E) and awrite(9E) entry points provide asynchronous I/O capabilities. The DDS web site contains example drivers (see for example bst and sst) which provide these entry points.

To exercise them, explicit calls to aioread(3) and aiowrite(3) must be made. Consult the aioread(3) man page for information on the libaio library, entry points, and the SIGIO asynchronous I/O notification signal.

Standard read(2) and write(2) system calls do not use these entry points.

4.14 When does the kernel unload loadable modules?

Auto unloading is triggered when the system is so short of memory that it needs to swap one or more processes out of memory. The precise condition involves comparing 'freemem' with various combinations of VM tuning parameters, though it's all a bit arbitrary really. In practice it doesn't happen very often.

When auto unloading, the kernel tries to unload every module in the system by calling the _fini(9E) routine of every module that is known to be not in use.

One simple way to prevent a module from unloading (without changing the source code) is to ensure that it is manually loaded using 'modload' for example:

# modload /kernel/drv/st

The system will not auto-unload a manually loaded module, for more details see modunload(1M). Another way to prevent a module unloading is to start a background process that will use the module in some harmless way:

# sleep 10000 < /dev/foo &

Though of course open(9E)isn't always without side effects. Device drivers can also fail the _fini(9E) or detach(9E) the request to prevent unloading.

Note: Part of testing a new unloadable device driver should be to put the system into a fairly tight loop, loading and unloading the driver to check that there aren't any resource leaks, for example:

#!/bin/sh
>while (/bin/true) do

# use the device in some way which will cause it to load and attach
sleep 3 < /dev/foo
# tell everything auto-unloadable to auto-unload
modunload -i 0
# verify that it really went away
modinfo | fgrep foo
done

These tests should be run on an otherwise unloaded machine running and your script should execute the load/unload loop at least for a thousand iterations while checking on VM resources using vmstat(1M) and crash(1M).

DMA and interrupt resource leaks and double-frees are much more likely to lead to kernel panics or attach failures and are thus easier to find. Memory leaks will show up as reducing quantities of free memory.

Freeing memory already freed is more difficult to find without using a debug kernel. However even without a debug kernel, running a stress test (e.g. sundiag) at the same time as running the above unloading test is a good way to verify that a driver's load and unload routines are working correctly.

Also note, that if you do get a panic, and the backtrace includes 'kmem*', don't assume that the problem is in the kernel memory allocator. The overwhelming majority of 'kernel memory allocator' bugs are caused by double-frees or misaligned or missized kmem_frees in some other part of the kernel or in your device driver.

4.15 How Do I make my driver support PCI Hotplug?

See the PCI Hot Plug White Paper.

4.16 How do I make my driver support SCSI Hotplug?

See the SCSI Hotplug information page.

4.17 Is D_HOTPLUG necessary?

Solaris 7 requires drivers to set D_HOTPLUG in cb_flag (see cb_ops(9S)) if the driver supports detaching individual instances with DDI_DETACH. In Solaris 8 it is assumed that all drivers provide that functionality. Beginning in Solaris 8, it does not matter whether D_HOTPLUG is set or not.

4.18 How do I set breakpoints to stop at the beginning of my driver's probe?

Please see the Deferred Breakpoint Feature documented in the kadb(1M) man page. An excerpt:

Deferred Breakpoint Feature

Since the kernel is dynamically loaded, not all modules may be loaded when a breakpoint is set. kadb can set deferred breakpoints which will be dynamically inserted when the corresponding module is loaded. The corresponding module and the location must both be specified when referring to a deferred breakpoint, as follows:
module_name#location:b

4.19 Solaris Driver Installation and Packaging

4.19.1. Should I get the "Solaris Ready" brand?
IHVs should be getting the "Solaris Ready" brand as applicable.

4.19.2. Where can I find general information about packaging for Solaris software and device drivers in particular?
See the Application Packaging Developer`s Guide at http://docs.sun.com.

The book has a case study on "Installing and Removing a Driver With Procedure Scripts".

4.19.3. Where should drivers be installed on the system?
The preferred location for drivers depends on the extent of platform support the driver provides. There are a number of possible locations to install drivers. Some locations are in the default kernel modules search path (see kernel(1M) and boot(1M) for details).

Maintenance Commands

kernel(1M)

...
The moddir variable contains a colon-separated list of directories that the kernel searches for modules. moddir can be set in the /etc/system file. The minimal default is
/platform/platform-name/kernel:/kernel:/usr/kernel, but this default may be overridden by a specific platform. It is common for many systems to override the default path with /platform/platform-name/kernel:/platform/hardware-class-
name/kernel:/kernel:/usr/kernel, where platform-name can be
found using the -i option of uname(1), and hardware-class-name can be found using the -m option of uname(1).

The kernel configuration can be controlled using the /etc/system file (see system(4)).

Here are some guidelines for placing drivers in any of the default kernel modules search path directories:

/platform/`uname -i`/kernel/drv/
Driver required for booting and designed, tested, and supported for a specific hardware implementation (e.g., "SUNW,Ultra-1", "Compaq Proliant 6000")

/platform/`uname -i`/kernel/drv/sparcv9/
64-bit driver required for booting and designed, tested, and supported for a specific 64-bit hardware implementation (e.g., `isalist` includes "sparcv9")

/platform/`uname -m`/kernel/drv/
Driver required for booting and designed, tested, and supported for a given hardware class (e.g., "sun4m", "sun4u")

/platform/`uname -m`/kernel/drv/sparcv9/
64-bit driver required for booting and designed, tested, and supported for a given hardware class that supports a 64-bit native instruction set (e.g., `isalist` includes "sparcv9")

/kernel/drv/
Driver required for booting and designed, tested, and supported for all systems of a given instruction set architecture or processor type (`uname -p`)

/kernel/drv/sparvc9/
64-bit driver required for booting and designed, tested, and supported for all systems of a given native instruction set (e.g., `isalist` includes "sparcv9")

/usr/kernel/drv/
Driver that can be loaded after initial boot and designed, tested, and supported for all systems of a given instruction set architecture or processor type (`uname -p`)

Another approach is to define an IHV-unique directory to install the drivers and modify /etc/system accordingly to tell the kernel to look in the directory to load the driver module (see system(4), drv and moddir namespaces), for example:

"IHVdrvr" package prototype(4) file:
i pkginfo
i depend
i copyright
d usr ? ? ?
d usr/IHVname ...
d usr/IHVname/drv ...
f usr/IHVname/drv/IHV,driver_name ...
i checkinstall
i postinstall
i preremove
e etcsystem etc/system ...
i i.etcsystem
i r.etcsystem

where "IHV,driver_name" is the file name of the driver, the "checkinstall" procedure scripts see whether the driver is already installed and/or applicable to the target system, the "postinstall" procedure script does the add_drv, the "preremove" procedure script does the rem_drv, and the "etcsystem" class defines the necessary modifications to /etc/system for driver addition (i.etcsystem) and removal (r.etcsystem) via class action scripts. See the Application Packaging Developer's Guide at docs.sun.com for further information.

4.19.4. Do drivers for Solaris need to be uniquely named?
Yes. The driver name, aliases, name-to-major entries, and so on, must be guaranteed to be unique in the Solaris distribution, for all ISAs (SPARC and IA). This is necessary to avoid namespace collisions on servers supporting (SPARC and/or IA) clients as an exported service. When the driver package is installed on the server, there is only one (1) flat namespace to record the driver name, aliases, class, minor node, and major number:

/etc/name_to_major
/etc/minor_perm
/etc/driver_aliases
/etc/driver_classes

See also the add_drv bug noted in add_drv(1M). A common naming convention is <IHV_name>,<driver_name>, where <IHV_name> can be a stock market identifier.

4.19.5. How do I add a new device driver to the system?
All drivers must be added using add_drv(1M), instead of directly modifying the device registration files using procedure scripts. This approach will dynamically create major and minor nodes. However, since the full path name to the driver is not registered with the system, your driver name and alias(es) need to be unique to the system.

Maintenance Commands

add_drv(1M)

...
BUGS
add_drv will accept a full pathname for device_driver. However, the kernel does not use the full pathname; it only uses the final component and searches the internal driver search path for the driver. This can lead to the kernel loading a different driver than expected.

For this reason, it is not recommended that you use add_drv with a full pathname. See kernel(1M) for more information on the driver search path.

In your driver package, this is done using the procedure scripts 'postinstall' and 'preremove' (or 'postremove'). Ideally, the package would use the "checkinstall" packaging script to check if the driver is already installed. See the Application Packaging Developer's Guide at docs.sun.com for further information.

4.19.6. Why isn't there a common location for drivers to be installed on the system?
Ideally, there would be a unique location in the kernel search path for IHV/OEM drivers to distinguish them from Solaris software-delivered platform independent, hardware-class dependent, and platform-specific drivers, but that is not defined. If you see this as a problem, please call your Sun Software Support Provider and ask to escalate Solaris BugID #1226960.

4.19.7. How should I name the software package that contains my driver?
The software package (see pkgmk(1M)) abbreviation (see pkginfo(4), PKG parameter description) naming heuristic is the prefix "SUNW" if Solaris software is providing first-line support or doing the packaging of the driver. If this is not the case, use your vendor stock ID or similar unique prefix for the first three or four characters in the name:

PKG=<COMPANY_ID><pkg_id>
where:
<COMPANY_ID>=: 3 or 4-character uppercase alphanumeric
company identifier; company stock ID is preferred
<pkg_id>=: 5 or 6-character alphanumeric package identifier

4.19.8. If I am allowed to put my driver package on the Solaris operating environment distribution media, do I have to do anything different in my driver package?
For driver packages being distributed on the Solaris operating environment media, the driver package may have a dynamic clustertoc test defined (see parse_dynamic_clustertoc(1M) and clustertoc(4) for details). Using a dynamic clustertoc test makes the package available for installation only on machines with the appropriate hardware attached. Only the SUNWCXall metacluster should be static, and all the other metaclusters should have dynamic entries. The SUNWCXall metacluster is used to install everything on a server necessary to support the clients of any hardware configuration.

5. SunOS 4.x to SunOS 5.x

5.1 I used to pass flags to my driver with the 'flags' keyword in SunOS 4.x. How do I do this in SunOS 5.x?

You can create arbitrary properties in your driver.conf(4) file.

These can be retrieved with one of the ddi_getprop(9F) routines in the driver, which is the preferred way to get information to the driver.

5.2 Is there a SunOS 5.x replacement for the SunOS 4.x dev_info command?

Similar functionality is provided by prtconf(1M).

5.3 My driver needs to keep track of all the processes using it. This can be done in SunOS 4.x by looking at proc structure information in open(9E), but I can't "forget" about processes that are no longer using it since close(9E) is only called when the last reference goes away. How can I do this?

You can't look at the proc structure in SunOS 5.x, so you might consider using the clone mechanism. Essentially, this allows you to allocate an "available" minor device on each open(9E), and since each open(9E) results in a new minor number, you get one close per process. This usually works fine, though processes that fork(2), dup(2), or send file descriptors to other processes may be problems.

There are two ways to do this:

STREAMS drivers can set the is_clone argument of ddi_create_minor_node(9F) to CLONE_DEV. When the node is opened, the clone driver calls the open(9E) routine of the driver with the CLONEOPEN flag set.
All drivers can decide to clone by changing the minor number of the dev_t pointer passed to them.

7. Compiler and Development Environment Questions

7.1 What compiler can I use to compile device drivers?

When you build Solaris kernel modules use the same compiler that Sun used to build the OS. SunPro C is the only supported compiler. The following table gives the SunPro C compiler versions used to compile Solaris releases (in descending chronological order).

Solaris Release	Compiler for SPARC	Compiler for x86
2.4	2.0.1 +patch 100961-01	2.0.1 +patch 101205-01
2.5	2.0.1 +patch 100961-08	2.0.1 +patch 101205-08
2.5.1	2.0.1 +patch 100961-08	2.0.1 +patch 101205-08
2.6	4.2	4.2
S7 (32 bit) FCS and quarterly updates	4.2	4.2
S7 (64 bit) FCS and quarterly updates	5.0 Alpha 06-Apr-98	N/A
Solaris 8	5.0 +patches 107311-03 107357-02 107390-02 Forte 6 +patches 109482-03 109491-02 109578-01	5.0 +patches 107311-05 107357-02 Forte 6 +patches 109492-01 109477-01
Solaris 9 (64-bit)	Forte 6 + patches 111679-14 115117-01 5.0	N/A 5.0
Solaris 9 (32-bit)	Sun Studio 8 Forte 6 + patches 111679-14 115117-01 5.0	Sun Studio 8 Forte 6 + patches 115976-01 5.0
Solaris Express (64-bit)	Sun Studio 8 Forte 6	TBD
Solaris Express (32-bit)	Sun Studio 8 Forte 6	Sun Studio 8 Forte 6

The SunPro compiler does not work in compliant mode (-Xc), because the system header files are not fully compliant.

Note: To ensure that compilers do not optimize away accesses to device registers, you must use volatile properly.

7.2 What compiler do I use to build 64-bit applications?

Sun WorkShop 5.0 was the first compiler from Sun to generate 64-bit code. The latest WorkShop compiler is now part of the Sun Studio line of developer tools.

7.3 Which option should I provide the compiler to generate 64-bit code? What is the difference between -xarch=v9, -xarch=v9a, and -xarch=v9b?

Using the Forte (WorkShop) C compiler, you specify the instruction set architecture (ISA) using -xarch.

-xarch=v9 will generate 64-bit code.
-xarch=v9a adds the Visual Instruction Set (VIS) and extensions specific to UltraSPARC processors to the SPARC-V9 ISA, and enables the compiler to generate code for good performance on the V9 SPARC architecture.
-xarch=v9b adds UltraSPARC-III extensions and VIS version 2.0 to the V9a version of the SPARC-V9 ISA. Compiling with this option uses the best instruction set for good performance in a Solaris UltraSPARC-III environment.

Consult the Forte (WorkShop) documentation for detailed information.

Warning: Code compiled with -xarch=v9b will generate instructions unique to the UltraSPARC-III platform and thus cause errors on the UltraSPARCII platform. The recommended option for generation of a 64-bit driver is -xarch=v9.

7.4 Why do I get "/kernel/drv/sparcv9/mumble: undefined symbol" when I try to load my 64-bit driver?

You should use the -xregs=no%appl compiler option to avoid this. This problem was introduced in Solaris 7 and has been fixed in Solaris 8.

7.5 Do I need to recompile my Solaris 7 driver to be able to run on Solaris 8?

No. However, do a sanity check and verify that the driver runs properly under Solaris 8.

7.6 The Forte Developer compiler specifies the -xchip=ultra3 / xtarget-ultra3 options. Do I need to recompile for the UltraSPARCIII processor?

32-bit drivers
The UltraSPARCIII processor only runs a 64-bit Solaris kernel. If your driver is a 32-bit driver, you will need to recompile your code to produce a 64-bit driver to be able to run on an UltraSPARCIII based machine. Several whitepapers are available with information on writing 64-bit drivers.

64-bit drivers
You do NOT need to recompile your 64-bit driver code to run on an UltraSPARCIII processor. Because the bus architecture of UltraSPARCIII based systems has changed, verify that the driver operates correctly on an UltraSPARCIII system.

Platform-specific compiler options
The Forte Developer Compiler provides several options which allow you to accurately describe the target computer hardware (-xtarget, -xchip, -xcache, etc). When program performance is critical, the proper specification of the target hardware could be very important. However, for most programs and older SPARC processors, the performance gain is negligible. For portability reasons, do not use platform-specific options.

7.7 Can I use the Forte C compiler -fast option to optimize the driver performance?

The -fast option is really a macro option that invokes several different options. Among them are -native, which includes -xchip, -xcache, and -xarch. For portability reasons, compile with as few platform-dependent options as possible. However, for performance reasons, it might be worthwhile to experiment with the -fast option. In such a case, overwrite the -xarch option with the preferred instruction set architecture (ISA) setting (e.g. -xarch=v8plusa for 32-bit drivers; -xarch=v9 for 64-bit drivers) because the -native setting depends on the version of the compiler.

7.8 What compiler options should I typically use?

32-bit drivers
-cc -D_KERNEL -xarch=v8plusa -x02

_KERNEL : produces kernel (driver) code
-cc -D_KERNEL -xarch=v8plusa : produces 32-bit code, taking advantage of UltraSPARC features
-x02 : basic local and global optimization

64-bit drivers
-cc -D_KERNEL -xarch=v9 -x02

_KERNEL : produces kernel (driver) code
-cc -D_KERNEL -xarch=v9 : produces 64-bit code that will run on all UltraSPARC processors
-x02 : basic local and global optimization

7.9 Can I write a device driver in C++?

You might be able to, but it isn't supported. The system header files are not guaranteed to work with both __cplusplus and _KERNEL set.

Also, unlike C, C++ requires significant runtime support. As libc isn't available in the kernel, you can't use object initialization, object constructors, and object destructors. You can circumvent the libc requirement by avoiding most uses of objects, but if you do that, why use C++ at all?

7.10 When using the 3.x SunPro C compiler on a SCSI driver, why are all the SCSI symbols undefined?

The problem is that the driver is defining _depends_on like this:

static char _depends_on[] = "misc/scsi";

_depends_on is used by the kernel to determine that this driver requires another module to be loaded into memory. The 3.x SunPro C compiler determines that since the driver does not reference _depends_on, and it is static, it is not needed in the executable. Since the compiler removes the _depends_on string, the kernel doesn't load the misc/scsi module, and the driver fails to load.

There are two solutions to this problem:

Don't use the 3.x compiler. Sun only supports using the 2.x and 4.x compilers for device drivers at this time.
Remove the 'static' from the definition of _depends_on, to make it a global symbol. This prevents the compiler from removing it, regardless of the compiler used.

The documentation has been changed to indicate that _depends_on should not be static.

Note that _depends_on is not a general mechanism to support loading other modules. It can only be used in certain well-defined areas, such as in SCSI drivers.

8. System Messages

8.1 What does "panic: Lock Held and only one CPU" mean?

This only occurs on a uniprocessor. It means that a spin mutex is held and it will spin forever because there is no other CPU to release it. This could be because the driver did not release the mutex on one code path, or blocked while holding it.

A common cause of this panic is incorrect usage of a spin-type mutex used by the device's high-level interrupt handler. If the interrupt is high-level (see ddi_intr_hilevel(9F) and Intro(9F)), the mutex is a spin-type mutex which blocks the interrupt while held. The interrupt handler cannot use an adaptive mutex or a condition variable since these might effect dispatcher state, and the dispatcher doesn't block high-level interrupts. If the spin-type mutex is held while calling a routine that blocks via cv_wait(9F) or via mutex_enter(9F) on an adaptive (normal non-high-level driver mutex), then the interrupt could occur during that time and spin forever on the mutex (on an MP) or get this panic on a uniprocessor.

8.2 What does it mean when prtconf(1M) returns "(no driver)"?

I loaded the driver with modload(1M), and prtconf(1M) said the same thing.

Referencing the driver by opening it and sending it data causes prtconf(1M) to remove the "(no driver)".

At some point, the installation of modules was decoupled from the loading of modules, which was a good idea. modload(1M) just loads a module and does not install it. Basically, if you use this command, you are just wasting memory. The command does have other valid uses.

If you want to load and install your driver, there are two ways to do it.

Reference the device. (echo < /dev/<device_name> or /devices/...) and the system framework will autoload and install your driver.
drvconfig -i <name> loads and installs the named driver and build the device nodes for the driver.

You'll have to use the second method if the device nodes for the device are not yet in the file system. add_drv(1M) does a drvconfig(1M) on your behalf.

Also note that the notion of the module subsystem installing a driver is slightly different than the notion of the driver/system frameworks installing a driver. drvconfig makes sure to do both.

8.3 What does this message mean: "panic: assertion failed: ibc<(void*) KERNELBASE, file: ../../common/mutex.c, line 427?

A common coding error is to pass the address of an iblock cookie to mutex_init rather than the value (possibly casted to (void*)). If you are running a DEBUG kernel, you will get this panic when you pass the address of the cookie to mutex_init.

The fix is to change:

mutex_init (..., &cookie);
to
mutex_init (...,(void*)cookie);

If you are running a non-DEBUG kernel (which is most likely the case), your driver will fail in unexpected ways. You may get the wrong type of mutex or assign the mutex the wrong priority.

9. Solaris OS on x86 Platforms

9.1 How do I boot with kadb on the Solaris OS for x86 Platforms?

General information on the x86 booting process can be found at the
http://docs.sun.com online documentation site, searching for

boot kadb

"b kadb"

The boot(1m), and eeprom(1m) commands also provide descriptive information; in general x86 and SPARC boot options are handled similarly. Here is some further clarification:

Specifically, the file the secondary boot loads can be controlled by setting of the 'boot-file' property, and by command line override.

During Solaris x86 boot, the /etc/bootrc file interprets user input. 'b kadb [boot-flags]' is the typical syntax for entering commands.

At the install prompt, typing 'kadb' is interpreted as a shorthand for 'boot kadb -d'.

Other alternatives effecting kadb boot are the setting of the 'boot-file' property, via either the devconf boot diskette menu system (see the Help Menu for the boot-properties editing menu), the eeprom(1m) command, or by direct editing of the bootenv.rc file, which stores the pseudo-prom properties which cannot actually be stored in non-volatile ram on the x86 platform.

NOTE: the '-D' boot option is not yet supported on the Solaris OS for x86 platforms.

9.2 How can I make a boot floppy that boots right into the Solaris OS on x86 Platforms?

Searching for 'auto-boot' at the http://docs.sun.com site brings up general information on control of automatic booting of Solaris (mostly related to SPARC). The 'auto-boot?' property (see the eeprom(1m) man page) generally controls whether Solaris automatically boots.

Additional properties introduced in the Solaris x86 boot process allow choosing a timeout value for allowing the menu system to be entered, and to select a 'Saved Configuration' (see the self documenting boot diskette menu subsystem for more details). Here are typical
default settings:

setprop auto-boot? true
setprop auto-boot-cfg-num -1
setprop auto-boot-timeout 5

These properties can be edited via either the devconf boot diskette menu system (see the Help Menu for the boot-tasks and boot-properties editing menus), the eeprom(1m) command, or by direct editing of the bootenv.rc file, which stores the pseudo-prom properties which cannot actually be stored in non-volatile ram on the x86 platform.

In summary, the boot diskette menu system for the Solaris OS on x86 Platforms can be used to control auto-booting, via these properties, via either the explicit menu to select the boot device, or the general property editing menu. The the boot diskette menu system for the Solaris OS on x86 Platforms is largely self documenting, with a context-sensitive help system.

9.3 How does one mount a dos IDE HD on the Solaris OS on x86 Platforms?

Searching for 'pcfs' at the http://docs.sun.com online documentation site provides general information on the pcfs file system and mounting pcfs filesystems.

Specifically, the pcfs(7FS) manpage describes the special syntax required to mount dos filesystem partitions on the Solaris OS on SPARC and x86 Platforms. On the intel platform, dos logical partition qualifiers are required on the mount command. From the man page:

F File Systems pcfs(7FS)

x86: pcfs is mounted from the hard disk with the command:

mount -F pcfdevice-special:logical-drive directory-name

NOTE: support for ide removable drives such as zip drives is not complete in Solaris at the time of this writing.

9.4 How do I allocate and access physical memory for the Solaris OS on x86 Platforms?

9.4.1 Solaris OS on x86 Platforms: How do I allocate physically contiguous memory?

Solaris provides several DDI/DKI routines prefaced with ddi_dma_ which abstract the characteristics of DMA, such that a single driver can be written to work across all Solaris supported platforms.

Searching for

alloc handle dma

at http://docs.sun.com, or this site, will locate information on the Solaris DMA Model. See the Writing Device Drivers book for detailed information.

Specifically, ddi_dma_mem_alloc(9F) is the preferred routine to allocate driver memory suitable for DMA. The ddi_dma_attr(9S) structure controls the allocation of memory. Note that setting the dma_attr_sgllen to a length of 1 informs the DDI framework that physically contiguous memory is required. On Solaris Intel, each ddi_dma_cookie(9S) structure returned when the allocated memory is bound with the various ddi_dma*_bind(9F) contains the physical address of the memory, suitable for use in constructing scatter-gather lists. See the example isp(7D) device driver source, available for download for examples of using various ddi_dma_ prefixed routines.

NOTE: some versions of the DDI framework may not handle all rounding and size issues. In general, it's better to allocate the minimal number of contiguous pages required, in sizes falling on page boundaries.

Other allocators: on the Solaris OS on x86 Platforms, ddi_iopb_alloc(9F) by default allocates from a a very restricted pool of pages allocated at boot time. It's best not to use this allocator unless necessary (note that most modern adapters can perform DMA into the entire 32- and/or 64-bit address range). Also note that the kernel virtual memory allocated by kmem_alloc(9F) can be bound using ddi_dma_addr_bind_handle(9F), but this memory is not physically contiguous, and may appear above 32-bits on 64-bit or PAE enabled systems.

ddi_dma_mem_alloc(9F) is a recommended interface to use.

TIP: allocating very large blocks of physically contiguous memory:

The Solaris kernel memory model does not support large blocks of physically contiguous memory: as virtual/paged memory usage fragments the physical memory, there is no management performed to defragment physical memory. Also, various kernel data structures are sized dynamically at boot time, to match the system's memory size.

(One method to allocate a large physically contiguous memory block is to perform the allocation very early (when the driver is first attached), as memory resources are relatively unfragmented at this time. Note that kmem_alloc(9F) will typically also return memory which is physically contiguous early in system boot).

9.4.2 Solaris OS on x86 Platforms: Why does ddi_iopb_alloc() fail with larger allocations?

On the Solaris OS on x86 Platforms, ddi_iopb_alloc(9F) by default allocates from a a very restricted pool of pages allocated at boot time. It's best not to use this allocator unless necessary (note that most modern adapters can perform DMA into the entire 32- and/or 64-bit address range).

It is possible to increase the number of pages available for use by ddi_iopb_alloc(9F) by placing the following kernel variable edit into /etc/system:

set lomempages=140

However it is preferable to use the new ddi_dma_ DMA Model routines.

9.4.3 Solaris OS on x86 Platforms: How do I convert a kernel virtual address to a physical address?

The ddi_dma_addr_bind_handle(9F) and ddi_dma_buf_bind_handle(9F) routines provide the method to allocate DMA resources for kernel buffers. Note that on the Solaris OS for x86 Platforms, each ddi_dma_cookie(9S) returned for each DMA window, will contain a physical address suitable for use as in constructing a scatter-gather list.

9.5 How can the kernel recognize a non-self-identifying device?

You need to write a realmode driver. See the realmode environment sources and White Papers ("Solaris Realmode Environment", "Realmode Drivers", and "External BEF Interface Specification for x86 Realmode DDI").

9.6 How do I make an x86 device bootable?


	About Sun \| About This Site \| Newsletters \| Contact Us \| Employment How to Buy \| Licensing \| Terms of Use \| Privacy \| Trademarks Copyright Sun Microsystems, Inc.		A Sun Developer Network Site Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License. Sun Developer RSS Feeds