Converting a SCSI Driver to Support Hotplugging of Targets

1. What is hotplugging?
2. How is hotplugging performed?
3. What happens to the SCSI bus when hotplugging?
4. How does Solaris do this?
5. What do these functions have to do?
6. How are these functions registered with the SCSA framework?

1. What is hotplugging?

Hotplugging of targets on a SCSI bus is the ability to add, remove, and swap SCSI targets while the system is up and running. The main advantage is that a system administrator can replace faulty drives without having to bring the system to a halt and then perform a reboot. It also means that new drives can be added to an almost full system without having to reboot.

2. How is hotplugging performed?

The tool (cfgadm) performs hotplug operations on a SCSI bus. The operation of cfgadm is beyond the scope of this document. (See cfgadm Extensions to Support SCSI Hotplugging.) cfgadm provides a (UNIX) user-friendly environment to hotplugging.

3. What happens to the SCSI bus when hotplugging?

Obviously, there can't be SCSI packets going over the bus when adding or removing targets. Some mechanism is needed to quiet the bus. In the Solaris operating environment, this is known as quiescing the SCSI bus. This involves draining all commands in the queue and not allowing any new commands to enter the queue.

The bus is now effectively frozen, and the hotplug operation can be performed; for example, plugging in a new target.

The bus is then unfrozen and allowed to continue as normal. This is known as unquiescing the bus.

4. How does Solaris do this?

The SCSA framework for Solaris 7 Update 5/99 and Solaris 8 provides support for hotplugging SCSI targets. When the cfgadm tool needs to quiesce (freeze) or unquiesce (unfreeze) a specific SCSI bus, the SCSA framework calls respectively the quiesce or unquiesce function provided by the HBA driver controlling that SCSI bus.

Converting a SCSI HBA driver to support hotplugging basically involves writing a quiesce function and an unquiesce function. It then involves registering them with the scsi_hba_tran structure in the driver attach function where all hba_tran entry points are registered.

The entry points in the scsi_hba_tran structure used for registering HBA-specific quiesce and unquiesce functions are:

int (*tran_quiesce)(
            dev_info_t *hba_dip);

int (*tran_unquiesce)(
            dev_info_t *hba_dip);

These entry points are called by the SCSA framework when the system administrator uses the cfgadm tool.

Back to Top

5. What do these functions have to do?

tran_quiesce(dev_info_t *hba_dip);

This function must perform a number of steps to quiesce the SCSI bus:

• It must prevent new SCSI commands from being queued. In some drivers (ncrs), this is known as setting the throttle to 0.
• The function checks to see how many commands are left in the queue; that is, commands sent to SCSI targets and not completed yet by those SCSI targets. If there are commands left, the function waits until either all queued commands complete, or a signal is received, interrupting the quiesce operation. Indicating the state of the quiesce operation might be necessary to prevent other operations from interfering with a queue draining process (for example, setting the DRAINING flag in the ncrs driver).
• If a signal is received while waiting for outstanding (queued) SCSI commands to complete, the quiesce is aborted, the state of the SCSI bus is marked as non-quiesced (in the ncrs driver the DRAINING flag in the softstate structure is unset), and the new SCSI commands are allowed to be executed (in the ncrs driver the throttle is restored to FULL_THROTTLE). The quiesce function returns -1 to show the framework that the operation was aborted.
• Waiting for queued commands to complete can be accomplished by periodically checking how many commands are left in the queue, or by other HBA-specific means.
• When all the queued commands are completed, the state of the bus is marked as quiesced (in the ncrs driver the DRAINING flag is unset in the softstate structure, and the QUIESCED flag is set). The function returns 0 to show success.

Following is an example of a quiesce function from the ncrs driver. The glm_scsi_quiesce function (registered as the tran_quiesce function in the scsi_hba_tran structure) is just a wrapper function that performs some checking and calls the glm_quiesce_bus function that does the real work. The glm_scsi_quiesce and glm_quiesce_bus functions could be merged into the single glm_scsi_quiesce function.

static int
glm_scsi_quiesce(dev_info_t *dip)
{
   struct glm *glm;
   scsi_hba_tran_t *tran;

   tran = (scsi_hba_tran_t *)ddi_get_driver_private(dip);
   if ((tran == NULL) || ((glm = TRAN2GLM(tran)) == NULL)) {
      return (-1);
   }
   return (glm_quiesce_bus(glm));
}

glm_quiesce_bus(struct glm *glm)
{
    NDBG28(("glm_quiesce_bus"));
    mutex_enter(&glm->g_mutex);

    /* Set the throttles to zero */
    glm_set_throttles(glm, 0, N_GLM_UNITS, HOLD_THROTTLE);
    /* If there are any outstanding commands in the queue */
    if (glm_check_outstanding(glm)) {
        glm->g_softstate |= GLM_SS_DRAINING;
        glm->g_quiesce_timeid = timeout(glm_ncmds_checkdrain,
                 glm, (GLM_QUIESCE_TIMEOUT * drv_usectohz(1000000)));
        if (cv_wait_sig(&glm->g_cv, &glm->g_mutex) == 0) {
            /*
             * Quiesce has been interrupted
             */
            glm->g_softstate &= ~GLM_SS_DRAINING;
            glm_set_throttles(glm, 0, N_GLM_UNITS, MAX_THROTTLE);
            glm_start_next(glm);
            if (glm->g_quiesce_timeid != 0) {
                mutex_exit(&glm->g_mutex);
                (void) untimeout(glm->g_quiesce_timeid);
                glm->g_quiesce_timeid = 0;

                return (-1);
            }
            mutex_exit(&glm->g_mutex);
            return (-1);
         } else {
                /* Bus has been quiesced */
            ASSERT(glm->g_quiesce_timeid == 0);
            glm->g_softstate &= ~GLM_SS_DRAINING;
            glm->g_softstate |= GLM_SS_QUIESCED;
            mutex_exit(&glm->g_mutex);
            return (0);
         }
     }
     /* Bus was not busy - QUIESCED */
     mutex_exit(&glm->g_mutex);
     return (0);
}

glm_check_outstanding and glm_ncmds_checkdrain are supporting functions.

• The glm_check_outstanding function returns the number of SCSI commands left to execute in the queue (commands not yet completed by SCSI targets).

  static int  glm_check_outstanding(struct glm *glm)
{
    uint_t unit_i;
    int ncmds = 0;

      ASSERT(mutex_owned(&glm->g_mutex));

      /* for every unit structure in the glm structure */
    for (unit_i = 0; unit_i < N_GLM_UNITS; unit_i++) {
        if (glm->g_units[unit_i] != NULL) {
            ncmds += glm->g_units[unit_i]->nt_tcmds;
        }
    }

      return (ncmds);
}

• The glm_ncmds_checkdrain function is called from the timeout initialized in the glm_quiesce_bus function. This function checks if the queue draining is completed. If so, it wakes up the suspended thread of the glm_quiesce_bus function. If draining is not completed, it re-initializes the timeout (sets up a delay for the next check).

  static void
glm_ncmds_checkdrain(void *arg)
{
   struct glm *glm = arg;

   mutex_enter(&glm->g_mutex);
   if (glm->g_softstate & GLM_SS_DRAINING) {
       glm->g_quiesce_timeid = 0;
       if (glm_check_outstanding(glm) == 0) {
          /* Command queue has been drained */
          cv_signal(&glm->g_cv);
      } else {
         /*
         * The throttle may have been reset because
         * of a SCSI bus reset
         */
         glm_set_throttles(glm, 0, N_GLM_UNITS, HOLD_THROTTLE);
         glm->g_quiesce_timeid = timeout(glm_ncmds_checkdrain,
            glm, (GLM_QUIESCE_TIMEOUT *
            drv_usectohz(1000 000) ));
      }
   }
   mutex_exit(&glm->g_mutex);
}

tran_unquiesce(dev_info_t *hba_dip);
This function must perform a few steps to unquiesce the SCSI bus:

• Indicate the state of the SCSI bus as not-quiesced (in the ncrs driver the QUIESCED flag is unset).
• Enable processing of the next SCSI command (in the ncrs driver the throttle is reset to FULL_THROTTLE).
• Start the next SCSI command if a command is waiting.

Following is an example of the unquiesce function from the ncrs driver. The glm_scsi_unquiesce function (registered as tran_unquiesce function in the scsi_hba_tran structure) is just a wrapper function that performs some checking and calls the glm_unquiesce_bus function that does the real work. The glm_scsi_unquiesce and glm_unquiesce_bus functions could be merged into a single glm_scsi_unquiesce function.

static int  glm_scsi_unquiesce(dev_info_t *dip)
{
    struct glm *glm;
    scsi_hba_tran_t *tran;

    tran = (scsi_hba_tran_t *)ddi_get_driver_private(dip);
    if ((tran == NULL) || ((glm = TRAN2GLM(tran)) == NULL)) {
       return (-1);
    }

    return (glm_unquiesce_bus(glm));
}

static int 
glm_unquiesce_bus(struct glm *glm)
{
    NDBG28(("glm_unquiesce_bus"));
    mutex_enter(&glm->g_mutex);
    glm->g_softstate &= ~GLM_SS_QUIESCED;
    glm_set_throttles(glm, 0, N_GLM_UNITS, MAX_THROTTLE);
    glm_start_next(glm);
    mutex_exit(&glm->g_mutex);
    return (0);
}

Back to Top

6. How are these functions registered with the SCSA framework?

In the driver's attach function, the scsi_hba_tran structure will be set up. This structure is just a struct of empty function pointers.

scsi_hba_tran_t *hba_tran;
hba_tran = scsi_hba_tran_alloc(dip, SCSIHBA_CANSLEEP);

The quiesce and unquiesce functions are registered as follows:

hba_tran->tran_quiesce = glm_scsi_quiesce;
hba_tran->tran_unquiesce = glm_scsi_unquiesce;

The scsi_hba_tran structure is registered with the SCSA framework:

if (scsi_hba_attach_setup(dip, &glm_dma_attrs,
     hba_tran, 0) != DDI_SUCCESS) {
         glm_log(glm, CE_WARN, "hba attach setup failed");
         goto fail;
}

Notes:

The SCSA framework supporting hotplugging is available in Solaris 7 Update 5/99 and Solaris 8. This SCSA framework also handles open and close calls to the driver, in addition to the ioctl calls related to hotplugging.

Previously pointers to HBA open, close, and ioctl functions were passed to the SCSA framework in the cb_ops structure, through the dev_ops and modldrv structures. With the new SCSA framework, the pointer to the cb_ops structure in the dev_ops structure must be set to NULL for a driver to use the framework supplied processing of the open, close, and ioctl calls.

Back to Top