iSCSI Under Rancher2

Steps required for iSCSI support on a Rancher v2 node
May 21, 2018
rancher kubernetes

Not obvious. When I installed v2 yesterday, I did so primarily so that I could have iSCSI support that doesn’t work properly under v1. It didn’t work out of the box, and I kicked it around for a bit before it did. I’m not sure what fixed it, but after a reboot today, it had gone back to not working.

The output was something like this:

MountVolume.WaitForAttach failed for volume "mqtt-data" : failed to get any path for iscsi disk, last err seen:

iscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target:, portal:,3260].

iscsiadm: initiator reported error (12 - iSCSI driver not found. Please make sure it is loaded, and retry the operation)

iscsiadm: Could not log into all portals

Logging in to [iface: default, target:, portal:,3260] (multiple) (exit status 12)

I looked at it more closely, being careful to collect data for a permanent fix. I determined the following:

From this I deduced that the iscsid service running on the host was in the way. I stopped and disabled the service on the host, and immediately new iscsid processes appeared. The output of docker top showed that these were started by the kubelet container, and the container that was unable to mount the iSCSI volume fired up cleanly.