Steps required for iSCSI support on a Rancher v2 node
May 21, 2018rancher kubernetes
Not obvious. When I installed v2 yesterday, I did so primarily so that I could have iSCSI support that doesn’t work properly under v1. It didn’t work out of the box, and I kicked it around for a bit before it did. I’m not sure what fixed it, but after a reboot today, it had gone back to not working.
The output was something like this:
MountVolume.WaitForAttach failed for volume "mqtt-data" : failed to get any path for iscsi disk, last err seen: iscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2006-04.us.monach:nas.mqtt-data, portal: 10.68.0.11,3260]. iscsiadm: initiator reported error (12 - iSCSI driver not found. Please make sure it is loaded, and retry the operation) iscsiadm: Could not log into all portals Logging in to [iface: default, target: iqn.2006-04.us.monach:nas.mqtt-data, portal: 10.68.0.11,3260] (multiple) (exit status 12)
I looked at it more closely, being careful to collect data for a permanent fix. I determined the following:
- the kubelet container is responsible for loading the iscsi kernel modules
- the host was running its own iscsid service
- the kubelet container has its own data under
/etc/iscsidthat differs from the hose, showing that it does not integrate with the host’s
iscsidprocess in any way
From this I deduced that the
iscsid service running on the host was in the way. I stopped and disabled the service on the host, and immediately new
iscsid processes appeared. The output of
docker top showed that these were started by the kubelet container, and the container that was unable to mount the iSCSI volume fired up cleanly.