Not obvious. When I installed v2 yesterday, I did so primarily so that I could have iSCSI support that doesn’t work properly under v1. It didn’t work out of the box, and I kicked it around for a bit before it did. I’m not sure what fixed it, but after a reboot today, it had gone back to not working.

The output was something like this:

MountVolume.WaitForAttach failed for volume "mqtt-data" : failed to get any path for iscsi disk, last err seen:

iscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2006-04.us.monach:nas.mqtt-data, portal: 10.68.0.11,3260].

iscsiadm: initiator reported error (12 - iSCSI driver not found. Please make sure it is loaded, and retry the operation)

iscsiadm: Could not log into all portals

Logging in to [iface: default, target: iqn.2006-04.us.monach:nas.mqtt-data, portal: 10.68.0.11,3260] (multiple) (exit status 12)

I looked at it more closely, being careful to collect data for a permanent fix. I determined the following:

• the kubelet container is responsible for loading the iscsi kernel modules
• the host was running its own iscsid service
• the kubelet container has its own data under /etc/iscsid that differs from the hose, showing that it does not integrate with the host’s iscsid process in any way

From this I deduced that the iscsid service running on the host was in the way. I stopped and disabled the service on the host, and immediately new iscsid processes appeared. The output of docker top showed that these were started by the kubelet container, and the container that was unable to mount the iSCSI volume fired up cleanly.