Sometimes apt goes south. Today it went south on all of my Ubuntu 14.04 systems, worldwide.

First, Ansible reported this:

TASK [common : Update apt cache] ***********************************************
fatal: [xx]: FAILED! => {"changed": false, "failed": true, "msg": "Could not fetch updated apt files"}

Running apt-get update gave variations of one or both of the following:

Hit http://ppa.launchpad.net trusty/main Translation-en
E: Method gave invalid 103 Redirect message

W: Failed to fetch mirror://mirrors.ubuntu.com/mirrors.txt/dists/trusty-updates/main/binary-amd64/Packages  Hash Sum mismatch

W: Failed to fetch mirror://mirrors.ubuntu.com/mirrors.txt/dists/trusty-updates/main/binary-i386/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

Poking at this leads to this bug report and this thread. The thread, while saying ‘SOLVED’ is solved by someone randomly changing their DNS and carrying their computer back and forth from work to home for a few months, reinstalling Ubuntu several times. That doesn’t sound like a solution. That sounds like the flailing of a drowning user who is in over his head.

Remember: a solution that doesn’t tell you why the problem happened is not a solution. It’s a random result that may only mask the reason that your problem exists.

Anyway, the bug report says the issue is fixed in a future update and recommends whacking the contents of /var/lib/apt/lists/partial/* as a workaround. That worked on one of the boxes, but the ones where apt-get threw an error about Hash Sum mismatch needed more serious intervention.

In the forum thread, filled with the drivel of useless community support, are two commands that point to corruption in /var/lib/apt/lists itself. Only after removing all of /var/lib/apt/lists/* was apt-get able to continue.

I get that sometimes things happen on a box, and we need to smack it to get it working again. I don’t like when the same error happens on multiple boxes deployed all around the world. That points to upstream errors, and the it’s not the box we need to smack. It’s the upstream admins who broke it.