tests before publishing changes

[dsschult]

There have been several instances of bad publishes in the last few days (python modules mysteriously disappearing).

Develop a test suite to run basic checks and catch these sorts of problems.

[claudiok]

Would an "index" file with a full directory listing and hashes of each file's contents help? You could have a script that keeps comparing them (that users could run on a node themselves if they run into trouble). This would make bug reports easy and you would immediately see which files changed, especially if a client is in an inconsistent state.

[dsschult]

If I just wanted to spot when changes occur, I can see that in the rsync while publishing. The problem is that most of the time the change was intended, and only some times it might break something.

Also, cvmfs already keeps hashes of everything, so the "index" probably already exists.

[dsschult]

I'm thinking more along the lines of running through the imports of all the python packages we need, and other basic tests like that.

[claudiok]

I guess you are right - if anything this is a different problem and only rarely occurs (but we've seen it happen on one of the "followup" nodes where a single file had the correct size but different contents). What you want are actual tests. Sorry for the noise!

[dsschult]

The correct size but different contents was an upstream cvmfs bug that is now fixed. So not my problem 😉