Checksums

[judebert]

Introduction:
We've had some trouble in the past -- and even recently -- with incomplete and corrupted uploads. And when things go wrong with the blog, it'd be nice to have some way to say "at least the files are OK".

Enter checksums. By keeping a list of the required files and their MD5 (or other algorithm) checksums, we can check at any time whether our files are intact.

Enter FTP, opposing. Since most of our files are ASCII, FTP can mangle them by translating the newlines to the local machine's variation. (DOS, Unix, and Mac all use different newlines.) The checksum algorithm would detect this perfectly normal and valid change as corruption.

Enter SuperJude! I've worked around this problem by reading the file into a buffer, changing the various newline variations to spaces, and checksumming that instead. Now FTP variations don't modify the checksum, and we can detect important modifications to the file.

I'm working on this in trunk/, and I've got it mostly working on one of my sandbox installations. It's surprisingly fast: generating checksums takes a few seconds, and verifying them takes two seconds or less.

The Questions:
There are lots of ways to work this. I've chosen to add serendipity_FTPChecksum (which calculates the FTP-impervious MD5 checksum for a single file) and serendipity_verifyFTPChecksums (which returns a list of files with incorrect checksums) in functions_installer.inc.php. I also updated the upgrader and the installer to call serendipity_verifyFTPChecksums when they're run. Finally, I'm modifying serendipity_admin.php to provide checksum validation from a button and a special URL. The checksums themselves are in the root directory, as checksums.inc.php.

But for generating the list of checksums, I've provided a serendipity_generateFTPChecksums script in deployment/. I'm not sure this is the best way to do it. What I'd really like is to have checksums automatically generated along with the nightly builds, and included in the build archive. How do I do that?

Finally, when I actually get this working, is there a chance that it'll get into the distribution, or is the whole thing just too radical?

[garvinhicking]

Hi!

That sounds great. Is checksum validation protected so that it's only executed for logged-in admins? I'd like to avoid some kind of
(D)DOS if users could run that intense functionality as a visitor.

Autogenerating the FTP checksums is technically not hard, but since Jannis provides the nightlies he would need to add that to his routine (and get contacted). Since he's writing his master thesis, I'm not sure if he'll find the time to add this.

For the s9y installation release tarballs, the bundled-libs/create_release.sh is used by me, so this can be modified to do that

I don't see why not to add this to the distro

Regards,
Garvin

[judebert]

Yes, I only allow checksum validation during installation, or from the admin page. I'm was toying with the idea of providing a URL with an encrypted query item, but that requires some record-keeping on the part of the admin. Besides, if login doesn't work, we're really messed up.

I've also made the validation dependent on a checksums.inc.php actually existing. That way we can keep it out of the Lite version, users who don't want to upload it don't have to, and if someone forgets to include it we don't wind up with a flurry of questions.

I'll modify create_release.sh, and it seems to me that serendipity_generateFTPChecksums.php should probably go in the bundled_libs/ directory instead of deployment/. (I really don't understand that particular bit of directory structure, so I'd appreciate some guidance on the correct location for it.)

[judebert]

I'm mostly finished. I changed my mind and decided to leave the checksum generation file in deployment/. I don't understand why the create_release.sh isn't there.

I allow only the installer, the upgrader, and users with siteConfiguration or blogConfiguration permissions to verify the checksums.

I tested on my own sandbox, and it correctly located the files I had modified.

I modified create_release.sh to generate checksums when you make a new release, and to check for php installed before it makes the attempt (on Debian, the php5-cli package takes care of it). The validation steps check for a checksum file existing before attempting to validate, so they won't throw errors for users without checksums.

I tried to use create_release.sh myself, but it failed miserably. I copied the trunk to a new directory and ran create_release.sh from the bundled-libs/ directory. It copied the .svn directories along with the actual code, creating a double-sized tarball. When I unarchived it on my server, I got 500 error messages, so I think it may have messed up permissions, too. I had to restore from a backup.

When I tar the trunk without create_release.sh, and exclude .svn/ directories, I can unarchive on my server and everything works as expected. I see an upgrade screen, and then go along my merry way.

What am I doing wrong with the create_release.sh script? How am I supposed to create a release?

[garvinhicking]

Hi!

deployment/ is only for shared installations. bundled-libs is always there for s9y.

In fact, create_release.sh would best fit into the "tests" subdirectory or a new subdirectory like "build". But I'm not so much for such a new directory and better leave the scripts in bundled-libs. I'd suggest to move your scrip there as well, deployment/ should stay clean only for stubs to be used for shared install.

I tried to use create_release.sh myself, but it failed miserably. I copied the trunk to a new directory and ran create_release.sh from the bundled-libs/ directory. It copied the .svn directories along with the actual code, creating a double-sized tarball. When I unarchived it on my server, I got 500 error messages, so I think it may have messed up permissions, too. I had to restore from a backup.

This is the script I use to release. It is placed in /home/garvin/cvs/serendipity/ and my SVN checkout is in /home/garvin/cvs/serendipity/trunk [but the script checks out its own install]:

Code: Select all

#!/bin/bash
export SVN_SSH="ssh -l garvinhicking"
rm serendipity-1.3.1.tar.gz
rm serendipity-1.3.1.zip
rm -rf serendipity
svn export svn+ssh://svn.berlios.de/svnroot/repos/serendipity/branches/1.3 serendipity
cd serendipity/bundled-libs
./create_release.sh serendipity-1.3.1.tar.gz serendipity nobody nogroup
cd ../../
zip -9 -r serendipity-1.3.1.zip serendipity
tar --owner=nobody --group=nogroup -cjf "serendipity-1.3.1.tar.bz2" serendipity
cd serendipity
mkdir templates_stripped
mv templates/default templates_stripped/
mv templates/carl_contest templates_stripped/
rm -rf templates
mv templates_stripped templates
rm -rf deployment
cd ..
zip -r serendipity-1.3.1-lite.zip serendipity
tar --owner=nobody --group=nogroup -czf "serendipity-1.3.1-lite.tar.gz" "serendipity"
tar --owner=nobody --group=nogroup -cjf "serendipity-1.3.1-lite.tar.bz2" "serendipity"

HTH,
Garvin

[judebert]

Thanks for explaining. I moved thumbnail generation to bundled-libs, and safeguarded it against hacking. I updated the release script and gave it a try.

I hate the 777 protections. POWWeb throws '500 invalid' errors on them, without actually putting anything in the CGI logs. As soon as I set them to 755, everything started working again.

And it did work, perfectly. I'm very pleased.

The resulting tarball was about 40K larger than the original without checksums. The checksums take up nearly 200K themselves. However, everything is designed to work seamlessly if no checksums are provided: the validation will not even be mentioned during upgrades or installs, and the admin item will simply inform the user that there are no checksums for comparison. So, for the LITE version, just leave the checksums out, and no grief will ensue.

I've checked everything into trunk. You might want to have a look at it before the next release; I'm pretty sure I secured it, but an independent review couldn't hurt.

If you want me to merge it into 1.3, let me know.