Pet Pixels - Techie

Basic Linux-HA heartbeat and pound config

marcus@bointon.com (Marcus Bointon) — Wed, 02 Dec 2009 17:05:36 +0000

I've had trouble in the past setting up Linux-HA a.k.a heartbeat to create a redundant front-end for a web service. The HA documentation is quite thorough and detailed, but it's lacking realistic examples to get a working system up and running. This is a quick guide to creating a 2-way cluster with one or more floating IP addresses and automatic failover on Ubuntu 9.04. I also include a quick config for pound, a reverse proxy / load balancer.

Continue reading "Basic Linux-HA heartbeat and pound config"

MySQL backups with Percona's XtraBackup

marcus@bointon.com (Marcus Bointon) — Fri, 11 Sep 2009 12:18:00 +0100

MySQL backup is sometimes very hard to do effectively. MySQL provides various options for backup, but many of them are simply unsuitable for large systems, particularly if they need to remain active during backups. Percona's XtraBackup is an open-source clone of InnoBase's InnoDB Hot Backup utility. So what makes XtraBackup a better solution, and how does it work?

Update: on December 10th 2009, Percona released Xtrabackup 1.0.

Continue reading "MySQL backups with Percona's XtraBackup"

Google Charts API Simple and Extended Encoders in PHP

marcus@bointon.com (Marcus Bointon) — Fri, 14 Aug 2009 09:34:27 +0100

Google's charting API has been around for quite a while now, but I've only just needed to actually look at it. It became immediately obvious that I needed a PHP encoding function, so off to google I went. Though I found several implementations, they were all incomplete or deficient in one way or another (and it didn't help that there was an error in google's extended encoding docs), so I've written my own based on several different ones. Both simple and extended encoders support automatic scaling, inflated maximum and lower-bound truncation, so you can pretty much stuff whatever data you like in, with no particular regard for pre-scaling and you'll get a usable result out. They have an identical interface, so you can use either encoding interchangeably according to the output resolution you need (contrary to popular belief, the encoding to use has very little to do with the range of values you need to graph). By default, the full range of possible values is used as it just seems silly not to. I deliberately omit the 's:' and 'e:' prefixes so that you can call these functions for multiple data series, and I include a function that does just that. You still need to generate your own URLs and other formatting, but that's a different problem. Read on for the code...
Continue reading "Google Charts API Simple and Extended Encoders in PHP"

Subversion 1.5 repository upgrade on SourceForge

marcus@bointon.com (Marcus Bointon) — Tue, 05 May 2009 11:19:04 +0100

I've just had a slightly tricky time upgrading a subversion repository on sourceforge. They have recently added support for subversion 1.5 at the server end. 1.5 brings major new features for merging, but as it's not backward compatible with older subversion clients, the upgrade is not done automatically. SF have also done a major rearrangement of their documentation while transferring everything to Trac, and it's not always easy to get the right info. Normally to upgrade a subversion repo, you just run the 'svnadmin upgrade /path/to/repo', however, it's not quite so simple on sourceforge as you don't have direct access to the repo, and the instructions they give are slightly wrong at the time of writing. You're likely to get an error like this (it's not obvious that this is a fatal error) when you reload a dump file:

svnadmin: File already exists: filesystem '/svnroot/projectname/db', transaction '443-0', path 'tags' \* adding path : tags ...

This is because load is intended to add files to an existing repo, not to replace those that are already there, so you need to wipe the repo and start from scratch. So, here is a working command sequence that needs to be run from a project login shell on sourceforge (it applies to the project you're logged in through, substitute your project's name for projectname):

adminrepo --checkout svn svnadmin dump /svnroot/projectname > svn.dump rm -rf /svnroot/projectname/\* svnadmin create /svnroot/projectname svnadmin load /svnroot/projectname < svn.dump adminrepo --save svn

Yes, you do need to delete the whole thing and re-import it, but it's quick and easy, and you have a backup in the dump file you take at the start. After the upgrade, make sure you get a new checkout of your project to ensure that you're using 1.5 all the way through. Now you'll find that commands like 'svn merge --reintegrate' work.

Speaker & room calibration

marcus@bointon.com (Marcus Bointon) — Fri, 20 Mar 2009 18:45:54 +0000

I was lucky enough to pick up a Behringer Ultracurve Pro DSP8024 for a mere £50 on eBay recently. It turned out to have a buggy OS version (1.2), and Behringer very kindly sent me a replacement EPROM with a new 1.3 OS on it, which works just fine. I now have it installed between my Soundcraft mixer and my Wharfedale active monitors. I used its "Auto-Q" calibration routine and put up with some quite loud pink noise to calculate a room correction curve. Because it knows the spectrum it's generating, it assumes that what it gets back has been altered by the combination of speakers, room and microphone, so it can calculate an eq map to compensate for it. It's quite fun to watch as it has a nice big LCD screen to display the 31 1/3 octave bands - the initial spectrum is fairly peaky, but as it iteratively applies corrections you see (and hear) it flattening out. It's also very obvious that my monitors don't put out much below 50Hz (it analyses down to 20Hz), but that's to be expected from a moderately sized box with a 6.5" driver. The results are really pretty good, sounds lovely and smooth, but the real surprise is when you've been listening to it for a while and you switch out the EQ - it's really quite a shock to hear the uncorrected version. Lots of purists don't like room correction by EQ, saying it's better to fix the room in the first place, and also that EQ calculated like this is highly dependent on the listening position (which it is). I have a lots of bookshelves facing my speakers; they make fantastic diffusers, and I have some Universal Acoustics absorber tiles on the sloping ceiling above my listening position. The longest room mode will be fairly undamped (I'm not about to start hanging duvets around the walls!), but the resulting EQ is below 6db in either direction across the whole range - I've heard of rooms with 24db peaks! Anyway, after all that, it sounds lovely, and I'm happy!

The web developer's holy vhost trinity

marcus@bointon.com (Marcus Bointon) — Fri, 20 Mar 2009 12:11:12 +0000

When you're developing web stuff, working with projects in path names (i.e. not at the top level of a domain) can be difficult (gets in the way of absolute links, rewrite rules etc), so you often need to set up a local apache virtual host, stick an entry in DNS and create an SSL certificate before you can get on with the serious business of doing some real work. This can get to be a drag when you do it a lot, but there is an extremely elegant solution that means you'll never have to do it again...
Continue reading "The web developer's holy vhost trinity"

Changing MacBook Bezel

marcus@bointon.com (Marcus Bointon) — Mon, 09 Mar 2009 13:15:11 +0000

I recently had cause to change the bezel on my rev A 13" MacBook. If you follow the guide on MacFixit, it is a long and complicated procedure, involving dismantling nearly everything. Luckily, most of it is unnecessary - it's really quite easy to simply remove the bezel without dismantling anything; just jump straight to step 35 and carry on from there.

Web Hooks, Callbacks and Distributed Observers

marcus@bointon.com (Marcus Bointon) — Wed, 04 Mar 2009 14:44:37 +0000

Someone at AMEE pointed out to me that there's been a flurry of activity around so-called "Web Hooks" when I referred to the concept. This is quite heartening as I thought of this a couple of years ago and implemented this in Smartmessages early last year! I call them callbacks, but the idea is the same - it's essentially a distributed observer pattern. I couldn't figure out why nobody seemed to understand what I was on about... When I get some interesting event (e.g. a message open, mailshot completion, clickthrough etc), I ping a user-supplied URL with the appropriate event data, pretty much the one-liner that Jeff alludes to. The reason we do it is that sync with external systems (usually CRM) is something that were always running into, and there seems to be no sensible, generic way of dealing with it other than this, so I'm surprised it has not been discussed in this context before. There's one downside as far as I can see - it is highly dependent on the receiver to be able to handle the event in a timely fashion. This isn't an issue if you're connecting say, Yahoo! to Google, but it could be a big problem if you connect Google to your Wordpress blog... My experience of CRM systems is that they are simply too slow to cope with the high rates of traffic that we are likely to generate, for example, if we point a stream of ~200 events per second at a CRM system, it will probably just bog down and fail (I'm thinking of the SalesForce API here which typically takes 1-2 sec to deal with a single SOAP API call). Retrying will only make this worse. I have two solutions for this: limit events to those that don't happen so often (kind of lame!), or alternatively, use an outbound message queue to rate-limit the sending (Amazon SQS and Memcacheq spring to mind). Queueing works, but you lose some of the real-time aspect. Ideally clients would implement their own incoming queue in order to allow them to process events at their leisure, but this is mostly beyond the vast majority of web authors (or at least those that host the CRM systems that we hear from!). Anyway, it's nice to know that I'm not completely barking...

Scalable irony

marcus@bointon.com (Marcus Bointon) — Mon, 02 Mar 2009 15:52:13 +0000

This article on highscalability.com is a really excellent rundown of some of the options available for scaling a site to the heights of Digg. Ironically enough, at the time of writing the highscalabilitycom web server shows this error:

Unable to connect to database server The MySQL error was: User highscal_admin already has more than 'max_user_connections' active connections.

I can point them at this really good article on how to avoid problems like this... oh, wait...

Hard Drive Story

marcus@bointon.com (Marcus Bointon) — Fri, 24 Oct 2008 15:24:43 +0100

So my MacBook was just sitting on a desk, idle. It was showing a screen saver or something. I woke it up and was greeted with a spinning beachball of death. This seemed to be an unusually persistent hang, so I killed the power and rebooted. I get a flashing system folder, no OS found. I reboot from a Leopard DVD and run disk utility. My HD does not even appear on the SATA bus. Around this time I notice a very quiet repeated clicking coming from the HD and I get a bad feeling about the dreaded "click of death". I power off and reboot from an external drive, but still no sign of the internal one. It seems the drive is dead. Now is the time to contemplate my backups. I have some recent enough Time Machine backups on an external drive, so I restore one of those onto the other external. That all works, but when I reboot into it and try to log in, I get an odd error about not being able to log in due to a FileVault problem, which I was indeed using. A bit of googling reveals the sad truth: Time Machine and FileVault do not play nicely together. Or more to the point, Time Machine just ignores everything protected by 'FailVault', that is to say if backs up everything except the bits that you actually want backed up. It turns out that it will back stuff up, but only if you log out. Hm. Logging out is for people that want to live without the instant-on we've come to love from Apple, and don't mind waiting 5 minutes to for their 10 apps to launch on login, i.e. those that don't have laptops. So I extract the internal HD and ship it off to a data recovery company. After a day or so I get the bad news:

We regret to inform you that we have tried our very best to recover data from the faulty Hard Disk Drive you sent to us, unfortunately we were unable to recover any data. Please accept our apologies for not bringing our services to your satisfaction, despite the fact that from a technical point of view, the damaged inflicted on HDD is beyond repair and it is absolutely impossible to recover any data as the HDD had suffered from severe internal mechanical failure accompanied by media damage, therefore the extreme nature of the damage made it impossible to recover any data. Despite the usage of different components to get the HDD to spin, the internals were too damaged to read any data from the HDD. The effect of the media damage is immediate on the magnetic information stored on the drive, jeopardising the stored data files and the logical structures. You can view the scratched area of the hard drive platter in the attached picture, it is the thick dark ring you see running around the inner part of the platter.

I've attached that very picture and you can see there is some fairly obvious mechanical damage to the drive - a great (for want of a better word) example of a head crash. The chances of a 50Gb encrypted volume file surviving that intact are pretty slim, as they say. I'm pretty surprised that a modern, state-of-the-art hard drive can suffer spontaneous catastrophic damage like this without having experienced any physical shock. For those that want to know (so you know what to avoid), the drive was a 160Gb Seagate Momentus 5400.3, a bit under 2 years old. It's still under guarantee, but a fat lot of use that is now. I need to reconsider my backup and encryption options...

eBay's new page design

marcus@bointon.com (Marcus Bointon) — Wed, 24 Sep 2008 11:40:28 +0100

I took a look at eBay's new page design, which they are currently previewing. Verdict: needs lots more work.
Continue reading "eBay's new page design"

ClamAV broken version check

marcus@bointon.com (Marcus Bointon) — Wed, 03 Sep 2008 13:40:20 +0100

If you've just built and installed the new version 0.94 of ClamAV (because it reminded you that your current version was out of date), and you're using the experimental features, it exposes a rather dumb bug that results in these messages in your freshclam log:

@4000000048be74fd0687f57c Software version from DNS: 0.94 @4000000048be74fd0688245c WARNING: Your ClamAV installation is OUTDATED! @4000000048be74fd06883014 WARNING: Local version: 0.94-exp Recommended version: 0.94

Apparently ClamAV's version comparison is not terribly clever, and the 'new feature' of marking experimental versions with '-exp' has tripped it up. I'm sure there are multiple workarounds, but I changed this in shared/misc.c:

#ifdef CL_EXPERIMENTAL /*#define EXP_VER "-exp"*/ #define EXP_VER #else #define EXP_VER #endif

as the EXP_VER string is what gets appended to the version number (apparently only for display purposes, but it does the trick). I then rebuilt it and now my log says:

@4000000048be848926da098c Software version from DNS: 0.94

Yay - no error!

Safari 4 preview

marcus@bointon.com (Marcus Bointon) — Tue, 26 Aug 2008 15:04:52 +0100

I just grabbed the new update to the Safari 4 preview, and I have to say it's very zippy indeed. I ran SunSpider benchmarks on it and FireFox 3.0.1 on my Mac Pro. Safari 4 is on average 75% faster, but has many results that are in the 200 - 500% range. This is only measuring javascript performance of course, but that, plus whatever other changes are in S4, translates to incredible speed in daily use. I use the SafariStand extension to remember my open tabs, and I can reopen about 40 tabs in 8 windows in about 3 seconds, including all their content! FF 3 was a huge improvement over 2, but Safari 4 is a different league. This is only a preview, so it will probably get faster still by release. The integration of the JS debugger is very elegant (Drosera has been rolled in), but I'm still looking forward to the appearance of FireBug for Safari, which someone is doing as a Google SoC project (S4 adds some FireBug hooks too). All the new CSS funkiness from recent webkit nightlies is there too, and I suspect many an iPhone and widget developer is drooling over them...

Apache mod_rewrite bug still lurks

marcus@bointon.com (Marcus Bointon) — Wed, 20 Aug 2008 22:02:43 +0100

There's this enormous apache mod_rewrite bug that I ran into back in 2005, and to my dismay, it's still there. Long-standing bugs are usually small edge cases that don't affect many people, but this one is a monster that I suspects pretty much everyone that's using mod_rewrite, and they've just been lucky in avoiding it. The basic issue is this: if you match params that require URL encoding to be safe, mod_rewrite will not rewrite the back-reference (that's $1 below) correctly. So take this very simple redirect: RewriteRule ^(.*)$ index.php?show=$1 [R] so you hit http://www.example.com/a%2Fb and mod_rewrite neatly rewrites it as http://www.example.com/index.php?show=a/b! Notice that it has urldecoded the matched parameter in the replacement string. Apache 2.2 introduced a new B flag to deal with this, but that apparently suffers the same problem! There are two workarounds I've used that are both horrible: double-encode the source string (if you are in control of both start and end points of the URLs) to survive the spurious urldecode, or base-64 encode (javascript flavour) it and do the decoding yourself. I did warn you they were horrible. I'll bet that there are a zillion mod_rewrites out there that suffer from this fundamental problem and haven't even noticed. If a few people voted for this to be fixed, it would probably go away...

Subversion MERGE error on commit nailed

marcus@bointon.com (Marcus Bointon) — Wed, 20 Aug 2008 17:35:32 +0100

Recently I've had a problem with subversion commits returning a '200 OK' result, accompanied by a peculiar MERGE error (when I'm not doing a merge), but weirdly, the commit had actually made it into the repo. I'd googled for this a while ago with no success, but I had another try today and found this post. Thinking about it, I'd had this problem since updating to the very-very-long-awaited trac 0.11 release. So I tracked down the latest release of the trac-post-commit-hook script and installed that, and amazingly enough, it just works. Thank you UberGeek!