Pet Pixels - PHP

MySQL backups with Percona's XtraBackup

marcus@bointon.com (Marcus Bointon) — Fri, 11 Sep 2009 12:18:00 +0100

MySQL backup is sometimes very hard to do effectively. MySQL provides various options for backup, but many of them are simply unsuitable for large systems, particularly if they need to remain active during backups. Percona's XtraBackup is an open-source clone of InnoBase's InnoDB Hot Backup utility. So what makes XtraBackup a better solution, and how does it work?

Update: on December 10th 2009, Percona released Xtrabackup 1.0.

Continue reading "MySQL backups with Percona's XtraBackup"

Google Charts API Simple and Extended Encoders in PHP

marcus@bointon.com (Marcus Bointon) — Fri, 14 Aug 2009 09:34:27 +0100

Google's charting API has been around for quite a while now, but I've only just needed to actually look at it. It became immediately obvious that I needed a PHP encoding function, so off to google I went. Though I found several implementations, they were all incomplete or deficient in one way or another (and it didn't help that there was an error in google's extended encoding docs), so I've written my own based on several different ones. Both simple and extended encoders support automatic scaling, inflated maximum and lower-bound truncation, so you can pretty much stuff whatever data you like in, with no particular regard for pre-scaling and you'll get a usable result out. They have an identical interface, so you can use either encoding interchangeably according to the output resolution you need (contrary to popular belief, the encoding to use has very little to do with the range of values you need to graph). By default, the full range of possible values is used as it just seems silly not to. I deliberately omit the 's:' and 'e:' prefixes so that you can call these functions for multiple data series, and I include a function that does just that. You still need to generate your own URLs and other formatting, but that's a different problem. Read on for the code...
Continue reading "Google Charts API Simple and Extended Encoders in PHP"

Subversion 1.5 repository upgrade on SourceForge

marcus@bointon.com (Marcus Bointon) — Tue, 05 May 2009 11:19:04 +0100

I've just had a slightly tricky time upgrading a subversion repository on sourceforge. They have recently added support for subversion 1.5 at the server end. 1.5 brings major new features for merging, but as it's not backward compatible with older subversion clients, the upgrade is not done automatically. SF have also done a major rearrangement of their documentation while transferring everything to Trac, and it's not always easy to get the right info. Normally to upgrade a subversion repo, you just run the 'svnadmin upgrade /path/to/repo', however, it's not quite so simple on sourceforge as you don't have direct access to the repo, and the instructions they give are slightly wrong at the time of writing. You're likely to get an error like this (it's not obvious that this is a fatal error) when you reload a dump file:

svnadmin: File already exists: filesystem '/svnroot/projectname/db', transaction '443-0', path 'tags' \* adding path : tags ...

This is because load is intended to add files to an existing repo, not to replace those that are already there, so you need to wipe the repo and start from scratch. So, here is a working command sequence that needs to be run from a project login shell on sourceforge (it applies to the project you're logged in through, substitute your project's name for projectname):

adminrepo --checkout svn svnadmin dump /svnroot/projectname > svn.dump rm -rf /svnroot/projectname/\* svnadmin create /svnroot/projectname svnadmin load /svnroot/projectname < svn.dump adminrepo --save svn

Yes, you do need to delete the whole thing and re-import it, but it's quick and easy, and you have a backup in the dump file you take at the start. After the upgrade, make sure you get a new checkout of your project to ensure that you're using 1.5 all the way through. Now you'll find that commands like 'svn merge --reintegrate' work.

The web developer's holy vhost trinity

marcus@bointon.com (Marcus Bointon) — Fri, 20 Mar 2009 12:11:12 +0000

When you're developing web stuff, working with projects in path names (i.e. not at the top level of a domain) can be difficult (gets in the way of absolute links, rewrite rules etc), so you often need to set up a local apache virtual host, stick an entry in DNS and create an SSL certificate before you can get on with the serious business of doing some real work. This can get to be a drag when you do it a lot, but there is an extremely elegant solution that means you'll never have to do it again...
Continue reading "The web developer's holy vhost trinity"

Web Hooks, Callbacks and Distributed Observers

marcus@bointon.com (Marcus Bointon) — Wed, 04 Mar 2009 14:44:37 +0000

Someone at AMEE pointed out to me that there's been a flurry of activity around so-called "Web Hooks" when I referred to the concept. This is quite heartening as I thought of this a couple of years ago and implemented this in Smartmessages early last year! I call them callbacks, but the idea is the same - it's essentially a distributed observer pattern. I couldn't figure out why nobody seemed to understand what I was on about... When I get some interesting event (e.g. a message open, mailshot completion, clickthrough etc), I ping a user-supplied URL with the appropriate event data, pretty much the one-liner that Jeff alludes to. The reason we do it is that sync with external systems (usually CRM) is something that were always running into, and there seems to be no sensible, generic way of dealing with it other than this, so I'm surprised it has not been discussed in this context before. There's one downside as far as I can see - it is highly dependent on the receiver to be able to handle the event in a timely fashion. This isn't an issue if you're connecting say, Yahoo! to Google, but it could be a big problem if you connect Google to your Wordpress blog... My experience of CRM systems is that they are simply too slow to cope with the high rates of traffic that we are likely to generate, for example, if we point a stream of ~200 events per second at a CRM system, it will probably just bog down and fail (I'm thinking of the SalesForce API here which typically takes 1-2 sec to deal with a single SOAP API call). Retrying will only make this worse. I have two solutions for this: limit events to those that don't happen so often (kind of lame!), or alternatively, use an outbound message queue to rate-limit the sending (Amazon SQS and Memcacheq spring to mind). Queueing works, but you lose some of the real-time aspect. Ideally clients would implement their own incoming queue in order to allow them to process events at their leisure, but this is mostly beyond the vast majority of web authors (or at least those that host the CRM systems that we hear from!). Anyway, it's nice to know that I'm not completely barking...

PHP London Conference 2009

marcus@bointon.com (Marcus Bointon) — Mon, 02 Mar 2009 15:58:36 +0000

From this truly excellent conference, I took away some good memories, some new ideas and a nasty bout of conference flu. There's nothing quite like being in close proximity to a few hundred people to really spread things around... Highlight for me was Aral Balkan's keynote. It's always nice to see someone showing plain enthusiasm, and I couldn't agree more with him about the "lost magic" of computing. Had a chat with him afterwards about AMEE and other things. He also seems to have put together some odd but dull things that I had noticed a need for - EU VAT codings and ISO language references as web services! I didn't really enjoy David Soria Parra's talk on sharding. It all came across as very negative and many of the ways of doing it and coping with the fallout were not really discussed. No mention of MySQL 5.1's partitioning (which is limited, but is at least a start), or more radical approaches like Sequoia. David Axmark's talk on Drizzle was more interesting than I expected, nice to see effort being put into this direction. Microsoft really does seem to be trying a bit harder these days - their CSS test suite for IE8 is very welcome, and the effort they are putting into PHP, apache and other projects benefits many people. It has to be said that while it's not a mainstream product, Surface is really pretty cool to play with. Chris Shiflett's talk was excellent too; his demos and examples were particularly good, and entertaining. The post-conference social was great fun, I met lots of nice new people. After our move to France I suspect it will be harder to get to events like this, so I should make the most of them while I can! I've had several ideas for talks that I'd like to do (I get sick of email sometimes!), so I guess I need to get a bit more proactive on actually submitting them to a call for papers.

Scalable irony

marcus@bointon.com (Marcus Bointon) — Mon, 02 Mar 2009 15:52:13 +0000

This article on highscalability.com is a really excellent rundown of some of the options available for scaling a site to the heights of Digg. Ironically enough, at the time of writing the highscalabilitycom web server shows this error:

Unable to connect to database server The MySQL error was: User highscal_admin already has more than 'max_user_connections' active connections.

I can point them at this really good article on how to avoid problems like this... oh, wait...

Genius on Experts Exchange

marcus@bointon.com (Marcus Bointon) — Sat, 01 Nov 2008 10:46:00 +0000

I finally got my 'Genius' rank in PHP on Experts Exchange today. It's taken about 3 years and in that time I've answered over 1,000 questions and participated in over 500 others, making about 3,300 posts. I think I've helped a lot of people, but I've learned a lot too. And I get a free T-shirt. Now why did I choose such a silly user name?

PHPLondon08 follow-up

marcus@bointon.com (Marcus Bointon) — Mon, 03 Mar 2008 16:49:28 +0000

I think it was Juliette that was asking me about spam filtering in PHP. I didn't think that was such a great idea (spamassassin is slow enough!), however, it seems that some others do. The framework shootout session was very interesting. It was nice to see Toby jumping into CodeIgniter with such ease - it contrasted markedly with my own difficult initiation into CakePHP! There was mention of Kohana which I'd only seen a couple of weeks previously, but looks pretty good - I had always been put off CI by its support for PHP4, so a PHP5-clean version is very appealing. This article spells out the differences. The shootout was, um, augmented by a heckler who had decided that all frameworks were "criminal" because they didn't conform to the letter of the "rules of MVC" (though MVC is really a pretty loose term AFAIAC). He wouldn't name his mystery preferred framework, but I later found out it was Agavi when I saw it demo'd by its creator. It is indeed very nice, and has distant ancestry in Mojavi (as used by Symfony). One of the things that I was impressed by is that it uses (of all things) HTML for marking up templates. This is an elegant reversal of what I've seen before, where form elements are typically generated by helpers (requiring syntax that you don't know). Why mark up a form input like this (vaguely CakePHP style): <?php echo $htmlhelper->textinput('name', 'name', 20, 50); ?> when you could do: <input type="text" id="name" name="name" size="20" maxsize="50" /> and still get automatic server and client-side validation and ajaxy feedback goodness, but also have it play nicely with HTML editors like Dreamweaver? It also makes for a great deal of sense when generating non-HTML output. I can't think why other frameworks have not done this before. Agavi also has very elegant routing so that a single controller function is available across all access methods, whether HTML, JSON, SOAP, XML-RPC or whatever. Makes CakePHP look very dumb. Anyway, so maybe this heckler had a point of sorts. I have to say a big thank you for the compliments I've received about my talk. Maybe it wasn't so bad after all! I'll get on with doing the remaining audio ASAP...

PHPLondon

marcus@bointon.com (Marcus Bointon) — Fri, 29 Feb 2008 23:45:00 +0000

PHPLondon08 was quite a success. I really enjoyed it, met lots of great people. The talks were generally good (I'll get to mine!). The only downer I thought was that there was really quite a bit of overlap between Ivo's intro and Mike & Scott's one immediately after it, though both were good in their own right (and I liked Ivo's bricks analogy). Mine went mostly ok, but I got the timing completely wrong, spent far too long on the first half, and I'd not even reached halfway through the second half when I spotted Richard (the room manager) holding up the "1 minute remaining" sign - I'd completely forgotten about time allocated for questions! That said, after being really quite nervous beforehand (which is generally unlike me!), I found it pretty easy once I got going, though taking it too easy is probably what made it take too long. Despite all that, there were plenty of good questions asked (no tumbleweed effect!), and I got some great compliments. Someone even said it had been their best talk of the day, so I must have been doing something right. The slides are linked below, and I'll also try to record audio of the bits I didn't get to, as otherwise my talk's MP3 will have a rather feeble ending! A real surprise appeared just before Derick's excellent keynote - a combination of PHP history and best-practice - three guys from PHP Barcelona invited me to speak at their conference in September! I think it's kind of provisional, but it was very flattering to be asked! I mentioned in the talk, and talked to various people, about writing an email book. One point came up that led me to think that there should be a guide for marketers - there simply isn't any good, accessible information on what's legal and what's not, and practical advice about what they should ask of their email service provider in implementing marketing campaigns. For those who want them, here are PDFs of my talk(s):

Part 1: mail(); (6.8Mb PDF)
Part 2: Life after mail();.pdf (7.6Mb PDF)

FYI, I ground to a halt on slide 13 of 28 in the second one.

GraphViz & SVG frustration

marcus@bointon.com (Marcus Bointon) — Sun, 10 Jun 2007 22:51:59 +0100

I thought I'd do a little warm-up hack for HackDay, so I wrote a little thing to interrogate the backnetwork API and build a network diagram of all the relations using GraphViz. It started out looking quite promising, but I soon discovered that lots of GraphViz features just don't work very well (even simple stuff like putting labels in boxes), and also that common SVG implementations suck. Firefox' SVG rendering is just awful - no anti-aliasing, lots of misdrawing, all text defaulting to 18px Courier bold and worse. WebKit's nightlies are in a different league in terms of accuracy and rendering quality, but since scrolling is broken, large diagrams are kind of hard to display. GraphViz's layout mechanisms are pretty cool (if rather unpredictable), but the errors it makes for labelling stuff just makes it unusable. Its output directly to PNG was worse than the SVG renderings. This was my first attempt at using both GraphViz and SVG, and I have to say I'm massively underwhelmed. I hope that I'm using a bad version, or there's some issue with my font configs as it really should be possible to get decent output from what I've gathered. Update Thanks to Bruno Pedro, I have something to compare against, and now something to point the finger at... It appears that the OSX version of GraphViz from fink is a waste of space, and simply doesn't support some vital features. Secondly, it's clear that Mac Firefox' SVG rendering is totally broken, but the windows version works ok (in Parallels). Thanks Bruno!

Me and Tony, we're like this...

marcus@bointon.com (Marcus Bointon) — Sun, 10 Jun 2007 22:34:55 +0100

I spent most of May building the back-end for Global Cool's MyCO₂ site, and I've just seen this little clip they put up. It's not every day you get a prime minister directly endorsing your stuff. Good stuff Global Cool!

CakePHP's invisible query cache

marcus@bointon.com (Marcus Bointon) — Wed, 23 May 2007 01:46:13 +0100

I just wanted to document something that was driving me nuts in Cake PHP. I do a simple find on a model: $thing = $this->Model->find(array('id' => 123), NULL, NULL, 1); Then later on, I make the same query again. Only the first query appears in the SQL debug output. Why call it again? Well, I've made some manual queries in between the calls using SQL that is inconvenient to emulate in Cake functions, in particular INSERT IGNORE. It turns out that Cake has a largely undocumented query cache, but Cake has no idea that its cache is dirty because of the manual queries. The _clearCache function is not yet implemented for the query cache, so the only choice is to disable it before the first query, and you can do that during a single instantiation by saying: $this->Model->cacheQueries = false; Note that the query has 1 level of recursion, which means that all the sub-queries are also cached. Setting cacheQueries to false on this one model means that all queries through it don't get cached, including those that may be in other models via relationships.

CakePHP hasManyAndBelongsTo relationship limitations

marcus@bointon.com (Marcus Bointon) — Mon, 14 May 2007 02:58:53 +0100

Cake has some nice support for modeling many to many relations using 'hasManyAndBelongsTo'. There's a simple example here. Though this works well for simple connections between classes, you can't do much else. Taking the classic Blog-style Posts and Tags example, this approach will easily let you link multiple tags to multiple posts. But what if you want to track the date that a tag was applied to a post? If you add a date field to the join table, it seems pretty much impossible to retrieve it, and setting it is even harder. You can search on it by twiddling with the conditions on the relation, but that's ugly. I think in that case, you have no option but to model the relation explicitly with two pairs of hasMany/belongsTo relations linked through a PostTag model, and then you'll have to set your recursion deeper and get busy with unbindmodel in order to keep the number of queries down. Who knows, I might even write up an example and post it here...

PHP London Take 4

marcus@bointon.com (Marcus Bointon) — Sat, 12 May 2007 01:41:47 +0100

A simplified London shape (Based on Sadiq's) with an exaggerated river:

It'd difficult to place the text without interference with the river, so I thought I'd try it offset.