Pet Pixels - Comments

Marcus Bointon: PHP Base-62 encoding

marcus@bointon.com (Marcus Bointon) — Thu, 11 Aug 2011 18:50:41 +0100

Yes, your approach is almost identical to my original posting on experts exchange (you might like to use my digit generator function). I'd say this type of encoding uses base conversions. For example I would have no trouble writing a base-32 encoder using gmp - it's possible to treat the input in chunks and get exactly the same result as if you had treated the whole thing as a single multiprecision number. However, you can't do it sensibly with base 62 because it will never fit into an exact number of bits, so you have to pad and chunk for a practical encoder.

ircmaxell: PHP Base-62 encoding

ircmaxell@gmail.com (ircmaxell) — Thu, 11 Aug 2011 18:35:31 +0100

It's not the fastest thing in the world. I'm working on updating that now to make it at least a little more efficient. (so far, I've reduced the runtime by a factor of 4). Also, it is exponential (O(n^2)). So doubling n increases runtime by 4. There are 2 problems with using the gmp function you specify. First, it's not an actual base62 *encode*. It's a base conversion between 16 and 62. That's a very different task. Encoding is taking an arbitrary string and converting it to a destination base. That's what my function does. gmp is unable to do that at all. The second problem, is that gmp only supports bases between 2 and 62. I need to convert to other bases (namely 256) often. So using it is sadly off the table for my needs...

Marcus Bointon: PHP Base-62 encoding

marcus@bointon.com (Marcus Bointon) — Thu, 11 Aug 2011 18:34:24 +0100

Yep, the page titles kept spilling out too. Picked another theme that's a bit more readable. Nobody seems to be doing decent themes for s9y any more.

M: PHP Base-62 encoding

marcus@bointon.com (M) — Thu, 11 Aug 2011 17:48:38 +0100

The coding colors u used in this website are terrible! use a white background vor the code pieces, my brains blow when i try to read this.

Marcus Bointon: PHP Base-62 encoding

marcus@bointon.com (Marcus Bointon) — Thu, 11 Aug 2011 12:16:03 +0100

True, and that's mentioned on the PHP manual page. Doesn't affect me as you won't find me anywhere near either of those distros... :)

Marcus Bointon: PHP Base-62 encoding

marcus@bointon.com (Marcus Bointon) — Thu, 11 Aug 2011 12:13:42 +0100

You're only the nth person to suggest that! There are no available base-64 encodings that are safe to use in all of URLs, domain names, pathnames and email addresses. Base-62 presents no opportunity for problems in any of those use cases (other than on a case-insensitive FS). This is why pretty much all of the url shorteners use base-62.

Anonymous: PHP Base-62 encoding

marcus@bointon.com () — Thu, 11 Aug 2011 11:26:07 +0100

Why not use base64-url encoding? Uses in - and _ insteaad of . and / respectively, and does away with = padding.

John Carter: PHP Base-62 encoding

comment@johnandlouise.com (John Carter) — Thu, 11 Aug 2011 11:07:26 +0100

Note this requires your PHP to be compiled against GMP 4.2.0 or above. If not you're limited to base 36 with this trick. For example RHEL/Centos 5 PHP is compiled against an older version.

Marcus Bointon: PHP Base-62 encoding

marcus@bointon.com (Marcus Bointon) — Wed, 10 Aug 2011 23:31:20 +0100

Apologies for my blog's comment formatting being so awful! Thanks for the pointer, I gave it a try. Your approach is similar to my old one, and for short strings like hashes, it's about 160x slower than using gmp for me, so I'd say it's worth special casing your base converter to trap more efficient conversions when possible. I suspect the time it takes is exponential - while a 1k string took 0.4 sec, 2k took 1.6sec, 4k 6.5 etc. I tried it on a 32k string and gave up waiting for a result after 10 mins! For long strings, you can't treat it as a single number, you need to step through it in small chunks. FWIW the internal base64encode function does 32k in 0.000005 sec :) Incidentally I noticed while trying it out that the gmp internal bases use [0-9A-Za-z] as their char pattern rather than [0-9a-zA-Z], which I guess makes sense as they are in ASCII order.

ircmaxell: PHP Base-62 encoding

ircmaxell@gmail.com (ircmaxell) — Wed, 10 Aug 2011 22:26:36 +0100

I've actually implemented this functionality more generically in my own PHP library (PHP CryptLib). https://github.com/ircmaxell/PHP-CryptLib/blob/master/lib/CryptLib/Core/BaseConverter.php So to do base62 encode/decode:

$characters = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';

$encoded = BaseConverter::convertFromBinary($data, $characters);

$decoded = BaseConverter::convertToBinary($encoded,$characters);

It will accept any binary data (base 256) including straight text, and output the properly encoded string... Basically, the $characters list is just a list of the identifiers for a specific position. For hex, you'd use 0123456789abcdef, for binary you'd use 01. For octal, 012345678. It will handle arbitrary length data (I've tested it out to about 5000 characters for the input, and all 256 possible bases)...

Vadim (Percona): MySQL backups with Percona's XtraBackup

vadim@percona.com (Vadim (Percona)) — Fri, 11 Sep 2009 19:49:53 +0100

And also I want to mention one more benefit over LVM. With LVM you have to allocate 10-20% of total space for snapshot, but with xtrabackup you do not need that. On 500Gb-1TB sizes, 10-20% is good addition.

Marcus Bointon: MySQL backups with Percona's XtraBackup

marcus@bointon.com (Marcus Bointon) — Fri, 11 Sep 2009 18:51:48 +0100

Thanks Vadim, I corrected it.

Marcus Bointon: MySQL backups with Percona's XtraBackup

marcus@bointon.com (Marcus Bointon) — Fri, 11 Sep 2009 18:49:46 +0100

Thanks Peter!

Vadim (Percona): MySQL backups with Percona's XtraBackup

vadim@percona.com (Vadim (Percona)) — Fri, 11 Sep 2009 17:49:05 +0100

Thank you for great article and support of our tool! I have small fix: about "2. Apply the current binary logs to them to bring into a consistent state at a known checkpoint" In real, the script does not work with binary logs, it applies innodb transactional logs. Thanks again !

Peter Zaitsev: MySQL backups with Percona's XtraBackup

pz@percona.com (Peter Zaitsev) — Fri, 11 Sep 2009 17:24:49 +0100

Marcus, Thank you for excellent article.