Why isn't UK voting anonymous?

I voted in the European election today. I was pre-registered and had received my voting card. I went to the polling station where they looked at my card, checked it against my name on the electroal roll, crossed it out, and noted its number on a list next to the unique ID of the ballot paper. I made my choice and put the slip in the ballot box. Much of this process is necessary in order to combat fraud and ballot rigging, however, it's clearly not anonymous - a simple reference back from the paper's ID to my voting card ID means that the government knows what I voted. Knowing what I voted gives no anti-fraud advantage I can see, and means that the collected data is way more sensitive than it needs to be. The simple fact that this knowledge exists is likely to distort voting patterns - people might be afraid of their political leanings being exposed. I'm not particularly worried about that personally, but those with more extreme views (like those that vote for the BNP) might not want this known, and it undermines the absolute right to freedom of association. Political affiliation is regarded as the highest form of 'personal data' under the data protection act, along with ethnicity, sexual preference, medical records etc, and the best way to avoid problems with this kind of data is not to keep it in the first place, as the numerous leaks/losses/exposures of government information have highlighted. So why is it done? Aren't elections meant to be done by secret ballot? Did I miss something? Why aren't more people disturbed by this?