My handbrake cable was a bit of a PITA to get out, its the most awkward bit about it tbh. I ended up using a centre punch to dislodge the end of the cable (tap it at the top, 12 o'clock. Move around the sides if needs be). Then use a pair of grips/pliers on the swing arm, twist it and the cable will pop out.
Soak everything in penetrating fluid the night before if you can, give everything a right good soaking.