hammer/suitable drift (screwdriver?). they can take some shifting, had the same bother before
i think it's because the caliper is alloy and will have reacted with the steel/whatever-it-is of the ferrule on the end of the outer cable
ed. need to be careful not to damage the caliper, as alloy's not tough