You've nothing to lose by getting the alternator back into its correct position, as mentioned the tensioner is spring loaded so will automatically tension the belt to the required load, it is adjustable for the purpose of changing the belt or components run by the aux belt.
There are a few guides on here that show how to remove the aux belt and tensioner, I'd personally take the tension off the belt and set the alternator back in place and give things a go, like I said previously I expect the alternator is past its best but it has to be worth trying before replacing