Personally, by hand I'd go with polish with fillers in so SRP (especially the new formula one) is exactly what you need, maybe go for a glaze (Poorboys Blackhole or similar) and then wax. That should leave you with a superb finish by hand.
To actually do full correction by hand would take some serious hard work but as said you'd need something more abrasive to actually remove them. I'd rather wait until I had access to a DA and have a go at them a bit more easily.
Lets face it, swirl marks are corrected for cosmetic purposes so by hand I'd aim to fill by machine I'd aim to correct.