Many Northerners were more than comfortable moving South directly after the Civil War (and up through the mid-1870s). History's motives are never so black and white as some people try to make them out to be. There have long been two conflicting stories about why so many Northerners (including many Union Generals even) went South for Reconstruction.
They were greedy, exploitative adventurers who manipulated the illiterate black vote for their own interests. They used their power to raise property taxes, confiscate farms, and did little to help the people they claimed to be helping (i.e. the freed slaves).
They were idealists who strove to ensure that the 13th, 14th, and 15th Amendments were enforced and that freed slaves were educated and given a legitimate opportunity to succeed. If it took higher taxes to build a real education system in the South, that was simply an unfortunate reality.
Really, it's a mess of both. It's not so different from what you see today with charitable, political, or religious leaders who get caught swindling the public. Some people are clean, upright, and idealistic. Others take the attitude of, "Hey, it's good to help people, and if you can become filthy rich in the process, all the better!"
My personal opinion is that Reconstruction was six of one, and a half-dozen of the other, though the Republicans really messed up the freed slaves by making themselves so hated and then leaving so quickly when the Southerners put up actual resistance.