Michael Cooper had put City ahead in the opening minutes at the Don Valley stadium, but the Millers equalised after 20 minutes courtesy of a Nick Daws free kick.
Rotherham won the game in the second half with Mark Robins giving them the lead, scoring from the rebound of Paul Warne's 58th minute shot that hit the post.
Warne sealed the 3-1 win with a goal late on to give Rotherham their second win in two games.