Former Rotherham striker Deon Burton struck the decisive goal for Wednesday at the start of the second half to condemn his former club to defeat.
The League Two side clearly fancied their chances of an upset against their local rivals, who had endured an opening day drubbing at Ipswich.
But it was Wednesday who threatened first on 14 minutes when Marcus Tudgay struck a post with a brave header.
Rotherham hit back immediately as Chris O'Grady curled a shot against the Wednesday crossbar as the Millers settled into the game.
Stephen Brogan brought a smart save from Lee Grant with a stinging drive before Rotherham's pressure told.
Brogan's free kick on 33 minites picked out Danny Harrison and the former Tranmere man converted from a Derek Holmes lay off.
Grant then made a point-blank save to deny Andy Todd as Rotherham threatened to double their lead.
But it was Wednesday who turned the game on its head on the stoke half time.
Wade Small swung over a cross from the left which Jermaine Johnson cushioned into the path of Glenn Whelan who fired the ball into the bottom corner from 25 yards.
The goal was to prove a turning point and just four minutes into the second half Burton fired Wednesday in front after Johnson's stinging shot was saved by Andy Warrington.
And six minutes later the comeback was complete when Small's attempted cross deceived Warrington and curled into the top corner.
Millers substitute Mark Newsham had Rotherham's best chance to pull them back into the game but by this time Wednesday were well in control and could have extended their lead when Leon Clarke ran clear but saw his shot saved by Warrington.