In a fairly even contest, the Sky Blues' clinical finishing put them in command although Chris Brunt's thunderbolt free-kick brought the visitors back into it late on.
Sheffield Wednesday had a great chance to open the scoring inside the first minute when Burton O'Brien crossed for Lee Bullen. His volley appeared to be heading for the top corner but was blocked by Ady Williams.
Coventry started poorly but felt they should have had a penalty in the 14th minute.
Dele Adebola robbed Drissa Diallo and as he headed towards goal the centre-back appeared to trip him. Referee Rob Styles waved away Coventry's claims for a spot-kick much to the disgust of boss Micky Adams.
Although they didn't get the penalty, the incident sparked Coventry into life. Dennis Wise cracked a shot from 25 yards which was turned onto the bar by Owls keeper Chris Adamson.
Michael Doyle almost gifted the visitors the lead on 18 minutes when he lost the ball on the edge of his box to Steven MacLean.
The striker, starting his first game of the season, looked rusty as he shot wide.
Bullen should have put the visitors ahead on 26 minutes when he latched onto a cross from O'Brien but fired wide with goalkeeper Marton Fulop out of position.
Diallo was forced out of the game when he appeared to pull a muscle as he chased for a ball with McSheffrey. It meant a reshuffle for Paul Sturrock as he introduced Chris Brunt and moved Bullen back to centre-back.
John had been quiet for most of the half but came into the game five minutes before the break when he controlled a long ball from Fulop and then cracked just wide of the post.
And then, in first-half injury-time, he got on the end of a cross from Wise and prodded home past Adamson from six yards.
Wednesday could have equalised immediately but Barry Corr blazed over from close range.
Adebola and John almost combined to double City's lead at the start of the second half but Frank Simek did well to clear the danger.
MacLean set-up substitute Brunt at the other end and he looked odds on to score as he raced through, but Fulop saved with his legs.
Wise fired in a sweet cross on 51 minutes but Adebola headed well wide.
O'Brien was twice denied in the space of a few seconds by blocks from Williams before MacLean curled wide as Wednesday started to catch the home team on the break.
Doyle should have finished the game off for Coventry with 18 minutes left after a great run from Adebola, but he fired over the bar from 12 yards.
McSheffrey doubled the home team's lead on 76 minutes when he raced onto Wise's quickly taken free-kick to slot under the body of Adamson.
It looked like Coventry would cruise to victory until Robert Page gave away a free-kick for a foul on Leon Best on the edge of his box.
Brunt stepped up and blasted it into the top corner with keeper Fulop only able to help it in on its way.