It had looked as though debut boy Liam Dickinson was set to hand County the three points when he powered home a 75th minute header, but Finnigan struck a sweet low strike into the corner to earn the Robins a point.
The home side created the first opening on seven minutes when Allen crossed for Rob Clare who headed narrowly wide.
At the other end JJ Melligan carved open the Stockport defence, but Steve Guinan's thumping low shot was superbly saved one-handed by James Spencer.
Stockport were making a bright start under their new boss, but on 25 minutes Melligan found space in the home box again only to see his header loop over the bar, while just seconds later Damian Spencer glanced a header just past County's far post.
As the opening half wore on the in-form visitors steadily increased the tempo, so it was against the run of play when Stockport opened the scoring on 35 minutes.
Harpal Singh floated in a trademark cross from the left, Cheltenham defender Michael Townsend failed to clear, and there was Matthew Hamshaw who swept home the loose ball from all of eight yards.
Cheltenham should have levelled on the stroke of half-time but Spencer was denied by a smart goal-line clearance from Williams, while at the other end Hamshaw fluffed a great chance to make it two when he rashly scooped a shot over from 12 yards.
Stockport had a let-off 10 minutes after the break when Jamie Victory disappointingly planted a free header over the crossbar, then goalkeeper Spencer was at his very best in racing out to thwart livewire Cheltenham substitute Kayode Odejayi.
However, the visitors did get their leveller on 67 minutes when Odejayi spun and crossed for the onrushing Brian Wilson who fired home from close range before clattering into the post.
Amazingly Wilson fired a shot against the same post minutes later, but Dickinson's dream debut goal did look as though it was going to be enough until Finnigan broke County's hearts.