The home side were forced into a substitution six minutes later when Chris Baird replaced full-back Gareth McAuley, who suffered a hamstring injury.
Northern Ireland offered virtually nothing in attack - David Healy volleyed well over with a rare opportunity.
Slovenia went close on the stroke of half-time when Novakovic's free-kick clipped off Chris Baird and over the bar.
George McCartney and Jonny Evans were booked in the first half while Zlatko Dedic was cautioned for diving.
McCartney had picked up a calf injury and was replaced at half-time by Ryan McGivern.
There was more urgency from Northern Ireland on the restart and they created their first clear opportunity on 57 minutes.
Grant McCann's corner found the unmarked Damien Johnson but the midfielder headed over from 12 yards.
Warren Feeney says Northern Ireland are capable of going all the way
Slovenia remained dangerous and West Brom midfielder Koren unleashed a fierce 25-yard shot after 67 minutes which smashed against the crossbar.
But it was Northern Ireland who made the breakthrough six minutes later with Dundee United striker Feeney sending his deft near-post header into the bottom corner from Sammy Clingan's cross.
David Healy was having a quiet night but he found space to hit a powerful shot which was blocked by keeper Jasmin Handanovic.
Novakovic burst through with five minutes left but blazed his shot over.
Baird was booked in the dying seconds for time wasting and is suspended for the next qualifier against Poland in September.
Northern Ireland stay a point clear of Slovakia, who defeated the Czech Republic 2-1.
Poland moved up to third after hammering minnows San Marino 10-0.
Bookmark with:
What are these?