Celtic were below par in their narrow Scottish Premiership win at St Mirren last weekend and, when they were undone early on, there were fears it could be another difficult night in Europe.
One long pass took out the visitors' defence and home captain Sem Steijn teed up Japan striker Ueda, who beat Kasper Schmeichel for his 14th goal of the season.
However, Celtic were unfazed by the early setback and roared back in fine style.
Yang arrived late in the box to flash a first-time finish past Timon Wellenreuther for the equaliser and the home goalkeeper gifted Celtic a second 12 minutes later.
Under pressure from Daizen Maeda, his attempted clearance went straight to Hatate, who rolled home into an empty net.
Feyenoord were rocked by the turnaround and Celtic carried their momentum into the second half, content to keep possession as the hosts sat off them.
Eventually, Robin van Persie's side did snap out of their funk and pushed for an equaliser.
However, Tsuyoshi Watanabe's effort was cleared off the line by Callum McGregor, Gaoussou Diarra had an effort blocked and a header bounce off the bar, while Anis Hadj Moussa fired wide on his left foot.
Having absorbed that pressure, Celtic made Feyenoord pay for their wasted chances, as Nygren smashed a left-footed shot in off the bar past a stranded Wellenreuther.
It is Celtic's first win in the Netherlands since 2001, when O'Neill was first in charge, when they knocked Ajax out of Champions League qualifying.
Before kick-off, O'Neill said he expected this match to be his last in interim charge. If that turns out to be the case, it is a fine way to bow out for the 73-year-old.
