It proved to be another frustrating evening for boss Nuno Espirito Santo, who has now collected only 11 points from his 16 games in charge of the Hammers since replacing the sacked Graham Potter in September.
That is the lowest total from any West Ham boss in the Premier League era - and the side now face an almighty battle if they are to stay in the top flight, which they have been in since 2012-13.
They were booed off at the end of the match by some of the remaining fans, though a large number had left before the final whistle.
Nuno, booked in the second half for complaining after a decision had gone against his side, had apologised to fans for what he called an "embarrassing" performance during their 3-0 loss to bottom club Wolves on Saturday.
He would have been pleased with his side's effort against Forest, though a lack of clinical finishing again proved decisive.
Leading 1-0 at the break, West Ham began the second half well with Jarrod Bowen firing one chance just over, before Summerville thought he had made it 2-0 only for VAR to rule that Castellanos was narrowly offside.
Argentina international Castellanos was making his Premier League debut after joining from Italian club Lazio earlier this week for an undisclosed fee. He worked hard up front, but he struggled to create chances.
His best opportunity came midway through the first half from Soucek's right-wing cross, but Castellanos got his volley all wrong, with the ball bouncing off him, and the chance vanished.
Nuno, who took Forest into European football before being sacked after three games of this campaign, brought on his other January signing: Brazilian forward Pablo Felipe, who joined from Portuguese club Gil Vicente.
Felipe's best effort was deflected over the top - but if West Ham are to have any chance of staying up, they will need their two new additions to find the goalscoring touch quickly.
Once again, the Hammers failed to hang on to a point and the pressure on Nuno increases.