Sheffield shooting guard Nate Reinking top scored with 21 points to lead his side into the BBL play-off final for the third time in four years. The Sharks were ahead for most of the game although London did fight back from an 11-point deficit at half-time to briefly lead 81-80.
However six points from Justin Phoenix and four from Lynard Stewart put Sheffield five points ahead.
And victory was sealed with a pair of free throws from Rob Yanders.
"Justin came up with some big baskets for us down the stretch - I don't know how he managed to get some of them in," said Sharks coach Peter Scantlebury.
"That's what you need - big players who make big plays when you need them. I the last two minutes, our defence was outstanding."
Sunday's final is live on BBC Five Live Sports Extra from 1800 GMT.