Eliota Fuimaono-Sapolu also scored in the 22-21 win over Leeds Carenegie
Samoan centre Eliota Fuimaono-Sapolu ran in his second try in as many games as Nicky Robinson kicked Gloucester to victory in a tense finish at Kingsholm.
After Gloucester went in 16-10 up at the break, the lead then changed hands four times in 13 second-half minutes.
Joe Simpson scored on his first start, adding to Dan Ward-Smith's first-half try, before Robinson's massive 53-metre penalty put Gloucester back in front.
But, after Dave Walder responded, cool Robinson's second long penalty won it.
The Wales international fly-half had been dropped to bench duty following last weekend's tame defeat against London Irish.
But he kicked his two late penalties under real pressure after coming on at half-time for the injured Tim Taylor, who had kicked 11 points before the break with three penalties of his own, as well as converting Fuimaono-Sapolu's first-half try.
Walder slotted two penalties and two conversions for a 10-point haul of his own but missed a long-range penalty with a minute left that would have clinched victory.
Gloucester had already lost to Exeter and London Irish in recent weeks, either side of an unimpressive home win against bottom club Leeds Carnegie, but Robinson held his nerve just when it looked as though Wasps might edge the verdict.
Although the lead changed hands seven times and a late goalkicking exchange ensured a dramatic finish, the game was riddled with mistakes and dominated by referee Andrew Small's fussy officiating.
But, while Gloucester will not particularly care how they conjured the win, Wasps must now reflect on a below-par opening month that has realised one victory, one draw and two defeats.
Scrum-half Joe Simpson scored Wasps' second try on his first start of 2010
Gloucester boss Bryan Redpath reacted to the Irish loss by making six changes, including demotion to bench duty for international trio Robinson, Mike Tindall and Alex Brown.
He also paraded a new front-row unit in Nick Wood, Olivier Azam and Paul Doran-Jones, while Wasps - the last away team to win at Kingsholm on Premiership business 10 months ago - were captained by former Gloucester favourite Phil Vickery.
Walder, who masterminded Wasps' fightback win against Leicester, opened the scoring with a 45-metre penalty before opposite number Taylor replied from much shorter range.
The opening quarter was punctuated by poor passing and wrong options. And, given all the errors, it was hardly surprising that frustration would kick-in at some point, a fact illustrated when Wasps' England prop Tim Payne and Gloucester number eight Brett Deacon traded blows.
After Walder rifled a drop-goal attempt wide, Taylor's second penalty nudged Gloucester ahead, only for Wasps to strike seven minutes before half-time when Ward-Smith powered through weak defence unopposed.
Walder converted, yet the home side remained firmly in contention when Taylor completed his penalty hat-trick before Fuimaono-Sapolu completed an impressive first-half revival.
Wasps stood off him way too long in defence, and he finished clinically, scoring a richly-deserved try that Taylor improved for a 16-10 interval advantage before giving way to a knee injury.
A poor third quarter failed to produce any further scoring until Wasps stung the Cherry and Whites with a breakaway try.
Substitute Richard Haughton made ground from deep and, as Gloucester's defence tried to regroup, the supporting Simpson finished off with a flourish.
Walder's conversion edged Wasps 17-16 in front, leaving Gloucester needing to rediscover their composure as their 10-month unbeaten home league record came under serious threat.
Robinson restored Gloucester's lead, courtesy of a penalty from just inside his own half, to set up a nerve-tingling finale.
But Riki Flutey remained a threat, carving Gloucester open three minutes from time to set up an attacking platform that Wasps capitalised on with a Walder penalty.
That lead proved short-lived, however, as Robinson stepped up to land another huge, this time match-winning, penalty.
Gloucester boss Bryan Redpath: "Nicky Robinson was confident with his kicking. He has the range. All credit to him, I am delighted for him. He and Tim Taylor did very well.
"We conceded 19 penalties against Irish and never got a chance to get in the game, but we had a bit more fire about us today. A lot of players performed well.
"The Gloucester fans are hugely passionate about the game, and it was key today that we showed them some passion. The expectation is massive every time we play here, and we are aware of that pressure, but you can't shrink from it."
Wasps rugby director Tony Hanks:
"We got ourselves right back into the game, so there was no lack of effort, but we need to be more clinical.
"We conceded a soft first-half try and our turnover rate was too high, but we put ourselves in a position to win the game and then we conceded it. We didn't manage the last couple of minutes well once we had gone into the lead.
"But you have got to make sure that you take something from your away games, and we managed that today with the losing bonus point."
E-mail this to a friend 
Printable version 
Bookmark with:
What are these?