Replacement fly-half Gareth Davies stepped off the bench to score the winning try three minutes from time.
Leigh Halfpenny converted to add to his earlier penalties after Rhys Patchell’s early kick had got things moving.
Toulon responded with five penalties by Jonny Wilkinson but the holders could not muster a late winner in a thrilling finale.
Cardiff, desperate to make amends from their opening loss at Exeter, were forced into a late change when captain Matthew Rees came down ill overnight.
But while the Arms Park’s plastic pitch stood up to the driving rain, the players stood up to the champions in a stop-start first half that ended all square at 6-6.
Patchell opened the scoring with a huge 60-metre kick from inside Cardiff’s own half inside just four minutes.
That lifted the home crowd and the team as Cardiff’s young backs broke free. Harry Robinson skipped past four tackles before Owen Williams was caught out wide while Robin Copeland also made a half-break.
Yet it was Toulon who thought they had snatched the first try on 18 minutes.
Josua Tuisova burst down the right and then won the foot-race to Steffon Armitage’s kick ahead but his pass was judged forward and the try was ruled out.
Instead it was left to the trusty left-boot of Wilkinson who kicked Toulon 6-3 ahead until Halfpenny equalised two minutes before half-time.
Toulon changed their entire front-row at the interval, including the arrival of former All Black Carl Hayman, as they looked to flex their muscles up front.
Wilkinson chipped away with a further two kicks, either side of a rare miss.
Yet amid a catalogue of mistakes by both teams, Halfpenny kicked two penalties to level the scores at 12-12 before he was penalised for a late tackle on Tuisova and Wilkinson nudged Toulon back in front.
Cardiff then threw caution to the wind and after Cory Allen beat four defenders, Davies darted between Maxime Mermoz and Wilkinson before riding Frederic Michalak’s last-ditch tackle to send the Arms Park wild.Â
