Trailing 22-13 at half-time, the Welsh region conjured up second half tries from Richard Fussell and Tom Grabham to get to within seven points of their hosts in a frantic finish.
It was a third successive defeat in Pool 4 for the Ospreys, who have to complete their campaign with a trip to Bath this week. Exeter Chiefs have qualified from Pool 4 for the play-offs.
Two of the four Gloucester tries came from former Welsh regional players, Aled Thomas and Dan Thomas, and the former also kicked a penalty. Centre Ben John scored a first-half interception try for the Ospreys and outside half Sam Davies kicked 10 points.
