The RBS 6 Nations champions were beaten by an immensely physical Springboks side that stormed into a 17-12 half-time lead.
Wales recovered and battled all the way but Fourie du Preez crossed for a crucial try to seal a 24-15 victory.
Phillips admits the defeat, which was compounded by injuries to Jon Davies, Liam Williams, Adam Jones and Scott Andrews, was bitterly disappointing.
But he has challenged his teammates to respond with consecutive victories over Argentina, Tonga and Australia.
“I’m bitterly disappointed. We are used to winning here in Wales and we really felt we could go out there and create an upset,” said Phillips.
“Everybody gave it everything. We came back really well in the second half and we just felt there were one or two lapses in concentration and the game was gone from us.
“We had a bit of bad luck and they capitalised when we had boys down injured as well. We didn’t have the rub of the green. It wasn’t our day really.
“We felt we were back but they killed us off with that last try. It was really flat in the changing room.
“We’ve got to learn from it, grow from it and move on. We’ve got to win the next three really to put this right.”
Wales crashed to a 26-12 defeat to Argentina last season. Phillips expects another hugely physical challenge at the Millennium Stadiums and admits Wales owe Los Pumas.
“They are very forward oriented and pride themselves on their scrum and driving lineout and we lost to them last year . We have to match them and get a victory this time.”
