This is my set up.
I have a cheap set of USB speakers (£8, I already owned)
A cheap USB Sound Card (£1.49 – Yes you read correct, less than £2.00)
A cheapish 8GB USB stick (£5.00, I already owned)
Everything is plugged into a USB Hub (£12, Again I already owned), which is also being used to power the Rpi.

Total cost £26.49, if you take away the cost for stuff already owned. The total is £1.49.

Sound quality is reasonble, obviously its not as good as it could be,
This is a WIP as I iam going to add a better USB DAC and A Proper set of HiFi speakers and possibly a differnt USB drive.