No, not that cheap GPS. This one:
I bought myself a Garmin Dakota 10 last year to use for road cycling. It does a great job for that - crucially, the basic UK road map it comes with is good enough that if you pre-load your route into it, you can usually see which way to turn at a junction without having to stop and consult your paper map. This really helps keep up the momentum. It also happens to run on AA batteries, which is handy as I carry a full spare set of those to feed my bike lights anyway.
However, setting off last Friday for a weekend in the Lake District, I threw it into my backpack just in case. It turned out to be extremely useful as we climbed Weatherlam in near-zero visibility caused by fog. You have to dig in to Settings > Position Format:
In the above example, "SP" is the OS grid square, and the two rows of numbers are the X and Y references within that square (I've blacked out most of the numbers so you can't see where I live).
This is very useful when your best efforts at map-reading start to come unstuck on the ground. Some semi-jokey remarks were made in our group about it being cheating, but I wouldn't go out without a GPS now - it could literally be a lifesaver if you get lost or stuck, and you can get one for less that £100.
Obviously, the reference is only as good as the accuracy of the GPS (on mine, you can tap the bars showing the satellite signal strength to see how many feet/metres it thinks it's accurate to) - but I certainly found the first three digits of each co-ordinate to be close enough for my purposes. What you have to be wary of is placing too much reliance on the difference between readings taken a few minutes apart, because the inaccuracy could be big enough to make it seem like you've gone the opposite way to what you actually did.
The compass, incidentally, is useless - slow to update and very dodgy [Update: A friend writes: The compass in cheaper GPS units often isn't a dedicated chip. It only works based on your direction of travel as reported by GPS co-ordinates. So you need to walk in a straight line for a few yards with the GPS unit held in front of you for it to do anything resembling the right thing.]. Just use a proper magnetic one instead.
Also, I can heartily recommend having a laminated OS map. "There's no such thing as bad weather, only bad equipment".