I remove the plastic casing and let it sit in warm water with some soap in it, then I scrub it with more soap and a stiff brush. If the case is in good condition (no yellowing) you will get a nice finish.
But if the case is already yellow it is still going to look ugly. The problem with physically rubbing the case is that you will erase some of the texture from the casing...