As it uses a Lithium cell ("button"/"coin" cell), you're probably safe enough. Those are designed for ridiculously long shelf and service lives, and they don't have the vents that make aging NiCds so dangerous. They design the things to survive swallowing by pets and dumb kids (yes, I met the latter once at summer camp), the corrosivity of the electrolyte apparently goes down with discharge, and were a rare dramatic failure to occur, you'd probably have a fire on your hands. (Literally, since you're probably wearing one in your wristwatch!)
As such, it can't hurt to pull it, but the capacitors are likely to dry up before the cell could fail corrosively. That's why all the manufacturers moved away from the "no maintenance" NiCds of the '80s to the "change once every 3-10 years" Li buttons of the '90s. Maybe this decade will give us wider use of
ultracapacitors - there's
one hidden in the XBox.