@motorollin
The one question I would ask is whether you are having relationship problems with your current partner. You say you've worked with your boss for a long time without having feelings for him, until a pang of jealousy polarised you into your current siutuation, which makes me wonder if you are looking wistfully over the fence because your b/f and you are having issues?
In my own humble experience I only ever think about other people when I'm with someone when I feel less than secure in the existing relationship for whatever reason.
Obv, you're sensible enough to realise that it could cause problems with a friendship if you overstep the line and you acknowledge it's not much more than a fantasy, but I'd have a long think about your current relationship and how you feel before I'd do anything rash.