Always a tricky one. Personally I would only call penetrative (be it anal or vaginal) sex, sex. Oral or general touching just doesn't quite seem the same to me. It doesn't really FEEL like sex.
Where it's a proper sorta... thrusting experience, it just feels in a slightly different world. There has to be some level of trust (well for me) for you to be able to let someone do that to you, to take a part of someone inside you. It's not every day that something goes inside those places, whereas your mouth and hands are occupied a lot of the time.