It’s certainly “sex”, and in the context I think the line makes good sense. Unless I’m mistaken Gary is talking with the only guy who’s welcomed him back to town with a big smile, the doctor (Raffaelli), in the doctor’s own house. Their sombre chat goes straight to the time after Gary was sent down, when his local defenders were intimidated by Acombar and his henchmen. Rosy’s husband was killed “accidentally”; the doctor was wounded in the leg in an “accident” that if he’d dared to call it anything else would have turned fatal. Having gone through the shit that happened to anyone who dared to stand up for Gary, the doctor asks him how he has been. His reply is the line you quote, which I hear as a bitterly ironic rejoinder: “Oh, for me it was one long party” [“All rest and sex,” i.e. exactly what ten years on a chain-gang isn’t, in this film anyhow]. Or: “That was my serving of shit.”
I like your link with his later rejection of Marcella Michelangeli’s steamy writhings, and think “reject” is exactly the right word - his desires, including the sexual ones, have all been channelled into vengeance, so he doesn’t have to resist them. The music suggests a moment of emotional nostalgia, but it’s soon gone.