I totally get what you're looking for—'Blackmail Sex' has this gritty, psychologically intense vibe that's hard to match, but I've got a few suggestions that might scratch that itch. 'The Collector' by John Fowles is a classic with a similar theme of obsession and power dynamics, though it leans more into the psychological thriller territory. The way Fowles explores the mind of the protagonist is chilling and immersive. Another one is 'Damage' by Josephine Hart, which delves into destructive passion and secrecy. It's less about outright blackmail but captures that same sense of moral unraveling.
If you're open to something more contemporary, 'The Girl on the Train' by Paula Hawkins has those elements of unreliable narration and twisted relationships. It's not identical, but the tension and dark undertones might appeal. For a wildcard pick, 'Exquisite Corpse' by Poppy Z. Brite goes into extreme territory with taboo themes—definitely not for the faint of heart, but if you're after raw, unfiltered darkness, it's worth a look. I’d say these books all share that unsettling, morally complex energy that makes 'Blackmail Sex' so gripping.