I have some thoughts about this, though I have to tread carefully given the subject matter. There's a whole subgenre of family-dynamic focused erotica that often gets lumped under the 'taboo' label, and some of it does aim for complexity. I've come across a few titles that try to build genuine suspense and psychological depth alongside the... physical elements.
One that often sparks discussion in niche forums is 'Forbidden Fruit'—not the biblical kind, obviously. It follows a protagonist returning to a fractured family after years away, and the tension between estranged siblings is built on a foundation of past trauma and misplaced loyalty. The sexual elements erupt from that pent-up emotional chaos rather than feeling gratuitous.
Another is 'Kin', which uses a dual timeline to explore how a hidden relationship between cousins affects their adult lives and choices decades later. The plot isn't just about the act itself; it's about the weight of the secret, the constant fear of discovery, and the corrosive effect that has on their other relationships. The writing can be surprisingly introspective.
I'd caution that a lot of what's marketed this way is pure fantasy fulfillment with minimal plot. The ones that feel 'intense and complex' usually weave the taboo element into a larger story about power, memory, or shared history. You have to sift through a mountain of simpler stuff to find those narratives.