I ran into this exact wall trying to track down books with that 'strangers trapped in a mansion solving a puzzle' vibe after reading 'The 7½ Deaths of Evelyn Hardcastle'. Generic keywords just pulled up thrillers or mysteries, which wasn't specific enough. What finally worked was using Goodreads lists, but in a roundabout way. I'd find one book that nailed the plot element, then I'd go to its Goodreads page and scroll way down to the 'Readers also enjoyed' section. It's not always perfect, but I found 'The Last' by Hanna Jameson and 'One by One' by Ruth Ware that way—both have that isolated-group, paranoia-driven core. Sometimes you get better results by describing the plot to a bookseller or librarian in plain English; they're wizards at the 'it's like that but with this' game. Another trick is searching for academic or blog articles analyzing the specific trope; they often cite multiple examples. I once found a whole subgenre of 'scientific expedition horror' just from a footnote in an essay about 'Annihilation'. It takes a bit more digging than a simple search bar, but stumbling onto those connections feels like uncovering a secret map.
Honestly, if you're just typing a plot summary into Google, you're gonna have a bad time. The metadata isn't that granular. I've had better luck on reddit, specifically the r/suggestmeabook and r/books subreddits. The key is to post your request with a very specific, almost absurdly detailed description of the plot beat you want, and mention a couple of books or even movies/shows that did it. People in those communities are fantastic at pattern-matching across genres. Someone once asked for 'books where the protagonist is slowly replaced by an imposter and no one else notices,' and the thread filled with deep cuts I'd never have found through a search engine. Library databases can sometimes be queried by subject headings that get weirdly specific, like 'Doppelgängers--Fiction' or 'Time loops--Fiction.' It's a clunky interface, but those librarian-assigned tags are a treasure trove.