The most immediate tool is to mirror the narrative structure. A protagonist reads a text that contains clues or a pattern which the reader must also decipher. In 'If on a winter's night a traveler', you're literally turning pages trying to read a novel that keeps fragmenting, experiencing the frustration of the character right alongside him. That layering makes the physical act of reading part of the plot.
Another method involves the author directly intervening or the characters gaining awareness of their fictional nature. It can be playful, like in 'The Princess Bride' with Goldman's 'abridged' commentary, or deeply unsettling, as in 'House of Leaves', where the academic footnotes about a film that might not exist create a terrifying labyrinth. The book-as-object becomes a haunted artifact.
I think the most interesting use is when the internal book acts as a virus, changing the reality of whoever reads it. The characters aren't just reading for clues; the text is actively rewriting their world's rules. That's where metafiction stops being a clever device and starts feeling dangerous, which is a fantastic narrative engine.