Man, that's a tough one. 'The Miserable Mill' is definitely where things start to feel heavier. I think whether it's worth it hinges entirely on your tolerance for the series' particular brand of bleakness mixed with absurdity. The Baudelaires in a lumber mill is a bizarre setting, and Handler introduces the hypnosis subplot which, frankly, feels like a slight narrative cheat to me—it's a convenient way to explain character actions that seem out of place. The book feels less about outsmarting Olaf through cleverness and more about enduring a weird, grim situation until the plot device kicks in to move them along.
That said, skipping it leaves some pretty crucial gaps. You miss the first real introduction of the Quagmire triplets, Isadora and Duncan, who become central to the emotional core of the next several books. Their connection with the Baudelaires, this shared experience of loss, is the closest thing to genuine warmth the series offers. Without Book 4, their later appearances and the tragedy that befalls them loses a lot of its punch. You also get more hints about the V.F.D., the sugar bowl, and the increasing sense that the parents' past is way more complicated than 'they died in a fire.' So as a piece of the overarching puzzle, it's necessary. The finale's payoff relies on appreciating how the pieces of this conspiracy slowly, painfully come together, and 'The Miserable Mill' adds a few key, if clunky, components.
I'd say power through. It's a weaker entry, but it's short, and the series finds its footing again with 'The Austere Academy' right after. The feeling of slogging through it almost mirrors the Baudelaires' own exhaustion, which, intentionally or not, makes you feel more aligned with their plight.