Man, I immediately think of 'The Exorcist' by William Peter Blatty, but the main antagonist there is Pazuzu, the demon possessing Regan. That’s not a human serial killer. You might be thinking of a subgenre where the antagonist is an exorcist who’s also a serial killer—someone who hunts demons but then starts killing the people they're supposed to save, or maybe sees the possessed as irredeemable and murders them.
I’m racking my brain and coming up a bit short on a perfect fit. There’s 'The Last Exorcism' type stories, but again, the antagonist is usually the demonic force. A closer angle might be in anime or manga, like 'Blue Exorcist' where the lines are blurred, but no serial killer there. Maybe 'The Rite' or 'The Exorcism of Emily Rose'? Nope, those are more about the legal or theological debate.
Honestly, a human exorcist-turned-serial-killer as the main antagonist feels like a niche yet-to-be-written premise. It’s a killer idea, pun intended. Someone should definitely write that. The internal conflict of a holy warrior gone rogue, using their knowledge of the spiritual to commit murders they frame as necessary purges… that’s some dark, psychological horror gold right there.