I always felt the broken tower was less about destiny and more about perspective. Most characters see it as this ominous ruin, a symbol of decay, but that's their own baggage coloring the view. Jaime sees it as the place where his life truly fractured, a site of infamy. For Bran, it’s a literal high point for his dreams, before the fall. It doesn't cause their fates so much as it holds up a mirror to them—their ambitions, fears, and failures are all projected onto those stones. The tower is passive; the characters project the meaning.
It's the setting for pivotal turns, sure, but the real impact is how it reframes their journeys afterward. Jaime’s act there defines him for years, until he begins to reject that definition. Bran’s fall from it sets his path, but he also gains a new sight. The tower breaks them, in a way, but the breaking is what allows for a different kind of becoming, if that makes sense. I don't think Martin believes in fixed fate, just consequential places.