Narratively it doesn't make sense for it to be Saruman. They already did that bait-and-switch in the first season where we were supposed to think the Stranger was Sauron for a minute. Doing another bait-and-switch making us think that it's Gandalf when really it's Saruman would be comically bad writing (not that the writing has been good, but that would just be on another level). They're clearly hinting as hard as they can that the Stranger is Gandalf- let's just assume he's Gandalf. We don't need to overthink this.
And yes I know the timeline doesn't work, and it means Gandalf shows up a few thousand years early, but they're already making a million changes to the timeline in this version of middle earth- whats one more?
