Scott Gimple himself said that any reference to the Whisperers in Season 3 and 4 is purely coincidental. The Whisperers weren't even in the comics when these episodes aired, and the way the TV timeline is established makes it impossible for the Whisperers to be around the same time frame of Season 3 or 4.