This makes a little sense when you think of the kingdoms that banded against joyboy. What if Imu granted them power and then eventually took it away and became king of the world in secret because only ones who knew of his existence were the 5 elders,which means they could've possibly been around during the void century and possibly were rulers of an ancient kingdom.