After building them up he knocks them back down though, he starts talking about how they had bought their position with their sweet trust. When he says “and the world was asleep/ to our latent fuss” He is trying to say that it did not matter what was said or done because the world was not ready to hear what was going to be