The main reason the Forerunners got overwhelmed by the flood was due to arrogance as they felt at the time humanity was no threat and only they were fit to hold the mantle. Then the human forerunner war distracted them from the threat of the flood even know they knew about it. Had the Didact been able to implement shield worlds as he wanted to, maybe things would have turned out a lot differently. The Master Builder's Halo array's that won the vote instead was always a last resort. The domain did allow them to see somewhat into the future but not everything and it was more a collective archive of past, present and the future that could be, they did lose connection with it too. If you saw the Didact's message on the terminal on Halo 3 on Halo mission (Legendary only). It goes someway to show 343 really just carried the story on and expanded on it vastly. It's worth watching all the terminals in Halo 4 too if you haven't as they explain a lot. However the Forerunner Saga is a must read for any Halo fan and I read it after Halo 4. It just really expands on Halo full stop but it also helps explain some of the issues you mentioned you had with the story. Forerunners could lie dormant for thousands of years in their Cryptums.