Have you beat all the lower level bosses too? Because it sounds like you should have it. Also, is this all on the same character? The fragments are not shared like the ghosts. They're like the Golden Chests, each needs to find them.