True, the particles that are in space tend to be very cold. That said, it is also very rare for those particles to come into contact with other objects in space. It's a vacuum, meaning that despite how cold they are, they'd come into contact with you so rarely that they wouldn't steal very much energy for a long time.