How does distance affect the time apart issue though?

Sound like you have a week connection. That in conjunction with long sleep times can cause an End device to be kicked off of the network and have issues connecting or associating again.