That 4 - 5 second wait sounds like you are sending a Broadcast packet or a few broadcast packets over the Xbee ZB network. Is that the case? If so, the only way to reduce the time is to send Unicast messages. The delay you are seeing on a broadcast is a required delay by the zigbee network to allow for propagation of the data.