ZigBee is by design of low bandwidth and high latency. It is designed to slowly move data around RF opaque obstacles by hopping the data from node to node to complete a data path. I never heard of it as being described as “fast” if you want fast look at Digi’s XB24 802.15.4 system for P2P/MP or the fancy new XB900HP that can do almost anything except cook your meal and handles DigiMesh, Repeater, and Point to Point/Multipoint in the same network on the fly. And can leap huge distances 29 miles at a hop.
With what you have use unicast 64 bit addressing. DL2 = SL1, DH2 = SH1 and the reverse… DL1 = SL2, DH1 = SH2.
Note that the special ZigBee case of DH/DL of 0,0 means that the Router or End Device will talk via unicast to the Coordinator whoever it is after finding its address via a automatic Node Discovery.
A broadcast in ZigBee is probably the single worst choice you could make.
I think, for what you are trying to do, you should take the plunge and switch to API mode. What you are trying to do becomes trivial in API mode because any device can freely address any other device for each operation.
Broadly, you just send API requests to your local XBee with the address of the target XBee.
E.g. if you want to change pin 1 on device Y from device X, you send X an API request for a remote AT command, where the destination is the address of Y, and the AT command is “D1”.