I enabled the encryption in XCTU software and i know it uses AES 128 bit encryption scheme. but my doubt is what is the key used ? this would be helpful for my research in ZigBee security.
XCTU does not have any Encryption in it but the XBee module you are using may.
The key would be what ever you entered into the modules KY value. If you do not know what that value is, you will need to reset the KY value on each module.