Is it possible to set the encryption mode like the AES-CTR on XBee Series 2?
No, it does not have an advanced crypto module on the processor. If you want to use advanced functions such as that, you need to add that function externally to the module and then send that encrypted data to the radios UART.
Oh, I thought it do have because I am allowed to enable or disable the encryption using the XCTU. I read somewhere that 802.15.4/Zigbee has encryption modes such as AES-CTR and AES-CCM. However, I did not see any encryption mode options available for me to choose in XCTU. So, I guessed that the XBee is fixed to one type of encryption mode and would like to know if there is other ways of setting it.Now it looks like kind of hard, but thanks for your answer!
I am not sure I understand what encryption options you are looking for but the radios and Zigbee do offer different encryption options. You can adjust these using the EO command. To get a better idea of what functions the EO command will allow you to Enable/Disable, I would strongly urge you to read over the Security section of the product manual for the radio in question. The manual explains the encryption options fully and how to use them.