what is the maximum baud rate over which an hdlc packet can be sent and received in rcm 6760?

To the best of my knowledge the maximum baud rate is 115200.Please check Rabbit 6000 processor reference manual at digi website.

On GitHub, I recently added an HDLC update for Dynamic C 10.72 that improves HDLC support on the Rabbit 6000 processor, and adds an HDLC sample program. You might want to take a look at it.

https://github.com/digidotcom/DCRabbit_10/commit/51e61a52ede15f432ad69ad223dd1637bed5b900

The README for that repository includes details on how to use Git to integrate updates into your installed copy of Dynamic C 10.72. Digi plans to release a “10.72A” version by the end of March that will include those HDLC changes.