Serial FIFO issue

Currently, I am implementing a 9-bit protocol on the ConnectME’s UART. I am directly controlling the serial registers (as described in the NS7520 hardware document) so that I can control the 9th bit.
Everything is working okay, except when the received packet is not divisible by 4. It seems that the FIFO register (NARM_SC1FIFO_ADDR) can only transfer data when there is at least 4 bytes in the serial FIFO. A slave device is sending 10 bytes. I am able to read 8 bytes, but cannot get the last 2 bytes.
On the next request, let’s say to a different slave device, I get those 2 bytes, plus the 10 bytes from that slave, totalling 12 bytes (which is divisible by 4).

Anyway to fix this or work around this problem?