SPI read delay in FT2232H with libMPSSE


I have been using the ft2232 board with libmpsse module for SPI reading in windows/raspberry pi. In both cases, I read 3KB of data with 20Mhz frequency. It takes about 4ms seconds for every read. My problem is that when I read 20 minutes one or two times the read time goes above 100ms. Is it related to ftdi driver issue?