Not sure what you mean -are you using Digi’s Connect Core i.mx53 Jumpstart kit board? i.mx53QSB is a freescale board and this is a Digi hardware forum.
U-boot - the official bootloader is essentially a baremetal system and it has limited amount of drivers, so the answer is “yes”