Yes, you will lose data. I tell people not to expect to move more than 10 packets per second (total, so about 2+ per router) unless they have very good low-level skills. Various RF collisions and back-off will greatly slow many-to-one behavior.
If the coordinator can poll for data, that can be used to maximize through put because it can eliminate radio collisions.