Add missing checks for a too-large packet, so we don't blow up trying to allocate a huge buffer; fixes bug 6668. Also add some other checks for invalid records.