Personally, I like my bullets to drop out about 0.002" bigger than the size-to diameter. That leaves some wiggle room for the occasional bullet that doesn't fill out so great, or if the diameters decrease as the mold gets hotter, or if I want to experiment with a larger sized diameter.
Sizing has gotten a bad rep because of all the crap sizing dies. If the die is made right, sizing does not hurt accuracy, and like you say, you gotta run them through the lubrisizer, anyhow.
BTW, how does your BFR shoot with the 503 throats ? Was there some logic behind making them oversize, or is this just another case of crap tolerances ?
It is your call, I am not going to tell you what to do, except to include a note with your order reminding me that you will be sizing to 0.504". That does help. Otherwise, I try to hit the middle of the tolerance range, with varying degrees of success.