The Reject unknown IP address option on the WebUI corresponds to the "reject_unknown_reverse_client_hostname" parameter in Postfix. When this option is enabled, IMSVA will reject the SMTP connection requests when the client IP address has no address > name mapping . But it will NOT check if the reverse DNS record matches the client IP address. Therefore, the option will NOT reject a mail as long as a reverse DNS record (PTR record) exists for the client IP address.
The response code, by default 450, will be sent to the upstream MTA.
For more details about this option and behavior, refer to the Postfix article: Postfix Configuration Parameters.
Because IMSVA does not accept the task of relaying the mail in such situation, it will NOT send any bounce notification/message to the sender mailbox, according to RFC5321. Rather, the upstream MTA is supposed to send "undeliverable mail" notification message to the sender.
As stated in RFC5321:
3.6.3. Message Submission Servers as Relays
If an SMTP server has accepted the task of relaying the mail and later finds that the destination is incorrect or that the mail cannot be delivered for some other reason, then it MUST construct an "undeliverable mail" notification message and send it to the originator of the undeliverable mail (as indicated by the reverse- path).
For more information about who should send the "undeliverable mail" notification, please refer to RFC5321.