Authenticating two BGP neighbors is usually done with a shared password:
router bgp 1
password clear TEST
IOS-XR (and also regular IOS) will ingest the clear text configuration password, perform some additional magic, and pass the resulting MD5 hash to the neighbor
But what a router is really doing underneath, is to enforce RFC 2385, by using TCP MD5 option during the three-way handshake.
The main goal is not just to authenticate the peer node, but also to secure the TCP connection itself by preventing any spoofing, especially RST packet, which may cause a connection reset.
The resulting hash is not just derived from the shared password configured on the two BGP speakers, but also from:
the TCP pseudo-header (source IP address, destination IP address, zero-padded protocol number, and segment length)
the TCP header
the TCP segment data
the password itself
Considering the hash of 18 bytes, plus an End of Options byte, a larger MTU might be taken into account, as the MSS will have additional 19 bytes.