Hi Alok,
There is no big difference between relay server and reverse proxy. Both are used to satisfy the requirement of extra security and load balancing.
Hence you could choose any of these.
But there are some dependencies with different types of apps SMP 3 supports (Odata based, Sylco and Mobiliser). Not all types of reverse proxies or relay server supports these different types of apps.
Hence you need to choose a reverse proxy or relay server based on the types of applications you are going to use.
You could have a look at this SAP note 1904213 that lists different reverse proxies it supports and it's limitations if any: https://websmp230.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/sno/ui_entry/entry.htm?param=69765F6D6F64653D3030312669765F7361…
Regards, Midhun
SAP Technology RIG