How to Set Up a Mail Relay Server on Windows Server

Introduction

In many corporate environments, applications and monitoring tools like ERP, CCTV, or PRTG need to send email alerts. However, directly sending emails through the internet can cause security or delivery issues.

A Mail Relay Server helps solve this by acting as a secure intermediary between your internal systems and the external mail server. In this guide, we’ll walk you through the complete process of setting up a Mail Relay Server on Windows Server 2019 or 2022 using the built-in IIS SMTP Server feature.

What is a Mail Relay Server?

A Mail Relay Server forwards emails from internal devices or applications to your main mail server (such as Microsoft 365, Google Workspace, or your company’s SMTP server).
It’s especially useful when:

  • You want to centralize outbound email communication.
  • Applications or printers don’t support authentication.
  • You need better control over email routing and security.

Step 1: Install the IIS SMTP Server Feature

  1. Open Server Manager → click Manage → Add Roles and Features.
  2. Choose Role-based or feature-based installation and click Next.
  3. Under Server Roles, expand Web Server (IIS).
  4. Check SMTP Server.
    • The wizard will automatically include IIS 6 Management Compatibility.
  5. Click Install and wait for the installation to complete.
  6. After installation, open IIS 6.0 Manager from Administrative Tools.

Step 2: Configure the SMTP Virtual Server

  1. In IIS 6.0 Manager, expand your server → right-click SMTP Virtual Server #1 → Properties.

General Tab

  • Click Advanced → specify the IP address of your server (or leave “All Unassigned”).
  • Enter your Fully Qualified Domain Name (FQDN) (e.g., relay.company.com).

Step 3: Set Relay Permissions

  1. Go to the Access tab → click Relay.
  2. Choose Only the list below.
  3. Click Add → enter the IP addresses of systems that can send mail through this relay (e.g., 172.16.x.x application servers).
  4. You can also enable Allow all computers which successfully authenticate if needed.

Tip: Never allow “All except the list below” — it opens your server to spam abuse.

Step 4: Configure Authentication

  1. Under the Access tab → click Authentication.
  2. Select Anonymous Access if you’re relaying only from trusted internal IPs.
  3. If you’ll use external SMTP (like Microsoft 365), choose Basic Authentication and enable TLS encryption later.

Step 5: Set Outbound Security (Optional)

If your mail relay will forward emails to an external SMTP service:

  1. Go to the Delivery tab → click Outbound Security.
  2. Choose Basic Authentication → enter your external SMTP credentials.
  3. Check TLS encryption if your mail provider requires it.

Step 6: Configure the Smart Host

  1. In the Delivery tab, click Advanced.
  2. Under Smart host, enter your mail provider’s SMTP address, for example:
    • Microsoft 365 → smtp.office365.com
    • Gmail → smtp.gmail.com
  3. Keep the default port 25 (or change to 587 if required).
  4. Set your server’s FQDN (e.g., relay.company.local).

Step 7: Restart the SMTP Service

After configuration:

  • Open Services (services.msc) → restart Simple Mail Transfer Protocol (SMTP).

Step 8: Test the Mail Relay

Use PowerShell to test the relay:

Send-MailMessage -From "test@company.com" -To "user@external.com" -Subject "Relay Test" -Body "This is a test email." -SmtpServer relay.company.local

Or test manually using Telnet:

telnet relay.company.local 25

Then type SMTP commands manually to simulate a message.

Step 9: Enable Logging

  1. In IIS 6 Manager → SMTP Virtual Server #1 → Properties → General → Enable Logging.
  2. Choose W3C Extended Log File Format and select a log directory.
  3. Logs can help track delivery and detect abuse.

Best Practices

  • Restrict relay access to specific IP addresses only.
  • Always use TLS when relaying to external SMTP servers.
  • Periodically review logs for unusual activity.
  • Keep Windows Server and SMTP features updated.