It’s not uncommon for Windows web servers to use the IIS SMTP Server as a mail relay. It’s a smart host in that it can do the MX record lookups to find the mail server handling the recipients’ mailboxes. But there is a problem. The big mail services such as Google and Microsoft are very picky about which mail servers they will communicate with as they try to mitigate the never-ending stream of spam. To this end, they will generally only accept mail from servers that speak ESMTP (Extended Simple Mail Transport Protocol) as currently defined in RFC 5321. Use of ESMTP is considered optional, with a fallback to the older SMTP commands if one mail server doesn’t support ESMTP. But because spammers are now the primary users of older SMTP, the big mail services simply drop the connection if the sending mail server doesn’t use ESMTP.

The SMTP Server fully supports ESMTP, but this support can be turned off for inbound and/or outbound connections via settings in the IIS 6 MetaBase. For outbound connections, this is controlled by the setting SmtpOutboundCommandSupportOptions. According to the documentation that can still be found, Microsoft says the default value is 7697601 which enables all ESMTP commands. In reality, the value is 7 which disables all ESMTP commands. This makes it nearly impossible for the SMTP Server to send email to recipients with the big mail services. Fortunately, this is easily fixed. Just change the value of SmtpOutboundCommandSupportOptions from 7 to 7697601.

There are two ways to change this setting. The easiest way is to use the adsutil.vbs scripting utility which is available if the IIS 6 Scripting Tools feature is installed. This feature is not installed by default when the SMTP Server feature is installed, but it’s easily added and does not require a restart. The other way is to directly edit the IIS 6 MetaBase using Notepad (or your favorite text editor), but that requires shutting down IIS. Using adsutil.vbs does not.

Using adsutil.vbs

adsutil.vbs is a VBScript utility for working with the IIS 6 MetaBase from the command line or scripts. It is installed in C:\inetpub\AdminScripts (which is not part of the system PATH environment variable). Since it’s a VBScript, it’s run by the Windows Script Host (WSH). It’s best to use cscript.exe as this is the command-line version. If the WSH program is not specified, wscript.exe is used and this sends all output to a pop-up dialog box. Regardless of which one is used, the command syntax is the same.

This command will get the current value. Note that you must use the full path for adsutil.vbs since it’s not in the system PATH.

cscript.exe C:\inetpub\AdminScripts\adsutil.vbs get SmtpSvc/SmtpOutboundCommandSupportOptions

This command changes the value to 7697601.

cscript.exe C:\inetpub\AdminScripts\adsutil.vbs set SmtpSvc/SmtpOutboundCommandSupportOptions 7697601

The change will not take effect until the SMTP Server service is restarted. This can be done from the Services administration tool (the service name is “Simple Mail Transport Protocol Server”) or the command line using the following commands.

net stop smtpsvc
net start smtpsvc

Editing The IIS 6 Metabase

Although IIS 7 and above no longer uses the MetaBase, to edit the MetaBase you must stop the IIS Admin service and this stops the World Wide Web Publishing Service, effectively taking all the IIS sites offline.  First use the iisreset utility to stop IIS.

iisreset /stop

Open the MetaBase in Notepad (or your favorite text editor). As this is a protected file, it must be done using “Run as Administrator” or from an elevated Command Prompt window.

notepad.exe C:\Windows\System32\inetsrv\MetaBase.xml

Search for SmtpOutboundCommandSupportOptions and change the value from 7 to 7697601 so reads as follows:

SmtpOutboundCommandSupportOptions="7697601"

Save the changes and then use the iisreset utility to start IIS.

iisreset /start

If the SMTP Server service startup type is not set to Automatic, you will need to start the service as iisreset only starts those services with automatic startup.