Exchange 2013 SP1 – problem #1 – Powershell virtual directory malfunction – HTTP error (500)

This is known issue, but to remember myself for next versions: If you run EMS for Exchange 2013 SP1. Error comes out:500error

It has 3 possible issues. Here are solutions:

Root cause 1:

Exchange server is out of sync with time of DC. You should always have the following hierarchy of time sync in your domain: Reliable time source -> PDC -> Other DCs -> Servers and clients

  • Disable windows time sync from physical host if it is virtual machine
  • Enable time sync with domain by the following commands:
  • On PDC
net stop w32time 
w32tm /config /syncfromflags:manual /manualpeerlist:0.pool.ntp.org 
w32tm /config /reliable:yes 
net start w32time

On other DCs and Servers:

net stop w32time
w32tm /config /syncfromflags:domhier /reliable:no /update
net start w32time

Root cause 2:

Exchange server path to kerbauth.dll is wrong / Powershell virtual directory is misconfigured. I have re-created virtual directory for Powershell on affected server:

Get-PowerShellVirtualDirectory -Server <AffectedServer> | Remove-PowerShellVirtualDirectory
New-PowerShellVirtualDirectory -Server <AffectedServer> -Name PowerShell
Get-PowerShellVirtualDirectory -Server <AffectedServer> | Set-PowerShellVirtualDirectory -BasicAuthentication:$false
IISReset

After virtual directory re-creation I have checked its modules in IIS and made sure, that Kerberos module is native and the path to its DLL is correct:

modules

Root cause 3:

There is a missing Windows feature WinRM IIS extension.The full description is here: http://technet.microsoft.com/en-us/library/dd759166.aspx This was the case in my lab and I feel it is the side effect of in-place upgrade of OS from Windows server 2012 to Windows Server 2012 R2 on Exchange server (Yes I know it is not good idea, but how to learn non standard issues in other way). Here is simple solution: Install this windows feature:

Get-WindowsFeature *IIS* #to check if it is installed
Add-WindowsFeature Winrm-IIS-Ext # to install

winrmext

RTF content archiving problem when using Mailstore against Exchange 2010 SPx – ErrorInternalServerTransientError

I have experienced problem in one of my customer´s Exchange environment after utilization of Mailstore archiving software. Mailstore is EWS and client based archiving solution for Exchange. All best practice configuration steps can be found here: http://en.help.mailstore.com/MailStore_Help

Environment:

  • Virtualized Exchange 2010 SP3 RUx environment with 2 node DAG, multirole servers. Both running on ESX 5.1. No Firewall and router between production Exchange and Mailstore virtual servers.

Symptoms:

  • RTF content messages cannot be archived using Mailstore via EWS
  • RTF messages can be easily simulated as new meeting request containing inline picture of any size. Meetings should not be answered to have error visible in 100 percent of cases
  • Error message in Mailstore job log as follows
08:36:58.874 [18] INFO Processing message: 23.1.2014 7:42:45 UTC 'FW: Problém s archivací meetingů', UID 1: @mail.domain.cz, UID 2: 
08:36:58.890 [18] INFO Retrieving message...
08:36:58.890 [18] INFO Sending EWS Request (GetMimeContent)
08:36:59.561 [18] INFO Sending EWS Request (GetMimeContent)
08:37:00.403 [18] INFO Sending EWS Request (GetMimeContent)
08:37:01.464 [18] INFO Sending EWS Request (GetMimeContent)
08:37:02.727 [18] INFO Sending EWS Request (GetMimeContent)
08:37:04.194 [18] INFO Sending EWS Request (GetMimeContent)
08:37:05.879 [18] INFO Sending EWS Request (GetMimeContent)
08:37:07.751 [18] INFO Sending EWS Request (GetMimeContent)
08:37:09.825 [18] INFO Sending EWS Request (GetMimeContent)
08:37:12.072 [18] INFO Sending EWS Request (GetMimeContent)
08:37:14.521 [18] INFO Sending EWS Request (GetMimeContent)
08:37:17.173 [18] INFO Sending EWS Request (GetMimeContent)
08:37:20.012 [18] INFO Sending EWS Request (GetMimeContent)
08:37:23.070 [18] INFO Sending EWS Request (GetMimeContent)
08:37:26.330 [18] INFO Sending EWS Request (GetMimeContent)
08:37:29.793 [18] INFO Sending EWS Request (GetMimeContent)
08:37:30.230 [18] EXCEPTION MailboxImportWorker:ProcessMailboxMessageWrapper
: Microsoft Exchange Server nedokázal dokončit úlohu. Detaily: An internal server error occurred. Try again later. EWS Error Kód: ErrorInternalServerTransientError.
  • Moving node to other ESX cluster or moving active database to another node solved error instantly, but after switch back error appeared again
  • User-generated load was also partly the problem

Solution:

We have tried everything from re-creation of throttling policies, moving databases between nodes, updates to latest RU and Mailstore versions, Disabling TCP chimney, RSS and AutoTuning features, re-creation of Exchange databases, re-creation of Mailstore database and many many others.

What has finally helped was to re-create EWS virtual directory and restart IIS:

Get-WebServicesVirtualDirectory SERVER\ID | Remove-WebServicesVirtualDirectory
New-WebServicesVirtualDirectory
Get-WebServicesVirtualDirectory SERVER\ID | Set-WebServicesVirtualDirectory -InternalURL <IURL> -ExternalURL <EURL>

I suspect 2 things. 1 is problematic IIS 7 metabase or utilization of CGI (Common Gateway Interface –http://technet.microsoft.com/en-us/library/cc753077(v=ws.10).aspx ) on EWS virtual directory. Uninstallation of CGI did not solve the problem. Problem has been solved by re-cration of EWS virtual directory on affected DAG node after uninstallation of CGI.

Exchange 2013 CU3 upgrade problem #3 – Failure cleaning up search foundation data folder

Another problem while upgrading Exchange 2013 CU3. During configuration of backend transport service search foundation folder error ocured:

Error:

error search foundationSolution:

Tricky one, but I have checked Exchange setup log and found the folder with search foundation installation script is located under %Exchangeinstalldirectory%\Bin\Search\Ceres\Installer\. I browse to the folder in powershell and run configuration script manually without arguments first, it gave me a list of available arguments and in my case it was needed to add configuration to existing one so “a” was the right choice.

.\InstallConfig.ps1 -action a

And voila. Next run setup of Exchange 2013 CU3 went ok without problems. Needed to say, that it has happened because of problem to connect to DC during setup.

solution1

Exchange 2013 CU3 upgrade problem #2 – IIS default web page incorrect binding

In this case I am not sure if it is problem or not, but just to remind myself for the future. After Exchange 2013 CU3 setup there was strange setting in IIS. Missing binding for all unassigned IP addresses for HTTPS on port 443. Solution is to remove incorrect binding and add new one.

ECP error IIS

Exchange 2013 CU3 upgrade problem #1 – EAC doesn´t work (HTTP 400, bad request)

I came into problem recently. After upgrade of Exchange 2013 CU2 to CU3 my ECP stopped to work. I tried to:

  • re-create ECP virtual directory
  • renew certificate (which expired recently)

none of those helped.

Error:

ECP error WEB

OWA - wrong authentication set

Then I checked authentication methods for my virtual directories and found, that Exchange 2013 CU3 setup messed up my OWA authentication methods. The goal here is to have both virtual directories set with the same authentication methods. In my case FBA and Basic.

Solution:

Set correct authentication methods and run IISreset on problematic backend exchange servers.

Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -FormsAuthentication $true -BasicAuthentication $true

OWA - correct authentication

Exchange 2013 / Exchange 2010, Windows Server 2012 – SChannel Event ID:36888 (1203) – TLS/SSL error – The root cause

I have problems in some environments, where these SChannel errors are generated. Well. It took me several days to find reasonable “why” it is logged.

Problem:

The event ID from the picture can be seen from time to time:

EventID-Error

Solution:

Based on several articles I have read and some discussions. First you have to make sure, that the process causing this error is LSASS.exe, which is by the way local security authentication server (authenticating users to winlogon service, using authentication such as msgina.dll and so on). To make sure it is LSASS.EXE. Open Event ID and check the Event ID details, Click on Details tab -> Expand System while friendly view is selected. Check Process ID.

EventID_Details

Then use powershell and run:

Get-Process | select name,id | sort id

Result should give you the name of the processes. It will be lsass.exe.

Why:

Reason is simple. Not standard or corrupted behavior of web browsers or users. The problem behind SChannel and Exchange 2012 is, that sometimes users use HTTP protocol, but on port 443, which expects certificates exchange rather than GET command.

How to test:

Option 1#:

Test is easy. For example you can input URL to your browser address bar, which is obviously wrong and see the results: HTTP://MAIL.DOMAIN.LOCAL:443/OWA – It says to use HTTP protocol (not HTTPS) on the 443 port and it generates errors immediately.

Option 2#:

Run Telnet and test command:

Telnet localhost 443 (to connect to HTTPS)

In Telnet window:

Get /index.htm (on HTTPS SSL must be established first so it will generate errors immediately. Result will not be seen in telnet window)

What is the solution?

Solution #1:

Some IT guys recommend to disable SCHannel logging to get rid of these events, but I cannot recommend that. To be honest. It is better to see, that somebody is trying to connect using HTTP on HTTPS port, because this might be some attempt to DoS attack or info, that users don´t know how to type OWA URL correctly. Shortly it is better to know something is wrong than disable logging.

Solution #2:

I suspect wrong redirect configuration for the websites from HTTP to HTTPS. I would check IIS if redirect is set correctly. For those having this issue without redirect I would suspect problem in web browser area.

Links:

To test SSL via command line:

http://www.bearfruit.org/2008/04/17/telnet-for-testing-ssl-https-websites/

LSASS description:

http://www.neuber.com/taskmanager/process/lsass.exe.html

ForeFront Protection 2010 for Exchange Server integration failure after installing of Exchange 2010 SP3 RU2 on hybrid server EventID:1007,EventID:1008, EventID:9581, EventID:9564

This article is continuation of the http://ficility.net/2013/10/16/exchange-2010-sp3-hybrid-server-in-organization-which-have-had-exchange-200x-before-freebusy-issues/ . After installation of RU2 for Exchange Server 2010 SP3 we have had problem to intefrate ForeFront to its new (mailbox role). The MSExchangeIS service stucked and started to Start/Stop in the loop. Here are the corrective actions (actions will remove ForeFront and its settings, so if you are not sure how to configure your ForeFront, don´t use it):

Problem:

The following Event IDs are circulating: 1007 (FSC moniror Initialize) -> 1008 (FSC moniror Termination) -> 9581 (MSExchangeIS – Virus scan cannot be loaded) -> 9564 (Cannost start Information Store because Virus Scan failed to load)

FFERRORS

Solution:

  • Stop services
Stop-service MSExchangeIS
Stop-service MSExchangeTransport
Stop-Service FSCController -force
FSCutility.exe /Disable
  • Uninstall ForeFront (From Programs and features menu)
  • Restart server
  • Install ForeFront from media again and it will be automatically integrated to MSExchangeIS again.
  • Restart server
  • Configure ForeFront again, because re-installation removed your settings.
  • Test server
    From FF management console run
     FSCUtility.exe /status

    FFSTATUS_After

  • If ForeFront is integrated, run the following to test services on the server and to check if DB is mounted.
    Test-ServiceHealthGet-MailboxDatabaseCopyStatus
    Get-MailboxDatabaseCopyStatus

Exchange 2013 – Replication problem when different disk configuration is used

My friend came with problem to add second copy of databese in Exchange 2013 DAG, because replication service failed to perform initial seed due to different configuration between source (first copy)  and target (second copy)disks. I wanted to test it, so here is case study:

LAB:

  • 2x DC, 2x Exchange 2013 – CAS and Mailbox role in DAG, HAProxy load balancer, Exchange 2010 SP3 multirole server
  • All running under VMWare player
  • TESTDB: First copy on one DAG member on mountpoint F: (GPT, SCSI, NTFS , 8kB / sector)

Getting info about NTFS:

I used the Powershell function presented in this article http://cornasdf.blogspot.cz/2010/03/using-powershell-to-get-ntfs-info-such.html?showComment=1362006532506#c4110837443209977959

Thanks to Conrad then gathering NTFS info is as easy as:

Get-NTFSInfo f

and result

Drive                           : f
NTFS_Volume_Serial_Number       : 0xfaf6f756f6f7121d
Version                         : 3.1
Number_Sectors                  : 20901887 (0x00000000013eefff)
Total_Clusters                  : 1306367 (0x000000000013eeff)
Free_Clusters                   : 1234271 (0x000000000012d55f)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : <Not Supported>
Bytes_Per_Cluster               : 8192
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 0
Mft_Valid_Data_Length           : 524288 (0x0000000000080000)
Mft_Start_Lcn                   : 393216 (0x0000000000060000)
Mft2_Start_Lcn                  : 1 (0x0000000000000001)
Mft_Zone_Start                  : 393248 (0x0000000000060020)
Mft_Zone_End                    : 418848 (0x0000000000066420)
RM_Identifier                   : 0E320AB6-7A27-11E2-B355-000C2940FA17

Plan:

Test replication issues while using the following setup:

  1. Create second copy of the database using SCSI, GPT, NTFS but larger size of the block (65kB)
  2. Create second copy of the database using SCSI, GPT, NTFS but lower size of the block (2kB)
  3. Create second copy of the database using IDE, MBR disk with the same NTFS config as the first copy of the database
  4. Create second copy of the database using IDE, MBR disk with the different NTFS config as the first copy of the database (65kB block)

Ad 1) Create second copy of the database using SCSI, GPT, NTFS but different size of the block (65kB)

I have preconfigured the disk with the same drive letter F: , GPT and now I will format the NTFS to 65kB block size. The info from NTFS:

[PS] C:\Windows\system32>get-ntfsinfo f

Drive                           : f
NTFS_Volume_Serial_Number       : 0x5ed0732ad0730793
Version                         : 3.1
Number_Sectors                  : 20901887 (0x00000000013eefff)
Total_Clusters                  : 163295 (0x0000000000027ddf)
Free_Clusters                   : 162095 (0x000000000002792f)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : 
Bytes_Per_Cluster               : 65536
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 0
Mft_Valid_Data_Length           : 65536 (0x0000000000010000)
Mft_Start_Lcn                   : 49152 (0x000000000000c000)
Mft2_Start_Lcn                  : 1 (0x0000000000000001)
Mft_Zone_Start                  : 49152 (0x000000000000c000)
Mft_Zone_End                    : 52384 (0x000000000000cca0)
RM_Identifier                   : 1749F9E1-7CB8-11E2-B6CB-000C29ECA938

Adding DB copy:

Add-MailboxDatabaseCopy -Identity TestDB -MailboxServer FrontEnd1 -ActivationPreference 2

Result:

Works normally. Database is seeded and also incremental seed works. Before point 2 I removed the mailbox database copy and formatted NTFS with lower block size.

[PS] C:\Windows\system32>Remove-MailboxDatabaseCopy testdb\frontend1

Confirm
Are you sure you want to perform this action?
Removing database copy for database "TESTDB" on server "FRONTEND1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: The copy of mailbox database "TESTDB" on server "FRONTEND1" has been removed. If necessary, manually delete
the database copy's files located at "f:\TESTDB" and "F:\TESTDB\Testdb.edb" on that server.
[PS] C:\Windows\system32>

AD 2) Create second copy of the database using SCSI, GPT, NTFS but lower size of the block (2kB)

I have preconfigured the disk with the same drive letter F: , GPT and now I will format the NTFS to 2kB block size.

Works normally. Database is seeded and also incremental seed works. Before point 2 I removed the mailbox database copy and formatted NTFS with lower block size.

AD 3)Create second copy of the database using IDE, MBR disk with the same NTFS config as the first copy of the database

Seeding works normally as in other configurations.

PS C:\Users\administrator.SALONOVI> get-ntfsinfo f

Drive                           : f
NTFS_Volume_Serial_Number       : 0x42f4703af47031f1
Version                         : 3.1
Number_Sectors                  : 20965375 (0x00000000013fe7ff)
Total_Clusters                  : 1310335 (0x000000000013fe7f)
Free_Clusters                   : 1300725 (0x000000000013d8f5)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : 512
Bytes_Per_Cluster               : 8192
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 0
Mft_Valid_Data_Length           : 262144 (0x0000000000040000)
Mft_Start_Lcn                   : 393216 (0x0000000000060000)
Mft2_Start_Lcn                  : 1 (0x0000000000000001)
Mft_Zone_Start                  : 393216 (0x0000000000060000)
Mft_Zone_End                    : 418848 (0x0000000000066420)
RM_Identifier                   : 700D6323-8444-11E2-9E80-000C29ECA938

Ad 4) Create second copy of the database using IDE, MBR disk with the different NTFS config as the first copy of the database (4kB block)

Works as other configurations.

Drive                           : f
NTFS_Volume_Serial_Number       : 0x0a262c1b262c0a71
Version                         : 3.1
Number_Sectors                  : 20965375 (0x00000000013fe7ff)
Total_Clusters                  : 10482687 (0x00000000009ff3ff)
Free_Clusters                   : 9682285 (0x000000000093bd6d)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : 512
Bytes_Per_Cluster               : 1024
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 1
Mft_Valid_Data_Length           : 524288 (0x0000000000080000)
Mft_Start_Lcn                   : 3145728 (0x0000000000300000)
Mft2_Start_Lcn                  : 8 (0x0000000000000008)
Mft_Zone_Start                  : 3145728 (0x0000000000300000)
Mft_Zone_End                    : 3350560 (0x0000000000332020)
RM_Identifier                   : 700D6344-8444-11E2-9E80-000C29ECA938

Result:

I haven´t found error or problematic configuration, however, there might be some stuff useful for others. It took so much energy to test, that I would still like to post this article for future refference.

 

Exchange 2010 SP upgrade failed (0x80070003)

Let me provide you fresh experience with Exchange 2010 SP3 upgrade.

In the first place, thank you Zbynek, because final solution was his idea!

Problem

Exchange 2010 SP3 upgrade unexpectedly failed for 2 servers from 9. Those servers had separated Exchange roles. So the following error occurred for MBX as well as HUB role.

[10/05/2013 18:58:41.0984] [2] Saving object "EXMBX02\PowerShell-Proxy (Default Web Site)" of type "ADPowerShellVirtualDirectory" and state "New".
[10/05/2013 18:58:42.0015] [2] Previous operation run on domain controller 'DC03.contoso.local'.
[10/05/2013 18:58:43.0481] [2] Searching objects "DEXMBX02\PowerShell-Proxy (Default Web Site)" of type "ADPowerShellVirtualDirectory" under the root "$null".
[10/05/2013 18:58:43.0497] [2] Previous operation run on domain controller 'DC03.contoso.local'.
[10/05/2013 18:58:43.0497] [2] Ending processing new-PowerShellVirtualDirectory
[10/05/2013 18:58:43.0497] [1] The following 1 error(s) occurred during task execution:
[10/05/2013 18:58:43.0497] [1] 0.  ErrorRecord: A failure occurred while trying to update metabase properties.
[10/05/2013 18:58:43.0497] [1] 0.  ErrorRecord: Microsoft.Exchange.Data.Common.LocalizedException: A failure occurred while trying to update metabase properties. ---> System.Runtime.InteropServices.COMException (0x80070003): The system cannot find the path specified.

As can be seen it was IIS related problem (a failure occurred while trying to update metabase properties) especially with PowerShellVirtualDirectory.

Root Cause

Only suspicion:

  • firewall or application (e.g. an anti-virus) was cutting the connection
  • an application was locking the IIS metabase (e.g. a backup solution)
  • not sufficient permissions

Solution

This solution is intended for separated (CAS, MBX, HUB) as well as multi roles.

1.  Remove corrupted PowerShellVirtualDirectory:

* remove all virtual directories whether CAS role

Get-PowerShellVirtualDirectory EXMBX02\* | Remove-PowerShellVirtualDirectory

2.  Recover Exchange server:

Setup /m:RecoverServer

Appendix

Extending schema for Exchange 2013 – error id: 8224 #2

I have faced another kind of Error ID: 8224, which was, however on virtual machine, not caused by TCP chimney issue, so here is the solution.

Problem

When running command

.\setup /PrepareSchema

I have received error 8224 as shown in picture:

In my previous article the problem was caused by incorrect settings of TCP interface (http://ficility.net/2012/09/06/extending-schema-for-exchange-2013-error-id-8224/), which caused delays in replication and change actually solved the problem. In this case TCP interface was set correctly and I still experienced delays in replication. Here is how to deal with this kind of problem.

Solution

Before extending schema domain controllers hosting FSMO roles (at least Schema master) must be accessible and replication should work correctly. In my case there was a problem replicating Schema and Forest zones to secondary DC, which was connected via VPN. I have resolved VPN connection problems and after that I forced replication between DCs. Note, that MS have incorrect syntax on their technet site. You have to use quotations to run command successfully.
Replication errors:

replication problems

To force replication run the following command from elevated command prompt. http://technet.microsoft.com/cs-cz/library/cc742152(v=ws.10).aspx

force replication

repadmin /replicate <SOURCE_DC> <DESTINATION_DC> "<DC=domain,DC=local>"

Result:

extending schema

After successful replication it is possible to extend schema.