URGENT: Hold off on deploying Exchange 2016 on Windows 2016

Exchange 2016 CU3 was the 1st version of Exchange supported on Windows 2016 when it was released on 9/20/2016 [Exchange 2013 & 2016 quarterly updates out]. This was a minor update, biggest changes are the support for Windows 2016 and indexing from passive copies.

Today officially Microsoft posted:

Microsoft recommends that customers delay deploying Exchange Server on Windows Server 2016 until this update is made available.

Source: https://blogs.technet.microsoft.com/exchange/2016/11/04/update-on-windows-server-2016-and-exchange-server-2016/.

The issue surfaces as an issue with IIS, that causes W3WP.exe to crash, which leads to “instability” on that Exchange server and the DAG. The issues does not seem to appear on stand-alone servers. You will see event IDs 1003, 4999, & 5011 in the application logs relating to IIS issues. Per Tony’s post, see below, this issue is not directly related to IIS, but IIS crashes as a result of it. This it only appears on DAG members, which use Windows Clusters, so it makes sense it is related to clustering.

Tony Redmond goes into more detail here: Exchange 2016 to Windows 2016: No Thanks – At Least, Not for the Moment (11/4/2016)

One of the first public post about the issues was on the TechNet forums on 10/19/2016: Exchange Server 2016 & Windows Server 2016 Datacenter IIS AppPool’s constantly crashing. ECP/Powershell inaccessible.

The release notes for Windows 2016 have also now been updated with:

If you attempt to run Microsoft Exchange 2016 CU3 on Windows Server 2016, you will experience errors in the IIS host process W3WP.exe. There is no workaround at this time. You should postpone deployment of Exchange 2016 CU3 on Windows Server 2016 until a supported fix is available.

I will update this post as more info comes in or when a fix is posted.


Posted in Exchange, Technical | Tagged , | 1 Comment

Don’t let your O365 tenant get hijacked! An example of this happening…

I recently helped a small non-profit recover from a disgruntled director hijacking their O365 tenant and domains, they were able to do this since they were a Global Admin for their tenant. This post is about what happened, from a O365\domain stand point only, and as a warning to other orgs out there.

When I was contacted by the interim director of the org I was asked to see what the ex-director had done. I quickly found out I was unable to login to their O365 tenant. I then tried a couple of other, test, accounts I had and none of them worked. My next step was to try to login to their GoDaddy account, but that also failed.

So I called Microsoft O365 support… And waited… And waited… When I got someone I described the issue to them, was transferred and repeated this to a few different people, before getting to the data protection team. As suspected, they wanted to verify who I was and the only method they would use was a domain validation (DNS record), since my account and any other account I had access to no longer existed. Because I didn’t have access to the GoDaddy account, where DNS was hosted, I had no way to validate I was the actual admin\owner of the domain. In addition, their policies require that they contact all Global Admins before doing anything else. Since the disgruntled ex-employee was the only Global Admin left I knew this wouldn’t help.

So I called GoDaddy. Since I didn’t have the password or call in PIN #, which were changed by the ex-employee, they initially said they couldn’t help. But I then got transferred to a manager and we went over the issue further. He said if we could get the domain record owner, the domain register contact, to send in a copy of their photo ID and a letter on the non-profit their letterhead he would reset the password. The problem with this is that the domain contact was a person who setup the domain years ago. The contact number for him was the number for the non-profit. So I sent him an email explaining the situation and asked him to contact me for more details, which I never heard back on.

So after spending a few hours with O365 and GoDaddy support I was dead in the water with this issue.

I called back the interim director and told her that currently I was stuck and asked about getting the passwords from the ex-employee. She let me know that they had already had a lawyer write up a letter to the ex-employee and it was to be delivered that day by certified carrier. She had until noon the next day to turn over several items she took and any passwords. If she did not the org was going to file charges against her for theft and other offences. So we decided to wait it out to see if she would turn over the passwords. The only other alternatives we had was to call back O365 & GoDaddy support and beg them to help us further and hope that the person that registered the domain years ago would help also.

Luckily, the ex-employee turned over the passwords to GoDaddy & her O365 account the next day.

With the new passwords, I logged into GoDaddy and did the following:

  1. Change the password on the account
  2. Change the contact info to my name, as the technical lead for the org
  3. Setup delegated access to my primary GoDaddy account
  4. Changed the contact info on the domain to me as primary & technical contact and the interim director for the administrative contact.
  5. Setup two-factor authentication to my cell phone
  6. Setup a new PIN
  7. I then shared the new login info and PIN with the interim director


On O365, I setup an account for me, recovered the other accounts this person has deleted and granted the interim director Global Admin rights also. The last I did just in case I got hit by a bus.


  1. Make sure your domain(s) contact info is current and multiple trusted parties are listed on it
  2. Set the password on your domain account to something complex and save it in a safe place
    • A safe place could be a piece of paper in a safe that only a few people have access to or better a password saving tool, like LastPass, that only a VERY few people have access to.
  3. Setup alerts, if possible, so key people are notified when the password is changed on the domain hosting account and when settings are changed on the DNS and\or hosting account
  4. Have at least two accounts with Global Admin rights that admin have access to and another with a password that is stored in a safe place, that management can access if needed

With access recovered I then recovered the deleted accounts & mailboxes, which just took a few click at https://portal.office.com/AdminPortal/Home by going to Users\Deleted Users and click the users and clicking Restore. By default, O365 will keep deleted users and mailbox for 30 days.

With the deleted mailboxes recovered I then delegated access to the ex-director’s mailbox to the interim director and ask her to look it over. She got back to me and told me there were very few emails in it. So I assumed the ex-director deleted email. But since this org was using the free O365 for Non-profit E1 we had no access to the eDiscovery tools to search or put mailboxes on retention old. So with authorization, I upgraded the ex-director’s account to an E3 license, which includes eDiscovery support.

After searching her mailbox, I found many hard deleted items. I placed the mailbox on retention hold, exported the search results to the Discovery mailbox and exported her entire mailbox, including deleted items to a PST file. The PST file provided a better format since it kept folder structure, even deleted ones in tack. With the Discovery mailbox all results are dumped into a single folder.

I plan on making a post that goes into more details on the eDiscovery processes I used above soon.

Posted in O365, Technical | Tagged , | Leave a comment

Exchange 2013 & 2016 quarterly updates out

11/7/16 Added a “Known & Possible Issues” list
11/4/16 Urgent Update: Do NOT deploy Exchange 2016 CU3 (or earlier) on Windows 2016 at this time. See URGENT: Hold off on deploying Exchange 2016 on Windows 2016 for more details.
Today, 9/20/2016 Microsoft release CU14 for Exchange 2013 and CU3 for 2016.

EHLO Blog post: Released: September 2016 Quarterly Exchange Updates

Exchange 2013 | Download CU14 | KB3177670

Exchange 2016 | Download CU3 | KB3152589

Exchange 2016 Key Updates

  • Windows Server 2016 Support
    • Also includes .Net 4.6.2 support, on Windows 2016 Only currently
    • .Net 4.6.2 support will be required by March 2017, which will be supported on Windows 2008 R2 and higher by then
  • Local indexing for search or “Read from Passive” support – Starting with CU3 the local DB, passive or active, will be used to index the content in the DB for search. Before CU3 servers hosting passive copies had to query the active copy of the DB to create the local search index. This change may result in up to a 40% reduction in bandwidth. This may also speed up failovers since the passive copy no longer need to query the active copy and make local updates before being made active. Lagged copies will still need to communicate coordinate with the active copy.
  • A data loss scenario was addressed with Public Folder migration (KB3161916)
  • AD scheme is updated with CU3
  • Pre-requisite install behavior changed so server is not placed in off-line monitoring state at the start of the install, now this is only done once the pre-requisite checks are done and the install of binaries is started

There are no key updates in Exchange 2013 CU14.

PS: Exchange 2007 End of Life (EOL) is now only seven months away, 4/11/2017, after this date 2007 will not longer be covered under extended support. Read more here. If you are still on 2007 you need to be migrating off of it NOW!

Known & Possible Issues

Posted in Exchange, Microsoft, Technical | Tagged , | 1 Comment

Calendar Sharing across Devices w/ Exchange or EXO/O365

Starting in Exchange 2010 you have been able to share your calendar, if enabled at the organization level, with anyone on the Internet anonymously. This can still be done today with Exchange 2016 and Office 365 in the ICS (iCAL), which can be used by most email\calendar clients, and HTML formats.

Personally, I use this support so I can view my wife and two teenager’s calendars and they can view my personal and work calendar. From a professional standpoint, this would be great for a manager to view his team member’s calendars or an admin assist to be able to view their manager’s calendar, from their mobile devices. The ActiveSync protocol, used by most mobile devices, does not support delegated mailbox or calendar access. So access team member or other people’s calendars from mobile devices has to be setup manually.

For this to work your Exchange or EXO instance must be setup for Internet calendar publishing. This can be done via PowerShell or EAC. See the steps in this TechNet article for EMS and ECP directions, ECP steps below. This support can also be enabled between Exchange\O365 orgs and to certain domains if you want to limit the support to partners, for example. In my case I have an Anonymous sharing rule so anyone, that I give the URL to can access our calendars.

To Enable individual calendar sharing, in ECP for Exchange 2013+ or EXO, at the org level

  1. Open ECP and goto the organization menu
  2. Under “Individual Sharing” click the + (plus) button
  3. Give the sharing policy a name, like Anonymous calendar sharing
  4. Under “Define sharing rules for this policy” click the + (plus) to create a new Sharing Rule
    1. Choose Sharing with a specific domain
    2. Enter “Anonymous” for the domain name, to allow anyone to be able to access the calendars shared by user
      Note: This rule only allows users to share their calendars but does not share them for them automatically, this is still an action the users have to take.

      1. Check Share your calendar folder
      2. Then choose the level of calendar info to share, for anonymous I recommend free/busy with time only or free/busy with time, subject, and location.
        Note: If you choose All attachment and details in the meeting body will also be shared
  5. Click Save

Now that calendar sharing is enabled, users will need to Opt-In if they want someone to be able to access their calendar anonymous from the Internet.

Enabling sharing for a user and get their ICS URL for the calendar

These steps are for Exchange 2016; this can also be done in Exchange 2010 & 2013 but the steps vary.

These steps should be carried out by the end user.

  1. Open the OWA Calendar options
    1. In OWA, goto the Setting “gear” and choose Options
    2. In Outlook 2013+, goto your calendar and right click on it and choose Publish This Calendar
  2. Once in OWA, expand Calendar and choose Publish calendar
  3. Under “Select permissions” choose Limited details or Availability only then click Save
  4. Note: Users can only share details up to the level set at the org policy level earlier
  5. After clicking Save two URLs will be displayed
    1. The HTML one will use OWA to display a read-only version of that person’s calendar
    2. ICS is the iCAL format that can be shared with others so they can view the calendar in their email clients
  6. Email\share the ICS URL with those you want to be able to view your calendar in their email client

Adding a shared calendar to an iOS device

Give the long URL, you will want to email the ICS URL to those that you want to share it with. In my environment I setup custom short URL for ours, like calendar.company.com/Jason, and one for my wife and each of our teenagers.

  1. Copy the ICS URL on the iOS device that are setting up access to another calendar on
  2. Goto Settings\Mail, Contacts, Calendars
  3. Click Add Account
  4. Choose Other
  5. Choose Add Subscribed Calendar
  6. Paste the ICS URL and click Next
  7. Enter a friendly name for the calendar when prompted, the other fields can be left blank
  8. This calendar should now show up in the Calendars app

Adding a shared calendar to in Outlook

These steps are only needed for calendars outside of the Exchange org your mailbox is in, also not required if there is an org sharing relationship setup between your org and theirs.

  1. Goto Outlook Calendar
  2. Right click on Other Calendars and choose Add Calendar\From Internet…
  3. Paste in the ICS URL

Adding a shared calendar to in Google Calendar

  1. Goto https://calendar.google.com
  2. Next to “Other calendars” click the down arrow and choose Add by URL
  3. Paste in the ICS URL, which should have been sent to the Gmail user via email

Using the steps above, you have enabled anonymous calendar sharing and given users the ability to share their calendar with anyone on the Internet. Furthermore, I covered how to add a calendar shared from Exchange\OWA to iOS, Outlook, and Gmail.

Posted in Exchange, Microsoft, O365, Technical | Tagged | Leave a comment

Exchange 2016 CU2 and 2013 CU13 are out

Microsoft released the latest update to Exchange 2013 and 2016 on 6/21/2016. No major changes, but the DAG auto rebalancing and finally .NET 4.6.1 support are much desired changes. The new DAG feature is important to larger environments and the .NET support is important since 4.6.1 is an automatically suggested update and will cause issues on Exchange, unless you update to CU2\CU13 before install it.

EHLO Blog Post on this: https://blogs.technet.microsoft.com/exchange/2016/06/21/released-june-2016-quarterly-exchange-updates/

Tony Redmon’s post: https://www.itunity.com/article/excitement-good-news-exchange-quarterly-updates-3412

Key Changes

Other Changes

  • 2016: Get-ExchangeServer cmdlet updated to include rule definitions. Not a big deal, since 2016 servers will either be a Mailbox of Edge server
  • 2016: Self-signed certificates will now use SHA-2

2016 CU2 Key Fixes

2013 CU13 Key Fixes

Posted in Exchange, Microsoft, Technical | Tagged , | Leave a comment

20 Years on Exchange and 30 years on email

I don’t recall the exact date, but when I worked at Digital Equipment Corporation (DEC), 1994-1996, I got a beta version of Exchange 4.0 in late 1995. Having worked with MS:Mail, VMS Mail (ALL-IN-1), and several other email systems I really wanted to see the new mail server from Microsoft.

So using my DEC AXP150, running a true 64-bit processor in 1995!, test box running Windows NT 3.51 64-bit. I installed my 1st version of Exchange 4.0 beta (64-bit also) and started down the path of being a Microsoft Exchange consultant and expert.

Initially, this was just a test system but our team of ~30 quickly moved it for our primary communications. We were on the Microsoft PC Apps support team and really didn’t like the character only messaging systems we had been using. Within a few months, it expanded to the PC hardware support and other teams and when I left DEC, in early 1996, we had over 200 mailboxes on two different Alpha based Exchange 4.0 beta servers. When I was leaving DEC I chatted with our corporate IT staff about their planned roll-out of Exchange, which had Tony Redmond (fellow Exchange MVP) on the team. It wasn’t until around early 2000s that Tony and I reconnected and realized we had talked about Exchange about 10 years earlier when we were both at DEC.

At home, where I ran a 12 node BBS and mini ISP, I continued to run NetScape Mail until Exchange 5.0 came out with OWA support in 1997.

My 1st email system I ever setup was actually about 30 years ago, in 1986 when I was sysop on multiple  WWIV BBSs that were integrated into the FidoNet. Back in those days you could still send an email around the county or world, but it would take many days to get to some locations. We were limited to 300 baud, or about 300 characters per sec, and long distance phone calls were expense, but the rates dropped in the middle of the night. So Fido net would store and queue up message during the day, then at a schedule time when long distance was cheaper it would call a BBS in another city. This process would repeat until your message got from your BBS to the user on the target BBS. It was common for a message to take days to get across the county initially, but hubs were setup and Delphia & Compuserve start providing quicker routing in the early 90s.

Today, we commonly see single emails that were many times the size of our entire mailboxes in Exchange 4.0. We started out with 10MB mailbox I believe on Exchange 4.0 at DEC, now I host mailbox for friends and family on Exchange 2016 out of my house that have a 10GB limit by default 🙂

Posted in Exchange, Technical | Tagged | 2 Comments

Odd Transport Issue: Mail Stuck in internal queues

A week or so ago I started to notice messages getting stuck in the queues on one of my Exchange 2013 servers. My troubleshooting included restarting services, applying the latest CU, statically setting IPv6 address, and looking though logs but I was unable to find the issue. What I did see in the logs, at the end of this post, were connection rejected and DNS errors that looked related to IPv6. So in an effort to see if IPv6 was really the issue I setup static IPv6 addresses on the Exchange servers, but that didn’t help. After exhausting many other things and getting tired of copying the mail.que files from one server to another to get the messages delivered, I finally called PSS\Microsoft Support.

PSS started out by checking basic name resolution, which was working. Then check IP and DNS settings on the NICs, where were fine. Then they checked for static DNS server settings on the transport services, I didn’t think of that and should have!

This is where they found the problem. Somehow the IPv6 address for the DNS servers was set on the backend transport service on my IZSRVEX01 server, the one were the queues were backing up. Messages both to be delivered to the Internet, via O365, and to internal mailboxes, even if on the local server, were getting stuck.

Here are the cmdlets PSS ran to find these settings:

Get-TransportService | ft  name, *DNSAdapterGuid
Get-FrontendTransportService | ft  name, *DNSAdapterGuid

On IZSRVEX01 the InternalDNSAdapterGuid value was set to something other than all zeros. So PSS cleared the values with this cmdlet:

Get-TransportService | ? {$_.Name -NotLike "*EDGE*"} | Set-TransportService -InternalDNSAdapterGuid 00000000-0000-0000-0000-000000000000 -ExternalDNSAdapterGuid 00000000-0000-0000-0000-000000000000

I’m not sure how these got set, the best wild guess I can make it that it got set somehow when moving VMs between Hyper-V servers. When doing this I’ve seen virtual NICs get lost and have had to reconfigure them, but still not sure why this would cause transport service to have a static IPv6 address set.

What makes troubleshooting this difficult is that starting with Exchange 2013 there is the Front-End (FE) transport services, which the *- FrontendTransportService cmdlets apply to and the Back-End (BE) transport service, which the *-TransportService cmdlets apply to. By default, the BE transport services do not have logging enable also. After I enabled logging, which I normally enable via a Transport configuring script, I did find which log had errors this should have led me to checking the DNS settings on the transport services, but I missed that.

Errors found using cmdlets

Using the [Get-Queue] cmdlet:

Identity    MessageCount NextHopDomain           Status LastError

——–    ———— ————-           —— ———

IZSRVEX01\4          115 edgesync – home to o365  Retry 451 4.4.0 DNS query failed. The error was: DNS query failed with error ErrorRetry

IZSRVEX01\5          497 mailboxes                Retry 451 4.4.0 DNS query failed. The error was: DNS query failed with error ErrorRetry

You can see above that message, that messages to both the EDGE server, which then delivers to Office 365, and to the “mailboxes” database were suck.

Below are further signs of the issues and log entries.

Using the [Get-Queue -Identity IZSRVEX01\5 | FL]:

RunspaceId            : 42ba65c4-de75-4a73-81c3-8c97f9a5a314
DeliveryType          : SmtpDeliveryToMailbox
NextHopDomain         : mailboxes
TlsDomain             :
NextHopConnector      : 500b24dd-bda7-49e5-816d-5e9ea8d9360b
Status                : Retry
MessageCount          : 1
LastError             : 451 4.4.0 DNS query failed. The error was: DNS query failed with error ErrorRetry


From BE connectivity transport, default path: C:\Exchange Server\V15\TransportRoles\Logs\Hub\Connectivity\CONNECTLOG<date>.LOG

2016-01-18T18:56:56.592Z,08D32039214A4CB4,SMTP,edgesync – home to o365,>,DNS server returned ErrorRetry reported by [Domain:Result] = IZSRVEDGE01.altered.com:ErrorRetry; IZSRVEDGE02.altered.com:ErrorRetry;

2016-01-18T18:56:56.592Z,08D32039214A4CB4,SMTP,edgesync – home to o365,-,Messages: 0 Bytes: 0 (The DNS query for  ‘SmtpRelayWithinAdSiteToEdge’:’edgesync – home to o365′:’54fa82f8-4b9d-49fe-acbd-2f968f11a3cd’ failed with error : ErrorRetry)

From C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\ProtocolLog\SmtpSend

2016-01-21T18:40:50.468Z,Intra-Organization SMTP Send Connector,08D322903B262F35,1,,[da8:6c3:ce53:a890::42]:2525,*,,”Failed to connect. Winsock error code: 10061, Win32 error code: 10061, Error Message: No connection could be made because the target machine actively refused it [da8:6c3:ce53:a890::42]:2525″

Posted in Exchange, Technical | Tagged | 1 Comment