Potential for database corruption after installing Exchange 2007 SP3 RU3

I recently posted about the availability of Update Rollup 3 for Exchange 2007 SP3 and Exchange 2010 SP1 and then followed up by posting about an issue impacting some customers which have RIM BlackBerry devices connecting to Exchange 2010 SP1 with RU3.

Over the weekend, the Exchange Product Group was made aware of an issue which may lead to database corruption if you are running Exchange 2007 Service Pack 3 with Update Rollup 3.

The issue was introduced in Exchange 2007 SP3 RU3 by a change in how the database is grown during transaction log replay when new data is written to the database file and there are no available free pages to be consumed and is of specific concern in two scenarios:

  1. When transaction log replay is performed by the Replication Service as part of ensuring the passive database copy is up-to-date and/or
  2. When a database is not cleanly shut down and recovery occurs.

When this issue occurs, the following similar events are logged in the Application Event log of the Mailbox server:

  • Event Type: Error
    Event Source: ESE
    Event Category: Logging/Recovery
    Event ID: 454
    Description: Microsoft.Exchange.Cluster.ReplayService (12716) Recovery E20 SG1\DB1: Database recovery/restore failed with unexpected error -4001.
  • Event Type: Error
    Event Source: MSExchangeRepl
    Event Category: Service
    Event ID: 2097
    Description: The Microsoft Exchange Replication Service encountered an unexpected Extensible Storage Engine (ESE) exception in storage group 'SG1\DB1'. The ESE exception is a read was issued to a location beyond EOF (writes will expand the file) (-4001) ().
  • Event Type: Error
    Event Source: MSExchangeRepl
    Event Category: Service
    Event ID: 2095
    Description: Log file D:\logs\SG1\E200006AFAE.log in SG1\DB1 could not be replayed. Re-seeding the passive node is now required. Use the Update-StorageGroupCopy cmdlet in the Exchange Management Shell to perform a re-seed operation

While only a small number of customers have been affected to date, the Product Group believe the risk is significant enough that they are recommending all customers to uninstall Exchange 2007 SP3 RU3 on all Mailbox Servers and Transport servers. Uninstalling the rollup will revert the system back to the previously installed version. They have also removed the Exchange 2007 SP3 RU3 download from the Microsoft Download Center and from Microsoft Update until we are able to produce a new version of the rollup.

It is strongly recommended that the below actions are taken to ensure that no data loss or outage is experienced.

For environments leveraging CCR and/or Standby Continuous Replication (SCR)

If you note the listed events in your environment the following steps must be taken in order to restore your high-availability configuration:

  1. Rollback the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
  2. Re-seed all database copies on the CCR Mailbox server and any SCR target Mailbox servers hosting the passive database copies.
  3. Verify the database copy status is healthy for all passive copies.
  4. Perform a switchover and rollback the remaining CCR Mailbox server to the previously installed version (e.g., Exchange 2007 SP3 RU2).

If you are not seeing these events in your continuous replication enabled environment, we recommend the following steps:

  1. Rollback the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
  2. Perform a switchover and rollback the remaining CCR Mailbox server to the previously installed version (e.g., Exchange 2007 SP3 RU2).

For environments leveraging Single Copy Clusters (SCC)

  1. Rollback passive nodes within the SCC environment to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
  2. Perform a switchover and rollback the remaining SCC Mailbox server nodes to the previously installed version (e.g., Exchange 2007 SP3 RU2).
  3. Restore and recover any damaged databases leveraging a last known good backup.

For environments leveraging standalone Mailbox servers

  1. Rollback the standalone Mailbox servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
  2. Restore and recovery any damaged databases leveraging a last known good backup.

For Hub Transport and Edge Transport servers

  1. Rollback the standalone transport servers to the previously installed version (e.g., Exchange 2007 SP3 RU2) by uninstalling RU3.
  2. Recover damaged mail.que databases by following the steps in Working with the Queue Database on Transport Servers.

BPOS/Office 365 and BlackBerry

Microsoft recently announced that they are in the process of updating pricing for the Hosted BlackBerry Service for Exchange Online to make it available free of charge. From the 16th of March 2011, BlackBerry mobile e-mail for new Exchange Online customers is now free through the Business Productivity Online Suite (BPOS).  For existing customers, a further announcement is expected later this month on how to take advantage of this change.

In addition, Hosted BlackBerry service will be available for free with Office 365 after it launches later this year.

Update Rollup 3 for Exchange 2010 – Download disabled

There have been reports of an issue impacting some customers which have RIM BlackBerry devices connecting to Exchange 2010 SP1 with RU3. The product team are working with RIM to identify the exact scenarios, narrow down the root cause of the problem and identify a suitable resolution for it. During this time the download page for Update Rollup 3 has been disabled.

If you are seeing duplicate messages being delivered when an email is sent from a BlackBerry device and you have RU3 installed within your Exchange 2010 environment, contact Microsoft Support for assistance in troubleshooting the issue you’re experiencing.

It is recommended at this time that everyone hold off deploying RU3 until they have identified and resolved the issues. If you have already deployed RU3 and you are not seeing any issues within your environment, leave RU3 in place at this time.

Update Rollup 3 for Exchange 2007 SP3 and Exchange 2010 SP1

The Exchange CXP team recently released Update Rollup 3 for Exchange Server 2007 SP3 and Exchange Server 2010 SP1 to the Download Center. Release via Microsoft Update will occur on March 22nd 2011.

These updates contain a number of customer reported and internally found issues since the release of RU2. Both updates resolve a number of issues, in particular for Exchange 2010 it resolves more than 40 issues.

For a detailed list of these issues, visit the following links:

Update Rollup 4 for Exchange Server 2007 SP3 and Exchange Server 2010 SP1  is currently scheduled to release in May 2011.

As always, those of you running Forefront, be sure you perform these important steps from the command line in the Forefront directory before and after this rollup's installation process. Without these steps, Exchange services for Information Store and Transport will not start back up. You will need to disable ForeFront via "fscutility /disable" before installing the patch and then re-enable after the patch by running "fscutility /enable" to start it up again post installation.

To download these updates, visit the following links:

Upgrading from Windows 1.0 to Windows 7

This one is slightly off topic, but I thought I would share it nonetheless. I recently came across a post about someone who upgraded through every version of Windows from 1.0 to 7.

What is truly impressive about this is that there is some 20 years of application compatibility to be witnessed. Here is the very impressive video:

The first thing I noticed when I watched this was that the author never installed Windows ME. I have since found his “FAQ”:

1) Why Didn't you Install ME?
Since Windows ME cannot be upgraded to 2000, I chose to install 2000 straight from 98 as it was chronologically the next release. I might do an "alternate history" version to see what going from 98 to ME to XP looks like.
2) Did Doom 2 and Monkey Island work in Windows 7?
Yes. Sorry I didn't point that out explicitly in the video.
3) Can you do the same thing with Mac OS?
I don't know enough about Mac OS to be able to say, and I would not be the best person to do that experiment as I have never been a Mac user.
4) Where are you from?
Scotland.

Exchange support for Windows Server 2008 R2 SP1

Last week the Windows Server team announced the Release to Manufacturing (RTM) of Windows Server 2008 R2 Service Pack 1 (SP1), along with Windows 7 SP1. On February 16, both will be available to current customers of the Windows Volume Licensing program, as well as subscribers to Microsoft Developer Network (MSDN) and TechNet. On February 22, both will be available to all customers through Windows Update and will also come preinstalled on new servers ordered.

"Is Exchange supported running on Windows Server 2008 R2 SP1?" The Exchange team have confirmed that the following versions of Exchange are supported to run on Windows 2008 R2 SP1 (the RTM version of SP1):

  • Exchange 2010 SP1
  • Exchange 2010 RTM
  • Exchange 2007 SP3

There are a couple of things to note:

  • Exchange 2007 was not supported to run on Windows 2008 R2 at all before Exchange 2007 SP3.
  • Windows 2008 R2 SP1 includes all the hotfixes required to install Exchange 2010 SP1 so if you're installing Exchange 2010 SP1 on a server running Windows 2008 R2 SP1, you don't need to install these hotfixes separately.

Exchange 2010 SP1 IPD released

The solution accelerators team has released a new Infrastructure Planning and Design (IPD) guide for Microsoft Exchange Server 2010 with SP1.

This guide provides actionable guidance for designing an Exchange Server 2010 infrastructure. The guide’s step-by-step process gives a straightforward explanation of the key steps required for a successful implementation. It can help you reduce planning time and costs, and ensure a successful rollout of Exchange Server 2010. The Exchange Server 2010 Guide includes the following content:

  • Step 1: Define the Business and Technical Requirements
  • Step 2: Define the Instances of Exchange Server 2010
  • Step 3: Design the Mailbox Server Infrastructure
  • Step 4: Design the Client Access Server Infrastructure
  • Step 5: Design the Hub Transport Server Infrastructure
  • Step 6: Design the Edge Transport Server Infrastructure
  • Step 7: Design the Unified Messaging Server Infrastructure
  • Step 8: Define the Active Directory Domain Services Requirements

image

For more information and to download the Exchange 2010 SP1 IPD, click here

Windows Phone 7 – My thoughts..

I don’t often have much to say about mobile devices.. it is a frequently discussed topic amongst my friends and colleagues though and everyone generally has their own opinion.

I spent the last month testing a Windows Phone 7 device, The LG E900 to be exact. Before I share my thoughts with you, perhaps I’ll give you a little background. I use an iPhone, and I make no excuses for the fact that I love my iPhone. Before you start hurling abuse my way (this is a Microsoft centric blog after all, right?) I should mention that I have probably tried every Windows Mobile device out there, from the very first HP Ipaq’s, WM 2003 through to WM 6.5 but at the end of the day it was the iPhone that really changed the way I use a mobile device.

lg-e900

Windows Phone 7 – In short, I think its good.. certainly better than all previous Windows devices I have used. It’s responsive and quick and visually the interface is very slick. I like how it brings your contacts and Facebook together. What I did not like was how it requires a live ID to setup the phone and then automatically syncs those contacts to your phone as well.. no biggie, except you can’t disable this so if you are like me and don’t want your live ID contacts to sync to your phone, you need to ensure that your live ID is not your hotmail account. It does email and calendar well, but that is to be expected.

A few other things I don’t like.. no cut and paste (and yes, I know the early iPhones never had cut and paste either, but we are supposed to be progressing here..) There is no OCS/Lync client available for Windows Phone 7.. I honestly can’t believe that but I’ve been unable to find one.

Windows Phone 7 devices only support a subset of all Exchange ActiveSync mailbox policy settings so if you have Windows Phone 7 devices in your organization, they will experience synchronization problems if certain Exchange ActiveSync mailbox policy properties are configured. To allow Windows Phone 7 devices to synchronize with an Exchange mailbox, either set the AllowNonProvisionableDevices property to true or only configure the following Exchange ActiveSync mailbox policy properties:

  • PasswordRequired
  • MinPasswordLength
  • IdleTimeoutFrequencyValue
  • DeviceWipeThreshold
  • AllowSimplePassword
  • PasswordExpiration
  • PasswordHistory
  • DisableRemovableStorage
  • DisableIrDA
  • DisableDesktopSync
  • BlockRemoteDesktop
  • BlockInternetSharing

Another thing I found is that Microsoft appear to have dropped support for SMS sync with Exchange Server 2010.

In summary, I think Windows Phone 7 is a great step up from previous versions of Windows devices.. but I don’t think its quite there yet.. the app community is nowhere near as large that for the iPhone or even Android. I don’t see this as a enterprise device, it seems to be more consumer centric and does not really compete that well against the likes of the iPhone which is after all a consumer device. If you are currently making use of the enterprise features in WM 6.5 you may be disappointed with Windows Phone 7. I look forward to seeing further development in this space.

Exchange 2010 Online Archive support in Outlook 2007

It’s been a long time coming.. Earlier this year it was announced that Exchange 2010 online archive support for Outlook 2007 would be coming. The good news is that it is finally here!

The December 2010 Cumulative Update for Office 2007 will allow users with Outlook 2007 to access their Exchange 2010 online archives. Organizations with Outlook 2007 deployed can now benefit from Exchange 2010 archiving and retention. There are a few BUT’s though.. the following functionality is not supported in Outlook 2007:

  • Search across primary and archive mailboxes: When a user searches the primary mailbox, and selects All Mailbox Items, Outlook does not search the archive mailbox. Similarly, when the user searches the archive mailbox, the primary mailbox is not searched.
  • Archive policies: In Outlook 2007, users can't use personal tags (also known as archive policies) to move items to the archive mailbox. Any default archive policies for the mailbox continue to be applied. Users can use Outlook Web App to see or apply archive policies. Archive policies are retention policy tags with the Move to Archive action. Organizations can use the Default Archive and Retention Policy, or apply custom retention policies to a mailbox. The policies can include a default policy tag (DPT) to move items from the primary mailbox to the archive mailbox, and personal tags which users can apply to messages or folders to move them to archive.

Scott Schnoll on Backup Strategy in Exchange 2010

This is a slightly old interview with Microsoft’s Scott Schnoll where he talks about backup strategy in Exchange 2010. While some customers may find this to be an outrageous concept, he talks about how Exchange 2010 database replication can address most, if not all of the reasons you would traditionally backup Exchange using legacy backup technologies.


Every organisation has different requirements, the challenge is balancing these requirements while we embrace new technologies and evolve the way we think about things.