Exchange Database Introduction
The database of an Exchange server is something that seems to raise a lot of questions with Exchange administrators. This article is designed to provide an overview of the Exchange database. It covers all versions in the same article, for simplicity.
A version of this article originally appeared on the blog of the director of Sembee Ltd.
First some terminology.
Where VERSION is mentioned, this is Exchange 2000, 2003, and 2007.
Where EDITION is mentioned, this is Standard or Enterprise. Standard edition that also applies to the SBS variant.
Unless stated otherwise, references to Exchange 2003 also apply to Exchange 2000.
Exchange 2007 references also apply to Exchange 2010 unless stated otherwise.
This is a background article, it does not tell you how to do anything (just in case you came here via Google expecting to be told how to do X with your database).
Myths of the Exchange Database
There are a lot of myths around the Exchange database size and limits which this article should help to dispel
- The store will dismount when you hit a physical size of 75gb
- Adding up the mailboxes listed in ESM should equal the size of the database
- Regular offline defrags are required.
Then there is the confusion with many administrators that the database doesn't shrink in size, even after the users have deleted lots of data. This is covered as well.
Exchange Database Basics
To begin with, some basics on the Exchange database.
With Exchange 2003, the database is made up of two separate files. An EDB and an STM file. These combined are referred to as a store and come in two flavours - Mailbox and Public Folder.
Mailbox and Public Folder stores can be grouped together in to Storage Groups.
The EDB file should be thought of as the MAPI database and will consist mainly of internal email.
The STM file should be thought of as the SMTP database and will consist mainly of external email.
Email sent by Outlook Express users or other internal non Exchange servers would be considered external email.
However some information from the mail in the STM file is held in the EDB file.
The two files should be treated as one and when the Exchange community refers to the Exchange database, they usually mean both files.
With Exchange 2007 and higher, the data is now stored in a single file, with the edb extension. Still separate databases for Mailbox and Public Folder data. This is a more efficient way of storing the mailbox content.
Mailbox Store and Storage Group Capacities
With Exchange 2000 and 2003 Standard edition you can have one storage group consisting of one database of each type.
With Exchange 2000 and 2003 Enterprise edition, you can have four storage groups consisting of a maximum of four mailbox stores in each group.
With Exchange 2007 and Standard edition you can have up to four storage groups with a single mailbox store or public folder store in each, or a single Storage Group with four mailbox stores.
With Exchange 2007 Enterprise the number of Storage Groups goes up to 50.
For Exchange 2010, storage groups have gone away. Now you can just have either 5 databases on Standard edition, or 100 databases on Enterprise edition.
Database Size
The size of the database is a source of much confusion with newcomers to Exchange.
The simple fact with the PHYSICAL size of the database is that it will never shrink without intervention from the administrator. When content is removed from the database then the Exchange server marks that space as white space, and should use that space first for new content before increasing the physical size of the database.
However in practise, that often does not happen. What you will usually find is that if users are asked to clean out their email, more external email will be removed (spam etc) but more internal email is generated. Therefore the size of the database can increase, and the size of the white space in the database can hardly change.
Database Limits
The database limits are probably the are that causes the most concern for the Exchange administrators, so lets clear that up to begin with.
Exchange 2000 Standard has a database limit if 16gb, which can be increased to 17gb via a registry hack.
Exchange 2003 Standard RTM and Service Pack 1 is also subject to the same limit.
Exchange 2003 Standard with Service Pack 2 has a soft limit of 18gb, which can be increased to 75gb via a registry change.
Exchange 2007 Standard has a soft limit of 50gb in RTM and 250gb in Service Pack 1 which can be removed/changed with a registry change.
Exchange 2010 Standard RTM has the initial limit is 250gb and 1024gb from Service Pack 1.
Enterprise edition of all versions have a technically unlimited database size, although if you are picky it is 8TB with Exchange 2000/2003, 16000GB with Exchange 2007 and 2010.
If you update Exchange 2003 from Standard edition to Enterprise edition, then the registry setting for the soft limit is not removed, so the database may still dismount when it hit the size stated. You need to remove the key completely for that to stop happening.
Soft Limit
Soft limits are basically a way for an administrator to ensure that the database doesn’t get out of control. The Exchange server will react when a soft limit is reached by dismounting the store.
Database Limit Enforcement
The way that the database limit is enforced changed with Exchange 2003 Service Pack 2 and subsequent versions.
With Exchange 2000 and Exchange 2003 RTM and Service Pack 1, the limit was simply the physical size of the two database files combined.
With Exchange 2003 Service Pack 2 and later, the limit is now a logical limit. The limit is the physical size of the two files, minus the white space.
The white space is reported by event ID 1221 during the night. (More information on this event)
The logical limit of the database is not reported by Exchange until you change the default limit of 18gb.
The registry keys for increasing the 18gb limit in Exchange 2003 are in Microsoft KB article 912375 (link at the end) however you should read the Technet Article on how to work with the limit and setting the registry key for the warnings.
When setting the check time, ensure that it is AFTER the maintenance window configured on the Exchange server (ie after event ID 1221 has reported) so that content removed that night is taken in to account.
If you hit the limit -whether it is a limit below 75gb or the maximum 75gb limit and the database dismounts, you can mount it again. However it will dismount again the next day.
Offline and Online Defragmentation of the Database
When it comes to the database size and reducing it, most Exchange administrators will be referred to an offline defrag. However Exchange also does an online defrag. While they are related there are some key differences to what they do.
The online defrag is part of the nightly maintenance that Exchange does on its databases and is what finds and marks the white space for use. Its results are reported by event ID 1221. If that process does not run, the space gained by deleting content will not be used.
Am offline defrag will take the database and create a new one, consisting of the same data, minus the white space. Therefore the physical size of the database will be reduced. An offline defrag is the only way to reduce the physical size of the database.
The offline defrag is not risk free, and can take a considerable amount of time. The process speed is hardware dependant and can vary between 1 and 4gb per hour. Therefore if you have a 50gb store you could be looking at anything between 12 and 50 hours for the process to complete. Once started, it cannot be stopped. If it is, then both the source and the destination files are useless and a copy will need to be put in place.
The Exchange services have to be stopped while the process runs - so requires total downtime of the server. If you have multiple databases on the server then you can dismount the store you are working on and allow the others to run, however if you are in a position to run multiple databases, then you do not need to do an offline defrag, as explained below.
Some Exchange administrators claim that a regular offline defrag is required to keep the server running at the peak of performance. This is not the case and Microsoft specifically state that an offline defrag should not be considered something that needs to be done regularly.
The reason why there can appear to be a performance gain is because an offline defrag creates a new database. As with many things, if you replace with new then you will see some performance gains. Minor imperfections in the database structure can be removed and things generally cleaned up. However because it will skip data that it cannot read, that can mean there will be data loss.
With Exchange 2007 and higher, and Exchange 2003 Service Pack 2, or Exchange Enterprise edition (any version) an offline defrag is not necessary and is a waste of time.
Why?
With Exchange 2003 SP2 standard, due to the way that the database is reported, you gain nothing by doing an offline defrag. All you could do is lose data during the process. If you hit the limit, you can remount the database and then remove content.
With Exchange 2007 and higher (all editions) and Exchange Enterprise Edition (all versions) the process is unnecessary. Simply create another mailbox store, move all of the mailboxes to that store and then drop the original one and delete the database file. You can then create a replacement and move the content back. Zero risk, zero downtime.
If the store you are replacing is the original first store, then it will also hold some system mailboxes. Those will be recreated in another database when the system attendant service is recreated, so you should do that as soon as possible after dropping the original store.
The only reason why you want to do an offline defrag is because you are tight on physical storage, however you will need considerable space to do the offline defrag (At least 110% the size of the store) which will mean additional storage somewhere, so you may as well add it to the original server.
Mailbox Size - Exchange 2000/2003 only.
Many Exchange administrators will be unaware that the list of mailboxes in ESM is not showing the true size of the mailbox. This is clearly shown by the number of questions on forums from administrators who add up the size of their mailboxes and then ask why there is a X gb difference between that total and the sum of their physical database sizes.
In Microsoft KB article number 828070 (link at the end), Microsoft state:
"When you view the space that a mailbox uses in Exchange System Manager, the amount only includes the space that is used by the Priv.edb file. The amount does not include the space that the Priv.stm file uses."
Therefore a significant difference between the size of the mailboxes and the total of the physical database size should be expected.
This difference is further increased when you take in to account single instance storage and deleted item retention.
Single Instance Storage is a mechanism used within the Exchange database to keep the size of the database down. If you send an email with a 5mb attachment to 10 users, rather than using 50mb of space, it only uses 5mb. The attachment is only removed from the store when the last of those ten recipients removes it from their mailbox.
Deleted Item Retention (aka dumpster) is a feature of the Exchange database, where an item that is deleted from the mailbox or public folder (including removal from the Deleted Items folder) is stored in the database where it can be recovered.
Conclusion
Day to day administration of the Exchange database is not something that most administrators should fear or have any concerns about. As long as you monitor the size of the database regularly, then issues around the size should not come as a surprise.
References
Exchange Server 2003 mailbox store does not mount when the mailbox store database reaches the 16-GB limit
http://support.microsoft.com/kb/828070/
Database Size Limit Configuration and Management (Exchange 2003 SP2)
http://technet.microsoft.com/en-us/library/aa998066.aspx
How to increase the Exchange Server 2003 Service Pack 2 18-gigabyte database size limit
http://support.microsoft.com/kb/912375
How to Modify a Database Size Limit (Exchange 2007)
http://technet.microsoft.com/en-gb/library/bb232092.aspx