I have encountered this several times now. I don’t know if the disks are being incorrectly sized or if something else is happening that is causing this, but the Exchange transaction logs are not being removed and they continue to fill up the disk until it is completely full, and then it causes mail to stop flowing or in sever cases the database will be dismounted and it can not be remounted until the disk has at least 1.5GB of space reclaimed.
Every change made to an Exchange database must first be written in a transaction log file. Most Exchange administrators keep Exchange transaction log files on a dedicated drive. If the drive fills up, all Exchange databases in an affected storage group will dismount. Before you can re-mount any of the databases, you must free up some space on the transaction log drive.
Circular logging is not enabled and full successful backups are not being performed on a regular basis.
- inbound email ceases to be delivered
- outbound email ceases to be sent
- Exchange database fails to mount or is dismounted and can not be remounted
- Determine which logs are not necessary for restore
- move *DO NOT DELETE* those files to another location
- remount database, and verify that the transaction logs needed for restore are available.
Determine which logs are not necessary for restore
There are two ways:
You can read the Checkpoint field in the checkpoint file, which lists the oldest log needed for recovery.
You can read the Log Required field in the database header, which lists the range of logs needed for recovery.
Read the checkpoint file
The first method works for all versions of Exchange. The second method only works for Exchange 2000 and later versions.
To read the checkpoint file, you first have to find it. The location of the checkpoint file is the System Path listed on the General page for each storage group’s properties. That same properties page also lists the transaction log file path and the prefix for the storage group. The name of the checkpoint file will always be [prefix].chk, for example, E00.chk.
Once the checkpoint file is located, it can be read by using Eseutil.exe. This utility is located in the Exchange bin directory (C:\Program Files\Microsoft\Exchange Server\V14\Bin for Exchange 2010 for example).
Issuing the following command will return quite a bit of information, but the key line is “Checkpoint”.
"C:\Program Files\exchsrvr\bin\eseutil.exe" /MK "C:\Program Files\exchsrvr\mdbdata\E02.chk"
The output will produce a line similar to the following:
where the first number is in the list is the important number. The first number lists the generation or sequence number of the oldest log file needed to recover the databases in the storage group. The other two numbers are offsets into the log file, and don’t really matter.
Remember: move log files, don’t delete them. If a mistake is made, or if the database has to be restored later from backup, you’ll want to get those log files back.
Read the “Log Required” field in the database header
In the case that the mailbox database has been unmounted or is not able to be mounted or if the database can be unmounted, the following command can be issued to determine what logs are necessary for recovery. The command looks at the database’s header and therefore will only work if the database is offline.
"C:\Program Files\exchsrvr\bin\eseutil.exe" /MH "D:\mdbdata\Mb1.edb"
This should return a large amount of information about the current state of the database back to the command prompt window. You will be looking for a line similar to the following:
Log Required: 8686-8690 (0x21EE-0x21F2)
where the 8686-8690 is the start and finish transaction log numbers required to restore. These are in decimal format and the logs are written in hexadecimal so you will want to use the numbers in parenthesis to located the required log files. In this example, the range of logs required is E02021EE.log to E02021F2.log.
Move unnecessary transaction logs to another partition or drive to free up disk space
Once it has been determined which files are no longer required to perform a full restore, the next step is moving those files to another partition or drive to free up disk space. Once the disk has reclaimed at least 1.5GB of space, the database should be mountable again. If a copy and then delete process was used, it may be important to empty the Recycle Bin as well if the disk space doesn’t seem to be changing. Recycle Bins are maintained on the disks they are associated with, so simple deleting the files to the Recycle Bin may not clear up any additional space.
Remount the database and verify mail flow and functionality
Once the available space is above 1.5GB, the database can be remounted and Exchange should start working as expected. It is probably important to run a few tests to verify. It is also important to remember that mail might have been building up in the queues and things may take a few moments to “catch up”.
Missing Transaction Logs
I have been adamant about not deleting any of the log files. This is because without the missing log file, there is no way to mount the database again without restoring it from backup or repairing it.
It may appear a log is missing because the most recent log doesn’t have a number associated with it yet. While Exchange is still filling up a log with data, the name of the log is set to only the log prefix (Exx) with no hexadecimal generation number (XXXXX). For example, E02.log would be the current log in use by the storage group with the log prefix E02. When this log is full, it might be renamed to something like E02021F2.log, and a new E02.log will be created.
You can tell what the current log will eventually be named by using Eseutil with the /ML switch:
"C:\Program Files\exchsrvr\bin\eseutil.exe" /ML "L:\mdbdata\E02.log"
Look for the “lGeneration” line in the output:
What this line shows is that the E02.log will become E02021F2.LOG once Exchange has finished filling the file.
Different steps is the “Log Required” field is 0-0
If the “Log Required” field is 0-0, this means the database has been shut down cleanly and has been detached from the log files properly. This database doesn’t need any transaction log files in order to be mounted again. But this does not mean you should remove all the transaction log files.
If Log Required is 0-0 for one database, the header of every other database in the storage group should be checked using (Eseutil /MH), and it should be verified that each database shows “Log Required: 0-0” and “State: Clean Shutdown”.
If this is the case, it is safe to move all numbered log files, but do not remove the current log file (Exx.log).
It is true that databases in “Clean Shutdown” state don’t need previous log files in order to mount them again. All logs could be removed without losing any data, but if all log files removed, Exchange will create a new series of log files starting over with log generation 1. This is called resetting the log series. If a restore from backup is required later, this will greatly complicate rolling the database forward because there will be two incompatible sets of log files to contend with.
However, if the current log is left in place, removing only numbered logs, then when the databases start up again, they will attach to the current log and continue the previous series. Remember that this only applies if “Log Required” is 0-0. If “Log Required” actually lists a range of logs, the entire range of logs must be available to the databases. Finally if “Log Required” is 0-0 for all databases in the storage group, the checkpoint is always in the current log file too.
As a general rule, don’t reset logs without a good reason. One good reason would be because you are getting close to the million log limit on log file generation numbers. If a log series is reset, take a full backup as soon as possible afterward.
How Exchange manages the transaction logs
There are two methods used by Exchange to “prune” old log files so that the disk doesn’t fill up:
Circular logging. By default, circular logging is enabled in Exchange 5.5. With circular logging turned on, as soon as the checkpoint passes through a log file, and the log file is therefore no longer needed by any database for crash recovery, then the log is deleted. Typically, if circular logging is turned on, there will be no more than half a dozen log files in existence at any given time. Circular logging was the Exchange 5.5 default specifically to prevent log drives from filling up. But there is a big drawback to circular logging.
If you have to restore from backup with circular logging enabled, then Exchange will not be able to roll the database forward with additional transaction logs because all those logs will have already been deleted.
In Exchange 2000, the default for circular logging was changed to be off, even though this means a log drive will eventually fill up if an administrator doesn’t take regular backups. This means that the problem of log drives running out of space happens more frequently than it did before. The great majority of Exchange administrators tell us that they prefer the new setting, because it means Exchange won’t delete any transaction log files without the administrator giving “permission” for it.
NOTE: Small Business Server 2003 includes Exchange 2003. Circular logging is enabled by default for Exchange 2003 running on Small Business Server. However, running the Small Business Server Backup Wizard will automatically turn off circular logging. If you use a third party backup solution with Small Business Server 2003, you should examine the storage group properties to verify that circular logging is not enabled.
Taking an online backup. After a Normal (Full) backup completes successfully, Exchange will automatically remove logs that are not needed to roll forward from the backup. Logs are also pruned after an Incremental backup.
An administrator can also manually remove log files, as long as the rules explained in this flash are followed.
Transaction Logs are not getting pruned
The most common reasons for this are:
- All the databases in a storage group are not being backed up regularly. All databases in a storage group share a single set of log files. Until no database in the storage group needs a particular log file to roll forward from backup, the log file will not be removed. So if one database hasn’t been backed up in a long time, all the log files that database needs will stay on disk, even if no other database requires them any longer.
- All databases in the storage group are not mounted when online backup is done. At the end of backup, Exchange checks with each database in the storage group to find out which logs it still needs for recovery. If a database is offline, Exchange cannot tell what it needs, and so, to be safe, no logs will be pruned.
- Copy or Differential backups are being performed instead of Normal (Full) or Incremental backups. By design, no logs will be pruned after a Copy or Differential backup.
Transaction Log disk recommendations
It is recommend that log drives be sized to hold at least 10 times as many logs as you generate on an average day. This gives some buffer room if the backup fails for several days in a row, or logs can’t be pruned for some other reason.
The extra space also helps if a sudden jump in database activity causes more logs to be generated than normal. For example, if a large number of mailboxes are moved between databases, that is likely to generate a large number of log files because in essence, all the mail is being “re-delivered” all at once.
Another measure that can be taken to make it easier to recover from running out of log drive disk space is to put a “spacer” file on the log drive. The size of this file should be a little more than the size of the log files you generate in a day. Put this file in the same folder as the log files, and name it something like “AAA DELETE ME IF YOU RUN OUT OF SPACE.SPACER.”
If the spacer file exists, it is likely that even an untrained administrator will delete this file instead of destroying essential Exchange log files.
Most of this information was taken from http://blogs.technet.com/b/exchange/archive/2004/05/12/130556.aspx.