Tuesday 01 November 2005
- The truth about the index.dat files: What, why and where?
The truth about the index.dat files: What, why and where?
by Steven Burn
Two of the most common questions I get asked are, "what are the index.dat files?" and "will deleting them harm my computer?", the answer to the second is quite simply, no - deleting them will not harm the computer, at all.
The answer to the first however, is not quite as simple.
Index.dat files, known around the internet and the tin-foilers as "the secret files", were introduced in Internet Explorer 4 as (according to Microsoft) a way of easing the workload on the system resources, when you surfed the internet. Prior to index.dat, these files were known as "Mm256.dat" and "Mm2048.dat", what do the numbers represent?, your guess is as good as mine.
When you surf the internet, a cache of each page is stored on your computer, and details of the page, stored in the index.dat files. This was supposedly so the page would load quicker the next time it was you visited it (this part is atleast partially true). However, the index.dat files themselves contain more than just your surfing history. Everything from the site's you've visited, to the e-mails correspondence, is logged in these files, even your computer usage history is stored here (i.e. information on files you've opened, created and deleted).
Microsoft themselves, have never provided an adequate explanation as to the use of these files, except to say;
"Index.dat files are binary files that Internet Explorer uses to save information about URLs. They are basically hash tables that have been optimized to be very fast when dealing with URLs. Given an URL, these files map the URL to a binary blob of data. The binary blob of data is different, depending on what the particular index.dat file is used for. Quoted from: http://blogs.msdn.com/jeffdav/archive/2004/11/18/266027.aspx
Index.dat files are for IE's internal usage only. They are not designed or tested to be used by others."
The contents of these files are stored in binary (not binary as in 1's and 0's (everything is 1's and 0's as far as the computer is concerned)) and ASCII, meaning whilst you can open the file in Notepad and view it's contents, the majority of the file will be unreable to you, using the file located in the hidden "Content.ie5" folder as an example;
Many tools exist, both freeware and commercial, that will
allow you to view the contents of these files, these will be covered later.
The location of these files will vary according to your system, but on an NT based system such as Windows XP, are typically;
[Drive]\Documents and Settings\<USERNAME>\Cookies\index.dat
[Drive]\Documents and Settings\<USERNAME>\Local Settings\History\History.IE5\index.dat
[Drive]\Documents and Settings\<USERNAME>\Local Settings\History\History.IE5\MSHIST~<NUMBER>\index.dat
[Drive]\Documents and Settings\<USERNAME>\Local Settings\Temporary Internet Files\Content.IE5\index.dat
[Drive]\Documents and Settings\<USERNAME>\UserData\index.dat
[Drive]\Documents and Settings\Local Settings\Cookies\index.dat
[Drive]\Documents and Settings\Local Settings\Local Settings\History\History.IE5\index.dat
[Drive]\Documents and Settings\Local Settings\Local Settings\Temporary Internet Files\Content.IE5\index.dat
[Drive]\WINDOWS\system32\config\SystemProfile\Local Settings\Temporary Internet Files\Content.IE5\index.dat
If you have a network, you will also have;
[Drive]\Documents and Settings\Network\Cookies\index.dat
[Drive]\Documents and Settings\Network\Local Settings\History\History.IE5\index.dat
[Drive]\Documents and Settings\Network\Local Settings\Temporary Internet Files\Content.IE5\index.dat
Index.dat files are not typically "hidden" from view, but are instead "cloaked" using the "System" attribute, which is why, unless you have deleted them before, you will not be able to see them in Windows, or in DOS.
The files are in constant use when explorer.exe (the Windows GUI (Graphical User Interface)) is running and ergo, cannot be deleted using normal methods. Attempts to use the normal deletion methods will result in an error message being presented that claims;
When you use the cleaning options provided by the Internet Options, whilst the Cookies, Temporary Internet Files and History folders are cleaned, the index.dat files are left completely untouched.
Some believe you can delete these files via a DOS command line (del <switches> index.dat) after dropping out of explorer, whilst this is certainly true, it should not be used. The problem with this method is that there are legitimate files with the name "index.dat" that if deleted, will tend to cause rather strange behaviour to occur, for example;
Deletion of this file will prevent the System Information tool working correctly (it will instead, load the Help and Support files) as it's contents are used by the Microsoft System Information tool (msinfo32.exe).
Instead, you must either employ more manual methods, or use a third party application such as Crap Cleaner or Index.dat Suite (there are some methods available that require payment before they will delete these files for you, and in my opinion, these should be avoided like the proverbial plague).
Although I cannot comment on alternatives such as Crap Cleaner (I do not use this one), Index.dat Suite provides the necessary functions to allow both the safe deletion, and viewing, of ALL index.dat files on your computer (including those in other user profiles) with the minimum of effort.
To answer the original question - what are the index.dat files?, the answer is actually quite simple. The index.dat files are the cause of an idea, an idea that quickly turned to the idea becoming an institution in and of itself, ergo, releasing the tin-foilers on an endless quest to spread a paranoia that is actually quite well-founded. The index.dat files are no more critical to or needed by, your computer, than cream is to an Irish coffee. Contrary to claims, they do not increase the loading times of Windows, installed applications, websites, e-mail or FTP et al, they are NOT cleaned when you use the built in "Internet Options cleaning methods" and cannot be cleaned, modified or deleted when explorer.exe (Windows - as it is most commonly known to non-techies) is running. This in and of itself should send a big red alert (preferably in the form of a dancing monkey) that something is not quite right.
Unfortunately however, attempts to complain about, or confirm this, will normally go completely ignored, both by Microsoft, and the fanatics (infact, the only time you will usually see someone taking notice of something like this is when they are looking for a reason to use "$" instead of "s").
IE options + index.dat files... really?