How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 1
0 Comments Published February 11th, 2012 in Programming, MVP Program, Windows 7You might as I have asked yourself many times – What is inside NTDS.dit? (Most experienced Active Directory admins knows that NTDS.dit is the database and the physical on disk store that Active Directory uses to store information – most of you have probably got in touch with NTDS.dit during backup and restore scenarios)
Long story in a short version – I wasn’t satisfy not knowing – neither was I after being reading the following article:
(That I actually think isn’t that bad – but is also probably the most detailed public available information on the subject)
[1] http://technet.microsoft.com/en-us/library/cc772829(WS.10).aspx
So I decided with a very good friend of mine Christoffer Andersson (long-time Microsoft MVP in AD) to go ahead and build a tool that could read NTDS.dit and decode its internals, and then we started a journey that has given us invaluable knowledge at this part of Active Directory, this is the first article in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer.
The illustration below has been presented in various documentations since Active Directory was initially released over 10 years ago; a similar illustration is also available in (However after this research project it’s actually turning out to be inaccurate in some aspects – in the way the DRA/REPL communicates with the DBLayer) [1]
Table 1: DSA Components (Simplified for the DBLayer)
|
Component |
Description |
|
Ntdsa.dll – Directory System Agent |
The DSA, which runs as Ntdsa.dll on each domain controller, provides the interfaces through which directory clients and other directory servers gain access to the directory database (the DBLayer). In addition, the DSA enforces directory semantics, maintains the schema, guarantees object identity, and enforces data types on attributes. |
|
Esent.dll – Extensible Storage Engine (ESE) APIs |
The Extensible Storage Engine (ESE) is an advanced indexed and sequential access method (ISAM) storage technology. ESE enables applications to store and retrieve data from tables using indexed or sequential cursor navigation. It supports denormalized schemas including wide tables with numerous sparse columns, multi-valued columns, and sparse and rich indexes. It enables applications to enjoy a consistent data state using transacted data update and retrieval. ESE was formerly known as Joint Engine Technology (JET) Blue, The DBLayer uses the ESE APIs documented here: http://msdn.microsoft.com/en-us/library/windows/desktop/gg269259(v=exchg.10).aspx |
|
NTDS.dit |
The on physical-disk file that represent the ESE/JetBlue database that holds the information store for the given DSA/Active Directory Domain Controller. |
Data Store Physical Structure / Inside NTDS.dit - Tables
Finally we can start looking into the content/internal structure of NTDS.dit – but first let’s take a look on what has been reveled before, the illustration below is from [1] and is accurate as far as outside the white box that represent the tables within the database, the tables do exist (Except for * “sd_table” on Windows 2000 DSAs) – but there is more tables that isn’t mentioned in this example.
So it’s about time to reveal the real table structure of an NTDS.dit database file – It’s time to use the tool we produced to first discover this:
Table 2: NTDS.DIT - Tables
|
Table |
Description |
Minimum DSA Version |
|
Datatable |
Contains all objects and phantoms [2.1] represented as rows (1 object/phantom = 1 row in the table) from any instanced naming context (NC) held as either writable or read-only by the Directory System Agent (DSA) hosting the database and where columns represent every [2:3] attribute present in the schema except linked attributes [2:2] [2.1]: phantoms are references to object’s hosted outside the given database (NTDS.DIT) and the given Directory System Agent (DSA) [2:2] Post-Windows Server 2003 the attribute “ntSecurityDescriptor” is stored in the “sd_table” rather than in the “datatable” [2:3] Some columns doesn’t reflect attributes and are columns pre-defined in the NTDS.dit template database generated by Microsoft (those are needed for internal states to the DSA) |
Windows 2000 Server Note: Windows Server 2008 R2 added a column to support the “is-Recycled” state |
|
Hiddentable |
Contains one row but several columns that defines the state of the database as well the [2:2] DNT (reference) of the NTDSA-Settings object that represents this DSA (used for finding config information specific to this domain controller.) [2:4] The concept of DNTs (Distinguished Name Tags) |
Windows 2000 Server |
|
Link_table |
Contains link-pair references (DNT, DNT), the link base (link id >> 1) and possibly a binary blob (In case of DN-binary, DN-string syntax) |
Windows 2000 Server Note: Windows Server 2008 R2 added a column to support deactivated links for recycle-bin |
|
Sd_table |
Contains single-instance-stored SDs (Security Descriptors) that pre-Windows Server 2003 was stored in the ntSecurityDescriptor attribute in the “datatable” – those are now instead referenced to the SDs in the “sd_table” that is, if more than one object has exactly the same security defined (Security Descriptor) both objects are referenced to the same row in the “sd_table”, hence the single-instance-storage and reducing the size needed to store Security Descriptors. |
Windows Server 2003. |
|
Sdpropcounttable |
Used by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database |
|
|
Sdproptable |
Used by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database |
Windows 2000 Server |
|
Quota_rebuild_progress_table |
Contains temporary information during quota tracking rebuild, for the Active Directory quota feature introduced in Windows Server 2003 – this allows the demon to keep track of processed objects. |
Windows Server 2003 |
|
Quota_table |
Contains quota tracking information, for the Active Directory quota feature introduced in Windows Server 2003, quota tracking is peer naming context (NC) and for a given security principal identified by its SID. |
Windows Server 2003 |
|
MSysObjects |
ESE Internals – out of scope for this article |
N/A |
|
MSysObjectsShadow |
ESE Internals – out of scope for this article |
N/A |
|
MSysUnicodeFixupVer2 |
ESE Internals – out of scope for this article |
N/A |
In the next article – we will take a deep-dive into the content and the structure of the “datatable” also known as the object-store.
Blocking unwanted advertisements and malware with a HOSTS file
0 Comments Published January 18th, 2011 in Uncategorized, MVP ProgramA fellow Microsoft MVP has assembled a large list (> 6MB) of domain names/IP addresses mainly used to serve advertisements, tracking cookies, and sometimes even malware.
You can find the full instructions at http://www.mvps.org/winhelp2002/hosts.htm. I personally have been using this list for almost two years now (it is frequently updated).
Because I have IIS set-up on my main machine I decided to change the redirect IP address to 0.0.0.0 instead of 127.0.0.1, so that IIS is not “bothered” with the failed requests.
P.S. This list also blocks the YouTube commercials, which is a nice touch!
Multithreading with Windows Forms in C#
0 Comments Published December 29th, 2010 in Microsoft, ProgrammingWhat are threads?
A thread executes code synchronously—a set of instructions processed by the CPU “first in first out”. This can include virtually any kind of code, be it updating of the GUI, processing calculations, or waiting for user input.
Windows Forms is arguably one of the most used aspects of .NET Framework. GUI programs typically rely on two layers:
- Processing stack, which includes any code or task executed in the background, and
- User interface, which is supposed to give the user a means for instructing the program to execute tasks, and receiving timely “updates” or indications of progress.
Multithreading
Multithreading is a programming paradigm, which has become popular during the past decade. In a nutshell, the CPU can switch between two or more thread contexts so two tasks (e.g. calculation, and GUI update) can be executed in parallel—or asynchronously.
For example, a program can make full use of a multi-core processor by instructing each core to process a separate task, or a chunk of the whole task. This results in improved performance.
In the case with Windows Forms, a separate thread can be used to process a task while allowing the main thread (GUI) to provide timely information about the progress, or to let the user continue their work during the lengthy task.
Честита Коледа (и поздравления на спечелилите)!
0 Comments Published December 26th, 2010 in MVP ProgramЧестита Коледа, и приятно посрещане на Новата година на всички читатели на блога!
Също така поздравления на Miguel Hernandez (@migue333) и Kim Nilsson (@no_substitute), които бяха избрани посредством случаен жребий от Bryant Zadegan и мен, за да получат по един подарък-карта за абонамент MSDN Ultimate.
Честито!Благодарности на Microsoft за подкрепата.
A simple backup solution using ImageX, a Windows Imaging tool
1 Comment Published September 18th, 2010 in Microsoft, ProgrammingPrevious readers of my blog might already know that I am a big fan of ImageX, a tool that has been supplied by Microsoft in the Windows Automated Installation Kit since the release of Windows Vista.
A quick introduction
ImageX is a command-line tool that enables original equipment manufacturers (OEMs) and corporations to capture, to modify, and to apply file-based disk images (.wim files) for rapid deployment.
WIM files can capture disk or directory snapshots in such a that identical files are only stored once in the internal structure of the image. In addition to this, compression can be used to minimize the size of the final data.
These are some of the reasons I have been using ImageX to take full “snapshots” of all of my code, private files, etc. from time to time, for backup and archiving purposes. When needed, I can extract different revisions of the same file, or restore a whole image/snapshot in case of disk damage or data loss.
The graph on the left visualizes the changes to my projects backup file spread across 16 months. The last image’s actual data weighs at 64GB, while the whole WIM is 59GB—and it contains all file changes, directory structures, even information that has long been deleted from my working copy.
Requirements
- The scripts package. Download and extract to a folder of your choice.
- ImageX.
- Download and install the Windows Automated Installation Kit (WAIK).
- Copy the ImageX binaries (both amd64 and x86 folders), located in \Program files\Windows AIK\Tools\ to the backup\app\imagex folder.
Continue reading ‘A simple backup solution using ImageX, a Windows Imaging tool’
Just wanted to post a quick update:
Thanks to the MVP Program I have received two MSDN Ultimate Subscription gift cards, worth $11,899 each.
These are going to be given away soon as I am working with Bryant on the giveaway details. Stay tuned.
FlacReader + WavWriter
As a follow-up to my previous article on FLAC and encoding uncompressed audio, I have further developed the Wav2Flac library and added FLAC decoding to the WAVE container. Some key points of this aspect of the WavFlacTest library are:
- Support for 16- and 24-bit audio streams
- Support for virtually all FLAC channel mappings: mono, stereo, 5.1, 7.1, etc.
- The resulting file is a bit-for-bit copy of the original audio stream
All source code and the compiled 32-bit FLAC library can be downloaded from here.
Known limitations
- Due to the wave format used being 32-bit, uncompressed files greater than 2GB in size might not play in full in some players.
- Even though FlacReader can receive metadata callbacks, the data is not interpreted. This can be further developed using the FLAC API documentation.
One thing that popped in my mind yesterday after having a conversation over twitter (admittedly, a service I had rarely used until recently), is that the more followers you have, the higher the odds you would have to use other languages in addition to a lingua franca. This ‘leaves out’ other friends who could have limited knowledge of foreign languages.
This is how the idea for the Tweet Balloon Translator was born. It’s a Greasemonkey script for Firefox, which uses the balloon/hover feature recently introduced in twitter plus the Google Translate service to translate tweets quickly and inline.
You can install the script from userscripts.org or the local copy here.
Development notes
The key phases I outlined for the script were as follows:
- Establish a working translation block of code using the Google Translate JSON API
- Research the possibilities of inserting a Translate link in such aspects of twitter as the profile page, direct messages, retweets. The Guest/logged out user scenario also had to be taken into account as twitter serves the pages in a different manner.
- Figure out a way to use the twitter internal API for the HoverCard balloon feature. This was probably the hardest of all three because of the way these are created and populated.
- Integrate 1 through 3 in a single script.
Needless to say, the script employs asynchronous requests to Google via the HTTPXMLRequest model so as to not break the UI thread, as well as exception handling.
Please feel free to try out, review, or contribute to the script at userscripts.org. I feel both international and English-speaking users could benefit from this by being able to communicate with more people on twitter, and understand international tweets.
Resizing a form while keeping aspect ratio is useful in many cases, like video playback or vector graphics. This way, the window can be resized while retaining the original ratio and avoiding the use of letterboxing or pillarboxing.
What’s needed is for the window function to be overriden (WndProc) and pre-process the target window rectangle used by the WM_SIZING message.
The new destination rectangle is calculated by taking into account the resizing handle and the window chrome size (title height, border width, etc.).
protected override void WndProc(ref Message m)
{
if (m.Msg == WM_SIZING)
{
RECT rc = (RECT)Marshal.PtrToStructure(m.LParam, typeof(RECT));
int w = rc.Right - rc.Left - chromeWidth;
int h = rc.Bottom - rc.Top - chromeHeight;
switch (m.WParam.ToInt32()) // Resize handle
{
// …
}
Marshal.StructureToPtr(rc, m.LParam, true);
}
base.WndProc(ref m);
}
You can find the full C# source code here, including a test program. The aspect ratio and initial client size is set to 16:9.
Why lossless?
Lossless audio is used on various media, including studio masters, CD, DVD-Audio (via MLP) and Blu-ray (via Dolby TrueHD, which is technically a rebrand of and an extension to MLP, and DTS-HD Master Audio). All of these, when decoded, will result in a pulse-code modulated signal identical to the source, unlike the popular MP3 format. MP3 performs a quality-file size trade-off by discarding or reducing frequencies less audible to human hearing.
PCM by design uses a constant bitrate, which is proportional to the sample rate, bit depth and number of audio channels, which results in very large file sizes with the increasing of each parameter, and/or duration of the audio track.
Solutions such as FLAC, TrueHD and DTS-HD MA are used to losslessly compress the source audio so that the rest of the medium (for example, a Blu-ray disc) can be used for more audio tracks, higher-bandwidth video or extras.
Out of the aforementioned, only FLAC is free to use—both TrueHD and DTS-HD MA encoders and decoders have to be licensed.
The first part of the series will explore the processing of uncompressed audio data with the FLAC API in C#. In case you prefer to use Visual Basic .NET, you can use this online converter.
Continue reading ‘Encoding uncompressed audio with FLAC in C#’
Search
About |

Stanimir Stoyanov is a programmer, Microsoft MVP, and Windows enthusiast. Read More...
He's currently working on an array of projects using Visual Studio 2010 on Windows 7.
Latest
- How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 1
- Blocking unwanted advertisements and malware with a HOSTS file
- Multithreading with Windows Forms in C#
- Честита Коледа (и поздравления на спечелилите)!
- A simple backup solution using ImageX, a Windows Imaging tool
- Soon: MSDN Ultimate Subscriptions Giveaway
- Decoding FLAC audio files in C#
- Inline Tweet Translator
- Resizing forms while keeping aspect ratio
- Encoding uncompressed audio with FLAC in C#
Also visit
Friends
- Andre Da Costa is Teching It Easy
- Christoffer Andrersson Executive Consultant at TrueSec
- Olcay Buyan Software developer and Media Center enthusiast
- Rafael Rivera Within Windows
- Steven Troughton-Smith iPhone Developer (Doom, Lights Off) and hacker extraordinaire
- Zack Whittaker iGeneration at ZDNet.com


