Forensics: Automating Active Directory Account Lockout Search with PowerShell (an example of deep XML filtering of event logs across multiple servers in parallel)

Article
08/31/2015

Overview

Today we learn how to efficiently filter event log queries, going beyond simple event ID filtering into the specific values of the XML message data. Then we will run this filter against multiple servers in parallel for faster data collection.

This posts meets the following objectives:

Add some efficiencies to my previous popular post for parsing XML event message data.
Apply the concept to Active Directory account lockout troubleshooting, formerly posted in the sample code of this post.
Provide a working example for your future event log forensic scripts, regardless of which logs or events or data you are mining. Our example today is Active Directory lockouts and bad password attempts.

Someone else may have done this already, but I have not searched for other examples. This is my own approach to the solution.

“Back in my day…”

During the Windows Server 2003 era the Account Lockout Tools became very popular for tracking down those notorious account lockout scenarios. You would use LockoutStatus.exe to find the DCs targeted with failed password attempts, then you would use EventCombMT.exe to harvest the security log lockout events on those DCs. Eventually you would find the culprit causing the lockout, and sometimes it was your own fault for leaving an RDP session logged in for days. Surprise! Smile

This process was half-automated. You still had to jump through multiple tools with manual effort to get the results. But that was before PowerShell.

Solution Overview

Today’s script mimics these steps entirely with PowerShell:

Get a list of locked out accounts using the AD cmdlet Search-ADAccount –LockedOut
Query the lockout count for each account across all DCs to see where the lockouts are occurring.
Retrieve the related event log entries from the DCs where the lockouts occurred (in parallel).
Review the data in Out-GridView and CSV.

The previous solution using multiple tools would prompt for an account name and ask you to pick DCs to query. In this case I have automated these steps into one big button to get it all. It takes the thinking out of the problem solving and makes sure you don’t miss anything.

Locked Out Accounts

Finding the currently locked out accounts is now really easy with the Active Directory cmdlets:

 PS C:\> Search-ADAccount -LockedOut

AccountExpirationDate : 
DistinguishedName     : CN=Jim,CN=Users,DC=CohoVineyard,DC=com
Enabled               : True
LastLogonDate         : 
LockedOut             : True
Name                  : Jim
ObjectClass           : user
ObjectGUID            : d13287cb-5725-4e21-ba75-2acfe383fc46
PasswordExpired       : True
PasswordNeverExpires  : False
SamAccountName        : Jim
SID                   : S-1-5-21-2999376440-943117962-1153441346-7287
UserPrincipalName     :

We have an array of rich PowerShell data objects coming back for each locked out account.

Lock Out Counts

When bad password attempts occur, you will see the lockout count incremented on the local DC processing the logon attempt and also the PDC Emulator (PDCe). This ensures that in case the logon attempts are tried across different DCs, a grand total bad password count is enforced on the central PDCe. For this reason I added a column to the output to note which DC is the PDCe.

While I was adding value to the DCs in the lockout count query, I also added columns for IPv4Address and Site. This data is helpful to identify cases where clients may be authenticating outside of their intended site. You can also loosely compare the DC IP address with the client IP address that comes up on the later report of bad password attempts.

In the “Lockout Status By Account By DC” report notice that you may see DCs repeated in the list. If you have multiple locked out accounts, then you will get a row for every DC where their logons were attempted. When multiple accounts are locked out you will always see at least the PDC listed for each account.

There are multiple DateTime columns in this output for determining when the account was created, last changed, last logon, last bad password, etc. DateTime values are not human-friendly sometimes in AD, so I converted them using this trick from a former post:

@{name='badPasswordTimeConverted';expression={[datetime]::fromFileTime($_.badPasswordTime)}}

The fromFileTime() method is your best friend here.

Event Log Collection

This solution builds on the XML event log script from the past post, but this is more efficient by doing deep XML filtering on the message body. This took some trial-and-error, but I finally crafted the correct XML filter syntax. Please study the code and use this technique across other event log scenarios you may encounter in the general world of Windows Server.

In the event viewer graphical interface we see the lockout event details.

Click the Details tab, and them XML View to reveal the data within the event.

Here you find the username that was locked out, and the computer name where the lockout originated. However, the names of the fields are not very intuitive (TargetUserName, TargetDomainName).

In my Microsoft Virtual Academy series on Active Directory PowerShell, module four covers forensics. In that segment I demo tracking down account lockouts based in part on some code from an earlier post about parsing XML event log message data. In that earlier post I offered a couple caveats: it is terribly inefficient and I couldn’t get it to work remotely using the Get-WinEvent cmdlets directly. There were also some helpful comments worth noting in that previous XML post.

Problem #1: Remote Event Retrieval

PowerShell remoting serializes output before it is returned from remote sessions, and this removes methods like .ToXML() that we need here. I addressed the remote event retrieval issue in two ways:

I incorporate the Get-WinEvent inside Invoke-Command. Inside a remote session we get the XML event data, append the properties to the event object, and then return it. We can get the details we need by executing the XML event data collect prior to bringing the data back across the wire. This code is passed as a scriptblock to Invoke-Command.
Another snag with Get-WinEvent –ComputerName is that the Remote Event Log Management firewall rule must be enabled on the server first. This may not always be practical, so using Invoke-Command uses the standard, already-open remoting port to retrieve the event data without having to open another firewall port.

 # Only get event logs from the DCs that show a lockout count            
$DCs = $report |            
    Where-Object {$_.badPwdCount -gt 0} |            
    Select-Object -ExpandProperty DC -Unique            
            
$Milliseconds = $Hours * 3600000            
# Script block for remote event log filter and XML event data extraction            
# Logon audit failure events            
#   Event 4625 is bad password in client log            
#   Event 4771 is bad password in DC log            
#   Event 4740 is lockout in DC log            
$sb = {            
[xml]$FilterXML = @"
<QueryList><Query Id="0" Path="Security"><Select Path="Security">
*[System[(EventID=4740 or EventID=4771)
and TimeCreated[timediff(@SystemTime) &lt;= $Using:Milliseconds]]]
$Using:UserFilterXML
</Select></Query></QueryList>
"@

     Try {            
        $Events = Get-WinEvent -FilterXml $FilterXML -ErrorAction Stop            
            
        ForEach ($Event in $Events) {            
            # Convert the event to XML            
            $eventXML = [xml]$Event.ToXml()            
            # Iterate through each one of the XML message properties            
            For ($i=0; $i -lt $eventXML.Event.EventData.Data.Count; $i++) {            
                # Append these as object properties            
                Add-Member -InputObject $Event -MemberType NoteProperty -Force `
                    -Name  $eventXML.Event.EventData.Data[$i].name `
                    -Value $eventXML.Event.EventData.Data[$i].'#text'            
            }            
        }            
            
        $Events | Select-Object *            
    }            
    Catch {            
        If ($_.Exception -like "*No events were found that match criteria*") {            
            Write-Warning "[$(hostname)] No events found"            
        } Else {            
            $_            
        }            
    }            
                
}            
            
# Clear out the local job queue            
Get-Job | Remove-Job            
            
# Load up the local job queue with event log queries to each DC            
Write-Verbose "Querying lockout events on DCs [$DCs]."            
Invoke-Command -ScriptBlock $sb -ComputerName $DCs -AsJob | Out-Null

Note the use of the –AsJob switch with Invoke-Command. Obviously querying event log data from multiple DCs is going to take some time. The ThrottleLimit parameter defaults to 32, so that should be plenty of parallel threads for most people. While there are multiple methods for parallel execution in PowerShell, this one is the most convenient for our purposes. If you have not read about Jobs in PowerShell, they have been around since version 2.0. Check out the help topic about_Jobs.

Problem #2: Deeper XML Filtering Inside the Message Data

In my last attempt at this solution I used this inefficient method:

Retrieved all the lockout events
Looped through all of them again to extract the XML data
Then passed them to Where-Object for filtering for the username

Since that time, my Connect item calling out the lack of MSDN links for XPath in the help for Get-WinEvent has been closed. The help has been updated with relevant links that are helpful in understanding the nuances of XPath and FilterXML.

Query Schema
XPath Reference
XML Event Queries in Event Selection <— Most helpful article from MSDN

However, most helpful to me was a series of forum posts over at StackOverflow by some people who understand XML way better than me. For more background I suggest you go read this handy post by the Scripting Guy on how to use FilterXML by copying from the Windows Event Viewer: Use Custom Views from Windows Event Viewer in PowerShell. I love this trick, and I use it all the time, including this script.

Here is the updated FilterXML syntax for deeper message data filtering with the new part highlighted:

<QueryList><Query Id="0" Path="Security"><Select Path="Security">
*[System[(EventID=4740 or EventID=4771) and TimeCreated[timediff(@SystemTime) <= 43200000)]]]
and *[EventData[(Data[@Name= 'TargetUserName' ] = 'alice') or (Data[@Name='TargetUserName'] = 'bob') or (Data[@Name='TargetUserName'] = 'charlie')]]
</Select></Query></QueryList>

I am not going to explain the XML syntax here, but the point is that we are adding another criteria to the filter reaching inside EventData/Data to the property TargetUserName. We are actually repeating the property filter for each username we need to capture for lockouts in the logs. Note that this works well when matching exact values or doing simple greater than or less than comparisons.

After this tweak, our filtering is much more efficient:

Query the log and only return the smaller set of entries in scope.
Then process the XML properties of each one.

This significantly reduces the operations required to achieve the same end.

While tweaking the FilterXML I added a parameter to my function to specify the number of hours of event log history to search. The highlighted number 43,200,000 above is 12 hours (12 hours * 60 minutes * 60 seconds * 1,000 milliseconds).

View the Results

When the script is finished you will get three reports in Out-GridView. These are also exported as CSV files.

 Directory: C:\Users\administrator.COHOVINEYARD\Documents

Mode         LastWriteTime Length Name                         
----         ------------- ------ ----                         
-a---  8/31/2015   1:02 PM   1121 LockoutEvents_AccountData.csv
-a---  8/31/2015   1:02 PM  12346 LockoutEvents_BadPassword.csv
-a---  8/31/2015   1:02 PM   2105 LockoutEvents_Lockouts.csv

These reports should contain all the forensic data you need to track down those lockouts in your environment:

LockoutEvents_AccountData.csv – the lockout counts by account by DC
LockoutEvents_BadPassword.csv – event ID 4771 details, one event for each bad password attempt, IP and attempted reverse lookup of hostname, authentication failure audit event
LockoutEvents_Lockouts.csv – event ID 4740 details, lock out success audit event, hostname of lockout machine

Enjoy!

One Big Button

Now you can retrieve your lockout data with one function call like this:

 # Default is events in last 24 hours, no IP to name resolution            
Get-ADAccountLockoutData            
            
# Last 12 hours, IP to name resolution for bad password events            
# Verbose for progress information            
Get-ADAccountLockoutData -ResolveIP -Hours 12 -Verbose

Assumptions and Prerequisites

Essentially you need to run this from a Windows Server 2012 R2 member server or domain controller. A Windows 8.1 workstation with RSAT installed should also suffice. Here is the full disclaimer on compatibility.

This code was labbed on Windows Server 2012 R2 using PowerShell version 4.0.
You will need the RSAT for the Active Directory PowerShell module.
Currently this makes no provision for specifying an alternate domain name or credentials. You should run this in the domain where the lockouts are occurring using an account with Domain Admin credentials. Feel free to tweak the code for your own purposes to reach across domains.
This code will return no results if no accounts are currently locked out.
The DNSClient module on Windows Server 2012 R2 includes the Resolve-DNSName cmdlets.
Reverse DNS zones are needed for IP to name lookups.

I have not tried running this against Windows Server 2012, but it should work. Windows Server 2008 R2 should work as long as WMF 3.0 or WMF 4.0 is installed. You must run it from 2012 R2, but it should be able to target these other domain controller operating systems for event log collection.

Make It Your Own

Like all of my published scripts this is intended as sample code for your to use as-is or refactor for other specific needs in your environment. This is a valuable example that reaches far beyond Active Directory applications. I hope that you have learned a little bit more about deep XML event filtering with Get-WinEvent and remote log collection efficiently. Use the comments area below to post questions or share how you have used the code.

Download the Code

You can get today’s script solution over at the TechNet Script Gallery. Please download and study to learn the techniques I discussed here today.

Comments

Anonymous
September 01, 2015
Awesome !!!
Anonymous
September 01, 2015
Do you prefer sorting through the xml ( $eventXML = [xml]$Event.ToXml() ) rather than using something like this - Select-Object -Property @{n="User SID";e={$_.properties[4].value}}
Anonymous
September 01, 2015
Great post Ashley!
Anonymous
September 02, 2015
Great question, Tim. Your technique is effective if you only need to target one or two properties of a specific event ID. I prefer the precision of the property names. I also prefer extracting all of the properties by name for reporting, regardless of the event schema.
Anonymous
September 02, 2015
Useful post and some good links to MSDN articles for referencing xPath, but Ashley running as domain admin, no please we need to stop doing this in the world of pTH, pTT and lateral movement.

How about a follow-up post covering Windows Event Log forwarding to a central member server using xml filtering in the subscriber (you might also highlight the limitations of xPath as implemented). Then you can run the script from a workstation and delegate privileges on the member server, plus you can keep logs for longer and in one location rather than having to query each DC.
- Anonymous
  November 08, 2016
  I have the forwarded events logs ,from all the domain controllers, but struggling to find a way to filter it and extract only the user lockout scenarios, since the 4771 contains computer account password failures as well which is flooding it....any help is appreciated.
Anonymous
September 03, 2015
Hi Paul,
I agree with your comments. The only trouble is that I work with customers who have not been proactive as you suggest. When we are trying to figure out what happened after an event, we have to use what the customer gives us.
I wish all of my customers would adopt the advice you recommend here.
Thanks,
Ashley
GoateePFE
Anonymous
September 07, 2015
208 Microsoft Team blogs searched, 57 blogs have new articles. 263 new articles found searching from
Anonymous
September 30, 2015
I'm trying to export EventData to csv. This script works well, but it's not pulling all the data for all types of events. Like, if the first event it scans doesn't have an "ObjectName", then it won't put that in my csv for any events. Any way to do that?
Anonymous
October 09, 2015
What is the lowest version of PS that can be used with this script?
BTW: Loved your Webcast on What's New in PS v5 on 10/9/15!!!
Anonymous
October 16, 2015
Hi Jasen,
I reviewed the code, and this may work on PSv2. For sure it should work on v3 and above. You will need the RSAT for Active Directory installed to get the AD cmdlets.
Thanks,
Ashley
GoateePFE
Anonymous
October 16, 2015
Hi Gene,
That makes sense as a general PowerShell convention. I have seen that in other cases. Feel free to adjust the code to send the results to XML or any other format that will capture everything. You could also try modifying the code to explicitly name the columns for output.
Thanks,
Ashley
GoateePFE
Anonymous
October 14, 2016
I have setup event forwarding for 4740,4771 events from all domain controllers to a collector THe logs are under "forwarded events" ,. trying to work out a way to use your script against this.
Anonymous
November 11, 2016
i somewhat managed to create a script which can parse the forwarded event log for the past 30 minutes and find all the locked users in the past 30 minutes. then the xml filter is changed to parse the forwarded event log again for bad passwords and retrieve the sources.. this post really helped me understand ways to query and enhance certains logics for working with event logs...
Anonymous
November 13, 2016
I managed to learn and use the methods described by Ashley and create a webpage for the L1 team, to find the sources of the lockout. Sharing it here ,please modify it accordingly. I know my scripts lack a bit structure, as im still in the process of learning many things... But im sure experts can use and modify as needed.Requirements:IIS to be installed if you want the webpage to be accessible to users (by default an HTM file is created for users to see)powershell remoting and event forwarding already setup for 4740 and 4771 events in your domain to a centralized serverTask scheduler can be utilized and you dont need any domain admin rights to run this,, just run as systemDoug Finke has created an out-datatableview module to display the output in a searchable web page, this has been utilized. Link is inside the script. Do let me know if you had any questions at nannnu@gmail.com#script Import-Module activedirectory# Count.txt file is filled with numbers from 1 to 100000, to run the script that many times ( you can change this to while and variable...i was just too tired to #change it now $counts = get-content C:\count.txtforeach($count in $counts){$Eventdataforhtml = null$FindReplace=@{}$FindReplace['<DomainControllerNAMEHERE'] = "(Domain Controller)"Function ChangeDesc($Desc){ foreach($key in $FindReplace.Keys) { if($Desc.ToLower().Contains($key.ToLower())) { $Desc+=" $($FindReplace[$key])" } } Return $Desc}$date = get-date#XML filter for getting the locked out user list from the last 30 minutes from the 'forwardedEvents' Log[xml]$FilterXML = @" *[System[(EventID=4740) and TimeCreated[timediff(@SystemTime) <= 1800000]]] "@#Select samaccountname of the user from the 4740 events in the past 30 minutes$lockedoutusers = Get-WinEvent -FilterXml $filterxml | Select-Object @{Name="User";Expression={($.Properties[0].Value) } } | Sort-Object user -unique | select -ExpandProperty user write-host " $lockedoutusers "$lockedoutusers | Measure-Object | select count#find the source of lockout for each user from the 4740 and 4771 eventsforeach($user in $LockedOutUsers){write-host "$count"#XML filter to get the source of lockout for the user from the events[xml]$FilterXML = @"*[System[(EventID=4771 or EventID=4740)]] and *[EventData[(Data[@Name='TargetUserName'] = '$user') ]] "@$eventdata = $null$eventdata = Get-WinEvent -FilterXml $filterxml | Select-Object @{Name="User";Expression={($.Properties[0].Value) } }, @{Name="DNSResolvedName";Expression={ changedesc(Resolve-DnsName (($_.Properties[6].Value) -replace '::ffff:') | select -ExpandProperty namehost) } },@{Name="NonResolvedIP";Expression={($_.Properties[6].Value) -replace '::ffff:'} },TimeCreated,@{Name="Source";Expression={($_.Properties[1].Value) } }#Import-Module D:\Nara\BadPasswordData\PSModules\OutDataTableView\OutDataTableView.psm1 $eventdata = $eventdata | Sort-Object Timecreated,User,DNSresolvedname,NonresolvedIP -Unique #"Users locked during $date are $LockedOutUsers" | out-file D:\nara\BadPasswordData\BadPasswordData-.csv -append $eventdata | export-csv -Path D:\Nara\badpassworddata\BadPasswordData-.csv -append -NoTypeInformation $Eventdataforhtml = $null $eventdataforhtml += $eventdata } $Eventdataforhtml | Out-DataTableView -Properties User,dnsresolvedname,nonresolvedip,timecreated,source -PageHeader "Account Lockout sources - Domain Only Please Enter the username in the Search Field below. Please contact team for any questions on this This list is updated every 20-30 minutes minutes and was last generated at <b>$("{0:h:mm t}M " -f $date)</b> on $("{0:M/d/yy}" -f $date). " -OutFile D:\nara\BadPasswordData\default.htm -Deploy }
- Anonymous
  May 12, 2017
  The output kicks out two out grid views of Lockout Status By Account by DC only. I really wanted to see the Bad Password Events file. Is there something I'm not doing?
  - Anonymous
    June 29, 2017
    If you look at this portion of the script you can see that it will only output those files if there are events that have occured within the specified time frame. # Display results by event ID, because each event ID has a unique set of columns # Optimize column selection for display. Feel free to tweak. $ReportOut = $Output | Where-Object {$.Id -eq 4771} | Select-Object TimeCreated,@{name='DC';expression={$.PSComputerName}},TargetUserName,IpAddress,IpPort,IPtoHostname,ProviderName,LogName,Id,RecordId,LevelDisplayName,PreAuthType,KeywordDisplayNames,ServiceName,Status,TargetSid,TaskDisplayName,TicketOptions,UserId | Sort-Object TimeCreated $ReportOut | Out-GridView -Title 'Bad Password Events' $ReportOut | Export-Csv -Path .\LockoutEvents_BadPassword.csv -NoTypeInformation -Force $ReportOut = $Output | Where-Object {$.Id -eq 4740} | Select-Object TimeCreated,@{name='DC';expression={$.PSComputerName}},TargetUserName,TargetDomainName,ContainerLog,Id,TaskDisplayName,KeywordsDisplayNames,LevelDisplayName,LogName,OpcodeDisplayName,ProviderName,RecordId,SubjectDomainName,SubjectLogonId,SubjectUserName,SubjectUserSid,TargetSid,UserId | Sort-Object TimeCreated $ReportOut | Out-GridView -Title 'Lockout Events' $ReportOut | Export-Csv -Path .\LockoutEvents_Lockouts.csv -NoTypeInformation -Force } Else { Write-Warning "No events returned for the specified hours."
Anonymous
May 25, 2017
An impressive share! I have just forwarded this onto a colleague who was doing a little homework on this. And he in fact ordered me breakfast due to the fact that I discovered it for him... lol.So allow me to reword this.... Thanks forr thee meal!! But yeah, thanks for spending the time to talk about this issue here on your blog.
Anonymous
December 22, 2017
Thanks for the lovely piece of code!Added this bit at the end to add the capability to unlocking the accounts as well[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")[void] [System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")[System.Windows.Forms.Application]::EnableVisualStyles();$Form = New-Object System.Windows.Forms.Form$Form.Text = "Account Unlock console"$Form.Size = New-Object System.Drawing.Size(300,200)$Form.StartPosition = "CenterScreen"$LockedUsers = (Search-ADAccount -LockedOut).samAccountName$DropDownBox_LockedUsers= New-Object System.Windows.Forms.ComboBox $DropDownBox_LockedUsers.Location = New-Object System.Drawing.Size(70,40) $DropDownBox_LockedUsers.Size = New-Object System.Drawing.Size(120,20) $DropDownBox_LockedUsers.DropDownHeight = 200foreach ($LockedUser in $LockedUsers) { $DropDownBox_LockedUsers.Items.Add($LockedUser) }$button_unlock = New-Object “System.Windows.Forms.Button”;$button_unlock.Location = New-Object System.Drawing.Size(70,100)$button_unlock.Size = New-Object System.Drawing.Size(120,20)$button_unlock.Text = “Unlock”;$button_unlock.Add_Click({Unlock-ADAccount -Identity $DropDownBox_LockedUsers.Text})$form.Controls.add($DropDownBox_LockedUsers)$form.Controls.add($button_unlock)$form.ShowDialog();
Anonymous
February 06, 2019
Hmm, and-ing two paths together, like "* and "? I don't think this is valid xpath, and yet it works for log filtering.[System[(EventID=4740 or EventID=4771) and TimeCreated[timediff(@SystemTime) <= 43200000)]]] and *[EventData[(Data[@Name='TargetUserName'] = 'alice') or (Data[@Name='TargetUserName'] = 'bob') or (Data[@Name='TargetUserName'] = 'charlie')]]

Share via