Thursday, February 15, 2018

Powershell: Indeterminate Inline Progress Bar

I am currently working on removing a replicas from all of our on-premise public folders. There is quite possibly 40,000 public folders nested inside of the structure. We are looking to remove 2 copies from all replicas and the Exchange 2010 tools, while working didn't provide any progress.

So, I grabbed all the 'good' replicas.

$firstPF = get-publicfolder ".\Top" 

$GoodReplicas = $firstPF.Replicas | ?{$_ -notlike "remove this db name"} 

Get-PublicFolder -recurse | ?{$_.replicas -ne $GoodReplicas} | Set-PublicFolder -replicas $GoodReplicas

This was taking for ever and I couldn't tell if it was even running. I modified one of my previous progress bars, so that it would simply count to 100, then reset back to 1. Next I modified the code, to return the object I am currently parsing. This allows me to put the progress bar in-line with my code above and it keeps running.

Get-PublicFolder -recurse | ?{$_.replicas -ne $GoodReplicas} | %{wp3 -passthru $_} | Set-PublicFolder -replicas $GoodReplicas

You can see I use a few global variables. This helps maintain the counter between iterations. This also means that it will start at where you left off with the last run.

function wp3 {
 [CmdletBinding()] param(  

 $envVar = get-Variable -Name $JobName -Scope Global -ErrorAction SilentlyContinue -ValueOnly
 $Times = $JobName+"_count"
 $envVar2 = get-Variable -Name $Times -Scope Global -ErrorAction SilentlyContinue -ValueOnly
 if ($EnvVar -eq $null) { 
  #Global Variable doesn't exist, create one called based on $JobName
  $Env_WPIndex = 0
  New-Variable -Name $JobName -Scope Global -Value 0 #-Visibility Private
 } else {
  #Use current global variable value.
  $env_WPIndex = [double]$EnvVar
 if ($envVar2 -eq $null) { 
  #Global Variable doesn't exist, create one called based on $JobName
  $envTimeThru = 0
  New-Variable -Name $Times -Scope Global -Value 0 #-Visibility Private
 } else {
  #Use current global variable value.
  $envTimeThru = [double]$envVar2
 Write-Progress -Activity ($JobName+"("+$envTimeThru+")") -Status $([string]$Env_WPIndex+"%") -PercentComplete $Env_WPIndex 
 $env_WPIndex = $env_wpIndex + 1
 if ($env_wpIndex -lt 100) { 
  #if less then max object count, increment the global variable by one
  Set-Variable -Name $JobName -Scope Global -ErrorAction SilentlyContinue -Value $env_WPIndex
 } else {
  $envTimeThru = $envTimeThru + 1
  #if already greater than max, remove the global variable from machine. 
  Set-Variable -Name $JobName -Scope Global -Value 0 # -ErrorAction SilentlyContinue

 Set-Variable -Name $Times -Scope Global -ErrorAction SilentlyContinue -Value $envTimeThru
 return $Passthru

Monday, January 8, 2018

MegaMillions Script - i.e. playing with Invoke-webrequest

My co-worker likes to read through the powershell reddit and found this interesting little challenge. 

I had an idea for a function to generate Powerball and Megamillions numbers - multiple ticket generation segmented into separate objects and then converted into valid JSON; I did come up with something but figured this would be a nice short script challenge since it doesn't rely on an external API.
Enjoy! (source)
I thought I'd take a crack at it. So far, of the 12 responses, most of them are variants of the random number generator. After one post about 'unique' numbers, the values started getting error checking.

Results returned looks like this:
Balls:  1(16%) 58(12%) 6(16%) 61(12%) 64(12%) Mega: 22

$MegaMillionsWebPage = Invoke-WebRequest
#Extract table from web page
$mmTable = ($MegaMillionsWebPage.ParsedHtml.getElementsByTagName("TABLE") | % {$_.innertext} | % {$_.split("`n")})[1..25]
#Return mid-five columns from table (ball column)
#DrawDate Balls MegaBall Megaplier Details
$balls = $mmTable | % {$_.split(" ")[1..5]}
#Get only megaball values
$Megaball = $mmTable | % {$_.split(" ")[6]}
#Group appearance of mega ball by appearance on table
$GrpBalls = $balls | group | Sort-Object -property count -Descending 
#Base line statistics of mega ball number occurence.
$BallStats = $GrpBalls | Measure-Object -Property count -min -max -average
# Have won more than 'average' number of times. 
$avg = [int]$BallStats.average
While ($mostPopular.count -lt 5 -and $avg -gt 0) {
    $mostPopular = $GrpBalls | ? {$_.count -gt $avg} | select -ExpandProperty Name    
    $avg-- #broaden scope if less than 5 results returned. 
#Return 5 numbers from the most popular results.
$MyBalls = $mostPopular | Sort-Object {Get-Random} | select -first 5 | % {[int]$_} | Sort-Object
#Show number of appearance of each value.. 
$WeightedBalls = $GrpBalls | ? {$myballs -eq $} | select name, @{Name = "Weight"; Expression = {[string](($_.count / 25) * 100) + "%"}}
$BallReport = $WeightedBalls | sort -Property name | %{$"("+$_.weight+")"}
write-host -NoNewline "Balls: ", ($BallReport -join (" "))
$megaGrp = $megaball  | group | Sort-Object -property count -Descending 
$MegaBallStats = $megaGrp | Measure-Object -Property count -min -max -average
$MegaAVG = [int]$MegaBallStats.average
#Randomly pick one of the most popular mega numbers.. 
$MegamostPopular = $megaGrp | ? {$_.count -eq $MegaBallStats.maximum} | select -ExpandProperty Name | Sort-Object {Get-Random} | select -first 1
write-host " Mega:",$MegamostPopular

Monday, September 11, 2017

Powershell CSV Join

I am currently working on a fairly large project migrating mailboxes from on-premise Exchange 2010 upto Microsoft's O365. One of the issues that we're having is that mailbox permissions don't always migrate correctly. So I've developed process to capture the mailbox permissions in one form or another in a CSV prior to migration, then I reapply them after reapplying them.

One of the issues experienced is that on-premise account information doesn't always match up with what's in O365. For example, I have the local Exchange / AD account information, but not necessarily what the account looks like in o365. So I wrote this script to join two CSV files based on similar properties. This way I can join a CSV containing mailbox information (i.e. SamAccountName) to the Get-MailboxPermission "USER" field.

.\CSVJoin.PS1 -CSV1Data "c:\bin\All-Mailboxes.CSV" -CSV1Property Samaccountname -CSV2Data "c:\bin\All-MailboxPermissions.csv" -CSV2Property USER -JoinedCSV "c:\BIN\Joined-CSV.csv"

The  script will export a CSV containing all fields from CSV1 plus all the fields from CSV2 that don't overlap in name.

    [parameter(Mandatory=$true,HelpMessage="Path and filename for source CSV 1")][ValidateScript({Test-Path $_ })][string]$csv1Name,
    [parameter(Mandatory=$true,HelpMessage="Propery from CSV1 to join files on.")][string]$csv1Property,
    [parameter(Mandatory=$true,HelpMessage="Path and filename for source CSV 2")][ValidateScript({Test-Path $_ })][string]$Csv2Name,
    [parameter(Mandatory=$true,HelpMessage="Propery from CSV2 to join files on.")][string]$csv2Property,
    [parameter(Mandatory=$true,HelpMessage="Path and Name for combined CSV file")][string]$JoinedCSV

$csv1Data = Import-CSV $csv1Name | ?{$_.$CSV1Property -ne $null}
$csv2Data = Import-csv $Csv2Name | ?{$_.$CSV2Property -ne $null}

#Capture all the column values for each CSV file and compare them.
$csv1Members = $csv1Data[0] | Get-Member | ?{$_.membertype -eq "NoteProperty"}
$csv2Members = $csv2Data[0] | Get-Member | ?{$_.membertype -eq "NoteProperty"}
$AddCSV2members = Compare-Object $csv1Members $csv2Members | ?{$_.sideindicator -eq "=>"} | %{$_.inputobject}

#Populate HashTable with First CSV based on JOIN fields
$csv1HashTable = @{}
$csv1Data | %{$csv1HashTable[$_.$csv1Property.trim()] = $_}

#Loop through Second CSV and join fields to first CSV. 
$newCSV = @()
ForEach ($c in $csv2Data) {
    $Row = $csv1HashTable[$c.$csv2property.trim()]
    if ($row ) {
        ForEach ($m in $AddCSV2members) {
            $Row | add-member -Membertype NoteProperty -Name $ -value $C.$($ -force
        $newCSV += $Row

if ($newCSV) {
    $newCSV | Export-csv $JoinedCSV -notypeinformation

Thursday, July 6, 2017

Delete and Compress old Log Files using DOS Batch file.

On all of our Exchange Client Access servers, I've been running my Compress and Delete script. Unfortunately, randomly the scheduled task will fail to run the script and the log files don't get purged. This requires kicking off the process by hand to avoid server meltdown.

I am thinking, that powershell may be partially at fault on some of these boxes. Permissions, or run time exceptions, may be causing the script to fail at running. So I've managed to put together this DOS Batch script that does basically the same thing.

  • Deletes all log (and zip) files in the folder older than 30 days
  • (if finds 7-zip) It will compress all log files older than 7 days, then delete them
The one disadvantage that I see is date-stamping. In Powershell, I was going through and stamping the original log files 'last modified' date on the ZIP. This allowed me to easily trigger 30-day deletes on any file in the folder because it would maintain it's original date. I figure that if I run this scheduled task daily, it will only offset the date by 7 days (i.e. an 8 day old log file will take today's date).

I am saving the following as "DOSPurgeOldLogFiles.CMD" and running it from a daily scheduled task.

@Echo off
REM Folder for Log Files
Set InetLogsFolder=c:\inetpub\logs\LogFiles\W3SVC1
if NOT EXIST %InetLogsFolder% Goto :NoLogs

REM Purge all files in log folders older than 30 days old
forfiles -p %InetLogsFolder% /s /m *.* /d -30 /c "cmd /c del @path"

if NOT EXIST "c:\program files\7-Zip\7z.exe" Goto :NoZIP

REM ZIP all files older than 7 days old
for /F %%G in ('forfiles -p %InetLogsFolder% /s /m *.LOG /d -7') DO "c:\program files\7-Zip\7z.exe" a -tzip -mtc=on "%InetLogsFolder%\" "%InetLogsFolder%\%%~G"

REM delete all files that are now ZIP'd
forfiles -p %INETLogsFolder% /s /m *.log /d -7 /c "cmd /c del @path"


Echo Cannot locate log files at %InetLogsFolder%

Echo Install 7-ZIP to compress log files.


Friday, May 12, 2017

Pull All WU Patches from Servers

With all the concern in the news lately, we went through all our on-premise servers and reviewed the patching. This script reads all of our Exchange servers (you could replace that for a CSV of server names) and does a remote call for the Windows Update patching. The script will return an object containing each installed patch and if it was successful or not.
$scriptBlock = {
 $Session = New-Object -ComObject "Microsoft.Update.Session"
 $Searcher = $Session.CreateUpdateSearcher()
 $historyCount = $Searcher.GetTotalHistoryCount()
 $Searcher.QueryHistory(0, $historyCount) | ?{$_.title -notlike "*definition update for*"} |Select-Object Title, Description, Date,
 @{name="Operation"; expression={switch($_.operation){
    1 {"Installation"}; 2 {"Uninstallation"}; 3 {"Other"}
 @{name="Status"; expression={switch($_.resultcode){ 1 {"In Progress"}; 2 {"Succeeded"}; 3 {"Succeeded With Errors"};4 {"Failed"}; 5 {"Aborted"} }}}

$serverList = get-exchangeserver
$Patching = @();$serverCount = $serverList.count;$index=1

forEach ($server in $ServerList ) {
    write-progress -activity "reading Windows Update" -Status $ -percentcomplete (($index/$serverCount)*100);$index++
    $LastUpdates = Invoke-Command -ScriptBlock $scriptBlock -ComputerName $ -ErrorVariable $failedWINRM
    $Patching+= $lastupdates
return $Patching

.\report-WindowsUpdatePatching.ps1 | ?{$_.title -like "*4012212*" -and $_status -eq "Failed"}

Thursday, March 30, 2017

Removing all the bad (email addresses)

With moving to o365, it comes time to get serious about cleaning out all of the invalid smtp domains that we've inadvertently stamped on all our mailboxes. Those other customers where their policy was applied incorrectly and stamped across domains. Objects that moved between customers and took on both domains. That 'default' smtp domain that isn't route-able, but is out there because your customers don't really want your smtp domain stamped on everything.

For this, I've developed two scripts. This first script is a bit of a blunt hammer. I will remove all domains that don't belong to this customer, plus leave any domains.


  1. All customers are sorted into their own OU. I am limiting my search to only their container to avoid removing email domains from objects that belong to other customers. 
  2. All objects have at least one valid email address that should be kept. Since this first one uses native Exchange tools, it won't remove the primary smtp address. 
#Remove all but the primary smtp domain and any domains.


if ($ou -notlike "ou=*") {
    $OuDN = get-organizationalunit $ou
    $ou = $oudn.distinguishedname

write-host "Reviewing objects in: ",$ou

$DomainFilter = "*@"+$PrimarySMTPDomain
$AllRecipients = get-adobject -filter {mail -like $DomainFilter} -properties ProxyAddresses -searchbase $ou -resultsetsize $null

$Index=1;$objCount = $AllRecipients.count; $ModifiedDateStr = "Modified: "+$(get-date).toshortdatestring()

write-host "Found $objCount with $DomainFilter"

Foreach ($m in $AllRecipients) {
    write-progress -Activity "reviewing objects" -Status $ -PercentComplete (($index/$objCount)*100);$Index++    
    $removeThese = $m.ProxyAddresses | ?{$_ -like "SMTP:*" -and $_ -notlike $DomainFilter -and $_ -notLike "*" -and $_ -notLike "*"} | %{$_}
    if ($removeThese) {
        $o = get-recipient $m.distinguishedname
        $n = $null
        if ($o.recipientType -eq "UserMailbox") {
            $n = get-mailbox $o.identity
        } elseif ($o.recipientType -like "mailUniversal*") {  #Some type of group
            $n = get-distributiongroup $o.identity
        } elseif ($o.recipientType -like "mailuser") {  #Some type of group
            $n = get-mailuser $o.identity
        if ($n -ne $null) {
            $removeThese | %{write-host "removing $_ from $n"}
            if ($removethese -is [string]){
                #was getting errors that email address was NULL 
                #  when trying to use foreach loop with single domain.
                $Results = $n.emailaddresses.remove($RemoveThese)
            } else {
                $results = $removeThese | %{$n.emailaddresses.remove($_)}
            if ($o.recipientType -eq "UserMailbox") {
                set-mailbox -identity $n.identity -emailaddresses $n.emailaddresses -customattribute8 $ModifiedDateStr
            }  elseif ($o.recipientType -like "mailUniversal*") {  #Some type of group
                set-distributiongroup -identity $n.identity -emailaddresses $n.emailaddresses -customattribute8 $ModifiedDateStr
            } elseif ($o.recipientType -like "mailuser") {  #Some type of group
                set-mailuser -identity $n.identity -emailaddresses $n.emailaddresses -customattribute8 $ModifiedDateStr
        } else {
            write-host "Error: $o not mailbox or group"
    } else {
        #write-host "nothing to change for $m"

Tuesday, January 10, 2017

MemberOf in O365

Now that we have user's in o365, I've been trying to reproduce my on-premise Exchange 2010 scripts using technology available in O365. The biggest issue so far has been the lack of  AD cmdlets. No longer can I run "get-qaduser" to pull the MemberOf and AllMemberOf properties. After numerous searches to find a way to do this, I finally decided it was going to take some powershell recursion with get-distributiongroupMember.. Ugh. I was that close, when I stumbled on this post on Spiceworks. It's a basic brute-force search, but it only took a tiny portion of the time of my recursive process. (of course, I am reading in all the groups into memory with this version, instead of live reads each group).

My starts where Raven left off. My intention is to also reproduce 'allmemberof' which includes all parent groups the user is a member of.  For example:

Windows Engineering -> IT Staff -> California Staff -> All Staff

#Finds all groups $identity belongs to in local Active Directory.
write-verbose "confirming identity"
$U = get-user $identity -ea silentlycontinue
if ($U -eq $null) { write-error "Can't find $identity. Try again.";Break}

write-verbose "reading all groups into memory"
$groups = get-group -resultsize unlimited | select name, members

write-verbose "reviewing first level groups that user belongs to."
$MemberOf = $groups | ?{$_.members -contains $u}
$childgroups = $MemberOf
$AllMemberOf = $MemberOf
Do {
    write-verbose "reviewing parent groups that user belongs to."
    $parentGroups = ForEach ($cg in $childgroups) {      
         $groups | ?{ $_.members -contains $}
    write-verbose "found some parent groups, let me check those as well. Keep climbing up the tree.."
    $AllMemberOf += $parentGroups
    $childgroups = $parentGroups
} While ($ParentGroups.count -gt 0) #Only stop when I reach the top.

Write-Verbose "#Create pretty answer, add to User object that we started with."
$amoj = ($AllMemberOf | select -unique Name | %{$}) -join(";")
$moj = ($MemberOf | select -unique Name | %{$}) -join(";")
$u | Add-Member -Name "AllMemberOf" -MemberType NoteProperty -Value $amoj
$u | Add-Member -Name "MemberOf" -MemberType NoteProperty -Value $moj

write-verbose "#Return a modified user object to requestor."
Return $u

To use:
Example 1:

.\o365MemberOf.ps1 -identity "Joe User" | select MemberOf

- echo first-level group membership to screen

Example 2:

$UserInfo = .\o365MemberOf.ps1 -identity "Joe User"
$UserInfo.AllMemberOf.split(";") | get-distributiongroup

- grab all group membership and find associated distribution groups.