pkisolutions / psfciv Goto Github PK
View Code? Open in Web Editor NEWPowerShell native implementation of former Microsoft's File Checksum Integrity Verifier (FCIV)
License: Microsoft Public License
PowerShell native implementation of former Microsoft's File Checksum Integrity Verifier (FCIV)
License: Microsoft Public License
Hello, hope all is well. I was testing PsFCIV on some real-world examples, My Documents containing 16,535 Files, 1,050 Folders. All seemed OK at first, "Total files processed" and "Total new added files" both indicated 16,535, and the Verbose output also printed that many lines.
But when I passed DB.xml through a program to count <FILE_ENTRY> tags, 16,534 Files came back. I traced the issue to one file name containing something that seems like a hyphen but is not. Attached is a smaller data set containing the problem file name. As a matter of fact, the github form isn't even displaying the character as I try to paste it, so check the attached .zip
Total : 3
New : 3
Ok : 0
Bad : 0
Missed : 0
Locked : 0
Unknown : 0
Deleted : 0
Notice only Two <FILE_ENTRY>'s are actually written to DB.xml
<FCIV> <FILE_ENTRY> <name>Barn Owl.txt</name> <Size>12</Size> <TimeStamp>01/18/2022 07:49:28</TimeStamp> <MD5>HIkS05fEaGZwazcMovokbQ==</MD5> </FILE_ENTRY> <FILE_ENTRY> <name>Barn_Owl.txt</name> <Size>16</Size> <TimeStamp>01/18/2022 07:49:41</TimeStamp> <MD5>NYX+pTjx8Ee6bzjBC+hyLQ==</MD5> </FILE_ENTRY> </FCIV>
Thanks again for making this program available, it is still very useful even with current issues.
Not sure how you possibly tested this before publishing in the gallery...no matter what directory I give it after I hash my temp folder, it adds no new files doesn't honor recurse simply doesn't work. Would love this if it worked!
I am looking for an alternative to FCIV which is more robust and capable of recording errors to an output file (akin go the fciv.err
file produced by FCIV), for the purposes of identifying corrupted files which are not yet part of a hashed integrity database.
FCIV was useful in this respect in that when it encounters a file that produces a Cyclic Redundancy Check error, that a record of this gets added to the fciv.err
file output. However, it was limited in that it would fail and halt if a file name had unexpected characters or a directory path was too long.
When running PsFCIV on a directory for the first time with known bad files, the generated XML file will add a <FILE_ENTRY>
element detailing the <name>
, <Size>
, and <Timestamp>
, and hashes such as <MD5>
if successful.
For corrupted files, it instead produces these errors within the PowerShell window which fail to identify which file produced the error:
Exception calling "HashFile" with "2" argument(s): "Data error (cyclic redundancy check).
"
At C:\Program Files\WindowsPowerShell\Modules\PsFCIV\1.1\PsFCIV.psm1:62 char:17
+ ... $hashBytes = [PsFCIV.Support.CryptUtils]::HashFile($file, ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : IOException
Exception calling "FormatBytes" with "2" argument(s): "Value cannot be null.
Parameter name: inArray"
At C:\Program Files\WindowsPowerShell\Modules\PsFCIV\1.1\PsFCIV.psm1:67 char:21
+ ... $object.$hash = [PsFCIV.Support.CryptUtils]::FormatBytes( ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : ArgumentNullException
The result is that a <FILE_ENTRY>
will be added to the XML DB as a new entry without any hashes.
This is difficult to parse for identifying bad files.
The tool in its current state is suitable for identifying changes in data compared to a previously built database, but not as well suited to discovering corrupted files.
Would it be possible to modify the functionality so that when it encounters a hashing failure, that it records an explicit failure to the XML DB instead of omitting the hash object? Or alternatively, writing its own fciv.err file.
When initially creating the XML database, this program seems to not create the XML database if there is no file in the PATH directory even if -Recurse is selected and there are folders within the directory tree that do have files. Adding even a zero length file to the PATH folder makes it work. This problem does not occur for folders deeper in the directory tree.
The code for actually writing out the XML DB is commented out in the code file for Start-PsFCIV. It looks like someone started working on removing the $host item for that section but did not complete it and lines 450-455 have been commented out.
PSPKI/Client/Start-PsFCIV.ps1
Adding those lines back in allowed the initial DB to be written again.
I have a solution that may resolve this, but here is the issue first from a test run:
PS C:\working_directory\Documents> Start-PsFCIV -Path .\ -XML "C:\working_directory\DB.xml" -HashAlgorithm SHA256 -Recur
se -Verbose -Debug -Action Rename
DEBUG: Native PsFCIV mode ON
VERBOSE: Perform file 'Directory1\Another Test File.txt' checking.
DEBUG: Selected hash name : SHA256
DEBUG: Selected hash value: 6978413F9D61E2AA69A31CB7750B80B7FFF5C9F9DF5336A1D0612EFB6C54A58A
DEBUG: File hash: 6978413F9D61E2AA69A31CB7750B80B7FFF5C9F9DF5336A1D0612EFB6C54A58A
VERBOSE: File 'Another Test File.txt' is ok.
VERBOSE: Perform file 'a text file.txt' checking.
DEBUG: Selected hash name : SHA256
DEBUG: Selected hash value: 837CCB607E312B170FAC7383D7CCFD61FA5072793F19A25E75FBACB56539B86B
DEBUG: File hash: 837CCB607E312B170FAC7383D7CCFD61FA5072793F19A25E75FBACB56539B86B
VERBOSE: File 'a text file.txt' is ok.
VERBOSE: Perform file '[Torgo].gif' checking.
DEBUG: Selected hash name : SHA256
DEBUG: Selected hash value: 0201C1264D23B9EA3183CF5AFA953CA470A3A786F79B039C84C0AB05FEF171AC
DEBUG: File '[Torgo].gif' failed hash verification.
Expected hash: @{HashName=System.String[];
HashValue=0201C1264D23B9EA3183CF5AFA953CA470A3A786F79B039C84C0AB05FEF171AC}.HashValue
Actual hash: 7C462C54798E9CA1A737A91E7F4B120460EF8754D00B7F7A1FFE09F395F1AFBC
Rename-Item : Cannot rename because item at 'C:\working_directory\Documents\[Torgo].gif' does not exist.
At C:\Program Files\WindowsPowerShell\Modules\PsFCIV\1.1\PsFCIV.psm1:95 char:19
+ "Rename" {Rename-Item $file $($file.FullName + ".bad")}
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Rename-Item], PSInvalidOperationException
+ FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.RenameItemCommand
-----------------------------------
Total files processed : 3
Total good files : 2
Total bad files : 1
Total unknown status files : 0
Total missing files : 0
Total locked files : 0
-----------------------------------
Total : 3
New : 0
Ok : 2
Bad : 1
Missed : 0
Locked : 0
Unknown : 0
Deleted : 0
It would seem that both the Rename-Item and Remove-Item cmdlets require the -LiteralPath paramenter to be able to interpret square brackets or other special characters.
I would suggest modifying this function:
Line 92 in dbc00cf
possible changes below. I have also included two calls to Write-Verbose since I feel better knowing when a file is being modified.
# process -Action parameter to perform an action against bad file (if actual file properties do not match the record in XML).
function __takeAction ($file, $Action) {
switch ($Action) {
"Rename" {
Write-Verbose "Action: Rename file '$file' to '$($file.FullName + ".bad")'"
Rename-Item -LiteralPath $file -NewName $($file.FullName + ".bad")
}
"Delete" {
Write-Verbose "Action: Delete file '$file'"
Remove-Item -LiteralPath $file -Force
}
}
}
Would it be possible to push version 1.2 to the powershell gallery later? It makes managing versions and re-deployment a bit easier. Thank You!
Hello, hope all is well. This issue is actually a duplicate of the main issue mentioned in issue #1 I am separating it out so it is not bundled with any other problem.
Sorry in advance for any bad formatting in this report.
I've been researching it and have come up with an explanation and possible solution.
I ran into this problem when trying to hash a data set containing 25 folders, 166 Gigs of data. Each folder had its own sets of files and subfolders of course, but the problem was that the program expects at least 1 file at the Path location, or the program exits without hashing.
I traced the issue to a strange behavior of the Get-ChildItem cmdlet. If Get-ChildItem is called with BOTH the -Recurse and -File parameters it will return nothing and not traverse any subdirectories unless 1 or more files is present at the working directory level.
As a small example lets say this is my data set:
C:\working_directory\folder1\file1.txt
C:\working_directory\folder2\an empty subfolder
C:\working_directory\folder3\file3.txt
I try to recurse and nothing is returned:
PS C:\working_directory> Get-ChildItem -Path .\* -Filter * -Recurse -Force -File
PS C:\working_directory>
Now I try again without the -File paramenter:
PS C:\working_directory> Get-ChildItem -Path .\* -Filter * -Recurse -Force
Directory: C:\working_directory\folder1
Mode LastWriteTime Length Name
-a---- 1/14/2022 4:39 PM 12 file1.txt
Directory: C:\working_directory\folder2
Mode LastWriteTime Length Name
d----- 1/14/2022 6:12 PM an empty subfolder
Directory: C:\working_directory\folder3
Mode LastWriteTime Length Name
-a---- 1/14/2022 4:39 PM 12 file3.txt
This time the Recursion is successful but a directory is also returned which we do not want.
Finally I try:
PS C:\working_directory> Get-ChildItem -Path .\* -Filter * -Recurse -Force | where { ! $_.PSIsContainer }
Directory: C:\working_directory\folder1
Mode LastWriteTime Length Name
-a---- 1/14/2022 4:39 PM 12 file1.txt
Directory: C:\working_directory\folder3
Mode LastWriteTime Length Name
-a---- 1/14/2022 4:39 PM 12 file3.txt
And here the recursion is successful and the directories are omitted.
Supposedly the $_.PSIsContainer property is only false in Files, not directories or symlinks so this can be used to obtain the File list instead of using the -File parameter itself.
Line 28 in 7ef7b1f
Original:
function dirx ([string]$Path, [string]$Filter, [string[]]$Exclude, $Recurse, [switch]$Force) {
Get-ChildItem @PSBoundParameters -File -ErrorAction SilentlyContinue
}
Modified:
function dirx ([string]$Path, [string]$Filter, [string[]]$Exclude, $Recurse, [switch]$Force) {
Get-ChildItem @PSBoundParameters -ErrorAction SilentlyContinue | where { ! $_.PSIsContainer }
}
Hello Mr. Podans,
I am very interested in migrating away from Microsoft's FCIV and over to your PsFCIV due to its support of more secure hashing algorithms, i.e. SHA256.
I was initially testing PsFCIV on a data set of 130 megabytes, 267 Files, 89 Folders. The DB.xml generated successfully but a second run for Checking the files was crashing. In this case there were Two zero byte files hiding in different locations in the data, not even next to one another.
I'm able to reproduce the crash on a small scaled down example of just 3 text files, one having text, the other two having zero bytes. Below are the commands and output. Attached is tmp.zip containing the samller example files and db.xml
Thank You!
PS C:> Start-PsFCIV -Path "C:\tmp" -XML "C:\Users\Owner\Desktop\DB.xml" -HashAlgorithm SHA1 -Recurse -Verbose
VERBOSE: Perform file 'C:\tmp\file1_does_have_text_inside.txt' checking.
VERBOSE: Perform file 'C:\tmp\file2_ZERO_bytes.txt' checking.
VERBOSE: Perform file 'C:\tmp\file3_ZERO_bytes.txt' checking.
Total files processed : 3
Total new added files : 3
Total good files : 0
Total bad files : 0
Total unknown status files : 0
Total missing files : 0
Total locked files : 0
Total : 3
New : 3
Ok : 0
Bad : 0
Missed : 0
Locked : 0
Unknown : 0
Deleted : 0
PS C:> Start-PsFCIV -Path "C:\tmp" -XML "C:\Users\Owner\Desktop\DB.xml" -HashAlgorithm SHA1 -Recurse -Verbose
VERBOSE: Perform file 'C:\tmp\file1_does_have_text_inside.txt C:\tmp\file2_ZERO_bytes.txt C:\tmp\file3_ZERO_bytes.txt'
checking.
VERBOSE: File file1_does_have_text_inside.txt file2_ZERO_bytes.txt file3_ZERO_bytes.txt is locked. Skipping this file..
PS C:>
Hello,
i had different problems with your cmdlet:
when the startfolder has no files and only a subfolder it seems that it doesnt work. (also with -Recurse)
when i specify the xml file in c:\ folder (c:\test.xml) its not working. (with admin rights).
often PsFCIV show no results, it show nothing.
Please help.
Thank you
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.