velocidex / go-ntfs Goto Github PK
View Code? Open in Web Editor NEWAn NTFS file parser in Go
License: Apache License 2.0
An NTFS file parser in Go
License: Apache License 2.0
i wrote the following code for file-log search on my windows,
but it get "Invalid magic" error, how can i fix up this demo code ? thank you~
`
package main
import (
"context"
"fmt"
"io"
"log"
"os"
"strings"
"github.com/Velocidex/go-ntfs/parser"
)
const (
record_directory = "out.txt"
)
func getReader(reader io.ReaderAt) io.ReaderAt {
if record_directory == "" {
return reader
}
return parser.NewRecorder(record_directory, reader)
}
func main() {
rd, er := os.Open(\\.\D:
)
if er != nil {
log.Fatalln(er)
}
defer rd.Close()
reader, err := parser.NewPagedReader(getReader(rd), 1024, 10000)
if err != nil {
log.Fatalln(err)
}
ntfs_ctx, err := parser.GetNTFSContext(reader, 0)
if err != nil {
log.Fatalln(err)
}
for record := range parser.ParseUSN(context.Background(), ntfs_ctx, 0) {
fmt.Println(record.Usn())
fmt.Println(record.Offset)
fmt.Println(record.Filename())
fmt.Println(record.FullPath())
fmt.Println(record.TimeStamp())
fmt.Println(strings.Join(record.Reason(), ", "))
fmt.Println(strings.Join(record.FileAttributes(), ", "))
fmt.Println(strings.Join(record.SourceInfo(), ", "))
}
}
`
@scudette
Thank you for sharing the code of this library. As a newbie to MFT, could you write an example? I would greatly appreciate it if you could read the $MFT file directly without copying it, and then parse out the file name, file size, modified time of each file.
The current paradigm of accessing data streams via their Attribute IDs is incorrect and results in wrong data being fetched in certain scenarios. The library relies on the inode notation (MftEntry-AttributeType-AttributeID) to uniquely identify data streams. Eg: 38-128-3
However in case of complex MFT entries where an attribute for the main MFT entry is stored in another another one, the attribute ID is no longer unique as it is likely going to be zero in the second one (while in the base entry, id 0 would always be STDINFO). If there are more than one streams, there is no way to distinguish them. This scenario is commonly seen in large fragmented files, most commonly $USNJRNL:$J.
This is not easily noticed as it only occurs if the file is very large and highly fragmented like the $MFT and $USNJRNL:$J. The current implementation partially works, but gives incorrect results that are otherwise difficult to validate without a lot of advanced testing (creation of large and fragmented files!).
The attached disk image charlie.zip demonstrates this problem via a handcrafted MFT entry that simulates the same fragmented behaviour by adding an ATTRIBUTE_LIST and storing some $DATA attributes in other MFT entries. The file /Nine.txt has the following streams:
Filename | Info |
---|---|
Nine.txt | Default $DATA stream, Non-resident |
Nine.txt:111 | Non-resident Alternate $DATA stream |
Nine.txt:222 | Resident Alternate $DATA stream |
Nine.txt:333 | Non-resident Alternate $DATA stream |
The picture below shows the MFT layout of this file.
It is not possible to access the streams 111 and 333 via the inode notation. To access alternate data streams, we need to address them by name. In NTFS, no other unique identifier exists.
Below is a listing of the contents and file sizes of the streams in Nine.txt.
Filename | Size | Content |
---|---|---|
Nine.txt | 5000 | 9999999999999999...<snipped> |
Nine.txt:111 | 5005 | "111111111111111111...<snipped> |
Nine.txt:222 | 56 | "222222222222222...<snipped> |
Nine.txt:333 | 6005 | "33333333333333...<snipped> |
To demonstrate the issue, we run the compiled exe against this image on the entry Nine.txt.
C:\go-ntfs>ntfs.exe cat "C:\temp\vr\tests\images\charlie_edited.dd” Nine.txt
"111111111111111111..._<snipped>_
Are you planing to write some documentation?
I would love to reuse the package, but apparently I will have to read all the code.
e.g. v0.2.0
, the current v0.1
does not seem to work
Adhoc errors like "Not found" from here https://github.com/Velocidex/go-ntfs/blob/master/parser/mft.go#L123 are hard to check against.
Instead we should use or wrap the standard io/fs
errors like ErrNotFound
in cases like that. https://pkg.go.dev/io/fs#pkg-variables.
Hi!
There's good support here for LZNT1 compression (created from Windows Compress contents to save space
option or compact.exe /c
).
In Windows 10 / Windows Server 2016, some extra compression algorithms XPRESS4K
, XPRESS8K
, XPRESS16K
, and LZX
were added (created from compact.exe /c /EXE LZX
). These have higher compression ratios but do not support in-place modification.
On disk these files show up as reparse points, with the compressed data stored in a WofCompressedData
ADS stream. In real Windows decompression is performed by a filesystem filter driver above ntfs. So it's arguable if it belongs in this project or not. But to correspond to the existing lznt1 support, there should be support for the other compression algorithms too.
Some links:
Hi,
I have an NTFS image that contains a compressed (LZNT1) file. When trying to read its content with go-ntfs, i get the Decompression error - shift is too large
error message.
7-Zip's ntfs library is successfully able to extract this file from the disk image.
Steps to reproduce:
for i in {1..512} ; do dd if=/dev/zero bs=4K count=10 >> combofile.bin ; dd if=/dev/urandom bs=4K count=10 >> combofile.bin
compact.exe /c filename
)Here is an affected filesystem: http://ms11.ivysaur.me/pub/raw.ntfs.xz . Try to extract the interior compresstest/combofile.bin
file
Hello :),
I'm using the cloned and compiled version of go-ntfs and it works fine even with partition offset in "ls command".
I noticed that the current release doesn't work the same way.
There is an idea of when we will be able to have the new release based on the current code replacing the current one from 2020 :)
Thanks & Best regards
Hi, thanks so much for this project, it's incredibly useful.
I have an NTFS disk image where the MFT is sufficiently large and fragmented, such that all the extents are not immediately in the $DATA
attribute. The $DATA
attribute contains only a single extent (4 long x cluster size 4096 / MFT entry size 1024 = 16 MFT IDs) and the rest of the MFT IDs have overflown into the $ATTRIBUTE_LIST
(0x20).
Go-ntfs is only able to read MFT IDs up to 16. However, the NTFS implementation in Sleuthkit handles this edge case and is able to see all the files inside this disk image. It checks the $ATTRIBUTE_LIST
, finds an additional $DATA
run, and treats it as a continuation of the first short $DATA
run.
In this case when reading the MFT, go-ntfs should also check if the $ATTRIBUTE_LIST
contains an additional $DATA
attribute, and add it to the runs
in the reader, probably somewhere near https://github.com/Velocidex/go-ntfs/blob/master/parser/mft.go#L466 . There is already support in go-ntfs for parsing the $ATTRIBUTE_LIST
but it just doesn't seem to be applied in this particular case.
I can confirm this issue on both v0.1.1 and master (b97c856),
I am working with a velociraptor offline collector and go-ntfs seems to break on my machine reproducible. I haven't done a deep dive into the code yet but this is my stack-trace. There seems to be an off by one error or similar thing happening.
panic: runtime error: index out of range [216] with length 216
goroutine 389 [running]:
www.velocidex.com/golang/go-ntfs/parser.(*NTFS_ATTRIBUTE).RunList(0xc003080ba0)
/go/pkg/mod/www.velocidex.com/golang/[email protected]/parser/attribute.go:158 +0x313
www.velocidex.com/golang/go-ntfs/parser.joinAllVCNs(0xc00bf20360, {0xc001646100?, 0x3, 0x4})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/parser/easy.go:393 +0x1d3
www.velocidex.com/golang/go-ntfs/parser.OpenStream(0xc00bf20360?, 0xc0015d5980?, 0xc00263e180?, 0xea1a?)
/go/pkg/mod/www.velocidex.com/golang/[email protected]/parser/easy.go:356 +0x207
www.velocidex.com/golang/velociraptor/accessors/ntfs.(*MFTFileSystemAccessor).OpenWithOSPath(0xc00263c4f0, 0xc0006d6ea0?)
/velociraptor-build/velociraptor/accessors/ntfs/mft.go:139 +0x127
www.velocidex.com/golang/velociraptor/vql/filesystem.(*ReadFileFunction).Call(0xc000ee7ae0?, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?}, 0x2149a20?)
/velociraptor-build/velociraptor/vql/filesystem/filesystem.go:353 +0x2eb
www.velocidex.com/golang/vfilter.(*_SymbolRef).callFunction(0xc0011600c0, {0x26c1e20?, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0}, {0x26b76e8?, 0x34ff2d0})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1691 +0x583
www.velocidex.com/golang/vfilter.(*_SymbolRef).Reduce(0xc0011600c0, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6ea0})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1545 +0x1c6
www.velocidex.com/golang/vfilter.(*_Value).Reduce(0xc00034cc00, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6ea0})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1433 +0x14f
www.velocidex.com/golang/vfilter.(*_MemberExpression).Reduce(0xc001a55e00, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1120 +0x56
www.velocidex.com/golang/vfilter.(*_MultiplicationExpression).Reduce(0xc001a55e40, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1369 +0x53
www.velocidex.com/golang/vfilter.(*_AdditionExpression).Reduce(0xc001a55e80, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1284 +0x53
www.velocidex.com/golang/vfilter.(*_ConditionOperand).Reduce(0xc001fe1410, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1322 +0x85
www.velocidex.com/golang/vfilter.(*_OrExpression).Reduce(0xc000592040, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1246 +0x56
www.velocidex.com/golang/vfilter.(*_AndExpression).Reduce(0xc000592200, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6ea0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1214 +0x4a
www.velocidex.com/golang/vfilter.(*_AliasedExpression).Reduce(0x0?, {0x26c1e20?, 0xc0013e4880?}, {0x26dc670?, 0xc0006d6ea0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:679 +0x9f
www.velocidex.com/golang/vfilter.(*_SelectExpression).Transform.func2({0x26c1e20, 0xc0013e4880}, {0xc00690c330?, 0xa?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:916 +0x5b
www.velocidex.com/golang/vfilter.(*LazyRowImpl).Get(0xc0001c8e00, {0xc00690c330, 0xa})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/lazy.go:60 +0x7d
www.velocidex.com/golang/vfilter/protocols.(*AssociativeDispatcher).Associative(0xc002252f98, {0x26dc670, 0xc0006d6fc0}, {0x1ecd480?, 0xc0001c8e00}, {0x1d97a20?, 0xc00263c130?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/protocols/protocol_associative.go:52 +0x3f5
www.velocidex.com/golang/vfilter/scope.(*Scope).Associative(...)
/go/pkg/mod/www.velocidex.com/golang/[email protected]/scope/scope.go:276
www.velocidex.com/golang/vfilter/scope.(*Scope).Resolve(0xc0006d6fc0, {0xc00690c330, 0xa})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/scope/scope.go:551 +0x176
www.velocidex.com/golang/vfilter/protocols.(*AssociativeDispatcher).Associative(0xc002252f98, {0x26dc670, 0xc0006d6fc0}, {0x2149a20?, 0xc0006d6fc0}, {0x1d97a20?, 0xc00263c120?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/protocols/protocol_associative.go:46 +0x482
www.velocidex.com/golang/vfilter/scope.(*Scope).Associative(0xc00690c330?, {0x2149a20?, 0xc0006d6fc0?}, {0x1d97a20?, 0xc00263c120?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/scope/scope.go:276 +0x53
www.velocidex.com/golang/vfilter.(*_SymbolRef).getFunction(0xc001161f80, {0x26dc670, 0xc0006d6fc0})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1505 +0x4de
www.velocidex.com/golang/vfilter.(*_SymbolRef).Reduce(0xc001161f80, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6fc0})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1534 +0x65
www.velocidex.com/golang/vfilter.(*_Value).Reduce(0xc00034cd00, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6fc0})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1433 +0x14f
www.velocidex.com/golang/vfilter.(*_MemberExpression).Reduce(0xc000592740, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1120 +0x56
www.velocidex.com/golang/vfilter.(*_MultiplicationExpression).Reduce(0xc000592780, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1369 +0x53
www.velocidex.com/golang/vfilter.(*_AdditionExpression).Reduce(0xc000592a00, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1284 +0x53
www.velocidex.com/golang/vfilter.(*_ConditionOperand).Reduce(0xc001fe15c0, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1322 +0x85
www.velocidex.com/golang/vfilter.(*_OrExpression).Reduce(0xc000a6e740, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1246 +0x56
www.velocidex.com/golang/vfilter.(*_AndExpression).Reduce(0xc000a6e800, {0x26c1e20, 0xc0013e4880}, {0x26dc670?, 0xc0006d6fc0?})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1214 +0x4a
www.velocidex.com/golang/vfilter.(*_CommaExpression).Reduce(0xc000a6eac0, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0006d6fc0})
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:1177 +0x5a
www.velocidex.com/golang/vfilter.(*_Select).processSingleRow(0xc001393220, {0x26c1e20, 0xc0013e4880}, {0x26dc670, 0xc0011a7680}, {0x2113680, 0xc0021cb180}, 0xc0020650e0)
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:585 +0x229
www.velocidex.com/golang/vfilter.(*_Select).Eval.func3()
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:548 +0xe5
created by www.velocidex.com/golang/vfilter.(*_Select).Eval
/go/pkg/mod/www.velocidex.com/golang/[email protected]/vfilter.go:533 +0x2ca
If the pageSize
is not aligned with the content (e.g. content: "abcd", pagesize: 3) the data from the last page is not read by the PagedReader.
r, _ := NewPagedReader(bytes.NewReader([]byte("abcd")), 3, 100)
buf := make([]byte, 1)
c, err = r.ReadAt(buf, 3)
// returns: c=0, buf=[]byte{0x00}, err=io.EOF
// should be: c=1, buf=[]byte{0x64}, err=nil
The redirect at https://www.velocidex.com/golang/go-ntfs/parser is not working, so e.g. https://godoc.org/www.velocidex.com/golang/go-ntfs/parser will not work.
I did some testing with extraction of files from NTFS. I figured out, that PagedReader
can be a bottleneck if configured incorrectly. A page size of 1024 is slowing down copies of large files.
I'm now using parser.NewPagedReader(r, 1024*1024, 100*1024*1024)
which yielded quite good results. I've not thoroughly tested implications on memory though.
Is there any reason for 1024 being used basically everywhere (https://grep.app/search?q=NewPagedReader)?
It may be possible to get into a recursion loop by expanding ATTRIBUTE_LIST_ENTRY into an MFT_ENTRY with another ATTRIBUTE_LIST.
Hi,
I'm looking for a way to add more precise timestamp for SI and FN fields.
If I'm right, the actual resolution of the SI and FN timestamps in NTFS is 100 nanoseconds counting from Jan 1, 1601.
Could we get such an output in MFT module output?
Thanks in advance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.