kashav / fsql Goto Github PK
View Code? Open in Web Editor NEWSearch for files using a fun query language
License: MIT License
Search for files using a fun query language
License: MIT License
Duplicate entries are shown when any of the directories has a leading/trailing slash. This only applies to queries which select from multiple directories with at least one being a child directory of another.
Assume structure is as follows:
.
├── foo
│ ├── ...
│ ├── ...
And you run the following queries
$ fsql "select all from ., foo"
...
$ fsql "select all from ., foo/"
...
$ fsql "select all from ., ./foo"
...
$ fsql "select all from ., ./foo/"
...
You should see that the first doesn't include foo
twice, but the next 3 do.
This can easily be solved with a new Excluder
implementation which maintains a set (map[string]bool
) of seen directories to determine if the directory has already been traversed. I'm fairly certain that the best solution for this would be to use regex, but it may actually be more efficient to just add multiple entries to the map per file (no trailing/leading slash, trailing, leading, and both). There's a naive implementation of this in Query.execute
.
Very open to hearing other ideas!
It would be better to support some function to convert the size unit, e.g. kb/mb/gb. So the sql would be as below
fsql "select name, mb(size) from . where ...."
Would be nice if -v
and --version
switches were supported, and printed the version and build date...
aside: should probably publish update instructions go get -u -v ...
on the site, since I re-tried the -v alone and nothing happened... ;-)
It would be nice to do something like
SELECT name FROM /myDir LIMIT 100
This should limit the number of results to maximum 100. This is an interesting feature when working with a huge amount of files.
Maybe even without the need to give a WHERE clause.
Alternative syntax could be:
SELECT TOP 100 name FROM /myDir
Example ese case: I would eventually like to be able to select a maximum of 100 files where the name starts with a certain letter. Ideally it would be possible to give a regex for the name (e.g. like in this library: https://github.com/gahag/FSQL).
Would you mind add update support to this tool.
eg.update dest = 'xxxx' to copy select result file to dest folder.
or delete from where XXX
I just want this tool long ago. Thanks in advance.
A simple SELECT query should be able to identify the relative path of a file, which should include its folder name.
Example:
fsql "SELECT * from /Users/---/vagrant/jenkins"
prints:
`-rw-r--r-- 5248 Dec 7 14:28:14 b874568 config.xml`
instead of:
-rw-r--r-- 5248 Dec 7 14:28:14 b874568 jobs/1/config.xml
If I run a query like in the README gif it works:
$ fsql "SELECT * from . WHERE name LIKE mkv AND size > 1gb"
...lots of results
but taking away the name LIKE mkv
clause breaks it:
$ fsql "SELECT * from . WHERE size > 1gb"
2017/06/09 22:52:28 strconv.ParseFloat: parsing "1gb": invalid syntax
Allow SELECT
clause to be omitted (attribute should default to all
) (14344414, 14345753).
$ fsql "name, size, ... FROM . WHERE ..."
$ fsql "FROM . WHERE ..."
Make WHERE
clause optional (#8).
Quote-less queries:
$ fsql SELECT \* FROM . WHERE ...
$ fsql all FROM . WHERE ...
Make FROM
clause optional, directory should default to the current directory (./
).
Hi could you implementa path toen in order to get it in the output format or in the where clause ?
I know there's intentionally WHERE name LIKE
, but would be nice to have glob support in the FROM clause directly.... FROM ./foo/**/*.js ...
This might be out of scope, but it would be handy to generate the SHA-1 (or other hashes) to output and/or compare with another directory of files (when subqueries are implemented).
Introduce subqueries to be used in condition statements, example:
"SELECT name FROM foo WHERE name NOT IN (SELECT name FROM ../bar WHERE date < ...)"
This shouldn't be too hard to implement.
First step is to introduce an IN
operator for strings/numbers. The operator will return true if current the value is in the provided list.
The subquery should be evaluated individually. It returns a []Result
for all files that passed the query (a Result
struct holds each of the applicable attributes for a single file). Now we can apply IN
to this result list.
very cool project
Sorry if this is listed somewhere, but will this work on Windows?
This works...
fsql "SELECT * FROM ./_src WHERE name LIKE %.js"
this does not...
fsql "SELECT * FROM _src WHERE name LIKE %.js"
Thank you for a great tool and sharing with the community; this is awesome.
Do you mind creating a binary/executable for Windows?
As the pathname can contain -
, it would be nice if there's explicit method of seeking in such directories.
Or at least noted in README that to search in directory -oops
you should do ./-oops
.
Darwin builds are failing with:
vendor/golang.org/x/sys/unix/syscall_darwin.1_13.go:29:3: //go:linkname must refer to declared function or variable
This can be fixed with go get -u golang.org/x/sys
, but then the compilation fails on slices which require go >= 1.17.
Could you bump both, please?
For example, excluding .git
(with -.git
) results in .gitignore
not being listed.
This can be resolved with excluding with -.git/
instead, but then .git
is listed.
Hola!
This seems to have a reasonable amount of overlap with the OSQuery project. Why write it anew?
Most of the use cases where I can see fsql being useful, normally involve doing something with the files found in the query, it would be really useful to know their file system location, at the moment additional manipulation is needed to get the path easily.
Adding path attribute as a fully resolved path, opens up a lot of possibilities.
This is a fantastic repo, and I'd love to add it to homebrew, but it needs a version to be added. If you could tag and release a version, that would be fantastic.
Query:
select * from / where size > 10GB
Panic:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x4c56d1]
goroutine 1 [running]:
main.compare(0xc420012408, 0x4, 0x12, 0xc420012440, 0x2, 0x0, 0x0, 0x0, 0x0)
/home/michael/Projects/go/src/github.com/kshvmdn/fsql/main.go:55 +0x2e1
github.com/kshvmdn/fsql/query.(*ConditionNode).Evaluate(0xc42000c600, 0x0, 0x0, 0x4e0b20, 0x505c38, 0x7fae1cd0b400)
/home/michael/Projects/go/src/github.com/kshvmdn/fsql/query/query.go:113 +0x80
main.main.func1(0xc4203db550, 0xf, 0x0, 0x0, 0x56d620, 0xc4203d5ce0, 0x0, 0x0)
/home/michael/Projects/go/src/github.com/kshvmdn/fsql/main.go:120 +0x162
path/filepath.walk(0xc4203db4b0, 0xd, 0x56e720, 0xc420251520, 0xc42000c620, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:372 +0x2fe
path/filepath.walk(0xc4203db1f4, 0xa, 0x56e720, 0xc4202505b0, 0xc42000c620, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:376 +0x414
path/filepath.walk(0xc420013880, 0x5, 0x56e720, 0xc42042dad0, 0xc42000c620, 0x0, 0x0)
/usr/lib/go/src/path/filepath/path.go:376 +0x414
path/filepath.walk(0xc4200123a8, 0x1, 0x56e720, 0xc420078680, 0xc42000c620, 0x0, 0x20)
/usr/lib/go/src/path/filepath/path.go:376 +0x414
path/filepath.Walk(0xc4200123a8, 0x1, 0xc42000c620, 0x1, 0xc420096000)
/usr/lib/go/src/path/filepath/path.go:398 +0x14c
main.main()
/home/michael/Projects/go/src/github.com/kshvmdn/fsql/main.go:144 +0x3bb
It outputted two files before crashing. Go version 1.8.1 linux/amd64, do you need more information?
A lot of suggestions have revolved around modifying attributes in some way (#3, #15).
Attribute modifiers will apply some change to the attributes in the SELECT
and WHERE
clauses.
I'm most fond of the following syntax right now. Of the ideas suggested, it's the easiest to parse (particularly for chained modifiers), while still being intuitive. I'm open to ideas if anyone has a suggestion.
SELECT name::upper::fullpath, size::mb FROM . WHERE time::iso > 2017-05-17T06:07:28Z
As I mentioned in #15 (comment), one of the downsides of a syntax like this is that we're moving away from SQL (i'm not entirely sure that this is even a problem).
would be good to have a test FS and have test that run on it.
The test FS can be created from the test start.
this will make it easy to spoke regressions as the lib grows.
really useful library.
Just a heads up that Go 1.17 (and the just-released 1.16 without first changing GO111MODULE
) won't support builds that aren't in module-aware mode.
The master build should work but the last tag (v0.3.1) does not. Is it possible you can tag a new release for this?
what if i want to find which file contains the specific string? just like in IDE.
I seem to have installed fsql using go get
, but I’m still not able to begin commands with fsql
. I followed the steps as written. To use the program I have to change my directory to the bin folder where I installed Go and then drag the fsql Unix executable from Finder into Terminal. In other words, I ended up using ./bin/fsql <query>
instead of fsql <query>
. Could there be any steps missing that would make fsql <query>
valid?
Super cool tool you made , thanks @kshvmdn
I would like to see some plugins, (how to extend the fsql
)
for example
git
: can find uncommit / unstage files ref: https://github.com/src-d/go-git https://github.com/cloudson/gitql
tag
: can find with macOS file/folder with tag
exif
: can find photos with exif query
Various enhancements and bugs that I've come across, feel free to comment below (or create a new issue) for suggestions or bugs.
.git
results in .gitignore
not being listed).... ENDSWITH jsx?
).OR
/ AND
/ ()
(for precedence) in condition statements (lexing is already done for these, just need to add the parsers).os.FileInfo
supports)."... WHERE time > May 1 ..."
is broken since we're splitting conditionals by space. Fix by allowing single quotes and backticks in query strings, so something like "... WHERE time > 'May 1' ..."
works and evaluates the conditional to have value of "May 1"
.NOT
operator for negating conditionals.Moved from the README to this issue as of b78a987.
In the scenario:
fsql "select FULLPATH(name), hash from...
I would expect the return to be:
/path hash
instead it's hash /path
running the latest version available via brew.
% fsql -v fsql version 0.3.x, built off master
fsql "SELECT name, size, time FROM ~/Documents WHERE name LIKE %.js AND time > 'May 15 2017 00 00'"
2017/05/16 07:42:28 no such file or directory: ~/Documents
(exists with newer files there also namely fsql src)
/home/user/blah works fine.
Would you be willing to start using the Go vendor directory or one of the other Go vendor systems so that we can avoid go resource blocks in the Homebrew formula?
I noticed while doing the upgrade from 0.1.1 to 0.2.0 that fsql no longer builds unless I add a go resource block for gopkg.in/oleiade/lane.v1
.
It would be much appreciated if we could avoid this in the formula, as we're trying to eliminate the use of go resource blocks, so having to add them to a formula that doesn't currently need them is :(
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.