eapache / starscope Goto Github PK
View Code? Open in Web Editor NEWSmart code search for Ruby, Go, and JavaScript
Home Page: https://rubygems.org/gems/starscope
License: MIT License
Smart code search for Ruby, Go, and JavaScript
Home Page: https://rubygems.org/gems/starscope
License: MIT License
The end is missed somehow, so subsequent code shows up as occurring inside it rather than in global scope (check MessageTooLarge
definition, specifically the call to New
).
Hi Evan,
I'm not a Ruby user, but I'm interested in the idea of a cscope for Ruby and Go. I have not found a similar project.
cscope produces info for C, C++ and Java e.g. "functions calling this function", but not object-oriented info. Meanwhile, ctags can do object-oriented info but not "functions calling this function". So, my project CodeQuery combines these two and adds a GUI tool as well. It takes the cscope and ctags databases as inputs. https://github.com/ruben2020/codequery
Meanwhile I found pycscope that can read Python code and produce a cscope database, that is readable using the cscope program. I modified CodeQuery to be compatible with this database, and now CodeQuery can work with C, C++, Java and Python. https://github.com/portante/pycscope
Now, if starscope can export to a cscope database that is readable using the cscope program just like what pycscope does, then I can make this work together with CodeQuery.
ERB, Markdown and others all support nesting other languages internally using special blocks. We should figure out what's necessary for extractors to dispatch to each other if we ever want to properly support that.
The latest oj
gem supports an interesting new streaming mode which may or may not lead to faster DB read/write times. Investigate the fastest method, pending a few bugs:
Ruby has a bunch of methods for defining things (such as alias_method
) which are technically dynamic but which are typically pretty easy to statically parse anyways. Rails adds even more. It would be great for the ruby extractor to know about these.
The table swapping added for #33 is awful, need to refactor that.
Just noticed, logging it so I don't forget to investigate at some point.
I'm actually going to need tests soon...
After reviewing a canonical cscope database: cscope permits "untagged" tokens, which presumably show up for show-all-uses-of-symbol searches but not in any of the special caller/callee/assignment-type searches. We could/should probably export a bunch of these.
Rather than passing verbose/quiet to the db and the db instantiating an output object, the script itself should instantiate an output object, use it, and pass it to the db.
Such as -q calls,new,lang:ruby
. Probably just comma-separated key:value
pairs on the end of the query string is the simplest. Escaping commas may or may not be a problem.
Perhaps via https://github.com/grosser/parallel or some other gem.
Parsing files (in particular large Ruby files) is slow, so parallelizing it would be a big speed-up on multi-core systems.
I have a deeply nested class, Entity::Animal::Cat
for example. Unfortunately, the code usage of this class is its full path, Entity::Animal::Cat.new
. When searching for Cat::new
using starscope, it cannot pick up the fully qualified namespace usage. Is there was a way to fuzz this? *Cat::new
and is there a way to fuzz this with cscope and ctags usage?
I find myself falling back to :grep
search which can resolve the fuzzy matching with a little human interaction.
I'm trying to use starscope with the ruby project (https://github.com/ruby/ruby) and exporting fails for ctags and cscope. I traced the error back to line_for_record, which seems to be returning nil
.
I'm not sure of the root cause of this problem. Perhaps it is because the source code is a mix between Ruby and C. This might not be an issue if starscope is not meant to be used for mixed source projects, although I've tried it on a gem with a C extension and exporting didn't fail.
In golang code like the following:
type a struct {
var1, var2 int
}
sometimes(?) the definition of var2
is not picked up. Need to reproduce this reliably and fix.
For very large projects, loading the entire DB into memory before anything else is painful and slow. Investigate DB formats that can stay on disk and be loaded in pieces or lazily or ...? to ameliorate this. Just switch to sqlite?
With the recent fixes for nested functions, proper token matching and the __fake_global shim, the cscope export code is now horrendous.
Hi, just found your gem and it looks quite powerful. How would I use this to create a "Find References.." features in vim. I'm imagining my cursor is on a method name, eg, I press a hotkey that I've mapped to something in starscope, and it opens up a quickfix window with the references. Is something like this already supported or would I have to build it on top of starscope and, if so, any tips on doing that?
for general symbol searches, regardless of definition, call, assignment, etc.
Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec,
via e.g.
spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']
Even for projects that already specify a license, including a license in your gemspec is a good practice, since
it is easily discoverable there without having to check the readme or for a license file.
For example, there is a License Finder gem
to help companies ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec.
This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.
If you need help choosing a license (sorry, I haven't checked your readme or
looked for a license file), github has created a license picker tool.
In case you're wondering how I found you and why I made this issue, it's because
I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too,
and make issues for gemspecs not specifying a license as a public service :).
I hope you'll consider specifying a license in your gemspec. If not, please just close the issue and let me know.
In either case, I'll follow up. Thanks!
p.s. I've written a blog post about this project
As I'm starting to setup my workflow with starscope
I noticed the tool doesn't work automatically. A user has to type starscope -e cscope
to refresh the DB after each commit, rebase etc.
So, how about adding instructions to make starscope work with git hooks?
Here are the related instructions for ctags by Tim Pope. I'm pretty sure the same can be done for starscope...
I'm a lazy user so I'd be happiest to just copy/paste or follow instructions for this, but I can probably help too.
This has been floating at the back of my mind for a while, but it really wouldn't be too difficult to hand-write a proper lexer/parser for Go since the syntax is so uniform and simple (possibly eventually as a separate gem).
This would permit proper parsing of all the weird corner cases that the current version breaks on, and much better recognition of variable uses, etc. Go provides a very nice spec with a full formal grammar etc (https://golang.org/ref/spec) so the information is available, I just need to bother to write it.
Need to create a generic metadata hash in the database so that we can implement things like ignoring files matching certain patterns etc.
Also should probably store dates as "integers" to avoid the complicated parsing overhead.
Ruby is so dynamic, there ought to be a way to get rid of the manually populated LANGS
array at the top of db.rb
.
Potential security concerns though?
It would be nice to be able to version individual extractors so that a tweak to ruby (for example) could force rebuilding ruby files but leave golang files untouched.
The matching algorithm right now sometimes provides too many unrelated results, and sometimes doesn't find the results you want. Perhaps use a fuzzy gem like https://github.com/seamusabshere/fuzzy_match
Need to define exactly how scopes should work as well, since that is a mess of unclear heuristics right now.
line
values from recordsHi,
the promise of this tool looks amazing! Thanks a lot for working on it.
I found this repo by browsing for cscope alternative for ruby - and this might just do it!
Now, what is the best way to integrate this plugin with the editor (vim?) ?
It's not mentioned anywhere, but the starscope is intended to be used primarily through the editor, right?
What are the mappings you're using? (couldn't find your dotfiles/vimrc)
How about adding more info, or a separate doc/wiki for editor integration.
Thanks
With the new DB format (#17) landed, it should be pretty straightforward to exclude certain files/directories/patterns from the scan. This will come in very handy when scanning compiled languages (C et al) which generate a binary object file for each source file. Right now we scan all the .o files as well, which is silly.
This should come with a sane set of defaults (.o
files, .starscope.db
, etc) as well as a flag to add custom ones (and probably a flag to remove the defaults, if you really want to scan .o
files for some reason).
Possibly using https://rubygems.org/gems/coffee-script or similar.
When I try to scan jekyll, I get this error:
/usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `readline': end of file reached (EOFError)
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `block in match_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `open'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `match_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:253:in `block in add_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:252:in `each'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:252:in `add_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:72:in `block in add_paths'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:71:in `each'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:71:in `add_paths'
from /usr/local/share/gems/gems/starscope-0.1.7/bin/starscope:123:in `<top (required)>'
from /usr/local/bin/starscope:23:in `load'
from /usr/local/bin/starscope:23:in `<main>'
Exuberant ctags defines a bunch of additional fields starting with a ;" that can be used to embed additional information in a tag file, see section 6 of: http://ctags.sourceforge.net/FORMAT
Most if not all of the required information is in the starscope db already, we just need to translate it appropriately.
Now that we store a language tag for each file, it makes sense to fill in a whole bunch of languages with a valid matcher and a stubbed extractor.
Need more proper docs above and beyond what's provided by starscope -h
. Look into what Ruby provides in the way of tooling, and/or just add some markdown files in docs
.
E.g. choosing which babel transformers to use (for javascript), choosing which ruby version to parse with parser, etc.
Hi,
I'm setting up starscope for use on a new project. I'm all good on the "technical" side (ie setting git hooks etc).
But frankly, the usage part is still a bit fuzzy for me. I never fully grokked original cscope
because I never used c. Extending upon that, I'm not sure what are the "answers" starscope can provide while browsing a new ruby codebase?
I'm sure the above question is badly formed, so try to clarify with a familiar analogy.
By using ctags
with ruby (other languages as well), I can quickly jump to the method definition and look up what the method does. Then jump "back".
I use it often for this single, clearly defined purpose and it's perfect.
Back to cscope. What are the common usage scenarios (analogous to the ctags example above) starscope can be used for? What is it good for? Where does it fall short?
Here are a couple specific questions:
I'm still investigating other searches, but it's probably time to stop writing :)
Help appreciated (maybe in a form of a wiki page for other to see as well) and thanks for working on this tool ๐
Need to make sure the above makes sense, but it seems that's how the parser is reporting this sort of thing to us...
Cscope in particular is getting large, using several cscope-specific helpers, and should be moved into its own module. Might as well do ctags at the same time. Mixin to database with extend
or some other ruby magic?
It would be nice to be able to specify some sort of global configuration (~/.starscope.config
?) so that I don't have to keep ignoring cscope.out in all of my databases
It might be easier to manage in combination with the global config if it's stored in a separately editable JSON blob .starscope.json
beside the DB instead of inside the DB itself.
I added a quick 80/20 hack for 1.1.2 but it's pretty easy to generate cases with string escapes like "\\"
and "\""
which will break it completely. Need to write a proper string scanner.
Running current starscope master (54c6f53) on current rails master (84b4260e6c6238a8bd8ec5880daa8f7b10cfcb14) generates a cscope database which basically doesn't work - searching for symbols like initialize
result in Internal error: cannot get source line from database%
They are currently awful, fragile, messes of instance_eval
because I didn't know what I was doing when I first wrote them.
TODOs include at the very least:
StringIO
instead of that awful Tempfile
hackinstance_eval
There are several tests (ctags/cscope export, database queries) which exercise the code paths to check for exceptions and syntax but don't actually verify the output is sane. They should be filled in.
Settle on Golang
everywhere
Extended maintenance of Ruby versions 1.8.7 and 1.9.2 ended on July 31, 2014. There are probably still people using them, but...
So users can get a list of files etc, and so I don't have to keep digging through the json while debugging.
Make sure all operations (query, export, etc) produce some output even in corner cases (such as no results for a query). Replace --no-progress
with the more general --quiet
. Potentially be more verbose when --verbose
is specified.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.