After reviewing a canonical cscope database: cscope permits "untagged" tokens, which presumably show up for show-all-uses-of-symbol searches but not in any of the special caller/callee/assignment-type searches. We could/should probably export a bunch of these.

Hi, just found your gem and it looks quite powerful. How would I use this to create a "Find References.." features in vim. I'm imagining my cursor is on a method name, eg, I press a hotkey that I've mapped to something in starscope, and it opens up a quickfix window with the references. Is something like this already supported or would I have to build it on top of starscope and, if so, any tips on doing that?

Need to make sure the above makes sense, but it seems that's how the parser is reporting this sort of thing to us...

Need more proper docs above and beyond what's provided by starscope -h. Look into what Ruby provides in the way of tooling, and/or just add some markdown files in docs.

It would be nice to be able to specify some sort of global configuration (~/.starscope.config ?) so that I don't have to keep ignoring cscope.out in all of my databases

Exuberant ctags defines a bunch of additional fields starting with a ;" that can be used to embed additional information in a tag file, see section 6 of: http://ctags.sourceforge.net/FORMAT

Most if not all of the required information is in the starscope db already, we just need to translate it appropriately.

With the recent fixes for nested functions, proper token matching and the __fake_global shim, the cscope export code is now horrendous.

Now that we store a language tag for each file, it makes sense to fill in a whole bunch of languages with a valid matcher and a stubbed extractor.

So users can get a list of files etc, and so I don't have to keep digging through the json while debugging.

Settle on Golang everywhere

Such as -q calls,new,lang:ruby. Probably just comma-separated key:value pairs on the end of the query string is the simplest. Escaping commas may or may not be a problem.

When I try to scan jekyll, I get this error:

/usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `readline': end of file reached (EOFError)
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `block in match_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `open'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/langs/ruby.rb:7:in `match_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:253:in `block in add_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:252:in `each'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:252:in `add_file'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:72:in `block in add_paths'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:71:in `each'
from /usr/local/share/gems/gems/starscope-0.1.7/lib/starscope/db.rb:71:in `add_paths'
from /usr/local/share/gems/gems/starscope-0.1.7/bin/starscope:123:in `<top (required)>'
from /usr/local/bin/starscope:23:in `load'
from /usr/local/bin/starscope:23:in `<main>'

Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec,

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Even for projects that already specify a license, including a license in your gemspec is a good practice, since
it is easily discoverable there without having to check the readme or for a license file.

For example, there is a License Finder gem
to help companies ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec.
This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

If you need help choosing a license (sorry, I haven't checked your readme or
looked for a license file), github has created a license picker tool.

In case you're wondering how I found you and why I made this issue, it's because
I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too,
and make issues for gemspecs not specifying a license as a public service :).

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue and let me know.
In either case, I'll follow up. Thanks!

p.s. I've written a blog post about this project

E.g. choosing which babel transformers to use (for javascript), choosing which ruby version to parse with parser, etc.

Ruby has a bunch of methods for defining things (such as alias_method) which are technically dynamic but which are typically pretty easy to statically parse anyways. Rails adds even more. It would be great for the ruby extractor to know about these.

f4e0351#commitcomment-13431734

Running current starscope master (54c6f53) on current rails master (84b4260e6c6238a8bd8ec5880daa8f7b10cfcb14) generates a cscope database which basically doesn't work - searching for symbols like initialize result in Internal error: cannot get source line from database%

for general symbol searches, regardless of definition, call, assignment, etc.

Ruby is so dynamic, there ought to be a way to get rid of the manually populated LANGS array at the top of db.rb.

Potential security concerns though?

It would be nice to be able to version individual extractors so that a tweak to ruby (for example) could force rebuilding ruby files but leave golang files untouched.

Just noticed, logging it so I don't forget to investigate at some point.

I'm actually going to need tests soon...

Possibly using https://rubygems.org/gems/coffee-script or similar.

As I'm starting to setup my workflow with starscope I noticed the tool doesn't work automatically. A user has to type starscope -e cscope to refresh the DB after each commit, rebase etc.

So, how about adding instructions to make starscope work with git hooks?
Here are the related instructions for ctags by Tim Pope. I'm pretty sure the same can be done for starscope...

I'm a lazy user so I'd be happiest to just copy/paste or follow instructions for this, but I can probably help too.

For very large projects, loading the entire DB into memory before anything else is painful and slow. Investigate DB formats that can stay on disk and be loaded in pieces or lazily or ...? to ameliorate this. Just switch to sqlite?

I added a quick 80/20 hack for 1.1.2 but it's pretty easy to generate cases with string escapes like "\\" and "\"" which will break it completely. Need to write a proper string scanner.

This has been floating at the back of my mind for a while, but it really wouldn't be too difficult to hand-write a proper lexer/parser for Go since the syntax is so uniform and simple (possibly eventually as a separate gem).

This would permit proper parsing of all the weird corner cases that the current version breaks on, and much better recognition of variable uses, etc. Go provides a very nice spec with a full formal grammar etc (https://golang.org/ref/spec) so the information is available, I just need to bother to write it.

Very sorry to report this, but it just doesn't seem to be working properly:

This is the incorrect reference, as directed by starscope:

That reference is actually to a private method, so it is certainly incorrect.

This is the correct reference, as directed by rubymine:

Need to create a generic metadata hash in the database so that we can implement things like ignoring files matching certain patterns etc.

Also should probably store dates as "integers" to avoid the complicated parsing overhead.

Extended maintenance of Ruby versions 1.8.7 and 1.9.2 ended on July 31, 2014. There are probably still people using them, but...

• update travis config
• get rid of backports gem
• use foo: 1 syntax for all symbolic hash keys
• revert 368ac67, fc401a4, 44e7e55, 690ec8d, 5c736ff
• replace hackery with require_relative (manually revert 4e50729)
• bump required version to 1.9.3
• use each_line

Cscope in particular is getting large, using several cscope-specific helpers, and should be moved into its own module. Might as well do ctags at the same time. Mixin to database with extend or some other ruby magic?

Hi,
the promise of this tool looks amazing! Thanks a lot for working on it.
I found this repo by browsing for cscope alternative for ruby - and this might just do it!

Now, what is the best way to integrate this plugin with the editor (vim?) ?
It's not mentioned anywhere, but the starscope is intended to be used primarily through the editor, right?
What are the mappings you're using? (couldn't find your dotfiles/vimrc)

How about adding more info, or a separate doc/wiki for editor integration.

Thanks

The latest oj gem supports an interesting new streaming mode which may or may not lead to faster DB read/write times. Investigate the fastest method, pending a few bugs:

• ohler55/oj#158
• ohler55/oj#160

Make sure all operations (query, export, etc) produce some output even in corner cases (such as no results for a query). Replace --no-progress with the more general --quiet. Potentially be more verbose when --verbose is specified.

I'm trying to use starscope with the ruby project (https://github.com/ruby/ruby) and exporting fails for ctags and cscope. I traced the error back to line_for_record, which seems to be returning nil.

I'm not sure of the root cause of this problem. Perhaps it is because the source code is a mix between Ruby and C. This might not be an issue if starscope is not meant to be used for mixed source projects, although I've tried it on a gem with a C extension and exporting didn't fail.

Rather than passing verbose/quiet to the db and the db instantiating an output object, the script itself should instantiate an output object, use it, and pass it to the db.

In golang code like the following:

type a struct {
    var1, var2 int
}

sometimes(?) the definition of var2 is not picked up. Need to reproduce this reliably and fix.

The table swapping added for #33 is awful, need to refactor that.

It might be easier to manage in combination with the global config if it's stored in a separately editable JSON blob .starscope.json beside the DB instead of inside the DB itself.

Hi,
I'm setting up starscope for use on a new project. I'm all good on the "technical" side (ie setting git hooks etc).

But frankly, the usage part is still a bit fuzzy for me. I never fully grokked original cscope because I never used c. Extending upon that, I'm not sure what are the "answers" starscope can provide while browsing a new ruby codebase?

I'm sure the above question is badly formed, so try to clarify with a familiar analogy.
By using ctags with ruby (other languages as well), I can quickly jump to the method definition and look up what the method does. Then jump "back".
I use it often for this single, clearly defined purpose and it's perfect.

Back to cscope. What are the common usage scenarios (analogous to the ctags example above) starscope can be used for? What is it good for? Where does it fall short?

Here are a couple specific questions:

• "find this definition" search seems to jump to the method definition. Is this the same as ctags method definition lookup? Any diff here?
• "find functions called by this function" - when I invoke this search on a method name, all it seems to do is go over method calls within the current method. I'm not sure what is the practical purpose this could be used for? Any ideas?

I'm still investigating other searches, but it's probably time to stop writing :)
Help appreciated (maybe in a form of a wiki page for other to see as well) and thanks for working on this tool 👍

ERB, Markdown and others all support nesting other languages internally using special blocks. We should figure out what's necessary for extractors to dispatch to each other if we ever want to properly support that.

I have a deeply nested class, Entity::Animal::Cat for example. Unfortunately, the code usage of this class is its full path, Entity::Animal::Cat.new. When searching for Cat::new using starscope, it cannot pick up the fully qualified namespace usage. Is there was a way to fuzz this? *Cat::new and is there a way to fuzz this with cscope and ctags usage?

I find myself falling back to :grep search which can resolve the fuzzy matching with a little human interaction.

There are several tests (ctags/cscope export, database queries) which exercise the code paths to check for exceptions and syntax but don't actually verify the output is sane. They should be filled in.

Possibly via https://github.com/whitequark/rlua or https://github.com/oxyc/luaparse and https://github.com/sstephenson/execjs

Perhaps via https://github.com/grosser/parallel or some other gem.

Parsing files (in particular large Ruby files) is slow, so parallelizing it would be a big speed-up on multi-core systems.

• let extractors return arbitrary hashes with additional data
• deduplicate line values from records
• ???
• profit!

Hi Evan,

I'm not a Ruby user, but I'm interested in the idea of a cscope for Ruby and Go. I have not found a similar project.

cscope produces info for C, C++ and Java e.g. "functions calling this function", but not object-oriented info. Meanwhile, ctags can do object-oriented info but not "functions calling this function". So, my project CodeQuery combines these two and adds a GUI tool as well. It takes the cscope and ctags databases as inputs. https://github.com/ruben2020/codequery

Meanwhile I found pycscope that can read Python code and produce a cscope database, that is readable using the cscope program. I modified CodeQuery to be compatible with this database, and now CodeQuery can work with C, C++, Java and Python. https://github.com/portante/pycscope

Now, if starscope can export to a cscope database that is readable using the cscope program just like what pycscope does, then I can make this work together with CodeQuery.

They are currently awful, fragile, messes of instance_eval because I didn't know what I was doing when I first wrote them.

TODOs include at the very least:

• mock calls (Timecop? Mocha?) instead of manually farting around with date/time literals as integers
• split the export tests into their own file, and use StringIO instead of that awful Tempfile hack
• add more database files as fixtures instead of generating them on-the-fly with instance_eval
• refactor database API to provide sane accessors and use those where appropriate

With the new DB format (#17) landed, it should be pretty straightforward to exclude certain files/directories/patterns from the scan. This will come in very handy when scanning compiled languages (C et al) which generate a binary object file for each source file. Right now we scan all the .o files as well, which is silly.

This should come with a sane set of defaults (.o files, .starscope.db, etc) as well as a flag to add custom ones (and probably a flag to remove the defaults, if you really want to scan .o files for some reason).

The matching algorithm right now sometimes provides too many unrelated results, and sometimes doesn't find the results you want. Perhaps use a fuzzy gem like https://github.com/seamusabshere/fuzzy_match

Need to define exactly how scopes should work as well, since that is a mess of unclear heuristics right now.

The end is missed somehow, so subsequent code shows up as occurring inside it rather than in global scope (check MessageTooLarge definition, specifically the call to New).