burrito-brothers / shiba Goto Github PK
View Code? Open in Web Editor NEWCatch bad SQL queries before they cause problems in production
Home Page: https://shiba-sql.com
License: Other
Catch bad SQL queries before they cause problems in production
Home Page: https://shiba-sql.com
License: Other
Can someone help?
2 problematic queries detected
Report available at /Users/dhurba87/apps/trip101/log/shiba_results/shiba_results-1554780870.html
Finished in 1.86 seconds (files took 11.02 seconds to load)
19 examples, 0 failures
/Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/shiba-0.9.2/lib/shiba/activerecord_integration.rb:90:in `block (2 levels) in when_done': can't modify frozen NilClass (RuntimeError)
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/example.rb:287:in `instance_exec'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/example.rb:287:in `instance_exec_with_rescue'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/hooks.rb:356:in `run'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/hooks.rb:410:in `block in run'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/hooks.rb:410:in `each'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/hooks.rb:410:in `run'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/hooks.rb:485:in `run'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/runner.rb:114:in `block in run_specs'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/reporter.rb:54:in `report'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/runner.rb:108:in `run_specs'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/runner.rb:86:in `run'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/runner.rb:70:in `run'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/lib/rspec/core/runner.rb:38:in `invoke'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/gems/rspec-core-3.0.4/exe/rspec:4:in `<top (required)>'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/bin/rspec:23:in `load'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/bin/rspec:23:in `<main>'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:24:in `eval'
from /Users/dhurba87/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:24:in `<main>'
We have a handful of tables that don't have any indexes. Some use the BLACKHOLE
storage engine. Another had its indexes removed in preparation for dropping it. Regardless, we get the following stack dump due to the disparity of SHOW TABLES
and information_schema.statistics
$ bin/mysql_dump_stats ...
Traceback (most recent call last):
8: from shiba/bin/dump_stats:22:in `<main>'
7: from shiba/lib/shiba/fuzzer.rb:39:in `fetch_index'
6: from shiba/lib/shiba/connection/mysql.rb:30:in `each_column_size'
5: from shiba/lib/shiba/connection/mysql.rb:30:in `each'
4: from shiba/lib/shiba/connection/mysql.rb:39:in `block in each_column_size'
3: from shiba/lib/shiba/connection/mysql.rb:39:in `each'
2: from shiba/lib/shiba/connection/mysql.rb:40:in `block (2 levels) in each_column_size'
1: from shiba/lib/shiba/fuzzer.rb:40:in `block in fetch_index'
shiba/lib/shiba/index_stats.rb:169:in `set_column_size': couldn't find table: example_table_name (RuntimeError)
I have created a method that clearly is of high severity when I test it with shiba locally. But when pushing this mehtod + specs to CircleCI it returns me that no errors where found in the diff.
The diff includes the problematic lines in spec/../model_spec.rb
and models/model.rb
def find_something
Model.where('title like "%something%")
end
RAILS_ENV=test SHIBA_DEBUG=true bundle exec shiba review --verbose --submit
Finding default options from CI environment.
diff:
branch: 098b26375a707373f590910840ce900143b88c3a
pull_request: 919
origin/HEAD...098b26375a707373f590910840ce900143b88c3a
grep /tmp/shiba/ci.json -e .circleci/config.yml -e Gemfile -e Gemfile.lock -e app/models/model.rb -e db/migrate/20190517103657_add_indexes.rb -e db/schema.rb -e spec/models/model_spec.rb -e spec/rails_helper.rb
Updated lines: [[".circleci/config.yml", 68..68], [".circleci/config.yml", 119..122], ["Gemfile", 121..123], ["Gemfile.lock", 94..94], ["Gemfile.lock", 100..100], ["Gemfile.lock", 449..449], ["Gemfile.lock", 616..618], ["Gemfile.lock", 710..710], ["Gemfile.lock", 772..772], ["app/models/model.rb", 87..91], ["db/migrate/20190517103657_add_indexes.rb", 1..7], ["db/schema.rb", 13..13], ["db/schema.rb", 52..52], ["db/schema.rb", 103..103], ["db/schema.rb", 105..105], ["db/schema.rb", 126..128], ["db/schema.rb", 149..156], ["db/schema.rb", 179..179], ["db/schema.rb", 186..186], ["db/schema.rb", 300..300], ["db/schema.rb", 561..561], ["spec/models/model_spec.rb", 25..31], ["spec/rails_helper.rb", 11..13]]
No problems found caused by the diff
circleci/config.yml
run_tests:
<<: *build_defaults
parallelism: 2
steps:
- *attach_workspace
- *restore_bundle_cache
- run: bundle install --deployment --with development test
- run:
name: Set up test database
command: RAILS_ENV=test bundle exec rake db:create db:migrate
- run:
name: Run tests
command: |
if [[ `git log -1 --pretty=%B` == *"[hotfix]"* ]]; then
echo "Hotfix fast-laned. Skipping tests.";
exit 0;
else
RAILS_ENV=test CIRCLE_BUILD_NUM=$CIRCLE_WORKFLOW_ID bundle exec rspec $(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=filesize);
fi
- run:
name: Review SQL queries
command: RAILS_ENV=test SHIBA_DEBUG=true bundle exec shiba review --verbose --submit
Hi.
Getting this when trying to run tests with shiba in gemfile:
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc
runtime stack:
runtime.throw(0x4bd3b1, 0x9)
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/panic.go:596 +0x95
runtime.newosproc(0xc420030000, 0xc420040000)
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/os_linux.go:163 +0x18c
runtime.newm(0x4c3f40, 0x0)
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/proc.go:1628 +0x137
runtime.main.func1()
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/proc.go:126 +0x36
runtime.systemstack(0x529500)
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/asm_amd64.s:327 +0x79
runtime.mstart()
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/proc.go:1132
goroutine 1 [running]:
runtime.systemstack_switch()
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/asm_amd64.s:281 fp=0xc42002c788 sp=0xc42002c780
runtime.main()
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/proc.go:127 +0x6c fp=0xc42002c7e0 sp=0xc42002c788
runtime.goexit()
/usr/local/Cellar/[email protected]/1.8.3/libexec/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42002c7e8 sp=0xc42002c7e0
shiba: timed out waiting for fingerprinter on SELECT table_name FROM (SELECT table_name, table_type FROM information_schema.tables WHERE table_schema = database()) _subquery...
E
Error:
SpotsPaymentTest#test_insufficient_credit_error:
Errno::EPIPE: Broken pipe
test/unit/user_test.rb:31:in `block in <class:UserTest>'
Environment:
We should maybe download dist.html from somewhere or other...
on
shiba/lib/shiba/index_stats.rb
Lines 107 to 115 in a433c2d
there isn't a final else
condition. If the table is > 1 billion rows, we crash with:
$ bin/mysql_dump_stats ...
Traceback (most recent call last):
31: from shiba/bin/dump_stats:23:in `<main>'
30: from shiba/lib/shiba/index_stats.rb:224:in `to_yaml'
29: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/core_ext.rb:13:in `to_yaml'
28: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych.rb:441:in `dump'
27: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:118:in `push'
26: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
25: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `visit_Hash'
24: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `each'
23: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:360:in `block in visit_Hash'
22: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:134:in `accept'
21: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:518:in `dump_coder'
20: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `emit_coder'
19: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `each'
18: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:535:in `block in emit_coder'
17: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
16: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `visit_Hash'
15: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `each'
14: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:360:in `block in visit_Hash'
13: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:134:in `accept'
12: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:518:in `dump_coder'
11: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `emit_coder'
10: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `each'
9: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:535:in `block in emit_coder'
8: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
7: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:381:in `visit_Array'
6: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `visit_Enumerator'
5: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `each'
4: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `block in visit_Enumerator'
3: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:134:in `accept'
2: from /opt/rbenv/versions/2.5.3/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:517:in `dump_coder'
1: from shiba/lib/shiba/index_stats.rb:120:in `encode_with'
shiba/lib/shiba/index_stats.rb:120:in `>': comparison of Float with nil failed (ArgumentError)
Hello! Just wondering if this project is still actively maintained or roughly "archived"?
The old user
vs username
:
[1] [development] pry(main)> require 'shiba/console'
=> true
[2] [development] pry(main)> shiba User.all
Shiba::ConfigError: Required: --username
from /Users/gmartin-dempesy/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/shiba-0.9.1/lib/shiba.rb:48:in `configure_mysql_defaults'
[3] [development] pry(main)> Shiba.configure Shiba::ActiveRecordIntegration.connection_options
Shiba::ConfigError: Required: --username
from /Users/gmartin-dempesy/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/shiba-0.9.1/lib/shiba.rb:48:in `configure_mysql_defaults'
[4] [development] pry(main)> Shiba::ActiveRecordIntegration.connection_options
=> {"host"=>"zd-dev.com", "database"=>"foo_development", "user"=>"admin", "password"=>"", "port"=>3306, "server"=>"mysql"}
[5] [development] pry(main)> Shiba.configure Shiba::ActiveRecordIntegration.connection_options.merge('username' => 'admin')
=> true
I've been testing out Shiba with queries in a log file for a non-Ruby project. It seems like some queries are working with aliases but not others. Is this because there is no SQL parsing performed within Shiba yet?
Alternatively I could list the aliases as part of the /*shiba*/
metadata with each query.
dump_stats is unweildy and weird, let's try to figure out how to make it friendly and safe.
When running shiba
on travis I see the following error:
1.09s$ bundle exec shiba review --submit
fatal: ambiguous argument 'origin/HEAD...': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
/home/travis/build/OLIOEX/api/vendor/bundle/ruby/2.5.0/gems/shiba-0.9.4/lib/shiba/review/diff.rb:152:in `paths': Failed to read changes (Shiba::Error)
from /home/travis/build/OLIOEX/api/vendor/bundle/ruby/2.5.0/gems/shiba-0.9.4/lib/shiba/review/explain_diff.rb:98:in `changed_files'
from /home/travis/build/OLIOEX/api/vendor/bundle/ruby/2.5.0/gems/shiba-0.9.4/lib/shiba/review/explain_diff.rb:53:in `result'
from /home/travis/build/OLIOEX/api/vendor/bundle/ruby/2.5.0/gems/shiba-0.9.4/lib/shiba/review/cli.rb:54:in `run'
from /home/travis/build/OLIOEX/api/vendor/bundle/ruby/2.5.0/gems/shiba-0.9.4/bin/review:7:in `<main>'
I'm running the latest release (0.9.4) and shiba
is triggered by running the following in my .travis.yml
:
after_script:
- bundle exec shiba review --submit
Any idea what could be going wrong?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.