Code Monkey home page Code Monkey logo

sql_footprint's Introduction

SqlFootprint Build Status

This gem allows you to keep a "footprint" of the sql queries that your application runs. It's like logging all the sql you're executing except that we remove all the value parameters and dedupe similar queries. This footprint should be valuable in determining if changes you've made will significantly change the way you're querying the database.

Installation

Add this line to your application's Gemfile:

gem 'sql_footprint', group: [:development, :test]

And then execute:

$ bundle

Usage

Typically, you would want to run this while you're running your specs. For example w/ RSpec:

RSpec.configure do |config|
  config.before(:suite) { SqlFootprint.start }
  config.after(:suite) { SqlFootprint.stop }
end

Minitest (in test_helper.rb) add the following:

SqlFootprint.start
Minitest.after_run { SqlFootprint.stop }

You can also add a Custom rule to SqlAnonymizer before running start:

RSpec.configure do |config|
  SqlFootprint::SqlAnonymizer.add_rule(/SELECT (.+) AS (.+)/, 'SELECT [redacted] AS [redacted]')
  config.before(:suite) { SqlFootprint.start }
  config.after(:suite) { SqlFootprint.stop }
end

Outputs

After running your specs you'll find a 'footprint.*.sql' file in your project. Footprints are per-database. For example, if you're using DB1 AND DB2 in your app, you would end up with two footprint files. (footprint.db1.sql, footprint.db2.sql)

If you're using an in-memory database, you'll end up with footprint.:memory:.sql.

Excluding Setup Code

If you want to exclude queries that your tests generate for fixture data, you can use the .exclude method. For example:

before do
  SqlFootprint.exclude do
    Model.create!(args*) # this query will not be included in your footprint
  end
end

Or if you're using FactoryBot you could do something like this:

RSpec.configure do |config|
  module FactoryKid
    def create(*args)
      SqlFootprint.exclude { FactoryBot.create(*args) }
    end
  end
  config.include FactoryKid
end

Compatibility

  • For Rails < 6.0 compatibility, please use v2.0.1.
  • For Rails >= 6.0 compatibility, please use v3.0.0.
  • For Rails >= 7.0, please use v3.0.1

DO NOT run SqlFootprint in production!

sql_footprint's People

Contributors

amygurski avatar brandonjoyce avatar cartoloupe avatar cmmgurski avatar davidmyersdev avatar jrs40492 avatar ksearfos avatar mikegee avatar olleolleolle avatar schneiderderek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sql_footprint's Issues

Exclude some SQL

Our application should be read-only, except for its own bookkeeping. I don't want to see INSERT statements from the application specs when it's only the specs that create records.

I'm thinking this API would be nice:

before do
  SqlFootprint.exclude do
    Widget.create!
  end
end

Running subset of test suite results in subset of collected data

We only want to collect data when the entire test suite is ran. The problem is that when we run a single spec or a subset of the specs, SqlFootprint will incorrectly remove the data that was collected for the specs that were not ran. We frequently have to reset the changes made to the db/footprint.test.sqlite3.sql because we didn't run the entire test suite.

Some ideas to fix this:

  • Configure SqlFootprint to be able to opt-out:
RSpec.configure do |config|
  unless ENV['NOFOOTPRINT']
    config.before(:suite) { SqlFootprint.start }
    config.after(:suite) { SqlFootprint.stop }
  end
end

We can then disable SqlFootprint with the NOFOOTPRINT environment variable. Developers can disable SqlFootprint in their shell or whenever they run a subset of specs.

  • Do the opposite of above and opt-in to the behavior. SqlFootprint would be disabled by default. We can have our CI opt-in and leave build artifacts to be analyzed later. This would be preferred since developers wouldn't have to maintain the generated file.
  • Hook into Rspec runner (or rake task) and enable SqlFootprint if the entire suite is being ran.

LOWER('anything') not being anonymized

Looks like we need to improve the regex to account for this. Example query that's not being anonymized correctly.

SELECT  1 AS one FROM "customers"  WHERE LOWER("users"."first_name") = LOWER('brandon') LIMIT 1

Adding uniqueness constraint results in non-deduped entries

Added the following validation to our QueueItem model:

validates :incoming_fax_id, uniqueness: true

resulted in

+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 1) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 11) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 12) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 13) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 14) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 15) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 16) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 17) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 18) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 19) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 22) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 38) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 39) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 4) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 42) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 43) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 44) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 5) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 50) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 51) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 53) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 54) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 6) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 66) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 67) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 68) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 69) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 7) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 70) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 71) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" = number-redacted AND "queue_items"."id" != 84) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 1) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 15) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 16) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 17) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 18) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 2) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 28) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 29) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 3) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 30) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 31) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 32) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 33) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 34) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 35) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 36) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 4) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 5) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 68) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 69) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 70) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 71) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 72) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 73) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 74) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 75) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 76) LIMIT 1
+SELECT  1 AS one FROM "queue_items" WHERE ("queue_items"."incoming_fax_id" IS NULL AND "queue_items"."id" != 83) LIMIT 1

Old and new footprints: footprints.zip

Some junk doesn't get deduped

See more in attached files. Includes current footprint and the git diff.
footprint.zip

-Identifier Exists (1.4ms)
+Identifier Exists (0.5ms)
...
-SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE '0ea728%') LIMIT 20
-SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE '7b4637c7-9841-4557-bfa6-c695c1f%') AND ("NPPESProvider"."provider_last_name_legal_name" LIKE 'e6805e55-44f0-48fe-bc4b-862d4c1%') LIMIT 20
-SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE 'a0010f%') LIMIT 20
-SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE 'a8224293-6676-44c0-a178-45be168%') AND ("NPPESProvider"."provider_last_name_legal_name" LIKE '0dd7be56-13fc-4f99-a93f-7aec831%') LIMIT 20
-SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_last_name_legal_name" LIKE 'c00eac%') LIMIT 20
-SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_last_name_legal_name" LIKE 'fa3593%') LIMIT 20
+SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE '29ddc5%') LIMIT 20
+SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE '71568d%') LIMIT 20
+SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE '9cb072f0-8828-4fdb-b0a5-a353ed9%') AND ("NPPESProvider"."provider_last_name_legal_name" LIKE 'bf0902e3-a31f-413c-b54a-3db4c9a%') LIMIT 20
+SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_first_name" LIKE '9f526dfe-128e-4328-b96c-c1635a0%') AND ("NPPESProvider"."provider_last_name_legal_name" LIKE 'cacc8c97-7c13-453d-acf3-13c70be%') LIMIT 20
+SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_last_name_legal_name" LIKE '64ac45%') LIMIT 20
+SELECT  DISTINCT "NPPESProvider".* FROM "NPPESProvider"  WHERE ("NPPESProvider"."provider_last_name_legal_name" LIKE 'c01735%') LIMIT 20

Footprint changes when list of things is not in same order

-TRUNCATE TABLE "public"."queue_items", "public"."notes", "public"."sessions", "public"."incoming_faxes", "public"."outbound_faxes", "public"."users", "public"."status_histories", "public"."fax_poller_results" RESTART IDENTITY CASCADE;
+TRUNCATE TABLE "public"."queue_items", "public"."notes", "public"."sessions", "public"."fax_poller_results", "public"."users", "public"."status_histories", "public"."incoming_faxes", "public"."outbound_faxes" RESTART IDENTITY CASCADE;

also
screen shot 2016-04-26 at 2 42 53 pm

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.