Code Monkey home page Code Monkey logo

scan's Introduction

Scan

GoDoc go test Coveralls github Report Card

Scan standard lib database rows directly to structs or slices. For the most comprehensive and up-to-date docs see the godoc

import "github.com/blockloop/scan/v2"

Examples

Multiple Rows

db, err := sql.Open("sqlite3", "database.sqlite")
rows, err := db.Query("SELECT * FROM persons")

var persons []Person
err := scan.Rows(&persons, rows)

fmt.Printf("%#v", persons)
// []Person{
//    {ID: 1, Name: "brett"},
//    {ID: 2, Name: "fred"},
//    {ID: 3, Name: "stacy"},
// }

Multiple rows of primitive type

rows, err := db.Query("SELECT name FROM persons")
var names []string
err := scan.Rows(&names, rows)

fmt.Printf("%#v", names)
// []string{
//    "brett",
//    "fred",
//    "stacy",
// }

Single row

rows, err := db.Query("SELECT * FROM persons where name = 'brett' LIMIT 1")
var person Person
err := scan.Row(&person, rows)

fmt.Printf("%#v", person)
// Person{ ID: 1, Name: "brett" }

Scalar value

rows, err := db.Query("SELECT age FROM persons where name = 'brett' LIMIT 1")
var age int8
err := scan.Row(&age, rows)

fmt.Printf("%d", age)
// 100

Nested Struct Fields (as of v2.0.0)

rows, err := db.Query(`
	SELECT person.id,person.name,company.name FROM person
	JOIN company on company.id = person.company_id
	LIMIT 1
`)

var person struct {
	ID      int    `db:"person.id"`
	Name    string `db:"person.name"`
	Company struct {
		Name string `db:"company.name"`
	}
}

err = scan.RowStrict(&person, rows)

err = json.NewEncoder(os.Stdout).Encode(&person)
// Output:
// {"ID":1,"Name":"brett","Company":{"Name":"costco"}}

Custom Column Mapping

By default, column names are mapped to and from database column names using basic title case conversion. You can override this behavior by setting ColumnsMapper and ScannerMapper to custom functions.

Strict Scanning

Both Rows and Row have strict alternatives to allow scanning to structs strictly based on their db tag. To avoid unwanted behavior you can use RowsStrict or RowStrict to scan without using field names. Any fields not tagged with the db tag will be ignored even if columns are found that match the field names.

Columns

Columns scans a struct and returns a string slice of the assumed column names based on the db tag or the struct field name respectively. To avoid assumptions, use ColumnsStrict which will only return the fields tagged with the db tag. Both Columns and ColumnsStrict are variadic. They both accept a string slice of column names to exclude from the list. It is recommended that you cache this slice.

package main

type User struct {
        ID        int64
        Name      string
        Age       int
        BirthDate string `db:"bday"`
        Zipcode   string `db:"-"`
        Store     struct {
                ID int
                // ...
        }
}

var nobody = new(User)
var userInsertCols = scan.Columns(nobody, "ID")
// []string{ "Name", "Age", "bday" }

var userSelectCols = scan.Columns(nobody)
// []string{ "ID", "Name", "Age", "bday" }

Values

Values scans a struct and returns the values associated with the provided columns. Values uses a sync.Map to cache fields of structs to greatly improve the performance of scanning types. The first time a struct is scanned it's exported fields locations are cached. When later retrieving values from the same struct it should be much faster. See Benchmarks below.

user := &User{
        ID: 1,
        Name: "Brett",
        Age: 100,
}

vals := scan.Values([]string{"ID", "Name"}, user)
// []interface{}{ 1, "Brett" }

I find that the usefulness of both Values and Columns lies within using a library such as sq.

sq.Insert(userCols...).
        Into("users").
        Values(scan.Values(userCols, &user)...)

Configuration

AutoClose: Automatically call rows.Close() after scan completes (default true)

Why

While many other projects support similar features (i.e. sqlx) scan allows you to use any database lib such as the stdlib or squirrel to write fluent SQL statements and pass the resulting rows to scan for scanning.

Benchmarks

$ go test -bench=. -benchtime=10s ./...
goos: linux
goarch: amd64
pkg: github.com/blockloop/scan
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
BenchmarkColumnsLargeStruct-8           41527964               288.0 ns/op
BenchmarkValuesLargeStruct-8             6816885              1807 ns/op
BenchmarkScanRowOneField-8               5686971              2074 ns/op
BenchmarkScanRowFiveFields-8             4962622              2381 ns/op
BenchmarkScanTenRowsOneField-8           1537761              8598 ns/op
BenchmarkScanTenRowsTenFields-8           322106             50431 ns/op
PASS
ok      github.com/blockloop/scan       92.374s

scan's People

Contributors

abelzhou avatar blockloop avatar dependabot-preview[bot] avatar dependabot[bot] avatar emarj avatar kataras avatar max-stytch avatar mzahradnicek avatar ricci2511 avatar taronish-stytch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scan's Issues

Calling `Row` does not autoclose

When calling scan.Row() it does not autoclose the rows as scan.Rows() does.
Is this intentional? I now end up having to add defer rows.Close() everywhere I used scan.Row()

Memory leak in scan.columns and scan.values

The &sync.Map{} cache mechanism used in columns.go and values.go caches individual instances of structs, not the "kind" of struct. This causes memory leaks in long-running applications that call scan.Columns or scan.Values on many different struct instances.

// PrintMemUsage outputs the current, total and OS memory being used. As well as the number
// of garage collection cycles completed.
func PrintMemUsage() {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	// For info on each, see: https://golang.org/pkg/runtime/#MemStats
	fmt.Printf("Alloc = %v MiB", bToMb(m.Alloc))
	fmt.Printf("\tTotalAlloc = %v MiB", bToMb(m.TotalAlloc))
	fmt.Printf("\tSys = %v MiB", bToMb(m.Sys))
	fmt.Printf("\tNumGC = %v\n", m.NumGC)
}
func bToMb(b uint64) uint64 {
	return b / 1024 / 1024
}

type MyObject struct {
	Value string "db:value"
}

func TestMemoryLeak(t *testing.T) {
	PrintMemUsage()
	for i := 0; i < 1e6; i++ {
		obj := &MyObject{Value: "Some value goes here"}
		cols, err := scan.ColumnsStrict(obj)
		if err != nil {
			t.Fail()
		}
		_, err = scan.Values(cols, obj)
		if err != nil {
			t.Fail()
		}
	}
	runtime.GC()
	PrintMemUsage()
}

outputs

=== RUN   TestMemoryLeak
Alloc = 4 MiB	TotalAlloc = 7 MiB	Sys = 18 MiB	NumGC = 3
Alloc = 503 MiB	TotalAlloc = 724 MiB	Sys = 619 MiB	NumGC = 13
--- PASS: TestMemoryLeak (2.45s)
PASS

Additionally, the debugger shows the columns cache map contains 100k objects.
image

Suggest either

  • changing the cache key to something that is consistent across different instances of the same struct
  • removing the cache mechanism entirely (we aren't even hitting the cache unless scan.Columns is called on the same instance multiple times)

Feature request: Support embedded structs in `scan.Columns` and `scan.Values`

Essentially #16, but for scan.Columns and scan.Values

Today it is impossible to list all columns for nested structs.

type thingy struct {
	ColA int `db:"a"`
}
type thingyWrapper struct {
	thingy
	ColB string `db:"b"`
}

func TestEmbeddedBehavior(t *testing.T) {
	thing := &thingy{}
	cols1, err := scan.Columns(thing)
	assert.NoError(t, err)
	assert.Equal(t, []string{"a"}, cols1)

	thingWrapper := &thingyWrapper{}
	cols2, err := scan.Columns(thingWrapper)
	assert.NoError(t, err)
	assert.Equal(t, []string{"a", "b"}, cols2)
}

fails with

Expected :[]string{"a", "b"}
Actual   :[]string{"b"}

bug:scan.Values(userCols, &user) is slice but need a arry

sq.Insert(userCols...).
Into("users").
Values(scan.Values(userCols, &user)...).RunWith(db).Exec()
err converting argument $1 type: unsupported type []interface {}, a slice of interface
scan.Values(userCols, &user) return a slice of interface,but args ...interface{} is a arry interface.

Dependabot couldn't find a Gopkg.toml for this project

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.

You can mention @dependabot in the comments below to contact the Dependabot team.

Cannot upgrade to v2.0.0

Hi, I`m trying to update to version 2.0.0, but go get -u points always do 1.3.0

What I`m doing wrong?

Thanks :D

bug: cached columns are returned based on the first use

scan.Columns does not take into account the excluded columns when checking values from the cache.
This means that if it is called on the same struct with different exclusion conditions, the first set of columns is returned for all additional calls.

type thingy struct {
	StrCol string `db:"str_col"`
	IntCol int    `db:"int_col"`
}

func TestExcludedBehavior(t *testing.T) {
	thing := &thingy{}
	cols1, err := scan.Columns(thing)
	assert.NoError(t, err)
	assert.Equal(t, []string{"str_col", "int_col"}, cols1)
	cols2, err := scan.Columns(thing, "int_col")
	assert.NoError(t, err)
	assert.Equal(t, []string{"str_col"}, cols2)
}
=== RUN   TestExcludedBehavior
    sql_test.go:21: 
        	Error Trace:	sql_test.go:21
        	Error:      	Not equal: 
        	            	expected: []string{"str_col"}
        	            	actual  : []string{"str_col", "int_col"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,4 @@
        	            	-([]string) (len=1) {
        	            	- (string) (len=7) "str_col"
        	            	+([]string) (len=2) {
        	            	+ (string) (len=7) "str_col",
        	            	+ (string) (len=7) "int_col"
        	            	 }
        	Test:       	TestExcludedBehavior
--- FAIL: TestExcludedBehavior (0.00s)


Expected :[]string{"str_col"}
Actual   :[]string{"str_col", "int_col"}
<Click to see difference>


FAIL

Process finished with the exit code 1

Suggest changing the cache key to be a struct { value reflect.Value, cols string } where cols is strings.Join(expected, " ").
I can open up a PR later this week.

no rows in result set

rows, _ := db.Query("SELECT * FROM persons where name = 'brett' LIMIT 1")
var person Person
// if rows.Count==0
err := scan.Row(&person, rows)
// err:sql: no rows in result set

add support for embedded structs

type Struct1 struct {
Name1 string db:"nam_e1" // no val
Name2 string db:"name2" // has val
}
type Struct2 struct {
Struct1
Name3 string db:"nam_e3" // has val
Name4 string db:"name4" // has val
}
func main(){
sql="select nam_e1,name2,nam_e3,name4"
rows, err := conn.Query(sql)
var st2 []Struct2
err = scan.Rows(&st2 , rows)
// Name1 string db:"nam_e1" // no value
// st2.nam_e1 no value
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.