From the README, sonyflake uses 63 bits. This aligns with the design of Snowflake ID, which is specifically designed to fit in a signed integer:

Snowflakes are 64 bits. (Only 63 are used to fit in a signed integer.)

I notice that sonyflake returns uint64 rather than int64:

Should it return int64, instead?

I assume not since it has been around for a while. If that's correct, perhaps some documentation could be added to describe how to safely convert the value into an int64 so that it might more easily be used in databases that don't support uint?

in this scenario, NextID() could block very long time.

1. in a high load: (sf.sequence + 1) & maskSequence == 0
2. in the same time, system clock drift backwards, say 8 hours

this will block the function for 8 hour.

can we use some kind of Monotonic time or add a new function with timeout-control to avoid this?

Deleted due to my wrong testcase code

Sorry, my fault. My eyes blind...

how to ensure time synchronization In the cluster environment, have any good solution

Hi!

We would like to know if sonyflakes generated from now are a guaranteed sort order and if not then when is the next time that length will change. We have noticed an issue in our database where our sort order is being ill-computed based on that we sometimes store sonyflakes as strings. Could you kindly affirm whether the length of a sonyflake string can change? Or is there a guaranteed length.

In particular the following algorithm...

l = [
    "1",
    "9999",
    "2",
    "10000",
    "3",
    "4",
    "999",
    "6",
    "7",
    "8",
    "9",
    "5",
    "10",
    "100",
    "1000",
    "90",
]

print(l)
l.sort()
print(l)

Shows that say the number 2 > 1000000000000000 when doing string comparison. I would like to know how Sonyflakes generated reliably are affected by this as the impact of this could affect the sort ordering for some production data (may require migration). Due to the algorithmic complexity of the Sonyflake and how its generated (sonyflakes are always the same length I believe) I'm not sure if we would be impacted by this in the near future.

Go example

package main

import (
    "fmt"
    "sort"
)

func main() {
    sl := []string{"1", "9999", "2", "10000", "3", "4", "999", "6", "7", "8", "9", "5", "10", "100", "1000", "90"}
    fmt.Println(sl)
    sort.Strings(sl)
    fmt.Println(sl)
}

I appreciate that this is not your concern and represents a mistake made in our systems, however I would appreciate any help on the matter. It would be helpful if I can run unit tests locally to generate snowflake now to the range generate one years in the future, but I have no idea how to adjust the snowflake without changing my system time like this. Appreciate the easiest way to diagnose the impact!

func TestNewID(t *testing.T) {
	total := 100000
	m := &sync.Map{}
	s := sonyflake.NewSonyflake(sonyflake.Settings{StartTime: time.Now()})
	queue := make(chan int, 10)
	for i := 0; i < 1000; i++ {
		go makeID(s, queue, m)
	}
	for i := 0; i < total; i++ {
		queue <- i
	}
	close(queue)
	var num int64 = 0
	m.Range(func(key, value interface{}) bool {
		atomic.AddInt64(&num, 1)
		return true
	})
	t.Logf("a: %d, b: %d", total, num)
}

func makeID(s *sonyflake.Sonyflake,queue <- chan int, m *sync.Map){
	for  {
		select {
		case i := <- queue:
			id, _ := s.NextID()
			m.Store(id,i)
		}
	}
}

result：a: 100000, b: 99065

When using sonyflake on a host with an IP address in the 100.64.0.0/10 space, a nil is returned.

The isPrivateIPv4 function does not recognize IP in 100.64.0.0/10 as private IP addresses.

Found this bug when using sonyflake in an app running on Kubernetes with Calico as an overlay network. Calico uses IP addresses from the 100.64.0.0/10 space for its pods.

Hi,

Thanks for open sourcing this.
I've read the source code and it seems that here https://github.com/sony/sonyflake/blob/master/awsutil/awsutil.go#L16 the IP address / URL is hard coded and can't be changed by the user. Shouldn't this be part of some sort of parameter one initializes the package with?

I can work on a PR to do the following changes:

• change the URL to a variable
• create an exported package function called awsutil.Init(url string) which should receive the URL as an argument.

However this would be a breaking change and I'd like to know your thoughts on this one first.

Just curious to know, how practical is it to limit generation of 2^8 (256) unique IDs per 10ms on databases.
Can someone please explain me with example like if I have 4GB RAM to 16GB RAM database instance, how many insert load I can expect to be handled by that server?

It will help me use sonyflake as id generation at database level.

my benchmark result.

BenchmarkMustNewID-8   	   50000	     38982 ns/op	       0 B/op

var snowflake *sonyflake.Sonyflake

func init() {
	st := sonyflake.Settings{
		StartTime: time.Date(2019, 07, 29, 15, 43, 0, 0, time.Local),
		CheckMachineID: func(machineId uint16) bool {
			return true
		},
	}

        snowflake = sonyflake.NewSonyflake(st)
	if snowflake == nil {
		err := fmt.Errorf("Invalid snowflake settings %v", st)
                panic(err)
	}
}

func MustNewID() uint64 {
	id, err := snowflake.NextID()
	if err != nil {
		panic(err)
	}
	return id
}

func BenchmarkMustNewID(b *testing.B) {
	for i := 0; i < b.N; i++ {
		MustNewID()
	}
}

Seems like when I try to use the sonyflake.Decompose on the generated ID, I get a timestamp which is similar to 24869724975. How can I convert this into an epoch timestamp so a I can sort by date in the DB? What I need is to be able to generate a timestamp from the current date and compare to the timestamps stored in the DBs as primary keys

So I can't use it as chronological order?

an error occurred when I new Sonyflake.
the file is github.com/sony/[email protected]/sonyflake.go.

i'm use MacOs 14.4.1.

After debugging, I found that the isPrivateIPv4() check ip error under lower16BitPrivateIP() method.
i not init MachineID when i new Sonyflake, so it will run this code :
var err error if st.MachineID == nil { sf.machineID, err = lower16BitPrivateIP(defaultInterfaceAddrs) } else { sf.machineID, err = st.MachineID() } if err != nil { return nil, err }

error Info
`
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11a1e92]

goroutine 6 [running]:
testing.tRunner.func1.2({0x1265300, 0x14d52d0})
/usr/local/go/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1548 +0x397
panic({0x1265300?, 0x14d52d0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/sony/sonyflake.(*Sonyflake).NextID(0x0)
/Users/pkg/mod/github.com/sony/[email protected]/sonyflake.go:116 +0x32

`

How is the performance of generating IDs with this library compared to the others out there?

I have a init function:

func init() {
	sf = sonyflake.NewSonyflake(sonyflake.Settings{
		StartTime: time.Now(),
	})
	if sf == nil {
		panic("sonyflake is not create")
	}
	fmt.Println("Sonyflake is initialized")
}

I works right on my macOS(M2) system.

Now I have to compile it to linux/amd64 architecture, when I put the compiled executable program to the Ubuntu system, it can't not be run, and into the panic("sonyflake is not create").

So that means sonyflake is not compatible with linux/amd64 architecture.

func sleepTime(overtime int64) time.Duration { return time.Duration(overtime)*10*time.Millisecond - time.Duration(time.Now().UTC().UnixNano()%sonyflakeTimeUnit)*time.Nanosecond }

i think overtime is no need minus the now time, so i think below is right:
func sleepTime(overtime int64) time.Duration { return time.Duration(overtime) * 10 * time.Millisecond }

I'd like to use this code in an AWS Lambda. Is it possible to get a unique 16 bit id for the lambda instance it is running in? The equivalent of AmazonEC2MachineID() but for Lambdas.

As I understand it - Lambda's are limited to 1000 instances - so if we could get something like a "slot" number, ir a unique number from 0 to 999, 1 to 1000 etc - that would be even better and we could reduce the size of the server id field.

@yuokada (or anyone) please let me know if this is possible. I would pay for help with this.

So if each EC2 instance has a unique private IP address in AWS VPC, the lower 16 bits of the address is also unique

why?

At the moment, the sonyflake.NewSonyflake() function only returns a *sonyflake.Sonyflake. While there aren't many reasons for the return value to be nil, it would be great to know what caused a potential error.

Given this package may be in use by a lot of people and in order to keep BC, I propose a new function with two return values (*sonyflake.Sonyflake, error) and mark the current function as deprecated.

Thoughts?

this is my code

for i:=0;i<100;i++ {
	go func() {
		flake := sonyflake.NewSonyflake(sonyflake.Settings{})
                pid, _ = flake.NextID()
		pidStr := strconv.FormatInt(int64(pid),10)
		t := utils.Md5(pidStr + time.Now().Format("2006-01-02 15:04:05"))
		session_key := t
		openid := t
		tpRes, _ := models.PersionLogin(int64(pid), t, session_key, openid)
	}()
}

result, pid has repeat

Hi,

Thanks for the library. It would be nice to get the time{} struct the sonyflake is based off so that implementing services can prevent drift between recording current time and time allocated to the sonyFlake.

Please let me know if this is something you can do.

Cheers!

I see that it leaves most significant bit unused. That shall make it safe for use with 64 bit signed integer, because it will never go into the negative range from DB perspective (and will stay sortable).

Am I reading it right?