Code Monkey home page Code Monkey logo

ksuid-ruby's Introduction

KSUID for Ruby

This is the repository that holds all KSUID for Ruby gems. What is a KSUID, you ask? The original readme for the Go library does a great job of explaining what they are and how you can use them, so we excerpt it here.


What is a KSUID?

KSUID is for K-Sortable Unique IDentifier. It's a way to generate globally unique IDs similar to RFC 4122 UUIDs, but contain a time component so they can be "roughly" sorted by time of creation. The remainder of the KSUID is randomly generated bytes.

Why use KSUIDs?

Distributed systems often require unique IDs. There are numerous solutions out there for doing this, so why KSUID?

1. Sortable by Timestamp

Unlike the more common choice of UUIDv4, KSUIDs contain a timestamp component that allows them to be roughly sorted by generation time. This is obviously not a strong guarantee as it depends on wall clocks, but is still incredibly useful in practice.

2. No Coordination Required

Snowflake IDs and derivatives require coordination, which significantly increases the complexity of implementation and creates operations overhead. While RFC 4122 UUIDv1 does have a time component, there aren't enough bytes of randomness to provide strong protections against duplicate ID generation.

KSUIDs use 128-bits of pseudorandom data, which provides a 64-times larger number space than the 122-bits in the well-accepted RFC 4122 UUIDv4 standard. The additional timestamp component drives down the extremely rare chance of duplication to the point of near physical infeasibility, even assuming extreme clock skew (> 24-hours) that would cause other severe anomalies.

3. Lexicographically Sortable, Portable Representations

The binary and string representations are lexicographically sortable, which allows them to be dropped into systems which do not natively support KSUIDs and retain their k-sortable characteristics.

The string representation is that it is base 62-encoded, so that they can "fit" anywhere alphanumeric strings are accepted.

How do they work?

KSUIDs are 20-bytes: a 32-bit unsigned integer UTC timestamp and a 128-bit randomly generated payload. The timestamp uses big-endian encoding, to allow lexicographic sorting. The timestamp epoch is adjusted to March 5th, 2014, providing over 100 years of useful life starting at UNIX epoch + 14e8. The payload uses a cryptographically strong pseudorandom number generator.

The string representation is fixed at 27-characters encoded using a base 62 encoding that also sorts lexicographically.


Contents of this repository

Currently, there are two gems available:

  1. KSUID for Ruby provides the main data type and handling for KSUIDs. If you want to use them outside of a database, this will be all you need.
  2. KSUID for ActiveRecord handles integrating with ActiveRecord. If you're using ActiveRecord and want to serialize KSUID columns, you will want this gem.

ksuid-ruby's People

Contributors

dependabot[bot] avatar michaelherold avatar opti avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

ksuid-ruby's Issues

Cannot use the ksuid or ksuid_binary field types when adding a column

So I just started an empty rails project this morning and tested it all out. Here's the steps to reproduce all errors.

  1. rails new project with ruby 3.2.0 and rails 7.0.8.
  2. Add to Gemfile: (also tried w/out 'ksuid' same error)
gem 'ksuid'
gem 'activerecord-ksuid', require: 'active_record/ksuid/railtie'
  1. Run rails generate model Event my_field_name:ksuid => creates error.
  2. Create a migration file and use add_column => creates error.
  3. Create a migration file and use create_table => does not create error and works as expected.

Originally posted by @DataDaoDe in #21 (comment)

Reflect from model to avoid manually including KSUID::ActiveRecord

From the README:

Then, to add proper handling to the field, you will want to mix a module into the model:

class Event < ApplicationRecord
  include KSUID::ActiveRecord[:unique_id]
end

Would it be possible to reflect the fields on a table and automate this step? That way we would not have to explicitly write include KSUID::ActiveRecord[:unique_id].

Rails 6 encounters "type "ksuid" does not exist"

Hey!

Not sure what information you need, but when trying to add the KSUID type in an existing modal, I get the following error:

rake db:migrate
/Users/XXXXXX/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.2/lib/active_record/associations.rb:1370: warning: The called method `has_many' is defined here
== 20201215203500 AddKsuidToUserProjectDownload: migrating ====================
-- add_column(:user_project_downloads, :unique_id, :ksuid)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedObject: ERROR:  type "ksuid" does not exist
LINE 1: ALTER TABLE "user_project_downloads" ADD "unique_id" ksuid
                                                             ^
/Users/XXXXXX/db/migrate/20201215203500_add_ksuid_to_user_project_download.rb:3:in `change'

Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedObject: ERROR:  type "ksuid" does not exist
LINE 1: ALTER TABLE "user_project_downloads" ADD "unique_id" ksuid
                                                             ^
/Users/XXXXXX/db/migrate/20201215203500_add_ksuid_to_user_project_download.rb:3:in `change'

Caused by:
PG::UndefinedObject: ERROR:  type "ksuid" does not exist
LINE 1: ALTER TABLE "user_project_downloads" ADD "unique_id" ksuid
                                                             ^
/Users/XXXXXX//db/migrate/20201215203500_add_ksuid_to_user_project_download.rb:3:in `change'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

With a migration that looks like this:

class AddKsuidToUserProjectDownload < ActiveRecord::Migration[6.0]
  def change
    add_column :user_project_downloads, :unique_id, :ksuid
  end
end

Make the random generator pluggable

We're currently using SecureRandom by default, but some people might have different constraints for their pseudorandomness. As such, we should make the psuedorandom number generator pluggable.

Using `include KSUID::ActiveRecord` always initializes the attribute with a new KSUID

We have been using include KSUID::ActiveRecord[:project_id] to declare KSUID attributes in models.

That's no problem for primary keys ("id"). For foreign keys this leads to the attribute initialized with a default KSUID, because of the default. OK and necessary for primary keys; not necessary and misleading for foreign keys.

I will no suggest to declare ksuid foreign keys with

attribute :project_id, :ksuid

instead.

Theoretically it would be nice if this could be derived from introspecting the relationship.

Or am I missing something?

One solution might be to have a separate concern for primary and foreign keys or to make setting the default optional somehow.

How to make ksuid work with active storage & action text?

Per this guide, if one needs uuid support, both the active storage table and the record_id reference need to be of the uuid type.

How can I make this work with ksuid?

I'm guessing ksuid is not a type pg will recognize.

It also does not seem to have an impact when I append type: :ksuid to the references statement.

In the generated schema, the record_id is still a bigint.

Please advise

activerecord-ksuid available on Rubygems?

Hi,

the readme references adding the activerecord-ksuid gem to the Gemfile, but it can't be found on rubygems and looks like it's not released. Is this the case? Is it in a production ready state?

Thanks,
Daniel

Add support for prefixes on ActiveRecord-defined fields

From this request.

This should be straightforward:

  1. During table definition for a string-serialized column, detect whether a prefix was passed and adjust the limit by the length of the prefix
  2. During table definition for a binary-serialized column, throw an exception because it wouldn't make sense to have a prefix for a binary column
  3. Extend the model mixin to handle prefixes. This will necessitate a new ActiveRecord::Type class that combines the prefix with the KSUID.
    a. The mixin should throw an error if you combine prefix: with binary: true because, again, that doesn't make sense.
    b. The new composite type probably will need some massaging to interface with created_at: true.
  4. Write up documentation about it everything.

KSUID and ActiveRecord Associations

class House
  include KSUID::ActiveRecord[:id]
  has_many :people
end

class Person
  include KSUID::ActiveRecord[:id]
  belongs_to :house
end

Person.new(house: House.last)

This throws an error by default.

Document all of the public interface

Libraries are a lot friendlier to use and work on when their public interface is documented. We should add documentation to the entire public interface for the gem. This should be done in the YARD format. We should then wire up the documentation helpers into the default rake task.

Write documentation on how to use KSUIDs as a primary key in ORMs

The highest impact guide would be using these in ActiveRecord, due to Rails' influence on the Ruby community. But it would also be interesting to see the following:

  1. Sequel
  2. ROM
  3. Mongoid - This one is maybe not worth it since MongoDB already has their ObjectIDs.

On further reflection, I won't write a Mongoid integration without being asked (or preferably helped). I haven't used Mongoid in years and don't think I would do a good job of it. In addition, the aforementioned ability of their ObjectIDs semi obviates the need for KSUIDs in that context.

Activerecord associations (:includes) not working

This happens when trying to eager load associations:

Organization.includes(:users).each do |o|
    ...
 end

Throws:

NoMethodError: undefined method `first' for nil:NilClass

The problem is in:
activerecord-6.1.4.1/lib/active_record/associations/preloader/association.rb:122:in block in records_for'`

Specifically, this line:

owner = owners_by_key[convert_key(record[association_key_name])].first

from

        def records_for(ids)
          scope.where(association_key_name => ids).load do |record|
            # Processing only the first owner
            # because the record is modified but not an owner

            owner = owners_by_key[convert_key(record[association_key_name])].first
            association = owner.association(reflection.name)
            association.set_inverse_instance(record)
          end
        end

In order to lookup associated records, Activerecord looks for the key stored as KSUID::Type. For instance, value of owners_by_key:

{<KSUID(1ym0xC8YYVoho8JmB19Y5Rsdi3O)>=>[#<Organization id: <KSUID(1ym0xC8YYVoho8JmB19Y5Rsdi3O)>...}

And the value for convert_key(record[association_key_name]):

<KSUID(1ym0xC8YYVoho8JmB19Y5Rsdi3O)>

It's the same id. But when trying to fetch that key the result is nil. I was assuming it had to do with how KSUID converts to object_id, as the two 'same' KSUIDs convert to different object_ids. Leading to not finding the key as the two object_ids are different.

I tried to define #object_id in KSUID::Type to return a static value:

    def object_id
      self.to_i
    end

However, that doesn't fix the issue and still, the key is not found in Activerecord Association's class' owners_by_key hash.

Any ideas how to fix this?

Ensure we're compatible with Segment's KSUID library

I started doing some compatibility tests and noticed that we're incompatible with the main library, even though the max is decoded properly. I have a feeling this is due to some endianness issues. My use of Array#pack and String#unpack is probably incorrect.

String builder interface?

One thing I am finding that I do a lot of is build a new KSUID and immediately coerce it to a string.

Have you thought about a public interface for building a string directly? Is there any preferable nomenclature for this sort of thing?

Use of mutation testing in ksuid-ruby - Help needed

Hello there!

My name is Ana. I noted that you use the mutation testing tool mutant in the project.
I am a postdoctoral researcher at the University of Seville (Spain), and my colleagues and I are studying how mutation testing tools are used in practice. With this aim in mind, we have analysed over 3,500 public GitHub repositories using mutation testing tools, including yours! This work has recently been published in a journal paper available at https://link.springer.com/content/pdf/10.1007/s10664-022-10177-8.pdf.

To complete this study, we are asking for your help to understand better how mutation testing is used in practice, please! We would be extremely grateful if you could contribute to this study by answering a brief survey of 21 simple questions (no more than 6 minutes). This is the link to the questionnaire https://forms.gle/FvXNrimWAsJYC1zB9.

Drop me an e-mail if you have any questions or comments ([email protected]). Thank you very much in advance!!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.