Code Monkey home page Code Monkey logo

goscale's Introduction

SCALE codec implementation in Go compatible with a Tinygo based toolchain

codecov

The SCALE types in Go are represented by a set of custom-defined types that implement the Encodable interface. Each type also has a corresponding decode function using the convention Decode<TypeName>. Note that the type to which data should be decoded is inferred from the context and is not self-contained in the SCALE-encoded data.

One exception is the Tuple type. It doesn't have methods attached. Instead, there are EncodeTuple and DecodeTuple functions that can be invoked with any custom struct that embeds the Tuple interface.

Some quirks deserve mention. For example, the FixedSequence type, which has the same representation as the Sequence type, facilitates the encoding of arrays. As arrays are fixed-size sequences, they cannot be encoded as the Sequence type. Note that there are no type checks on the size.

The use of custom-defined types and generics reduces reliance on reflection, which isn't fully supported by TinyGo.


SCALE/Rust Go
bool goscale.Bool
SCALE/Rust Go
i8 goscale.I8
u8 goscale.U8
i16 goscale.I16
u16 goscale.U16
i32 goscale.I32
u32 goscale.U32
i64 goscale.I64
u64 goscale.U64
i128 goscale.I128
u128 goscale.U128
SCALE/Rust Go
Compact<u8> goscale.Compact
Compact<u16> goscale.Compact
Compact<u32> goscale.Compact
Compact<u64> goscale.Compact
Compact<u128> goscale.Compact
SCALE/Rust Go
bytes goscale.Sequence[U8]
[u8; u8] goscale.FixedSequence[U8]
string goscale.Str
SCALE/Rust Go
goscale.Dictionary[K, V]
SCALE/Rust Go
goscale.Empty
SCALE/Rust Go
Enumeration(tagged-union) goscale.VaryingData
SCALE/Rust Go
Option<bool> Option[goscale.Bool]
Option<i8> Option[goscale.I8]
Option<u8> Option[goscale.U8]
Option<i16> Option[goscale.I16]
Option<u16> Option[goscale.U16]
Option<i32> Option[goscale.I32]
Option<u32> Option[goscale.U32]
Option<i64> Option[goscale.I64]
Option<u64> Option[goscale.U64]
Option<i128> Option[goscale.I128]
Option<u128> Option[goscale.U128]
Option<bytes> Option[Sequence[U8]]
OptionBool OptionBool
None nil
SCALE/Rust Go
Result<EncodeLike, EncodeLike> goscale.Result[goscale.Encodable]

Go structs are encoded as SCALE Tuple, where each struct field is encoded in a sequence containing all the fields. To decode SCALE encoded structs, it is required to have prior knowledge of the destination data type.

SCALE Go
struct goscale.Tuple

Run Tests

go test -v

goscale's People

Contributors

radkomih avatar failfmi avatar nicksinch avatar

Stargazers

Jarrian Gojar avatar Daniel Ivanov avatar

Watchers

George Spasov avatar Tugay Emin avatar Christian Veselinov avatar  avatar  avatar  avatar

goscale's Issues

finish implementing the rest of the numeric interface methods.

SaturatingDiv(other Numeric) Numeric
SaturatingMod(other Numeric) Numeric

CheckedAdd(other Numeric) (Numeric, error)
CheckedSub(other Numeric) (Numeric, error)
CheckedMul(other Numeric) (Numeric, error)
CheckedDiv(other Numeric) (Numeric, error)
CheckedMod(other Numeric) (Numeric, error)

And(other Numeric) Numeric
Or(other Numeric) Numeric
Xor(other Numeric) Numeric
Not() Numeric
Lsh(other Numeric) Numeric
Rsh(other Numeric) Numeric

EncodeTuple always encodes Options' values

Regardless of whether the Option has value or not, it will always encode the value (default if empty)

type TupleOption struct {
	Tuple
	M0 Option[U8]
	M1 Option[Bool]
	M2 Option[Str]
}

func Test_EncodeTupleOption(t *testing.T) {
	var testExamples = []struct {
		label       string
		input       TupleOption
		expectation []byte
	}{
		{
			label: "TupleOption",
			input: TupleOption{
				M0: Option[U8]{true, 3},
				M1: Option[Bool]{true, false},
				M2: NewOption[Str](nil),
			},
			expectation: []byte{
				0x01, 0x03, // M0
				0x01, 0x00, // M1
				0x00, // M2
			},
		},
	}

	for _, testExample := range testExamples {
		t.Run(testExample.label, func(t *testing.T) {
			buffer := &bytes.Buffer{}

			EncodeTuple(testExample.input, buffer)

			assertEqual(t, buffer.Bytes(), testExample.expectation)
		})
	}
}

Decode methods does not return error, just panic.

Panicking is not particularly helpful, especially in the Runtime. Instead, errors should be passed on to the Host. To accomplish that, all Decode functions should be refactored to return an error in addition to their current output, rather than simply panicking.

Example:

func DecodeBool(buffer *bytes.Buffer) Bool

should be

func DecodeBool(buffer *bytes.Buffer) (Bool, error)

SCALE Sequence(fixed size) implementation

Array types have fixed size known at compile time, thus the size encoding is omitted.

From the spec:
In some cases, the length indicator is omitted if the length of the sequence is fixed and known by the decoder upfront. Such cases are explicitly stated by the definition of the corresponding type.

Go proposal that might help to make generic parameterized encoding of Array (Sequence with fixed size): golang/go#44253

Related: #4

hitting the max possible allocation size when U8 encoding with the custom gc

Since U8 is used extensively, e.g. Sequence[U8], invoking value.Bytes() to return the bytes causes a significant performance overhead with the custom gc, and it tries to allocate single allocation bigger than the max possible.

func (value U8) Encode(buffer *bytes.Buffer) {
	encoder := Encoder{Writer: buffer}
	encoder.Write(value.Bytes())
}

func (value U8) Bytes() []byte {
	return []byte{byte(value)}
}

Instead encode the bytes directly

encoder.EncodeByte(byte(value))

Specifying the error type E for type Result[T, E] + initializer

Declarations of Option[T] and Result[T, E] value types can specify custom-defined types for T and E.

Example:

type MyOk struct {}
type MyError struct {}

...

buffer := &bytes.Buffer{}
Result[MyOk, MyError]{}.Encode(buffer)

Currently, DecodeResult relies on the convention that for each type there is a Decode<TypeName> function. It invokes the decode function only for the predefined types, not for custom-defined ones.

res := DecodeResult[MyOk, MyError](buffer)

We need to be able to dynamically build/call the corresponding decode function for the specified type using reflection.

Approach 1: Obtain the full package name of the corresponding function. We could potentially derive it from the Type. However, we'll rely on the convention that the decode function is defined in the same package.

Approach 2: Include a Decode method as part of the interface, so there will be type checks, and each type will have a Decode method that can be called using reflection.

Related #62

helper function that converts any sequence type to bytes

Implement a generic function that converts any Sequence or FixedSequence to bytes

Previously there were some issues with the compiler #38.

type SequentialU8 interface {
	Sequence[U8] | FixedSequence[U8]
}

func SequentialU8ToBytes[S SequentialU8](bytes S) []byte {
	result := make([]byte, len(bytes))
	for i, v := range bytes {
		result[i] = byte(v)
	}
	return result
}

complete the numeric interface implementation and i128

  • finish the implementation of I128
  • revise the implementation of all Operations (Add, Mul, Div, etc.) for all types U/I 8/16/32/64 and U/I 128 to have similar behavior
    • Operation - wraps around
    • SaturatingOperation - saturates at the boundaries
    • CheckedOperation - returns error
  • rename NewU128FromUint64 to NewU128
  • add support for passing any numeric type to NewU128/NewI128
  • add support for passing string literals NewU128("12749827981791739187398173981")
  • finish the implementation of To[N] (revise type conversion)
  • add tests
  • inline docs
    * [ ] extract it into a separate package (num or numeric), for ease of use and rename NewNumeric to New
    ex: num.New[U128]
    ex: num.To[U64]

related #82

SCALE Tuple type implementation

  • encoding struct with fields of any SCALE type (using the default fields order)

  • decoding struct with fields of any SCALE type (using the default fields order)

    • panic("unimplemented: reflect.MakeSlice()")
    • panic("unimplemented: (reflect.Value).Addr()")
  • struct tags - allow modification of the struct field ordering, that preserve the encoding/decoding order.

Check if the bug is in Tinygo

func StringToSliceU8(s string) []U8 {
	result := make([]U8, len(s))
	for _, v := range []byte(s) {
		result = append(result, U8(v)) // panic: cannot convert pointer to integer -> /tinygo/interp/memory.go:541
	}
	return result
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.