Code Monkey home page Code Monkey logo

bitwise's Introduction

Announcing Bitwise

Feynman's blackboard quote "What I cannot create, I do not understand" never fails to give me goosebumps.

I've always been obsessed with how things work under the hood--physical contraptions as well as abstract concepts. Most engineers and programmers I've met are motivated by building things, and their knowledge and skills are first and foremost a tool for achieving that. But I'm motivated by learning how things work, and the process of solving problems and designing and implementing systems has been a tool for increasing my understanding. And of course there's the unrivaled thrill of seeing what you've built actually work and come alive!

After working as a game developer and systems programmer for over 15 years at places like Epic Games, NVIDIA, RAD Game Tools and most recently Oculus, I decided it was time to take a break from professional programming and spend a few years pursuing a long-time dream of mine, a project I've dubbed Bitwise, where I want to share my passion and try to demonstrate by example how to build systems from scratch, with a low-level computing focus.

Here are some examples of what you will learn to build:

  • Hardware description language compilers and simulators
  • FPGA-based hardware, including CPU, GPU, HDMI controller, Ethernet MAC, DDR3 PHY, etc
  • Kernels, including drivers, hardware abstraction layer, scheduler, virtual memory manager, file systems, TCP/IP stack, etc
  • Systems libraries, including GUIs, memory allocators, etc.
  • Systems applications, including compilers, assemblers, disassemblers, profilers, debuggers, etc
  • Test infrastructure, including property-based testing, directed randomized testing, fuzz testing, etc.

My goal with Bitwise is to show that these things can be done much more simply and quickly than people realize if we strongly favor simplicity over marginal gains in feature completeness or performance. The goal is not to outdo or compete with any existing product; the goal is to show how things work with real hardware and software.

Bitwise will consist of educational videos, articles, and software/hardware source code. The code will be released into the public domain for everyone to study and extend as they see fit. Everything produced as part of the project is free of charge. I'm doing this project as a service to the community and neither ask nor accept financial donations.

Despite the hardware focus, I want to make it clear almost nothing we do will require any special hardware or FPGA development boards for you to participate. All our development will be done with free tools that run on normal Windows, Linux or Mac computers. I will periodically demonstrate code running on real FPGA boards, but it will mostly be for debugging purposes. In fact, we want to minimize the time we spend dealing with real hardware since it's usually painful and time intensive compared to running in simulation.

Structure

NOTE: This is highly subject to change based on feedback from the community.

My plan is to treat Bitwise as a full-time commitment for several years. Avoiding burnout is a major priority.

I will be streaming daily or semidaily on Twitch, one or two hours per session. Videos will be posted later on YouTube.

When I'm not streaming, I will be working full-time on the project: programming, writing and other related activities.

I will be writing weekly or semiweekly blog posts summarizing the progress since the last update.

Code streams will begin with a code review and walk-through of the diffs that were committed since last stream. Thus, even though a lot of the code will necessarily be written off stream, you won't miss out on a single line of code. After each review, I will push the latest changes to the GitHub repository, where everyone can download them.

To keep things fun for myself and viewers, the streams will frequently alternate between different development tracks. In a given week, streams might alternate between the primary software and hardware development tasks, but you can expect streams on random side topics to be commonplace. If you're not interested in some topics, there will likely be others to your liking.

Most streams will be focused on live coding, while others will be in a more conventional presentation format.

Aside from the progress summary blog posts, I will try to distill what I cover on stream into standalone articles. Writing articles is a lot of work, but my hope is that by using my overviews from the stream as a template, I will be able to write them more quickly than if I had to start with a blank page. The audience for articles is much larger, so this will extend the project's reach, and it helps prospective or lapsed viewers to catch up to the streams; it's not practical for most people to catch up by watching archived streams.

Prerequisites

The intended audience are fluent C programmers. Python is used as a secondary language, but knowing Python in advance is not a prerequisite. The pace will be quick; while I will give overviews of many different topics, not everything can be covered in depth, so pointers to additional reading material will be provided whenever possible.

Schedule

The expected stream schedule is Monday through Friday, 6 PM PDT/9 PM EDT/2 AM CET.

The kick-off stream is planned for March 12. I expect it will mainly be an overview and Q&A.

The first major project will be the C-like systems language compiler. I expect the initial work on this to be a few weeks, after which we will start working on the hardware track as well.

I'm located in Thailand, so the schedule is my attempt at juggling time zone differences. I may balance it out by moving some streams to Saturdays or Sundays during morning or noon hours for Europeans. I will be spending a month in Europe each year in August, which will force a rotation of the schedule.

Roadmap

NOTE: This is subject to change, but it gives you an idea of what I have planned. The outline corresponds to an estimated six months of work, depending on my productivity and how much we decide to focus on the critical path versus side topics.

We will be building not only software but hardware.

On the hardware side of things, we will be designing a computer from scratch that can be synthesized and deployed on a real FPGA. This will include a RISC-V CPU, GPU, IO interface controllers for DDR3 memory, HDMI video/audio, Ethernet networking, and more.

But before getting there, we will need to learn about digital logic and how to design hardware with an HDL. In fact, we will designing our own HDL and the associated toolchain, including software-based simulators and debugging tools. Along the way we'll be doing lots of fun mini-projects, like designing a version of Pong entirely using logic gates, and implementing a variant of the toy parallel computer used in the Zachtronics game TIS-100.

In tandem, we will also be building the software stack. This includes both the host-side tooling as well as the software that runs on the Bitwise computer. Eventually most of the host tooling will be portable to the Bitwise computer itself! This toolchain will start with a simple but powerful C-like systems programming language that will initially be bootstrapped to run on the host and which we will use to construct all the other software. As we first bring up the CPU, we will be writing the emulator, assembler, disassembler and debugger in this language.

From there we'll be writing test programs in the assembly language to validate the CPU. Once the CPU is in a usable state, we will code a backend for our systems language compiler that targets our CPU and so start running simple compiled programs on it. At that point we will build a simple microcontroller-class task-switching operating system and some simple applications on top of it.

At first, the CPU will be a simple RV32I microcontroller-class core with limited features and performance, and the only IO peripheral will be a UART for communicating with the host. Over time we will make this more capable. We will extend the CPU to eventually support RISC-V's RV32G profile, which includes support for integer multiply/divide and floating point instructions. To improve performance we will add instruction pipelining and branch prediction, and caches once we have DRAM support. We will also be building out our peripherals to support audio/video output and networking, and the software to support this will be constructed in parallel as the hardware comes online.

Eventually we will end up with a simple desktop-class operating system with a graphical user interface that can run games and other applications we've written. We might eventually try to bring up Linux on the computer once we've implemented MMU support, but that is not a primary goal.

And this is only the beginning!

Links

Please follow on Twitch, YouTube and Twitter to see when streams go live and when new videos are uploaded.

There is a Discord chat server if you have questions or just want to hang out and chat.

bitwise's People

Contributors

cpl avatar croepha avatar davidegrayson avatar fierydrake avatar liorda avatar maqtech avatar mcavanagh avatar paralax avatar pervognsen avatar stevetranby avatar tbodt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bitwise's Issues

Please use 4K

I remember you said the recording setup is wip and will improve. So feel free to close.

Nevertheless Iโ€˜d like to point out that 4K would be a good option, makes the image/ text look much cleaner, YouTube supports it and especially you said this is a resource for future visitors and in 5 years 1080 will be a crutch and not appropriate for the quality content of the videos.

If 4K is not an option for some reason maybe at least 1440p would do for some time.

Thanks

Incorrect type info for void const * type

@foreign
func memcpy(dst: void*, src: void const *, size: usize): void*;

compiles ok, but outputs

[1] = &(TypeInfo){TYPE_VOID, .name = "void"},
  ...
[35] = &(TypeInfo){TYPE_CONST, .size = sizeof(void const ), .align = alignof(void const ), .base = 1},

(note the missing *), leading to the C compiler to report:

main.ion(292): warning C4034: sizeof returns 0
main.ion(292): error C2714: alignof(void) is not allowed

The line number is also off (past the end of the ion file), but this might be a knock-on effect.

ion second try

#24 (comment)

I'm currently a bit behind your schedule so I might have missed some information, but are there news about

  • defer statement
  • no ; needed
  • formatter
  • C output/ C single header output
  • completion plugins/ output
  • renaming
    .. being added to ion?

thanks

Is this project still alive?

Hi author,

I subscribed your youtube channel, and this repository. However, I haven't seen any activities on this project since last 4 months.

Can you let me know: will you continue on this project?

syntax error: 'cv-qualifier' (TypeInfo type_to_cdecl for const structs containing array)

With the following:

struct FooArray
{
    x : int[3];
}

struct FooWithConstArray
{
    array : FooArray const*;
}


func test_const_members() {
    foo := FooWithConstArray{};
    #assert(foo.array.x[0] == 0);
}

Then when TypeInfo gets written, ion generates:

    [53] = &(TypeInfo){TYPE_CONST, .size = sizeof(int (const [3])), .align = alignof(int (const [3])), .base = TYPEID(50, TYPE_ARRAY, int [3])},

the int (const [3]) is not parsed as a valid type expression

Somehow the type entry seems invalid to me, as all arrays in C are const. So the type info should probably not be generated at all (TYPE_CONST with base TYPE_ARRAY)

type_to_cdecl could be instrumented to assert that a TYPE_CONST should not be an array

TypeInfo generated for undeclared types

Steps to reproduce:

  1. Create a package libfoo with one file with the following contents:
struct Foo {
    x, y: int;
}

func bar() {
    foo: Foo;
    foo_p := &foo;
    foo_pp := &foo_p; // This triggers the type info generation for 'libfoo_Foo'
}
  1. Create a second package with a main method and import libfoo {...}

Ion compilation works, but the generated C will error on compilation with error C2065: 'libfoo_Foo': undeclared identifier.

You'll get a line similar to:
[190] = &(TypeInfo){TYPE_PTR, .size = sizeof(void *), .align = alignof(void *), .base = TYPEID(53, TYPE_PTR, libfoo_Foo *)},

If the struct Foo isn't directly used in the main package, the C struct def for Foo will never be generated.

void next_token() with switch instead of if ?

Why does the next_token() function use such a huge switch case when it could be just a few if statements? Just curious if this is just a personal preference of @pervognsen or something else.

// This is how I would see this done.
void next_token() {
    if (*stream >= '0' && *stream <= '9') {
        uint64_t val = 0;
        while (isdigit(*stream)) {
            val *= 10;
            val += *stream++ - '0';
        }
        token.kind = TOKEN_INT;
        token.val = val;
    } else if ((*stream >= 'a' && *stream <= 'z') || (*stream >= 'A' && *stream <= 'Z') || *stream == '_') {
        const char *start = stream++;
        while (isalnum(*stream) || *stream == '_') {
            stream++;
        }
        token.kind = TOKEN_NAME;
        token.start = start;
        token.end = stream;
    } else {
        token.kind = *stream++;
    }
}

Warnings about `foreign` and `static_assert` being unknown on Linux

When I compile Ion with GCC on Linux, it prints a bunch of warnings from builtin/types.ion. It's been happening for a while now. Here is a shell session reproducing the issue.

david@davidvm5 ~/bitwise (per_master)
$ git log -1
commit b5c2d0ad7132e666a69e402019e8731d204e2445 (HEAD -> per_master, origin/master, origin/HEAD)
Author: Per Vognsen
Date:   Wed Jun 6 16:15:23 2018 +0700

    #define __USE_MINGW_ANSI_STDIO 1 to support mingw-gcc
david@davidvm5 ~/bitwise (per_master)
$ cd ion
david@davidvm5 ~/bitwise/ion (per_master)
$ gcc main.c -o ion
david@davidvm5 ~/bitwise/ion (per_master)
$ cd ..
david@davidvm5 ~/bitwise (per_master)
$ echo $IONHOME
/home/david/bitwise/ion
david@davidvm5 ~/bitwise (per_master)
$ ./ion/ion testpkg
/home/david/bitwise/ion/system_packages/builtin/types.ion(1): warning: Unknown declaration #directive 'foreign'
/home/david/bitwise/ion/system_packages/builtin/types.ion(2): warning: Unknown declaration #directive 'foreign'
/home/david/bitwise/ion/system_packages/builtin/types.ion(4): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(5): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(6): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(7): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(8): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(9): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(10): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(11): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(12): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(13): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(14): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(15): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(17): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(18): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(19): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(20): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(21): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(22): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(23): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(24): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(25): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(26): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(27): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(28): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(29): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types.ion(30): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types_win32.ion(4): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types_win32.ion(5): warning: Unknown declaration #directive 'static_assert'
/home/david/bitwise/ion/system_packages/builtin/types_x64.ion(1): warning: Unknown declaration #directive 'static_assert'
Processed 100 symbols in 2 packages
Generated out_testpkg.c
david@davidvm5 ~/bitwise (per_master)
$ ls testpkg/
test.ion
david@davidvm5 ~/bitwise (per_master)
$ cat testpkg/test.ion 
func main() : int {
  return 1;
}

size_t isnt usize and usize isnt size_t (at least on Macos)

See repro at: uucidl@a96b036

ion -os osx -arch x64 bugs && \
  cc out_bugs.c -o bugs.elf && \
  ./bugs.elf ; echo "rc:$?"

Expected:

Processed 101 symbols in 2 packages
Generated out_bugs.c
rc:42

Got:

Processed 101 symbols in 2 packages
Generated out_bugs.c
<foo>/Desktop/bitwise/ion/bugs/size_bug.ion:10:25: warning:
      incompatible pointer types passing 'ullong *' (aka 'unsigned long long *')
      to parameter of type 'size_t *' (aka 'unsigned long *')
      [-Wincompatible-pointer-types]
    fetch_value(&(num), &(num_size));
                        ^~~~~~~~~~~
<foo>/Desktop/bitwise/ion/bugs/size_bug.h:3:41: note: passing argument
      to parameter 'dest_size_ptr' here
int fetch_value(void* dest_ptr, size_t* dest_size_ptr);
                                        ^
1 warning generated.
rc:42

I would have expected C apis expressed in terms of size_t to be
exposed in ion via usize. It appears that on Macos at least:

cc -E out_bugs.c | grep -E '(typedef.* uint64_t;|typedef.* size_t;)'
typedef unsigned long long uint64_t;
typedef long unsigned int size_t;

Since usize is defined as uint64, we see that these types differ.

Functional programming

Since you state that the structure will change based on the feedback:

I am highly interested to see this in functional styles also.

You can show and explain the different abstraction layers and how they are composed together. ๐Ÿ™‚

Thousands are very interested to see somebody with empathy and deep understanding doing this. ๐Ÿ˜„

There is so much material out there for the imperative approach and so less for the functional one (yet). ๐Ÿ™‚

Rust has very functional approaches and can go very low level, while its suitable for many of the tasks you describe, since its without garbage collection. ๐Ÿ™‚

Haskell is capable to do GPU programming and highly focused on education by its design.

F# can do GPU programming as well and is relevant in the .Net space, which performs amazingly well today on .Net Core. ๐Ÿ™‚

Compilers are a strenght of these languages so is the initial compiler from Rust written in OCaml ๐Ÿ˜ƒ

I think, since you aim to give educational courses is it interesting to show how things can be done in other ways as the traditions tell us.

I see a couple of assumptions in our community, who actually slow down our productivity and the personal development.

Portable low level code is possible with other languages as C, languages who are eventually more suited for the different tasks. ๐Ÿ˜ƒ

I appreciate what you are doing and hope you find a lot of use cases to demonstrate the beauty of functional languages.

And I am sure, thousands over thousands see it very similar. ๐Ÿ˜ƒ

Level of fluency in C Programming

You described being fluent in C as a prerequisite for Bitwise. Could you describe it in a little more detail? I only have a beginner's level of experience with C and with programming in general, so I'd like to know how much work there is ahead of me to be able to go with the flow.

Problems with literal floating point numbers in #assert

The following program illustrates the issue:

`import libc {...}

func main() {
x := 1.0d;
y := x + 1e-10d;
printf("x=%.12f y=%.12f\n", x, y);
printf("y-x = %.12f\n", y-x);

#assert(fabsd(y-x) < 2e-10d);

}`

The printf's work as expected, but the assertion fails. This is the output:
`x=1.000000000000 y=1.000000000000

y-x = 0.000000000000

ionprobl: /home/amoura/ion-probl/tst.ion:9: main: Assertion `(fabs((y) - (x))) < (0.000000)' failed.

Aborted (core dumped)
`
It looks like while scanning the assertion, the literal 2e-10d is scanned as 0.000000.

Using := on arrays leads to C compile-time error

 path : char[256];   // in C: char (path[256]);
 p := path;          // in C: char (p[256]) = path;

gcc -std=c11 test.c returns test.ion:4:21: error: invalid initializer, as initializing arrays such way is no-no.

Which leads to more general question - should p be an array(as now), pointer to char(as c++), pointer to array of chars or forbidden at all.

Importing enum doesn't import enum values

Package b:

enum Em{
    Em1
}

Package c:

  import b{Em}

  func main() : int {
      em : Em = Em1;
      return 0;
  }

Running:

 ~/src/bitwise/ion$ git pull && gcc main.c
 Already up-to-date.

 ~/src/bitwise/ion$ IONHOME=. ./a.out c
 /mnt/tera1/devel/src/bitwise/ion/system_packages/builtin/types_win32.ion(4): warning: Unknown declaration #directive 'static_assert'
 /mnt/tera1/devel/src/bitwise/ion/system_packages/builtin/types_win32.ion(5): warning: Unknown declaration #directive 'static_assert'
 /mnt/tera1/devel/src/bitwise/ion/c/c.ion(4): error: Unresolved name 'Em1'

Using import b{...} makes everything compiling fine, but importing enum should probably import values as well, as otherwise it's not very useful, but pretty surprising

(Also I'm on linux, I thought _win32.ions are not parsed)

Ion keyword clash

Some identifiers used in the ion c compiler are reserved words in ion, such as var or import. When controlling a whole api, this can be worked around by changing the identifier name. However, if someone needed to use a third-party api with naming conflicts, they'd have to make adapters in c before they could use it.

It might be worth making some escape mechanism (like a prefix such as $identifier) for such cases.

An added complexity is that this would allow expression of any identifier name in ion, which could then break a c back end, so I guess the c back end would then need to have a way to change those identifiers to some safe word.

lexing part

GCC I believe supports case ranges. It helps a lot.

void next_token()
{
        switch (*stream) {
        case '0'...'9':
                token.kind = TOKEN_INT;
                token.val = 0;
                while (isdigit(*stream)) {
                        token.val *= 10;
                        token.val += (*stream - '0');
                        stream++;
                }
                break;
        case 'a'...'z':
        case 'A'...'Z':
        case '_':
                token.kind = TOKEN_NAME;
                token.start = stream++;
                while (isalnum(*stream)
                        || *stream == '_') stream++;
                token.length = stream - token.start;
                break;
        default:
                token.kind = *stream++;
        }
}

It is 3 dots, not 2.

ion first try

this is just me trying out the compiler cause I was exited to see it working, if this kind if report is not yet appropriate feel free to close!

what I noticed:

  • The compiler does seem to forward declare everything. In that case you'd not need to do any reordering algorithm so I think this needs to be corrected/ enabled correctly.

This was just a bit strange to me cause after all this effort of reordering and traversing the effect does not really show through.

  • var declarations are reordered even if they are not used and thus there would be no reason to reorder them
  • line numbers in syntax errors do not always get propagated/ reported
  • the following produces an assertion error line 577 resolve.c which disappears if one does a := f();
func f(){

}

func f2(){
    f();
}
  • redefinition of a variable does not produce an error, the following compiles:
var c = 4
var c = "abd"
  • var declarations do not require a semicolon, the rest does

On the other hand I'm impressed how type inference is actually working!

I hope this was not disturbing, I was just trying to get an overview and give some basic feedback, not doing anything fancy or trying to brake ion.

I'm exited to see this working ๐ŸŽ‰ ๐ŸŽˆ

One thing you mentioned today was editor support for sublime/ vim? I could not yet find something like that yet but if someone could give a link to something (WIP) that would be nice as well!

Ion cast to void**

In ion, any pointer type seems to auto cast to void*, but they don't auto-cast to void**. For example, I have to cast from char** to void** here:

str: char* = NULL;
buf_push((:void**)&str, &val, 1);

Where I elsewhere have the following function:

func buf_push(b: void**, elem: void*, elem_size: usize) {
    buf_fit(b, 1 + buf_len(*b), elem_size);
    memcpy((:char*)*b + elem_size * buf__hdr(*b).len++, elem, elem_size);
}

If I leave out the cast, it says the following:

error: Invalid type in function call argument. Expected void**, got char**

I understand that there are bizarre edge cases for pointers on various ancient or strange systems. So maybe this is by design because of edge cases that I haven't studied well. But in my mind, this ought to work. If it should work, then it would be nice to automate it.

If it shouldn't work, then I clearly need to design the example function differently ...

functions returning pointers generate invalid c code

Currently functions that return a pointer generate c code that puts the * character inside brackets with the function name, like this:

int (*test)(void) {
return 0;
}

This doesn't seem to be valid syntax (cl, gcc and clang all complain).

How to build this?

This might be a dumb question, but how are you supposed to build this? There is no makefiles, shell scripts or cmake files. Sorry if that's an obvious question.

typo in print_test()

These expressions in print_test()

expr_binary('+', expr_int(1), expr_int(2)),
expr_unary('-', expr_float(3.14)),

currently print

(*= 1 2)
(%= 3.140000)

The char literals probably should be replaced by TOKEN_ADD and TOKEN_SUB.

Ion conditional compilation

Unless I've missed, ion currently has no conditional compilation. Tying that with some built-in consts for OS and such would be great. This requires falling out to c whenever this is needed, which is one place in the current ion compiler.

An implementation for this might possibly be tied up with some implementation of macros, if any.

(As an aside, I'm mostly against general compile time code execution, because I'm afraid of people getting carried away with it. But it's technically a potential solution for these sorts of things, too.)

Missing stdio_osx.ion & time_osx.ion for macOS

[edited]

The ion binary compiles fine with gcc main.c

However, the Ion tests don't compile unless I duplicate linux files:

cp stdio_linux.ion stdio_osx.ion
cp time_linux.ion time_osx.ion

Also, on OSX (10.13 High Sierra, XCode 9.3) clock_t is defined as unsigned long:

@foreign typedef time_t = long;
@foreign typedef clock_t = ulong;

from mac system headers

typedef unsigned long       __darwin_clock_t;   /* clock() */
typedef long                __darwin_time_t;    /* time() */
typedef __darwin_time_t     time_t;
typedef __darwin_clock_t    clock_t;

stdio_osx.ion:

@foreign const _IOFBF = 0;
@foreign const _IOLBF = 1;
@foreign const _IONBF = 2;

@foreign const BUFSIZ		= 1024;
@foreign const FILENAME_MAX	= 1024;
@foreign const FOPEN_MAX	= 20;
@foreign const TMP_MAX		= 308915776;
@foreign const L_tmpnam		= 1024;

from mac system headers

#define _IOFBF  0       /* setvbuf should set fully buffered */
#define _IOLBF  1       /* setvbuf should set line buffered */
#define _IONBF  2       /* setvbuf should set unbuffered */

#define BUFSIZ  1024    /* size of buffer used by setbuf */

                                /* must be == _POSIX_STREAM_MAX <limits.h> */
#define FOPEN_MAX   20          /* must be <= OPEN_MAX <sys/syslimits.h> */
#define FILENAME_MAX    1024    /* must be <= PATH_MAX <sys/syslimits.h> */

/* System V/ANSI C; this is the wrong way to do this, do *not* use these. */
#ifndef _ANSI_SOURCE
#define P_tmpdir    "/var/tmp/"
#endif
#define L_tmpnam    1024        /* XXX must be == PATH_MAX */
#define TMP_MAX     308915776

buf_free leaves dangling pointer

buf_free should set the pointer to NULL

#define buf_free(b) ((b) ? free(buf__hdr(b)) : 0, b = NULL)
int *asdf = NULL;
buf_push(asdf, i);
buf_free(asdf);
assert(asdf == NULL);

Tags for videos

Please create tags/ releases/ branches for videos.
I think tags are appropriate but anything would do.

When revisiting the videos it will make it WAY easier to look at the code from that time.

Thanks for your work, much appreciated!

Ion ternaries with string literals on one side

This doesn't work at present:

token.pos.name = name ? name : "<string>";

It gives the following error:

error: Left and right operands of ternary expression must have arithmetic types or identical types

Instead, I have to cast:

token.pos.name = name ? name : (:char const*)"<string>";

It would be nice if the compiler could auto-coerce string literals to char const*.

There might be other cases where this applies, too, beyond ternaries. It's just the case where I remember I've run across it.

"1.2.3.4" parses without error

This one's a little pedantic.
I was reading the lexer code (I've fallen a bit behind) and it seems there's nothing preventing "1.2.3.4" from parsing without error.

diff --git a/ion/lex.c b/ion/lex.c
index 56087ea..eb5e534 100644
--- a/ion/lex.c
+++ b/ion/lex.c
@@ -611,11 +611,14 @@ void lex_test(void) {
     assert_token_eof();
 
     // Float literal tests
-    init_stream("3.14 .123 42. 3e10");
+    init_stream("3.14 .123 42. 3e10 1.2.3.4");
     assert_token_float(3.14);
     assert_token_float(.123);
     assert_token_float(42.);
     assert_token_float(3e10);
+    assert_token_float(1.2);
+    assert_token_float(.3);
+    assert_token_float(.4);
     assert_token_eof();

Consider a podcast format?

I'm not sure if this will lend itself to the idea, but if the video streams are not heavily reliant on graphics, it would be nice to have an audio only version available that can be augmented with the articles and reviewing the code in the repo later on. This would be beneficial for those of us that don't have a lot of time to watch videos while at a screen, but spend several hours a day stuck in traffic or doing other tasks that don't require a lot of our attention.

Anonymous and ad hoc structs and unions

For kicks, I've been porting the ion compiler to ion. Mostly trying to retain things unchanged.

As far as I can tell, ion doesn't currently support anonymous and/or ad hoc aggregates, such as the following:

struct Typespec {
    TypespecKind kind;
    SrcPos pos;
    Typespec *base;
    union {
        const char *name;
        struct {
            Typespec **args;
            size_t num_args;
            bool has_varargs;
            Typespec *ret;
        } func;
        Expr *num_elems;
    };
};

I could imagine expressing that like so in ion:

struct Typespec {
    kind: TypespecKind;
    pos: SrcPos;
    base: Typespec*;
    union {
        name: char const*;
        function: struct {
            args: Typespec**;
            num_args: usize;
            has_varargs: bool;
            ret: Typespec*;
        }
        num_elems: Expr*;
    }
}

Meanwhile, I've been pulling out the nested types and giving the fields names where missing. That's doable, but some things, like Expr, get pretty intense about it. Do you have the intention to add these features to ion?

(And macros are another matter, but I've been working around those for now, too.)

Ion builtin conflicts

In porting the ion compiler to ion, I had conflicts with ion builtins, specifically the TYPE_... constants. I ended up renaming the TYPE_... constants in the compiler to CMPL_TYPE_..., though it's easy to mix things up, still.

Maybe this isn't likely to trip people up, but any ongoing additions to builtins can break existing code.

I'm not sure the right way to address this matter. Some possibilities:

  • Just ignore it. Maybe won't cause much trouble.
  • Promise never to add new builtins (including new constants on existing enums). Could freeze builtins and add a new std library or whatnot for anything new, and it needs explicitly imported.
  • Never auto-import anything. This would clearly be a breaking change and would make uses of the very basics more awkward.
  • Allow per-file opt-out of (particular?) builtins.

Maybe other options, too.

Ion macros

I don't know what plans exist on macros, and I know there are lots of ways to do them with lots of pros and cons. That said, they are used extensively in the ion compiler in c. They might possibly be worth adding to ion, and in a way familiar to c programmers, but better. Something perhaps to consider, at least.

Here's maybe some syntax for syntactic macros:

macro DO_SOMETHING(x, y) {{{
    // Replacement syntax between braces.
    // Semi heredoc-ish, allow arbitrary number of opening braces,
    // so we can ignore braces in the middle.
    // And yes, this particular macro is useless.
    }
}}}

This requires a prepass and presumably these macros are non-hygienic, like c macros. Some syntax for concatenation or string formatting would also be good, like c.

I'd also recommend they be automatically namespaced like everything else in ion.

One other more innovative idea follows, but it wouldn't cover all use cases of standard syntactic macros. For cases it does cover, it would be more embedded in the language. First start with any old function:

// ** here because this isn't a macro, and we might need to change the pointer.
func buf_push(b: char**, elem: char) {
    buf_fit(b, 1 + buf_len(*b));
    (*b)[buf__hdr(*b).len++] = elem;
}

This function can then be called normally, or else it can be called macro-expanded:

// Type explicit here just to emphasize macro-ized call.
field: CompoundField = parse_expr_compound_field();
// Here, I use a trailing bang to tell the compiler I want it to create an ad hoc
// variation of the original function.
buf_push!(&fields, field);

In this case, I'd still recommend it create a new name-mangled function to match the call site.

As a final possibility of my own, I could also imagine importing packages, including the current package, with redefinitions of types or constants. As for the ad hoc function type replacement, it wouldn't cover all use cases of syntactic macros.

Of course, one could go all out with custom parsers like in Rust or any number of other ideas, but I think all that goes beyond the scope of a c-like language.

Commits for day 62 and day 63

Is there a way to see the code after the changes occurred on day 62 and 63 yet?

(I apologize if this is the wrong place to ask)

Field of a function return value not treated as an l-value

struct Thing { a: int; }
func returns_ptr(): Thing*;

func set_value(): void
{
	returns_ptr().a = 5;
}

produces

error: Cannot assign to non-lvalue

Is this intended? Currently resorted to doing

func set_value(): void
{
    	thing := returns_ptr();
    	thing.a = 5;
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.