nidium / libapenetwork Goto Github PK

View Code? Open in Web Editor NEW

15.0 15.0 6.0 1.56 MB

Fast cross-platform async network library

License: Other

C 84.15% Python 3.19% C++ 12.66%

ape c epoll event-driven kqueue library

libapenetwork's Introduction

nidium

Introduction

nidium is an ongoing effort for a general purpose rendering engine to create apps and games that run on both desktop and mobile. It also offers a way to create server-side applications through nidium-server which shares all the non graphics related code base.

nidium helps you create graphical software with Javascript. It's not a NodeJS, QT, Chromium, or WebKit derivative. It has been designed from scratch and has a small codebase written in C++.

It leverages the combination of Skia Graphics from Google, and Mozilla's JavaScript Engine (SpiderMonkey) together with many more awesome libraries.

It supports various common and well known APIs such as:

WebGL
Canvas 2D Context
WebSocket (client & server)
Module loading (require())

And some other non standard:

UDP/TCP Socket (client & server)
HTTP (client & server)
File access
Threading
Threaded Audio
Videos
Fragment shader on 2D canvas
Local Key/Value database

It can seamlessly run various JavaScript libraries like Three.JS, PixiJS, Phaser (and probably a lot more without much modification).

What problem(s) is nidium trying to solve

To put it simply, nidium aims to expose something close to web technology but in an easily hackable environment.

We're not trying to copy the Web. Think of it as a sandbox where you can quickly prototype new things, without any dependency to blink/webkit (e.g. like Electron does). This allows us to add various optimizations and non standard things. Moreover, nidium is small (nidium statically linked with all of its dependencies is about 20 MB).

Also, nidium aims to target a large spectrum of devices and our short term goal is to run it on low-power devices (mobile, rpi, ...)

By the way, it uses SpiderMonkey as its JavaScript engine, and we believe that Mozilla needs some love too!

Work in progress

nidium is still a work in progress and several key features are missing:

App distribution (create actual app out of nidium)
Microsoft Windows support
Good documentation

Layout engine

nidium ships with its own layout engine. That is, every element has its own memory buffer (retained mode), pretty similar to HTML elements.

The layout engine allows several "layout" operations to be made on each element (relative position, opacity, drag'n'drop, margin, scrolling, overflow, and so on).

It also implements a simple HTML DOM compatility layer and is mature enough to run a MVVM framework like Vue.js and probably React with little to no effort.

Network and event loop

nidium uses its own high-performance library (libapenetwork) which handles all the networking operations in a non-blocking, async way.

Download nidium

You can download nidium binaries for Linux and OSX from the download page.

Building nidium

To build nidium you need at least 5.7GB of disk space. A build from scratch may take 30 to 90 minutes, depending of the speed of your computer.

On a debian based system, a few commands will get you started.

$ apt-get install libpci-dev python2.7 git make patch clang pkg-config libgtk2.0-dev libgtk-3-dev mesa-common-dev libglu1-mesa-dev libosmesa6-dev yasm libasound2 libasound2-dev libbz2-1.0
$ git clone https://github.com/nidium/NidiumTools.git
$ git clone --recursive https://github.com/nidium/Nidium.git
$ export PYTHONPATH=$(pwd)/NidiumTools/src
$ cd Nidium
$ ./configure_frontend
$ ./bin/nidium

On a mac-os based system you need xcode installed.

sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
$ git clone https://github.com/nidium/NidiumTools.git
$ git clone --recursive https://github.com/nidium/Nidium.git
$ export PYTHONPATH=$(pwd)/NidiumTools/src
$ cd Nidium
$ ./configure_frontend
$ ./bin/nidium

Documentation

We strive to have excellent documentation, both for our getting started guide and for the API Reference.

Videos - nidium in action

http://p.nf/nidium-vids/

Bug reports and Collaboration

Feel free to report any bug or issue to us.

If you feel that something is not clear, that could be an issue worth reporting and solving.

License

libapenetwork's People

Contributors

Stargazers

Watchers

Forkers

ezhangle happy-ferret zhangjinde linecode haifenghuang zeta1999

libapenetwork's Issues

USE_SELECT_HANDLER does not 'work'.

Today I tried the the select handler on linux but failed somewhat;

Forcing 'common.h' to use the select handler.

#define USE_SPECIFIC_HANDLER 1
#define USE_SELECT_HANDLER 1

This appears to be broken since bf30ece.

I've corrected the signatures, dropped ptr etc; return the info, and it compiles cleanly,

diff --git a/ape_event_select.c b/ape_event_select.c
index 6b5364c..5a46708 100644
--- a/ape_event_select.c
+++ b/ape_event_select.c
@@ -55,24 +55,22 @@ enum {

 typedef struct select_fdinfo_t
 {
-    void *ptr;
     int fd;
     char watchfor:8;
 } select_fdinfo_t;

-static int event_select_add(struct _fdevent *ev, int fd, int bitadd,
-        void *attach)
+static int event_select_add(struct _fdevent *ev,
+    ape_event_descriptor *evd, int bitadd)
 {
-    if (fd < 0 || fd > FD_SETSIZE) {
-        printf("cant add event %d\n", fd);
+    if (evd->fd < 0 || evd->fd > FD_SETSIZE) {
+        printf("cant add event %d\n", evd->fd);
         return -1;
     }

     select_fdinfo_t *fdinfo = malloc(sizeof(select_fdinfo_t));
-    fdinfo->fd = fd;
-    fdinfo->ptr = attach;
+    fdinfo->fd = evd->fd;
     fdinfo->watchfor = 0;
-  
+
     if (bitadd & EVENT_READ) {
         fdinfo->watchfor |= kWatchForRead_Event;
     }
@@ -81,13 +79,37 @@ static int event_select_add(struct _fdevent *ev, int fd, int bitadd,
         fdinfo->watchfor |= kWatchForWrite_Event;
     }

-    hashtbl_append64(ev->fdhash, fd, fdinfo);
+    hashtbl_append64(ev->fdhash, evd->fd, fdinfo);

-    printf("[++++] added %d\n", fd);
+    printf("[++++] added %d\n", evd->fd);

     return 1;
 }

+static int event_select_mod(struct _fdevent *ev,
+    ape_event_descriptor *evd, int bitadd)
+{
+    select_fdinfo_t * fdinfo;
+
+    fdinfo = hashtbl_seek64(ev->fdhash, evd->fd);
+    if (fdinfo == NULL ) {
+        return -1;
+    }
+    fdinfo->watchfor = 0;
+
+    if (bitadd & EVENT_READ) {
+        fdinfo->watchfor |= kWatchForRead_Event;
+    }
+
+    if (bitadd & EVENT_WRITE) {
+        fdinfo->watchfor |= kWatchForWrite_Event;
+    }
+
+    printf("[++++] modded %d\n", evd->fd);
+
+   return 1;
+}
+
 static int event_select_del(struct _fdevent *ev, int fd)
 {
     hashtbl_erase64(ev->fdhash, fd);
@@ -184,9 +206,9 @@ static int event_select_poll(struct _fdevent *ev, int timeout_ms)
     return i;
 }

-static void *event_select_get_fd(struct _fdevent *ev, int i)
+static ape_event_descriptor *event_select_get_evd(struct _fdevent *ev, int i)
 {
-    return ev->events[i]->ptr;
+    return (ape_event_descriptor *)ev->events[i];
 }

 static int event_select_revent(struct _fdevent *ev, int i)
@@ -246,12 +268,13 @@ int event_select_init(struct _fdevent *ev)
     ev->add               = event_select_add;
     ev->del               = event_select_del;
     ev->poll              = event_select_poll;
-    ev->get_current_fd    = event_select_get_fd;
+    ev->get_current_evd   = event_select_get_evd;
+    ev->setsize           = event_select_setsize;
     ev->revent            = event_select_revent;
     ev->reload            = event_select_reload;
-    ev->setsize           = event_select_setsize;
+    ev->mod               = event_select_mod;

-    printf("select() started with %i slots\n", ev->basemem);
+    printf("Event loop started using select() with %i slots\n", ev->basemem);

     return 1;
 }

But does not 'work' with network data. A breakpoint with gdb on 'event_select_poll' is never reached.
What did I miss or what am I doing wrong? Please advice.

ape_socket: No IPv6 support

The ape_socket.c#L282 is using the inet_addr method, which will convert a text to an IPv4 binary representation. Therefore the socket isn't IPv6 ready or compatible.

There's a very good ipv6 porting guide that compares old structs / methods with the newer ones: http://long.ccaba.upc.edu/long/045Guidelines/eva/ipv6.html

The reason why IPv6 support is important is mostly because of NAT breaking techniques. These days, most ISPs have carrier-grade NATs and IPv6 with global scopes allows peer-to-peer driven networks; as it can also be tunneled through other routing layers like TOR, HYPE, cjdns etc.

So, for the gaming scene, this might be an important feature to have to properly support multiplayer networking.

APE_ABS macro gcc's -Wparentheses or clang's -Wshift-op-parentheses

By default clang is very verbose as soon as the APE_ABS is used.
The fix is probably very easy.

#include <stdio.h>

#include <common.h>

static int usage(const char *name) 
{

        printf( "usage: %s abs\n", name);
        return 1;
}
int main( const int argc, const char ** argv)
{
        int val = 0, absolute = 0;

        if (argc != 2 ){
                return usage(argv[0]);
        }
        val = atoi(argv[1]);
        absolute = APE_ABS( val );

        if ((val < 0 && val * -1 == absolute ) ||
            (val >=0 && val == absolute)) {
                printf( "OK %d\n", absolute );
        } else {
                printf( "Wrong %d\n", absolute);
        }

        return 0;
}

cc -o abstest -I .. abs.cpp -Wall
abs.cpp: In function 'int main(int, const char**)':
abs.cpp:19:13: warning: suggest parentheses around '-' inside '>>' [-Wparentheses]
abs.cpp:19:13: warning: suggest parentheses around '-' inside '>>' [-Wparentheses]

clang -o abstest -I .. abs.cpp
abs.cpp:19:13: warning: operator '>>' has lower precedence than '-'; '-' will be evaluated first [-Wshift-op-parentheses]
        absolute = APE_ABS( val );
                   ^~~~~~~~~~~~~~
../common.h:56:29: note: expanded from macro 'APE_ABS'
#define APE_ABS(val) (val + _APE_ABS_MASK(val)) ^ _APE_ABS_MASK(val)
                            ^~~~~~~~~~~~~~~~~~
../common.h:55:52: note: expanded from macro '_APE_ABS_MASK'
#define _APE_ABS_MASK(val) (val >> sizeof(int) * 8 - 1)
                                ~~ ~~~~~~~~~~~~~~~~^~~
abs.cpp:19:13: note: place parentheses around the '-' expression to silence this warning
        absolute = APE_ABS( val );
                   ^~~~~~~~~~~~~~
../common.h:56:29: note: expanded from macro 'APE_ABS'
#define APE_ABS(val) (val + _APE_ABS_MASK(val)) ^ _APE_ABS_MASK(val)
                            ^~~~~~~~~~~~~~~~~~
../common.h:55:52: note: expanded from macro '_APE_ABS_MASK'
#define _APE_ABS_MASK(val) (val >> sizeof(int) * 8 - 1)
                                   ~~~~~~~~~~~~~~~~^~~
abs.cpp:19:13: warning: operator '>>' has lower precedence than '-'; '-' will be evaluated first [-Wshift-op-parentheses]
        absolute = APE_ABS( val );
                   ^~~~~~~~~~~~~~
../common.h:56:51: note: expanded from macro 'APE_ABS'
#define APE_ABS(val) (val + _APE_ABS_MASK(val)) ^ _APE_ABS_MASK(val)
                                                  ^~~~~~~~~~~~~~~~~~
../common.h:55:52: note: expanded from macro '_APE_ABS_MASK'
#define _APE_ABS_MASK(val) (val >> sizeof(int) * 8 - 1)
                                ~~ ~~~~~~~~~~~~~~~~^~~
abs.cpp:19:13: note: place parentheses around the '-' expression to silence this warning
        absolute = APE_ABS( val );
                   ^~~~~~~~~~~~~~
../common.h:56:51: note: expanded from macro 'APE_ABS'
#define APE_ABS(val) (val + _APE_ABS_MASK(val)) ^ _APE_ABS_MASK(val)
                                                  ^~~~~~~~~~~~~~~~~~
../common.h:55:52: note: expanded from macro '_APE_ABS_MASK'
#define _APE_ABS_MASK(val) (val >> sizeof(int) * 8 - 1)
                                   ~~~~~~~~~~~~~~~~^~~
2 warnings generated.

Unittest for blowfish appears to be incorrect.

This unittest is failing with ./configure --debug --unit-test -asume-yes

[ RUN      ] Blowfish.Simple
=================================================================
==23129==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc6e41bf11 at pc 0x0000005fb1e0 bp 0x7ffc6e41acb0 sp 0x7ffc6e41aca8
READ of size 1 at 0x7ffc6e41bf11 thread T0
    #0 0x5fb1df (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x5fb1df)
    #1 0x555edf (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x555edf)
    #2 0x630832 (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x630832)
    #3 0x61f900 (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x61f900)
    #4 0x60518a (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x60518a)
    #5 0x605fed (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x605fed)
    #6 0x606816 (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x606816)
    #7 0x60e900 (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x60e900)
    #8 0x62cb32 (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x62cb32)
    #9 0x621f50 (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x621f50)
    #10 0x60e59a (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x60e59a)
    #11 0x5fba90 (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x5fba90)
    #12 0x5fba5e (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x5fba5e)
    #13 0x7f68126bbb44 (/lib/x86_64-linux-gnu/libc.so.6+0x21b44)
    #14 0x53b9ec (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x53b9ec)

Address 0x7ffc6e41bf11 is located in stack of thread T0 at offset 4465 in frame
    #0 0x555e0f (/data/buildbot/nidium_slave/debian8.2_amd64/libapenetwork_Debug_debian8_2_amd64_master/build/tests/bin/nativenetwork-unittests+0x555e0f)

  This frame has 2 object(s):
    [32, 4200) 'ctx'
    [4464, 4465) 'key' <== Memory access at offset 4465 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow ??:0 ??
Shadow bytes around the buggy address:
  0x10000dc7b790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b7c0: 00 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2
  0x10000dc7b7d0: f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2 f2
=>0x10000dc7b7e0: f2 f2[01]f3 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000dc7b830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  ASan internal:           fe
==23129==ABORTING

ape_timers seems to be obsolete.

It seems that ape_timers_next is the preferred way and that ape_timers are deprecated.

The function process_tick wich fires the registered struct _tick_callbacks is not called in ape_events_loop:events_loop.

I am not sure if ape_timers is used in applications other then NativeStudio or NativeServer but as the process_tick is not called by default, I guess that it is obsolete now..

May I cleanup the ape_timers.c ape_timer.h and their #includes?
If not: several types of struct members and function parameters are void*; Should this be improved (typedef)?

APE_sendfile is not present when compiling under windows.

(it seems that 'writeev' has the same issue.)
https://github.com/nidium/libapenetwork/blob/master/src/ape_socket.c#L665-L747

Native support of LZ4 compression

Provide a way to enable client and server to natively support lz4 compression "on-the-fly".

Implement TCP TFO

https://en.wikipedia.org/wiki/TCP_Fast_Open

https://github.com/bradleyfalzon/tcp-fast-open/blob/master/server.go#L17-L33

Memory cleanup of a listening socket

There is something fishy with server sockets.

cleanup_listening.c

#include <native_netlib.h>
#include <signal.h>

#define IP "0.0.0.0"
#define PORT 1234

int ape_running;

static void signal_handler(int signal)
{
if (signal == SIGINT || signal == SIGTERM) {
    ape_running = 0;
    }
}

static void OnConnect(ape_socket *socket_server, ape_socket *socket_client, ape_global *ape, void *arg)
{
    ape_running = 0;
}

int main(const int argc, const char** argv)
{
    ape_global *g_ape;
    ape_socket *socket;

    ape_running = 1;
    g_ape = native_netlib_init();
    socket = APE_socket_new(APE_SOCKET_PT_TCP, 0, g_ape);
    socket->callbacks.on_connect = OnConnect;
    APE_socket_listen(socket, PORT, IP, 1, 1);

    signal(SIGINT, &signal_handler);
    signal(SIGTERM, &signal_handler);
    events_loop(g_ape);

    APE_socket_shutdown_now(socket);
    native_netlib_destroy(g_ape);

    return 0;
}

Compile with gcc -o test cleanup_listening.c ./build/out/Release/obj.target/gyp/libnativenetwork.a -I . -lz -lrt -L ./build/third-party/ -lssl -lcrypto -lcares -ldl

Scenario 1: not connected at all

kill SIGINT 6972

==6972== 
==6972== HEAP SUMMARY:
==6972==     in use at exit: 50,028 bytes in 3,120 blocks
==6972==   total heap usage: 3,297 allocs, 177 frees, 134,472 bytes allocated
==6972== 
==6972== 172 (132 direct, 40 indirect) bytes in 1 blocks are definitely lost in loss record 2 of 7
==6972==    at 0x4028308: malloc (vg_replace_malloc.c:263)
==6972==    by 0x804A7B1: APE_socket_new (ape_socket.c:127)
==6972==    by 0x8049E8D: main (in /home/peter/Development/nidium/NativeStudio/nativejscore/network/test)
==6972== 
==6972== 2,088 (8 direct, 2,080 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 7
==6972==    at 0x4028308: malloc (vg_replace_malloc.c:263)
==6972==    by 0x804C99F: ape_ssl_init_global_client_ctx (ape_ssl.c:80)
==6972==    by 0x4088E45: (below main) (libc-start.c:244)
==6972== 
==6972== LEAK SUMMARY:
==6972==    definitely lost: 140 bytes in 2 blocks
==6972==    indirectly lost: 2,120 bytes in 20 blocks
==6972==      possibly lost: 0 bytes in 0 blocks
==6972==    still reachable: 47,768 bytes in 3,098 blocks
==6972==         suppressed: 0 bytes in 0 blocks

Scenario 2: connected

telnet 127.0.0.1 1234
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

==6980== 
==6980== HEAP SUMMARY:
==6980==     in use at exit: 50,200 bytes in 3,122 blocks
==6980==   total heap usage: 3,299 allocs, 177 frees, 134,644 bytes allocated
==6980== 
==6980== 344 (132 direct, 212 indirect) bytes in 1 blocks are definitely lost in loss record 4 of 9
==6980==    at 0x4028308: malloc (vg_replace_malloc.c:263)
==6980==    by 0x804A7B1: APE_socket_new (ape_socket.c:127)
==6980==    by 0x804B96C: ape_socket_accept (ape_socket.c:983)
==6980==    by 0x804A643: events_loop (ape_events_loop.c:66)
==6980==    by 0x8049EFC: main (in /home/peter/Development/nidium/NativeStudio/nativejscore/network/test)
==6980== 
==6980== 2,088 (8 direct, 2,080 indirect) bytes in 1 blocks are definitely lost in loss record 7 of 9
==6980==    at 0x4028308: malloc (vg_replace_malloc.c:263)
==6980==    by 0x804C99F: ape_ssl_init_global_client_ctx (ape_ssl.c:80)
==6980==    by 0x4088E45: (below main) (libc-start.c:244)
==6980== 
==6980== LEAK SUMMARY:
==6980==    definitely lost: 140 bytes in 2 blocks
==6980==    indirectly lost: 2,292 bytes in 22 blocks
==6980==      possibly lost: 0 bytes in 0 blocks
==6980==    still reachable: 47,768 bytes in 3,098 blocks
==6980==         suppressed: 0 bytes in 0 blocks

Please neglect the ape_ssl_init_global_client_ctx for now.
I am more concerned about the APE_socket_new (ape_socket.c:127).
I gues that APE_socket_shutdown_now and APE_socket_shutdown are not with the correct state under all circumstances.