flomesh-io / pipy Goto Github PK

Pipy is a programmable proxy for the cloud, edge and IoT.

License: Other

CMake 0.81% Dockerfile 0.12% Shell 0.57% C++ 83.63% JavaScript 6.91% C 7.66% HTML 0.01% Batchfile 0.25% Makefile 0.04%

sidecar-proxy sidecar proxy socks-proxy http-proxy software-loadbalancer gateway api-gateway load-balancer reverse-proxy

pipy's Introduction

中文 | 日本語

Pipy

Pipy is a programmable proxy for the cloud, edge and IoT. It's written in C++, which makes it extremely lightweight and fast. It's also fully programmable by using PipyJS, a tailored version from the standard JavaScript language.

Why Pipy?

Versatile

Although Pipy is mostly used as a high-performance reverse proxy, the real power of Pipy relies on providing you a range of basic pluggable building blocks, aka. "filters", and not imposing any restrictions on how you combine them. It's entirely up to you. We've seen people using Pipy to convert protocols, record network traffic, sign/verify messages, trigger serverless functions, health-check servers, and more.

Fast

Pipy is written in C++. It leverages asynchronous network I/O. Allocated resources are pooled and reused. Data is transferred internally by pointers whenever possible. It's fast in every way.

Tiny

A build of Pipy with no built-in GUI frontend resources gives you an executable merely around 10MB with zero external dependencies. You'll experience the fastest download and startup times with Pipy.

Programmable

At the core, Pipy is a script engine running PipyJS, a tailored version from ECMA standard JavaScript. By speaking the planet's most widely used programming language, Pipy gives you unparalleled freedom over what you have in other products based on YAML configuration files and the like.

Open

Pipy is more open than open source. It doesn’t try to hide details in a black box. You'll always know what you are doing. It might sound a bit daunting but fear not, it doesn’t require a rocket scientist to understand how the different parts work together. In fact, you’ll only have more fun as you have complete control over everything.

Quick Start

Build

The following prerequisites are required to build Pipy:

Clang 5.0+
CMake 3.10+
Node.js v12+ (only if the builtin Admin UI is enabled)

Run the build script to start building:

./build.sh

The final product can be found at bin/pipy.

Run

Run bin/pipy with zero command line options, Pipy will start in repo-mode listening on the default port 6060.

$ bin/pipy

[INF] [admin] Starting admin service...
[INF] [listener] Listening on port 6060 at ::

Open the browser of your choice, point to http://localhost:6060. You will now see the Admin UI where you can start exploring the documentation and playing around with the tutorial codebases.

Documentation

Overview
Getting started
Tutorials
Reference
- API Reference
- PipyJS Reference
Learning Resources
- Blog
- Docker Desktop Extension - Flomesh Pipy Docker Desktop Extension
- InfoQ article - Brief Introduction
Copyright
License

Compatibility

Pipy is being constantly tested on these platforms:

RHEL/CentOS
Fedora
Ubuntu
Debian
macOS
FreeBSD
OpenBSD
OpenEuler
OpenWrt
Deepin
Kylin

Pipy runs on the following architectures:

X86/64
ARM64
LoongArch
Hygon

Copyright & License

Please refer to COPYRIGHT and LICENSE.

Contact

For security issues, please email to [email protected]
For legal issues, please email to [email protected]
For commercial, sales and marketing topics, please email to [email protected]
For other topics not suitable for the public, please email to [email protected]
For public discussions, please go to GitHub issues: https://github.com/flomesh-io/pipy/issues

pipy's People

Contributors

Stargazers

Watchers

Forkers

weibaohui linwb315 keveinliu redbonezhang skymysky yonyoung nkbai zjndglj org-mars blond2005 5itest startime-h ikingye tide999 j4ckzh0u jaydzhou uyuang zengyuxing007 laashub-soa shijunlee phenix-gao chenhtvip beaver-company dystudio longamu lorisunjunbin yuanfeng0905 watch-later i0r3k addozhang zhuhuahua168 wormon shader13 naqvis ethinx khas-flomesh nixff flyarong spurringworld saigp tain198127 daixiang0 chenqing24 bluepeople1 forifido genedna int2ecall tchaikov piano-wow wanpf gg-big-org david0718 zwjhuhu assetoverflow madlaxcb gengle ccliushou cybwan devopstoday11 zzl133 deltared1a jadenblack hexiaofeng sixinyiyu xyhbill mayo11 change-from-now preresearch-labs yyk808

pipy's Issues

after execute exec filter the pipy crash and restart

version -info:

Version : nightly-202207101146
Commit : 9a327f8
Commit Date : Sat, 9 Jul 2022 17:19:49 +0800
Host : Linux-4.19.90-17.ky10.x86_64 x86_64
OpenSSL : OpenSSL 1.1.1q 5 Jul 2022
Builtin GUI : Yes
Tutorial : Yes

pipy()

.listen(18080)
  .demuxHTTP('req')

.pipeline('req')
  .exec('ls -l').print()
  .replaceMessage(
    msg => (
      new Message(
      {
        headers: {
          'content-type': 'application/json'
        }
      },
      JSON.encode(msg.head))
    )
  )

result:

output the dir files and restart pipy

Cannot build on macOS with silicon chip

Got below error if execute ./build.sh

[ 61%] Building CXX object deps/leveldb-1.23/CMakeFiles/leveldbutil.dir/db/leveldbutil.cc.o
Created configdata.pm
Running configdata.pm
[ 61%] Linking CXX executable leveldbutil
Created Makefile.in
[ 61%] Built target leveldbutil
Created Makefile

**********************************************************************
***                                                                ***
***   OpenSSL has been successfully configured                     ***
***                                                                ***
***   If you encounter a problem while building, please open an    ***
***   issue on GitHub <https://github.com/openssl/openssl/issues>  ***
***   and include the output from the following command:           ***
***                                                                ***
***       perl configdata.pm --dump                                ***
***                                                                ***
***   (If you are new to OpenSSL, you might want to consult the    ***
***   'Troubleshooting' section in the INSTALL.md file first)      ***
***                                                                ***
**********************************************************************
make[3]: warning: -jN forced in submake: disabling jobserver mode.
In file included from In file included from ../apps/lib/app_libctx.c../apps/lib/app_params.c::910:
:
In file included from In file included from ../apps/include/app_libctx.h../apps/include/apps.h::1313:
:
In file included from In file included from ../include/openssl/types.h../include/internal/e_os.h::3216:
:
In file included from In file included from ../include/openssl/e_os2.h../include/openssl/e_os2.h::234234:
:
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/inttypes.h:21/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/inttypes.h::1521: :15fatal error: : 'inttypes.h' file not foundfatal error:
'inttypes.h' file not found
#include_next <inttypes.h>#include_next <inttypes.h>
              ^~~~~~~~~~~~

              ^~~~~~~~~~~~
1 error generated.
1 error generated.
make[4]: *** [apps/lib/libapps-lib-app_libctx.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: *** [apps/lib/libapps-lib-app_params.o] Error 1
make[3]: *** [build_sw] Error 2
make[2]: *** [/Users/linyang/workspace/flomesh-projects/pipy/deps/openssl-3.2.0/build/libcrypto.a] Error 2
make[1]: *** [CMakeFiles/OpenSSL.dir/all] Error 2
make: *** [all] Error 2

OS information:

uname -a
Darwin XXX-MBP-2021.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64

Suspicious memory usage

What happened

Run pipy proxy with script in sample/http, client send requests to route /, /api, /api/private, /hi, /home in http/https randomly. Run several rounds of the test, and I notice that the memory usage of pipy may ramp up at the time of new run starts.

Reproduce the issue

Start pipy with script in sample/http, two upstreams, and one http server for responding the pipy logging request
Run test by k6 and the attached test.js.gz script several times

while true; do ./k6 run -d 2m -u 1000 test.js; sleep 180; done

Expect behavior

Should some of the memory be recycled after testing, or should the mem usage be stable?

Screenshots

Version info

Version     : 0.50.0-37
Commit      : 5c115a842c4e98559fd83a6271be2d3ba79ef914
Commit Date : Tue, 26 Jul 2022 19:08:51 +0800
Host        : Linux-5.15.0-1014-azure x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No
Tutorial    : No

Compile Pipy GUI issue

When I build pipy gui, I got this problem, how to resolve this ?

admin-port visit not found

Bug Report

Make sure to review these points before submitting issues - thank you!

so bad, follow the instruction and do not find the visit the 6060 port;

Pipy process crashes when printing "__inbound.remoteAddress" in onEnd()

Pipy process crashes when printing "__inbound.remoteAddress" in onEnd().

Version : nightly-202208231036
Commit : 8436d3b
Commit Date : Mon, 22 Aug 2022 23:05:38 +0800
Host : Linux-5.15.0-43-generic x86_64
OpenSSL : OpenSSL 1.1.1q 5 Jul 2022
Builtin GUI : Yes
Samples : Yes

test pjs:
///////////////

((
) => pipy({
})
.listen(8081)
.onEnd(
() => console.log('bug ......', __inbound.remoteAddress)
)
.serveHTTP(
() => new Message('Hi, there!\n')
)
)()

`bind` override `socket.setRawOption`

TLDR

// pipy this_file.js --reuse-port

pipy().task().onStart(new Message('Turn around')).connect('127.0.0.1:8000', {
  bind: '127.0.0.1:1234',
  onState: (conn) => {
    if(conn.state === 'open') {
      conn.socket.setRawOption(1, 15, new Data([1]))
    }
  },
}).listen('127.0.0.1:1234').print()

The code above won't work correctly.

Reproduce

Setting up a server returning message.

pipy -e 'pipy().listen("127.0.0.1:8000").print().connect("127.0.0.1:1234")'

Using code in the tldr to get error log like:

❯ pipy error.js --reuse-port
2024-06-27 21:49:31.069 [INF] [config]
2024-06-27 21:49:31.069 [INF] [config] Module /error.js
2024-06-27 21:49:31.069 [INF] [config] ================
2024-06-27 21:49:31.069 [INF] [config]
2024-06-27 21:49:31.069 [INF] [config]  [Listen on 1234 at 127.0.0.1]
2024-06-27 21:49:31.069 [INF] [config]  ----->|
2024-06-27 21:49:31.069 [INF] [config]        |
2024-06-27 21:49:31.069 [INF] [config]       print -->|
2024-06-27 21:49:31.069 [INF] [config]                |
2024-06-27 21:49:31.069 [INF] [config]  <-------------|
2024-06-27 21:49:31.069 [INF] [config]  
2024-06-27 21:49:31.069 [INF] [config]  [Task #1 ()]
2024-06-27 21:49:31.069 [INF] [config]  ----->|
2024-06-27 21:49:31.069 [INF] [config]        |
2024-06-27 21:49:31.069 [INF] [config]       connect -->|
2024-06-27 21:49:31.069 [INF] [config]                  |
2024-06-27 21:49:31.069 [INF] [config]  <---------------|
2024-06-27 21:49:31.069 [INF] [config]  
2024-06-27 21:49:31.069 [INF] [listener] Listening on TCP port 1234 at 127.0.0.1
2024-06-27 21:49:31.069 [ERR] connect() at line 1 column 58 in /error.js: bind: Address already in use
2024-06-27 21:49:31.069 [ERR] connect() at line 1 column 58 in /error.js: bind: Address already in use
2024-06-27 21:49:31.069 [ERR] connect() at line 1 column 58 in /error.js: bind: Address already in use
^C2024-06-27 21:49:32.175 [INF] [shutdown] Shutting down...
2024-06-27 21:49:32.175 [INF] [shutdown] Waiting for workers to drain...
2024-06-27 21:49:32.175 [INF] [listener] Stopped listening on TCP port 1234 at 127.0.0.1

Also, I've made a python equivalent. If everything works fine in pipy, the pjs code should act just like this.

import socket

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

BIND = 1234
TARGET = 8000

try:
    client.bind(('127.0.0.1', BIND))
    server.bind(('127.0.0.1', BIND))

    server.listen(1)
    client.connect(('127.0.0.1', TARGET))

    client.sendall(b'Turn around')

    s, _ = server.accept()
    data = s.recv(16)
    s.close()

    print(data.decode())

finally:
    client.close()
    server.close()

Reason

I tried to add a print in src/socket.cpp line 885 where we define setRawOption, it will only be called when not set bind option.
So it comes out that bind will override setRawOption.

[filter] manually bound `connect` doesn't close tcp socket cleanly.

Env

The newest nightly build of pipy on Archlinux.

❯ pipy -v
  Version          : nightly-202406162205
  Commit           : 159d4e82db0367fc138f2427503dfc8ba62dde22
  Commit Date      : Sun, 16 Jun 2024 11:12:56 +0800
  Host             : Linux-6.9.4-zen1-1-zen x86_64
  OpenSSL          : OpenSSL 3.2.0 23 Nov 2023
  Builtin GUI      : Yes
  Builtin Codebases: Yes

Reproduce

First, start a tcp server. Here I use a brief pjs to make it.

pipy -e 'pipy().listen(8080)'

Then, connect to this port with a bound address. Here I use idleTimeout to speed up this reproduction.

❯ pipy -e 'pipy().task().onStart(new Data).connect("127.0.0.1:8080", {bind: "127.0.0.1:1235", idleTimeout: 1}).dump()'
2024-06-17 09:53:28.059 [INF] [config]
2024-06-17 09:53:28.059 [INF] [config] Module 
2024-06-17 09:53:28.059 [INF] [config] =======
2024-06-17 09:53:28.059 [INF] [config]
2024-06-17 09:53:28.059 [INF] [config]  [Task #1 ()]
2024-06-17 09:53:28.059 [INF] [config]  ----->|
2024-06-17 09:53:28.059 [INF] [config]        |
2024-06-17 09:53:28.059 [INF] [config]       connect
2024-06-17 09:53:28.059 [INF] [config]       dump -->|
2024-06-17 09:53:28.059 [INF] [config]               |
2024-06-17 09:53:28.059 [INF] [config]  <------------|
2024-06-17 09:53:28.059 [INF] [config]  
2024-06-17 09:53:29.060 [INF] [dump] [worker=0] [context=2] [ 1:105] StreamEnd, code = IdleTimeout

At the first time we run it, the program will exit as expected. But if we instantly run it again, it says:

❯ pipy -e 'pipy().task().onStart(new Data).connect("127.0.0.1:8080", {bind: "127.0.0.1:1235", idleTimeout: 1}).dump()'
2024-06-17 09:53:30.099 [INF] [config]
2024-06-17 09:53:30.099 [INF] [config] Module 
2024-06-17 09:53:30.099 [INF] [config] =======
2024-06-17 09:53:30.099 [INF] [config]
2024-06-17 09:53:30.099 [INF] [config]  [Task #1 ()]
2024-06-17 09:53:30.099 [INF] [config]  ----->|
2024-06-17 09:53:30.099 [INF] [config]        |
2024-06-17 09:53:30.099 [INF] [config]       connect
2024-06-17 09:53:30.099 [INF] [config]       dump -->|
2024-06-17 09:53:30.099 [INF] [config]               |
2024-06-17 09:53:30.099 [INF] [config]  <------------|
2024-06-17 09:53:30.099 [INF] [config]  
2024-06-17 09:53:30.100 [ERR] connect() at line 1 column 40: bind: Address already in use
2024-06-17 09:53:30.100 [INF] [dump] [worker=0] [context=2] [ 1:105] StreamEnd, code = RuntimeError, error = { name: "Error", message: "connect() at line 1 column 40: bind: Address already in use", cause: null, stack: undefined }

If not set the bind option, everything works fine.

Plus, from the packets cap by wireshark, the program indeed disconnected gracefully.

support jsonpath tools for get value from json

Discussion

for example:

response:
{
    "store":{
        "book":[
            {
                "category":"reference",
                "author":"Nigel Rees",
                "title":"Sayings of the Century",
                "price":8.95
            },
            {
                "category":"fiction",
                "author":"Evelyn Waugh",
                "title":"Sword of Honour",
                "price":12.99
            },
            {
                "category":"fiction",
                "author":"Herman Melville",
                "title":"Moby Dick",
                "isbn":"0-553-21311-3",
                "price":8.99
            },
            {
                "category":"fiction",
                "author":"J. R. R. Tolkien",
                "title":"The Lord of the Rings",
                "isbn":"0-395-19395-8",
                "price":22.99
            }
        ],
        "bicycle":{
            "color":"red",
            "price":19.95
        },
        "clothes":[
            {
                "name":"牛仔裤",
                "sizes":"S",
                "price":94
            },
            {
                "name":"背心",
                "sizes":"M",
                "price":48
            },
            {
                "name":"裙子",
                "sizes":["S", "M"],
                "price":1.24
            },
            {
                "name":"羊毛衫",
                "sizes":["XS", "XL"],
                "price":78.99
            },
            {
                "name":"Polo衫",
                "sizes":["XS", "XL", "M"],
                "price":18.99
            }
        ]
    },
    "expensive":10
}

 console.log(JsonPath.read(response, "$.store.book[*].author"))

output the all author

Allow listening on Virtual IPs

Feature Request

Is your feature request related to a problem? Please describe clearly and concisely what is it.

For some use cases, there is a need to have Pipy listens on Virtual IP(s) and ports.
Describe the feature you would like, optionally illustrated by examples, and how it will solve the above problem.

Would like to have Pipy support listening on Virtual IP Addresses, so that one can have the ability to start Pipy instance(s) on Virtual IPs and ports like:

.listen('vip1:port')
.listen('vip2:port')
.....

Describe considered alternative solutions, and the reasons why you have not proposed them as a solution here.

Current approach is to listen on specific IP or address bind to specific NIC.
Does it break backward compatibility, if yes then what's the migration path?
No

run `./build.sh` get `make: *** [Makefile:136：all] 错误 2`

Bug Report

Make sure to review these points before submitting issues - thank you!

Make sure a similar issue does not exist yet: use the search box.
Include reproducible code: we should be able to paste it into an editor, run it and get the same error as you. Otherwise it's impossible for us to reproduce the bug.
Write code in the issue itself.
Reduce code, if possible, to the minimum size that reproduces the bug.
If all of the above is impossible due to a large project, create a branch that reproduces the bug and point us to it.
Provide details about the environment like how you are running Pipy, upstream services, client software etc. The more the details, the better it will help us understand and diagnose the problem.
Include pipy version (pipy -v) and OS. If possible, try to see if the bug still reproduces on master.

node v16.18.1

g++ --version
g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

clang --version
Ubuntu clang version 14.0.0-1ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

cmake --version
cmake version 3.22.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

openssl version
OpenSSL 1.1.1q 5 Jul 2022

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

cat /proc/version
Linux version 5.15.0-56-generic (buildd@lcy02-amd64-004) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022

err info

-- The C compiler identification is Clang 14.0.0
-- The CXX compiler identification is Clang 14.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Deprecation Warning at deps/yajl-2.1.0/CMakeLists.txt:15 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


running /usr/bin/cmake -E copy_if_different /home/luo/devops/pipy/deps/yajl-2.1.0/src/api/yajl_parse.h /home/luo/devops/pipy/build/deps/yajl-2.1.0/src/../yajl-2.1.0/include/yajl  2>&1
running /usr/bin/cmake -E copy_if_different /home/luo/devops/pipy/deps/yajl-2.1.0/src/api/yajl_gen.h /home/luo/devops/pipy/build/deps/yajl-2.1.0/src/../yajl-2.1.0/include/yajl  2>&1
running /usr/bin/cmake -E copy_if_different /home/luo/devops/pipy/deps/yajl-2.1.0/src/api/yajl_common.h /home/luo/devops/pipy/build/deps/yajl-2.1.0/src/../yajl-2.1.0/include/yajl  2>&1
running /usr/bin/cmake -E copy_if_different /home/luo/devops/pipy/deps/yajl-2.1.0/src/api/yajl_tree.h /home/luo/devops/pipy/build/deps/yajl-2.1.0/src/../yajl-2.1.0/include/yajl  2>&1
CMake Warning (dev) at deps/yajl-2.1.0/reformatter/CMakeLists.txt:38 (GET_TARGET_PROPERTY):
  Policy CMP0026 is not set: Disallow use of the LOCATION target property.
  Run "cmake --help-policy CMP0026" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The LOCATION property should not be read from target "json_reformat".  Use
  the target name directly with add_custom_command, or use the generator
  expression $<TARGET_FILE>, as appropriate.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/yajl-2.1.0/verify/CMakeLists.txt:32 (GET_TARGET_PROPERTY):
  Policy CMP0026 is not set: Disallow use of the LOCATION target property.
  Run "cmake --help-policy CMP0026" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The LOCATION property should not be read from target "json_verify".  Use
  the target name directly with add_custom_command, or use the generator
  expression $<TARGET_FILE>, as appropriate.

This warning is for project developers.  Use -Wno-dev to suppress it.

!! doxygen not found, not generating documentation
CMake Deprecation Warning at deps/libexpat-R_2_2_6/expat/CMakeLists.txt:6 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for fcntl.h
-- Looking for fcntl.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for strings.h
-- Looking for strings.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for getpagesize
-- Looking for getpagesize - found
-- Looking for bcopy
-- Looking for bcopy - found
-- Looking for memmove
-- Looking for memmove - found
-- Looking for mmap
-- Looking for mmap - found
-- Looking for getrandom
-- Looking for getrandom - found
-- Looking for arc4random_buf
-- Looking for arc4random_buf - not found
-- Looking for arc4random
-- Looking for arc4random - not found
-- Looking for 4 include files stdlib.h, ..., float.h
-- Looking for 4 include files stdlib.h, ..., float.h - found
-- Looking for off_t
-- Looking for off_t - not found
-- Looking for size_t
-- Looking for size_t - not found
-- Performing Test HAVE_SYSCALL_GETRANDOM
-- Performing Test HAVE_SYSCALL_GETRANDOM - Success
-- Performing Test FLAG_NO_STRICT_ALIASING
-- Performing Test FLAG_NO_STRICT_ALIASING - Success
-- Looking for crc32c_value in crc32c
-- Looking for crc32c_value in crc32c - not found
-- Looking for snappy_compress in snappy
-- Looking for snappy_compress in snappy - not found
-- Looking for malloc in tcmalloc
-- Looking for malloc in tcmalloc - not found
-- Looking for fdatasync
-- Looking for fdatasync - found
-- Looking for F_FULLFSYNC
-- Looking for F_FULLFSYNC - not found
-- Looking for O_CLOEXEC
-- Looking for O_CLOEXEC - found
-- Performing Test HAVE_CLANG_THREAD_SAFETY
-- Performing Test HAVE_CLANG_THREAD_SAFETY - Success
-- Performing Test LEVELDB_HAVE_NO_MISSING_FIELD_INITIALIZERS
-- Performing Test LEVELDB_HAVE_NO_MISSING_FIELD_INITIALIZERS - Success
-- Performing Test HAVE_CXX17_HAS_INCLUDE
-- Performing Test HAVE_CXX17_HAS_INCLUDE - Failed
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
Enabling LTO
CMake Deprecation Warning at deps/zlib-1.2.11/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for unistd.h
-- Looking for unistd.h - found
CMake Deprecation Warning at deps/brotli-1.0.9/CMakeLists.txt:5 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Build type is 'Release'
-- Performing Test BROTLI_EMSCRIPTEN
-- Performing Test BROTLI_EMSCRIPTEN - Failed
-- Compiler is not EMSCRIPTEN
-- Looking for log2
-- Looking for log2 - not found
-- Looking for log2
-- Looking for log2 - found
-- Configuring done
CMake Warning (dev) at deps/zlib-1.2.11/CMakeLists.txt:189 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'zlibstatic'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/zlib-1.2.11/CMakeLists.txt:188 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target 'zlib'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/brotli-1.0.9/CMakeLists.txt:173 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'brotlicommon-static'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/brotli-1.0.9/CMakeLists.txt:174 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'brotlidec-static'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/brotli-1.0.9/CMakeLists.txt:167 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'brotlicommon'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/brotli-1.0.9/CMakeLists.txt:168 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'brotlidec'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/brotli-1.0.9/CMakeLists.txt:169 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'brotlienc'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/brotli-1.0.9/CMakeLists.txt:175 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'brotlienc-static'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at deps/brotli-1.0.9/CMakeLists.txt:218 (add_executable):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target 'brotli'.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /home/luo/devops/pipy/build
[  0%] Generating deps/version.h
[  0%] Building OpenSSL
[  0%] Building C object deps/brotli-1.0.9/CMakeFiles/brotlicommon-static.dir/c/common/constants.c.o
[  1%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl.c.o
[  1%] Built target GenVer
Operating system: x86_64-whatever-linux2
[  1%] Building C object deps/libexpat-R_2_2_6/expat/CMakeFiles/expat.dir/lib/loadlibrary.c.o
[  2%] Building C object deps/brotli-1.0.9/CMakeFiles/brotlicommon-static.dir/c/common/context.c.o
[  3%] Building C object deps/libexpat-R_2_2_6/expat/CMakeFiles/expat.dir/lib/xmlparse.c.o
[  3%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_lex.c.o
[  3%] Building C object deps/brotli-1.0.9/CMakeFiles/brotlicommon-static.dir/c/common/dictionary.c.o
Configuring OpenSSL version 1.1.1q (0x1010111fL) for linux-x86_64-clang
Using os-specific seed configuration
[  3%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_parser.c.o
[  3%] Building C object deps/brotli-1.0.9/CMakeFiles/brotlicommon-static.dir/c/common/platform.c.o
[  4%] Building C object deps/brotli-1.0.9/CMakeFiles/brotlicommon-static.dir/c/common/transform.c.o
[  5%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_buf.c.o
[  5%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_encode.c.o
[  5%] Linking C static library libbrotlicommon-static.a
[  5%] Built target brotlicommon-static
[  5%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_gen.c.o
[  5%] Building CXX object deps/leveldb-1.23/CMakeFiles/leveldb.dir/db/builder.cc.o
In file included from /home/luo/devops/pipy/deps/leveldb-1.23/db/builder.cc:5:
In file included from /home/luo/devops/pipy/deps/leveldb-1.23/./db/builder.h:8:
/home/luo/devops/pipy/deps/leveldb-1.23/include/leveldb/status.h:16:10: fatal error: 'algorithm' file not found
#include <algorithm>
         ^~~~~~~~~~~
1 error generated.
make[2]: *** [deps/leveldb-1.23/CMakeFiles/leveldb.dir/build.make:76：deps/leveldb-1.23/CMakeFiles/leveldb.dir/db/builder.cc.o] 错误 1
make[1]: *** [CMakeFiles/Makefile2:730：deps/leveldb-1.23/CMakeFiles/leveldb.dir/all] 错误 2
make[1]: *** 正在等待未完成的任务....
[  6%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_alloc.c.o
[  6%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_tree.c.o
[  6%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl_version.c.o
[  6%] Building C object deps/libexpat-R_2_2_6/expat/CMakeFiles/expat.dir/lib/xmlrole.c.o
[  7%] Linking C static library ../yajl-2.1.0/lib/libyajl_s.a
[  7%] Built target yajl_s
[  8%] Building C object deps/libexpat-R_2_2_6/expat/CMakeFiles/expat.dir/lib/xmltok.c.o
[  8%] Building C object deps/libexpat-R_2_2_6/expat/CMakeFiles/expat.dir/lib/xmltok_impl.c.o
[  8%] Building C object deps/libexpat-R_2_2_6/expat/CMakeFiles/expat.dir/lib/xmltok_ns.c.o
Creating configdata.pm
Creating Makefile

**********************************************************************
***                                                                ***
***   OpenSSL has been successfully configured                     ***
***                                                                ***
***   If you encounter a problem while building, please open an    ***
***   issue on GitHub <https://github.com/openssl/openssl/issues>  ***
***   and include the output from the following command:           ***
***                                                                ***
***       perl configdata.pm --dump                                ***
***                                                                ***
***   (If you are new to OpenSSL, you might want to consult the    ***
***   'Troubleshooting' section in the INSTALL file first)         ***
***                                                                ***
**********************************************************************
make[3]: 警告: jobserver 不可用: 正使用 -j1。添加 “+” 到父 make 的规则。
[  9%] Linking C static library libexpat.a
[  9%] Built target expat
ar: creating apps/libapps.a

ar: creating libcrypto.a
ar: creating libssl.a
[  9%] Built target OpenSSL
make: *** [Makefile:136：all] 错误 2

Can't compile as dynamic library in linux.

Use PIPY_SHARED to complied as dynamic library.
Reproduction method：

[I] user@mac ~/w/p/l/p/build ((c009821d))> git log -1
commit c009821dd1d4f0e0cfd7fda0720e6f0456e4525f (HEAD, main)
Author: pajama-coder <[email protected]>
Date:   Thu May 23 20:18:54 2024 +0800

    [fix] Crash in non-shared Quota due to missing of a null-check

[I] user@mac ~/w/p/l/p/build ((c009821d))> cmake -DPIPY_SHARED=ON ..
-- The C compiler identification is GNU 14.1.1
-- The CXX compiler identification is GNU 14.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Deprecation Warning at deps/yajl-2.1.0/CMakeLists.txt:15 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.
  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.
......
-- Build files have been written to: /pipy-rs/libs/pipy/build

[I] user@mac ~/w/p/l/p/build ((c009821d))> make -j4
[  0%] Building C object deps/brotli-1.0.9/CMakeFiles/brotlicommon-static.dir/c/common/constants.c.o
[  0%] Built target OpenSSL
[  0%] Built target GenVer
[  0%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl_s.dir/yajl.c.o
......
[ 12%] Building C object deps/yajl-2.1.0/src/CMakeFiles/yajl.dir/yajl_parser.c.o
[ 12%] Building C object deps/zlib-1.3/CMakeFiles/zlibstatic.dir/gzread.c.o
pipy-rs/libs/pipy/deps/yajl-2.1.0/src/yajl_parser.c: In function ‘yajl_do_parse’:
pipy-rs/libs/pipy/deps/yajl-2.1.0/src/yajl_parser.c:334:24: warning: this statement may fall through [-Wimplicit-fallthrough=]
  334 |                     if (yajl_bs_current(hand->stateStack) ==
      |                        ^
pipy-rs/libs/pipy/deps/yajl-2.1.0/src/yajl_parser.c:347:17: note: here
  347 |                 case yajl_tok_colon:
      |                 ^~~~
pipy-rs/libs/pipy/deps/yajl-2.1.0/src/yajl_parser.c:405:24: warning: this statement may fall through [-Wimplicit-fallthrough=]
  405 |                     if (yajl_bs_current(hand->stateStack) ==
      |                        ^
pipy-rs/libs/pipy/deps/yajl-2.1.0/src/yajl_parser.c:414:17: note: here
  414 |                 default:
      |                 ^~~~~~~
[ 12%] Building C object deps/zlib-1.3/CMakeFiles/zlibstatic.dir/gzwrite.c.o
......
[ 62%] Building CXX object CMakeFiles/pipy.dir/src/api/algo.cpp.o
In file included from pipy-rs/libs/pipy/src/pjs/pjs.hpp:29,
                 from pipy-rs/libs/pipy/src/event.hpp:29,
                 from pipy-rs/libs/pipy/src/filter.hpp:29,
                 from pipy-rs/libs/pipy/src/admin-link.hpp:29,
                 from pipy-rs/libs/pipy/src/admin-link.cpp:26:
pipy-rs/libs/pipy/src/pjs/types.hpp:1206:43: error: redefinition of ‘bool __tls_guard’
 1206 | template<class T> thread_local Ref<Class> ClassDef<T>::m_c;
      |                                           ^~~~~~~~~~~
pipy-rs/libs/pipy/src/pjs/types.hpp:1206:43: note: ‘bool __tls_guard’ previously declared here
pipy-rs/libs/pipy/src/pjs/types.hpp:1206: confused by earlier errors, bailing out
make[2]: *** [CMakeFiles/pipy.dir/build.make:76: CMakeFiles/pipy.dir/src/admin-link.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from pipy-rs/libs/pipy/src/pjs/pjs.hpp:29,
                 from pipy-rs/libs/pipy/src/api/http.hpp:29,
                 from pipy-rs/libs/pipy/src/admin-proxy.hpp:29,
                 from pipy-rs/libs/pipy/src/admin-proxy.cpp:26:
pipy-rs/libs/pipy/src/pjs/types.hpp:1206:43: error: redefinition of ‘bool __tls_guard’
 1206 | template<class T> thread_local Ref<Class> ClassDef<T>::m_c;
      |                                           ^~~~~~~~~~~
pipy-rs/libs/pipy/src/pjs/types.hpp:1206:43: note: ‘bool __tls_guard’ previously declared here
pipy-rs/libs/pipy/src/pjs/types.hpp:1206: confused by earlier errors, bailing out
make[2]: *** [CMakeFiles/pipy.dir/build.make:90: CMakeFiles/pipy.dir/src/admin-proxy.cpp.o] Error 1
In file included from pipy-rs/libs/pipy/src/pjs/pjs.hpp:29,
                 from pipy-rs/libs/pipy/src/api/algo.hpp:29,
                 from pipy-rs/libs/pipy/src/api/algo.cpp:26:
pipy-rs/libs/pipy/src/pjs/types.hpp:1206:43: error: redefinition of ‘bool __tls_guard’
 1206 | template<class T> thread_local Ref<Class> ClassDef<T>::m_c;
      |                                           ^~~~~~~~~~~
pipy-rs/libs/pipy/src/pjs/types.hpp:1274:50: note: ‘bool __tls_guard’ previously declared here
 1274 | template<class T> thread_local std::vector<Str*> EnumDef<T>::m_val_to_str;
      |                                                  ^~~~~~~~~~
pipy-rs/libs/pipy/src/pjs/types.hpp:1206: confused by earlier errors, bailing out
make[2]: *** [CMakeFiles/pipy.dir/build.make:118: CMakeFiles/pipy.dir/src/api/algo.cpp.o] Error 1
In file included from pipy-rs/libs/pipy/src/pjs/pjs.hpp:29,
                 from pipy-rs/libs/pipy/src/api/http.hpp:29,
                 from pipy-rs/libs/pipy/src/admin-service.hpp:29,
                 from pipy-rs/libs/pipy/src/admin-service.cpp:26:
pipy-rs/libs/pipy/src/pjs/types.hpp:1206:43: error: redefinition of ‘bool __tls_guard’
 1206 | template<class T> thread_local Ref<Class> ClassDef<T>::m_c;
      |                                           ^~~~~~~~~~~
pipy-rs/libs/pipy/src/pjs/types.hpp:1206:43: note: ‘bool __tls_guard’ previously declared here
pipy-rs/libs/pipy/src/pjs/types.hpp:1206: confused by earlier errors, bailing out
make[2]: *** [CMakeFiles/pipy.dir/build.make:104: CMakeFiles/pipy.dir/src/admin-service.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:472: CMakeFiles/pipy.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

system info, test in two machine.

# machine 1:
cmake version 3.29.5
g++ (GCC) 14.1.1 20240522
GNU Make 4.4.1
OS: Arch Linux on Windows 10 x86_64 
Kernel: 5.15.146.1-microsoft-standard-WSL2

# machine 2:
cmake version 3.18.4
g++ (Debian 10.2.1-6) 10.2.1 20210110
GNU Make 4.3
 OS: Debian GNU/Linux 11 (bullseye) x86_64 
Kernel: 5.10.210-201.855.amzn2.x86_64

where is benchmark

Discussion

What aspect of Pipy would you like to see improved?
What are the reasons?
Include practical examples to illustrate your points.
Optionally add one (or more) proposals to improve the current situation.

pipy crash when repo addr point to domain name which is configured in /etc/hosts

What happened

As subject

Reproduce the issue

Download the latest version(x86) from https://flomesh.io/nightly/
Start a pipy repo

2022-09-20 18:27:06.088 [INF] [admin] Starting admin service...
2022-09-20 18:27:06.088 [INF] [listener] Listening on TCP port 6060 at ::

=============================================

  You can now view Pipy GUI in the browser:

    http://localhost:6060/

=============================================

Set a host entry in /etc/hosts, e.g. 192.168.66.1 pipy-repo.com on the worker node
Start pipy and point to the repo, use domain name instread of IP addr.

pipy http://pipy-repo.com:6060/repo/tutorial/01-hello/

That's all. However if we compile pipy locally and run it on the same node, there is no problem.

May the issue relate to the compiled environment? Only x86 version meets this problem now.

I tried to get the backtrace via gdb as follows

Reading symbols from pipy...
(gdb) set args http://pipy-repo.com:6060/repo/tutorial/01-hello/
(gdb) start
Temporary breakpoint 1 at 0x5b5bf0
Starting program: /usr/local/bin/pipy http://pipy-repo.com:6060/repo/tutorial/01-hello/

Temporary breakpoint 1, 0x00000000005b5bf0 in main ()
(gdb) s
Single stepping until exit from function main,
which has no line number information.
[New LWP 57759]
[New LWP 57760]
[New LWP 57761]
[New LWP 57762]
[New LWP 57763]
warning: File "/usr/lib/x86_64-linux-gnu/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /usr/lib/x86_64-linux-gnu/libthread_db.so.1
line to your configuration file "/home/ubuntu/.config/gdb/gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/ubuntu/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Thread 6 "pipy" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 57763]
0x00007ffff5650a7f in __GI___pthread_enable_asynccancel () at ./nptl/cancellation.c:34
34      ./nptl/cancellation.c: No such file or directory.
(gdb) bt
#0  0x00007ffff5650a7f in __GI___pthread_enable_asynccancel () at ./nptl/cancellation.c:34
#1  0x00007ffff5606ca7 in __GI___getrandom (flags=1, length=8, buffer=0x7ffff57e04d8 <tcache_key>) at ../sysdeps/unix/sysv/linux/getrandom.c:29
#2  __GI___getrandom (buffer=buffer@entry=0x7ffff57e04d8 <tcache_key>, length=length@entry=8, flags=flags@entry=1) at ../sysdeps/unix/sysv/linux/getrandom.c:27
#3  0x00007ffff5661a66 in tcache_key_initialize () at ./malloc/malloc.c:3162
#4  ptmalloc_init () at ./malloc/arena.c:321
#5  0x00007ffff5665355 in ptmalloc_init () at ./malloc/arena.c:315
#6  __GI___libc_malloc (bytes=472) at ./malloc/malloc.c:3295
#7  0x00007ffff563f6ce in __fopen_internal (is32=1, mode=0x7ffff5798074 "rce", filename=0x7ffff579c262 "/etc/hosts") at ./libio/iofopen.c:65
#8  _IO_new_fopen (filename=0x7ffff579c262 "/etc/hosts", mode=0x7ffff5798074 "rce") at ./libio/iofopen.c:86
#9  0x00007ffff5714e32 in __GI___nss_files_fopen (path=path@entry=0x7ffff579c262 "/etc/hosts") at ./nss/nss_files_fopen.c:27
#10 0x00007ffff5719746 in internal_setent (stream=<synthetic pointer>) at nss_files/files-XXX.c:76
#11 __GI__nss_files_gethostbyname4_r (name=0x7ffff5ff3a80 "pipy-repo.com", pat=0x7ffff5ff3278, buffer=0x7ffff5ff3580 "", buflen=1024, errnop=0x7ffff5ff46b0,
    herrnop=0x7ffff5ff46e8, ttlp=0x0) at nss_files/files-hosts.c:389
#12 0x0000000000a1688f in gaih_inet.constprop ()
#13 0x0000000000a183d9 in getaddrinfo ()
#14 0x00000000005c9431 in asio::detail::resolve_query_op<asio::ip::tcp, pipy::OutboundTCP::resolve()::$_1>::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) ()
#15 0x0000000000535d81 in asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) ()
#16 0x00000000005357e1 in asio::detail::scheduler::run(std::error_code&) ()
#17 0x00000000005c84c6 in asio::detail::posix_thread::func<asio::detail::resolver_service_base::work_io_context_runner>::run() ()
#18 0x000000000053565d in asio_detail_posix_thread_function ()
#19 0x00000000008c00f9 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x0000000000a1dd53 in clone ()
(gdb)

Expect behavior

pipy start normally with any valid url.

Version info

Distributor ID: Ubuntu
Description:    Ubuntu 22.04 LTS
Release:        22.04
Codename:       jammy

pipy

Version     : nightly-202209191340
Commit      : 15bd2ef63098d73bcca2dccf4ff8a294ba19dd39
Commit Date : Mon, 19 Sep 2022 21:40:15 +0800
Host        : Linux-5.15.0-1019-azure x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No
Samples     : No

support php-fpm fastcgi protocol

Discussion

What aspect of Pipy would you like to see improved?
like nginx, it can forward http request to php-fpm service.
What are the reasons?
i wish to replace nginx
Include practical examples to illustrate your points.
RT
Optionally add one (or more) proposals to improve the current situation.

Observing high mem usage on specific use cases

Bug Report

During a test case it was discovered that pipy running as sidecar is having continuous increase in memory usage and not showing any indication of drop in that spike. Looking into Pipy Admin panel stats shows high number of HTTP2 Streams objects.

This pattern continues until allocated RAM is exhausted and we start seeing non 2XX response codes.

Test were performed on Pipy image 0.70.0-2

Tested Usecase

Emojivoto

error when running build.sh

root@service-mesh-011122063081:~/openSource/pipy# ./build.sh
error: unknown option `show-current'
usage: git branch [] [-r | -a] [--merged | --no-merged]
or: git branch [] [-l] [-f] []
or: git branch [] [-r] (-d | -D) ...
or: git branch [] (-m | -M) []
or: git branch [] (-c | -C) []
or: git branch [] [-r | -a] [--points-at]
or: git branch [] [-r | -a] [--format]

it seems pipyjs not implement JSON.parse(text, recevier?) ，recevier not work.

.pipeline('notify-pipeline')
// .onStart(()=> void(console.log('module====>(pipeline)[notify-pipeline]')))
.branch(
() => Boolean(__serviceObj.notify_services && __serviceObj.notify_services?.length > 0), (
$=>$.fork(() => __serviceObj.notify_services).to(
$=>$.onStart(r => void(console.log('on-start-invoked', r) , _nextService = r))
.replaceMessage(
e => (
(route, path, headers, body) => (
_target = _services[_nextService].select(),
path = e.head.path,
body = JSON.parse(e.body.toString(), (k,v)=>(
console.log(v), // not work
v
)),
new Message({
...e.head,
path: path,
headers
}, JSON.stringify(body? body : '') )
)
)()
).link(
'forward', () => Boolean(_target),
'bypass'
)
)
),
)

pipy容器启动时，可能会出现OOMKilled

此现象是按照https://github.com/flomesh-io/service-mesh-demo 搭建demo时出现的。

pipy镜像版本：flomesh/pipy-pjs:0.4.0-263

以pod的sidecar形式启动pipy容器时，有可能很快出现OOMKilled的情况，重启几次后，便不再报错了

查看pipy容器日志时，发现日志停在这里，如果是正常启动的pipy，后续会打印更多日志

查看pipy容器的内存指标，发现内存占用500多M，这个图显示内存占用稳定在500多M，但是有时会降下去，只占用几M内存

Use OpenSSL 3

OpenSSL 1.1.1 is already EOL this September, https://www.openssl.org/blog/blog/2023/09/11/eol-111/, needs to update the build to use OpenSSL 3

[filter] pipeline get stuck in the `demux()` filter

At the out req posistion, nothing output to the subsequent pipeline. The scripts works in the version 0.30.0-184

pipy()

.listen(8000)
  .link('forward')

.pipeline('forward')
  .decodeHTTPRequest()
  .dump('before req')
  .demux('req')
  .dump('out req')
  .muxHTTP('connection', ()=>'')
  .encodeHTTPResponse()

.pipeline('req')
  .handleMessage(
    msg=>(
      console.log("do nothing")
    )
  )
  .dump('end req')

.pipeline('connection')
  .connect('localhost:8080')

.listen(8080)
.serveHTTP(
    msg=>(
      new Message('hello\n') 
    )
  )

Version:

Version     : nightly-202207151451
Commit      : 1d835d44c571634e8e99bfda1890e56e16f0968e
Commit Date : Thu, 14 Jul 2022 21:13:30 +0800
Host        : Darwin-21.5.0 arm64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No
Tutorial    : No

Expected result:

curl http://localhost:8000 and return hello

on kylin system build error：error: variable length array of non-POD element type 'pjs::Value' Value layouts[n]

Bug Report

Make sure to review these points before submitting issues - thank you!

Make sure a similar issue does not exist yet: use the search box.
Include reproducible code: we should be able to paste it into an editor, run it and get the same error as you. Otherwise it's impossible for us to reproduce the bug.
Write code in the issue itself.
Reduce code, if possible, to the minimum size that reproduces the bug.
If all of the above is impossible due to a large project, create a branch that reproduces the bug and point us to it.
Provide details about the environment like how you are running Pipy, upstream services, client software etc. The more the details, the better it will help us understand and diagnose the problem.
Include pipy version (pipy -v) and OS. If possible, try to see if the bug still reproduces on master.

support Option Configuration for LoadBalancer Object

Discussion

for example:

_lb = new algo.RoundRobinLoadBalancer(["10.1.1.1:9001", "10.1.1.2:9000"])
_upstream = _lb .next()

$.connect(() => _upstream .id)

Improvement example:

_lb = new algo.RoundRobinLoadBalancer([{"id":"10.1.1.1:9001","weight":40,"options":{"retryCount":1,"retryDelay":1,"connectTimeout":2,"readTimeout":3,"writeTimeout ":2}},{"id":"10.1.1.2:9000","weight":60,"options":{"retryCount":1,"retryDelay":1,"connectTimeout":2,"readTimeout":3,"writeTimeout ":2}}])

_upstream = _lb .next()

$.connect(() => _upstream .id,
 {
      connectTimeout: _upstream.options?.connectTimeout ,
      readTimeout:_upstream.options?.readTimeout 
    }
)

the opiton is map ， user can transparent transmission the additional options for next pipeline
the upstream has customized configurations ;

[filter] Manually close the Inbound connection got the whole program terminated.

Env

The newest nightly build of pipy on Archlinux.

❯ pipy -v
  Version          : nightly-202406162205
  Commit           : 159d4e82db0367fc138f2427503dfc8ba62dde22
  Commit Date      : Sun, 16 Jun 2024 11:12:56 +0800
  Host             : Linux-6.9.4-zen1-1-zen x86_64
  OpenSSL          : OpenSSL 3.2.0 23 Nov 2023
  Builtin GUI      : Yes
  Builtin Codebases: Yes

Reproduce

First, start a tcp server. Here I use a brief pjs to make it.

pipy -e 'pipy().listen(8080)'

Then, run the pjs code as below: (I put them in a file named test.js)

pipy().task().onStart(new Data).connect("127.0.0.1:8080", {
		// bind: "127.0.0.1:1234", 
		onState: function(conn) {
			if(conn.state=="connected") {
			console.info("Connection: ", conn)
			conn.close()
		}
	}
})

And we shall get:

❯ pipy test.js
  2024-06-17 10:47:00.200 [INF] [config]
  2024-06-17 10:47:00.200 [INF] [config] Module /test.js
  2024-06-17 10:47:00.200 [INF] [config] ===============
  2024-06-17 10:47:00.200 [INF] [config]
  2024-06-17 10:47:00.200 [INF] [config]  [Task #1 ()]
  2024-06-17 10:47:00.200 [INF] [config]  ----->|
  2024-06-17 10:47:00.200 [INF] [config]        |
  2024-06-17 10:47:00.200 [INF] [config]       connect -->|
  2024-06-17 10:47:00.200 [INF] [config]                  |
  2024-06-17 10:47:00.200 [INF] [config]  <---------------|
  2024-06-17 10:47:00.200 [INF] [config]  
  2024-06-17 10:47:00.200 [INF] Connection:  { state: "connected", socket: {  }, localAddress: "127.0.0.1", localPort: 42718, remoteAddress: "", remotePort: 8080 }
  terminate called after throwing an instance of 'std::system_error'
    what():  set_option: Bad file descriptor
  zsh: IOT instruction (core dumped)  pipy test.js

The disconnection packets sent correctly.

I found the usage in ztm: agent/mesh.js
The function call there SHOULD work fine though I didn't strictly test yet. However, I think a single call causing the termination of program should still be considered as a bug.

reuse-port option not work?

Edit

After reading source code in connect.cpp and listener.cpp, I found that the option --reuse-port is only set for Listener sockets.
When the connect filter try to bind the same address without this option, the kernel would refuse.
Well, needless to say, this is a corner case indeed. However I DO need this to do something.

I'm trying to implement NAT traversal in pure PJS, which need to take full control of socket, so that I can cheat the firewall and punch a hole through NAT. Using a specific <ip: port> pair to send something then open a server on it is one of the fundamental step. (
For more information about NAT traversal, go check this!)

So I reopen this issue and expecting someone can help.

Issue

First I have a tcp server running on port 8080. And then I wrote some code in test.js

// filename: test.js
pipy().listen('127.0.0.1:1234').dump()
	.task()
	.onStart(new Message('client'))
	.connect('127.0.0.1:8080', {bind:"127.0.0.1:1234"}) // this is line 57 in file.

Run it with option --reuse-port:

❯ pipy test.js --reuse-port
  2024-06-18 13:33:42.611 [INF] [config]
  2024-06-18 13:33:42.611 [INF] [config] Module /test.js
  2024-06-18 13:33:42.611 [INF] [config] ===============
  2024-06-18 13:33:42.611 [INF] [config]
  2024-06-18 13:33:42.611 [INF] [config]  [Listen on 1234 at 127.0.0.1]
  2024-06-18 13:33:42.611 [INF] [config]  ----->|
  2024-06-18 13:33:42.611 [INF] [config]        |
  2024-06-18 13:33:42.611 [INF] [config]       dump -->|
  2024-06-18 13:33:42.611 [INF] [config]               |
  2024-06-18 13:33:42.611 [INF] [config]  <------------|
  2024-06-18 13:33:42.611 [INF] [config]  
  2024-06-18 13:33:42.611 [INF] [config]  [Task #1 ()]
  2024-06-18 13:33:42.611 [INF] [config]  ----->|
  2024-06-18 13:33:42.611 [INF] [config]        |
  2024-06-18 13:33:42.611 [INF] [config]       connect -->|
  2024-06-18 13:33:42.611 [INF] [config]                  |
  2024-06-18 13:33:42.611 [INF] [config]  <---------------|
  2024-06-18 13:33:42.611 [INF] [config]  
  2024-06-18 13:33:42.611 [INF] [listener] Listening on TCP port 1234 at 127.0.0.1
  2024-06-18 13:33:42.611 [ERR] connect() at line 57 column 10 in /test.js: bind: Address already in use
  2024-06-18 13:33:42.611 [ERR] connect() at line 57 column 10 in /test.js: bind: Address already in use
  2024-06-18 13:33:42.611 [ERR] connect() at line 57 column 10 in /test.js: bind: Address already in use

I wonder if it's the right way to use --reuse-port, but I do need a way to use the same port for both inbound and outbound connections in the same time.

Furthermore, I'm expecting a feature in connect or listen like:

pipy().listen(xxx, {reusePort: true })
pipy().task().onStart(new Message()).connect(xxx, {reusePort: ture})

So the socket option can be separately set.

Prompted wrong js filename when calling a function in a module with an error.

Steps to reproduce

// file main.js

1
2 ((
3 {
4 funcTest
5 } = pipy.solve('module.js')) => (
6
7 funcTest(),
8
9 pipy({
10 })
11
12 )
13
14 )()
15
16

// file module.js

1
2 (
3 (module) => (
4
5 module = {},
6
7 module.funcTest = () => (
8 console.log('hello', var1)
9 ),
10
11 module
12 )
13
14 )()
15
16

run pipy main.js

2022-11-01 14:50:12.628 [ERR] [pjs] File /main.js:
2022-11-01 14:50:12.628 [ERR] [pjs] Line 8:
2022-11-01 14:50:12.628 [ERR] [pjs] ^
2022-11-01 14:50:12.628 [ERR] [pjs] Error: unresolved identifier
2022-11-01 14:50:12.628 [ERR] [pjs] Backtrace:
2022-11-01 14:50:12.628 [ERR] In (anonymous function at line 7 column 23) at line 8 column 28
2022-11-01 14:50:12.628 [ERR] In (anonymous function at line 2 column 2) at line 7 column 11
2022-11-01 14:50:12.628 [ERR] In (root) at line 14 column 2
2022-11-01 14:50:12.628 [INF] [shutdown] Shutting down...
2022-11-01 14:50:12.628 [INF] [shutdown] Stopped.
Done.

HTTP response code based failovers with caching

Could I achieve HTTP response code based failovers with caching using pipy?

What I mean is

Pipy receives a request and forwards it to the primary backend
If the request fails with e.g. 404, pipy retries the request with a secondary backend
Failed primary backend request urls are cached e.g. on disc
For performance, request urls present in the list of cached urls are forwarded to the secondary backend skipping the primary

Is this something I could do with pipy?

after connect invoke onData twice

I want to know the life cycle of pipy ,

.listen(6081)
  .onSessionStart(
    () => (
      console.log('6081session start')
    )
  )
  .connect('127.0.0.1:6080')
  .onData(
    () => (
      _queue.push(Date.now() + 6081),
      console.log('0681 onData', JSON.stringify(_queue))
    )
  )

out put

2021-06-22 11:09:59 [info] [pjs] 6080 session end [1624331405455]
2021-06-22 11:09:59 [info] [pjs] 0681 onData [1624331405455,1624331405456]
2021-06-22 11:09:59 [info] [pjs] 0681 onData [1624331405455,1624331405456,1624331405456]

请问支持国密证书吗

HTTPS透传代理如何实现？

下文中描述的HTTPS透传代理，用Pipy如何实现？
https://blog.xmgspace.me/archives/nginx-sni-dispatcher.html

原理上，需要从握手包里提取SNI信息，用其中的域名信息做路由参考，原样转发对应后端的原始TCP报文到客户端，不需要解密HTTPS报文，所以也不需要源站的证书和私钥。

pipy的API中比较像的handleTLSClientHello接口回调参数只有两个数字版本号，貌似获取不到SNI。

这种场景能实现吗？

另外，文档中描述的代理场景主要讲的是HTTP代理，不知道MySQL/Redis等纯TCP协议能否用pipy当代理，可能有点折腾:)

metrics label is not well handled in multi-threading version

What happened

Use samples/gateway for testing, enable metrics plugin as following

{
  "listen": 8000,
  "listenTLS": 8443,
  "plugins": [
    "plugins/router.js",
    "plugins/metrics.js",
    "plugins/jwt.js",
    "plugins/cache.js",
    "plugins/hello.js",
    "plugins/balancer.js",
    "plugins/serve-files.js",
    "plugins/default.js"
  ],
  ...
}

The label of custom metrics(request_*, response_*) are incorrect in repo.

The label of custom metrics on the repo as following:

$ curl localhost:6060/metrics -s | grep -Ei 'request_|response_'
request_count{instance="pipy1-1"} 0
request_count{instance="pipy1-1",request_count="private"} 4
request_count{instance="pipy1-1",request_count="api"} 4
request_count{instance="pipy1-1",request_count="home"} 2
response_status{instance="pipy1-1"} 0
response_status{instance="pipy1-1",response_status="private"} 0
response_status{instance="pipy1-1",response_status="api"} 0
response_status{instance="pipy1-1",response_status="home"} 0
request_latency{instance="pipy1-1"} 0
request_latency{instance="pipy1-1",request_latency="private"} 0
request_latency{instance="pipy1-1",request_latency="api"} 0
request_latency{instance="pipy1-1",request_latency="home"} 0

Moreover, I try to get the metrics from proxy worker several times, the first label of the metrics may be lost ({,).

$ curl localhost:6060/metrics -s | grep -Ei 'request_|response_'
request_count 0
request_count{route="private"} 4
request_count{route="api"} 4
request_count{route="home"} 2
request_latency_bucket 0
request_latency_count 0
request_latency_sum 0
request_latency_bucket{1="private",le=""NaN""} 0
request_latency_count{1="private"} 4
request_latency_sum{1="private"} 0
request_latency_bucket{1="api",le=""NaN""} 0
request_latency_count{1="api"} 4
request_latency_sum{1="api"} 2
request_latency_bucket{1="home",le=""NaN""} 0
request_latency_count{1="home"} 2
request_latency_sum{1="home"} 0
response_status 0
response_status{route="private"} 0
response_status{route="private",status="200"} 4
response_status{route="api"} 0
response_status{route="api",status="200"} 4
response_status{route="home"} 0
response_status{route="home",status="200"} 2

$ curl localhost:6060/metrics -s | grep -Ei 'request_|response_'
request_count{} 0
request_count{,route="private"} 4
request_count{,route="api"} 4
request_count{,route="home"} 2
request_latency_bucket{,le=""NaN""} 0
request_latency_count{} 0
request_latency_sum{} 0
request_latency_bucket{,1="private",le=""NaN""} 0
request_latency_count{,1="private"} 4
request_latency_sum{,1="private"} 0
request_latency_bucket{,1="api",le=""NaN""} 0
request_latency_count{,1="api"} 4
request_latency_sum{,1="api"} 2
request_latency_bucket{,1="home",le=""NaN""} 0
request_latency_count{,1="home"} 2
request_latency_sum{,1="home"} 0
response_status{} 0
response_status{,route="private"} 0
response_status{,route="private",status="200"} 4
response_status{,route="api"} 0
response_status{,route="api",status="200"} 4
response_status{,route="home"} 0
response_status{,route="home",status="200"} 2

There is no problem on main branch.

How to reproduce

As aforementioned.

Expect behavior

labels are consistent in pjs and the final result.

Version info

Version     : nightly-202301041229
Commit      : 55c9e5d546f1e415688decfee7d823983577dd4c
Commit Date : Wed, 4 Jan 2023 11:37:10 +0800
Host        : Linux-5.15.0-39-generic x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No
Samples     : No

gui build fail

environment variable

os : deepin 20.2.1

uname info:

Linux sxyy-PC 5.10.29-amd64-desktop #2 SMP Mon Apr 26 09:48:13 CST 2021 x86_64 GNU/Linux

node: version v14.17.0
npm: version 7.16.0

Generating JavaScript bundles failed

Converting circular structure to JSON
    --> starting at object with constructor 'SymbolDef'
    |     property 'orig' -> object with constructor 'Array'
    |     index 0 -> object with constructor 'AST_SymbolFunarg'
    --- property 'thedef' closes the circle

File: node_modules/monaco-editor/esm/vs/editor/standalone/browser/standalone-tokens.css


 ERROR #98123  WEBPACK

Generating JavaScript bundles failed

Converting circular structure to JSON
    --> starting at object with constructor 'SymbolDef'
    |     property 'orig' -> object with constructor 'Array'
    |     index 0 -> object with constructor 'AST_SymbolFunarg'
    --- property 'thedef' closes the circle

File: node_modules/monaco-editor/esm/vs/platform/actions/browser/menuEntryActionViewItem.css


 ERROR #98123  WEBPACK

Generating JavaScript bundles failed

Converting circular structure to JSON
    --> starting at object with constructor 'SymbolDef'
    |     property 'orig' -> object with constructor 'Array'
    |     index 0 -> object with constructor 'AST_SymbolFunarg'
    --- property 'thedef' closes the circle

File: node_modules/monaco-editor/esm/vs/platform/contextview/browser/contextMenuHandler.css

the execution cycle of the main pipeline and fork pipeline ?

Version : 0.0.0
Commit : d54c0aa
Commit Date : Wed, 9 Jun 2021 13:56:10 +0800
Host : Linux-5.10.18-amd64-desktop x86_64
OpenSSL : OpenSSL 1.1.1g 21 Apr 2020

pipy({
})


.listen(9000)
  .fork('invoke-A')
  .fork('invoke-B')
  .decodeHttpRequest()
  .replaceMessage(
    () => new Message({ status: 200 }, '成功!\n')
  )
  .encodeHttpResponse()
  .onMessage(
    msg => (
      console.log('main service response')
    )
  ).onSessionEnd(
    () => (
      console.log('main service session end')
    )
  )

  .pipeline('invoke-B')
    .decodeHttpRequest()
    .onMessageStart(
      () => console.log('invoke B service ...')
    )
    .replaceMessage(
      () => new Message(
        {
          method: 'POST',
          path: '/A',
          headers: {
            host: '127.0.0.1:8080',
            'Content-Type': 'application/json',
          },
        },
        JSON.encode({
          cmd: 'add',
          module: 'keyworld',
        })
      )
    )
    .encodeHttpRequest()
    .connect('127.0.0.1:8080')
    .print()
    .decodeHttpResponse()
    .print()
    .onMessage(
      msg => (
        console.log('Get B service response' + msg.body)
      )
    )

  .pipeline('invoke-A')
    .decodeHttpRequest()
    .onMessageStart(
      () => console.log('invoke A service ...')
    )
    .replaceMessage(
      () => new Message({
        method: 'GET',
        path: '/hello',
        headers: {
          host: '127.0.0.1:8080',
        },
      })
    )
    .encodeHttpRequest()
    .connect('127.0.0.1:8080')
    .print()
    .decodeHttpResponse()
    .print()
    .onMessage(
      msg => (
        console.log('Get A service response' + msg.body)
      )
    )
---------console------

2021-06-30 10:26:22 [info] [pjs] invoke A service ...
2021-06-30 10:26:22 [info] [pjs] invoke B service ...
2021-06-30 10:26:22 [info] [pjs] main service response
HTTP/1.1 200 
Content-Type: text/plain;charset=UTF-8
Content-Length: 5
Date: Wed, 30 Jun 2021 02:26:22 GMT

Hello
2021-06-30 10:26:22 [info] [pjs] Get A service responseHello
2021-06-30 10:26:22 [info] [pjs] main service session end

the /A endpoint sleep for 5 seconds until retrun  response.

does the main  session end when the B pipeline got the response ? 

I want to know when it will print B resposne ? 

use wait block the code until B got resposne?

Assertion failure when running Repo in persistent mode

Bug Report

Pipy is generating abort signal at closure when running repo in persistent mode.

Error generated is:

Assertion failed: (dummy_versions_.next_ == &dummy_versions_), function ~VersionSet, file version_set.cc, line 755.

Steps to reproduce

pipy data

Where data refers to some existing directory which is used to store repo database.

Pipy Version

Version     : 0.70.0-2
Commit      : 0ba615d8f5fcce4d8bfd537397bb11e18aa00866
Commit Date : Mon, 31 Oct 2022 16:26:29 +0800
Host        : Darwin-22.1.0 arm64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : Yes
Samples     : Yes

duplicated builtin metric in pipy repo aggregated result

What happened

builtin metric pipy_outbound_count is duplicated in the pipy repo aggregated result.

Reproduce the issue

1. Start pipy repo, create a codebase with this simple script

((
  router = new algo.URLRouter({
    '/*': new algo.RoundRobinLoadBalancer(['192.168.66.64:80', '192.168.66.170:80']),
  }),

) => pipy({
    _target: undefined,
    _requestCounter: new stats.Counter('http_requests_count', ['method', 'status', 'host', 'path']),
    _reqHead: null,
    _resHead: null,
    _path: null,
    _method: null,
    _host: null,
    _reqTime: 0,
  })

  .listen(18000)
  .demuxHTTP().to(
    $=>$.handleMessageStart(
      msg => (
        _target = router.find(
          msg.head.headers.host,
          msg.head.path,
        )?.next?.(),
        _reqHead = msg.head,
        _path = (new URL(msg.head.path)).pathname,
        _method = msg.head.method,
        _host = msg.head.headers.host,
        _reqTime = Date.now()
      )
    )
      .branch(
        () => Boolean(_target), (
          $=>$.muxHTTP(() => _target).to(
            $=>$.connect(() => _target.id)
          )
        ), (
          $=>$.replaceMessage(
            new Message({ status: 404 }, 'No route')
          )
        )
      )
      .handleMessageStart(
        msg => (
          _resHead = msg.head,
          _resHead && _requestCounter.withLabels(_method, _resHead.status, _host, _path).increase()
        )
      )
  )

)()

2. Start multiple pipy instance and subscribe to the codebase, maybe one instance could reproduce the issue too.

#!/bin/bash
ulimit -SHn 655360

export REPO=http://192.168.66.1:6060/repo/test-metrics/

for i in `seq 1 4`
do
        export PIPY_NAME=01-$i

        if [ ! -f uuid.$i ]
        then
                uuidgen > uuid.$i
        fi

        nohup pipy --reuse-port --instance-uuid=$(cat uuid.$i) --instance-name=$PIPY_NAME $REPO 2>&1 > /dev/null &
done

3. Start load generator and make some requests to proxy, 3-5 seconds may be long enough for the pipy instance to create multiple connections to the upstream. Stop the loads and wait until the connections to the upstreams closed (check with `ss -tn`), then starting a new round of requests.

hmm... I use k6 with the test script, but I thinks it's not important:

$ ./k6 run metric.js -u 10 -d 5s

import http from 'k6/http';
import { check } from 'k6';
import {
  randomItem
} from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';

export const options = {
  summaryTrendStats: ['avg', 'min', 'med', 'max', 'p(95)', 'p(99)', 'p(99.9)', 'p(99.99)', 'count'],
  noConnectionReuse: false,
  insecureSkipTLSVerify: true,
};

const first_name = [
  "admiring",
  "adoring",
];

const last_name = [
  "kalam",
  "kapitsa",
];

const ids = [
  "",
]

const targets = [
  {
    name: 'hello',
    url: 'http://192.168.66.209:18000/',
    headers: {},
    checker: {
      'hello check': (r) => r.body.includes('Hi, there'),
    },
  },
];

export default function() {
  let target = targets[0];

  let resp = http.get(target.url + randomItem(first_name) + '-' + randomItem(last_name) + randomItem(ids), {
    headers: target.headers,
  });

  check(resp, target.checker);
}

4. you could see there are duplicated `pipy_outbound_count` entries on the repo/metrics page, after each round of test.

$ curl http://192.168.66.1:6060/metrics -s | grep -E 'outbound_count{'
pipy_outbound_count{instance="01-1"} 6
pipy_outbound_count{instance="01-1",peer="[192.168.66.1]:6060"} 2
pipy_outbound_count{instance="01-1",peer="[192.168.66.64]:80"} 2
pipy_outbound_count{instance="01-1",peer="[192.168.66.170]:80"} 1
pipy_outbound_count{instance="01-1",peer="[192.168.66.170]:80"} 2
pipy_outbound_count{instance="01-1",peer="[192.168.66.64]:80"} 2
pipy_outbound_count{instance="01-1",peer="[192.168.66.170]:80"} 2
pipy_outbound_count{instance="01-1",peer="[192.168.66.64]:80"} 2
pipy_outbound_count{instance="01-3"} 2
pipy_outbound_count{instance="01-3",peer="[192.168.66.1]:6060"} 1
pipy_outbound_count{instance="01-3",peer="[192.168.66.64]:80"} 2
pipy_outbound_count{instance="01-3",peer="[192.168.66.170]:80"} 1
pipy_outbound_count{instance="01-3",peer="[192.168.66.170]:80"} 1
pipy_outbound_count{instance="01-3",peer="[192.168.66.170]:80"} 1
pipy_outbound_count{instance="01-2"} 4
pipy_outbound_count{instance="01-2",peer="[192.168.66.1]:6060"} 1
pipy_outbound_count{instance="01-2",peer="[192.168.66.64]:80"} 1
pipy_outbound_count{instance="01-2",peer="[192.168.66.170]:80"} 2
pipy_outbound_count{instance="01-2",peer="[192.168.66.64]:80"} 1
pipy_outbound_count{instance="01-2",peer="[192.168.66.170]:80"} 2
pipy_outbound_count{instance="01-2",peer="[192.168.66.64]:80"} 1
pipy_outbound_count{instance="01-4"} 3
pipy_outbound_count{instance="01-4",peer="[192.168.66.1]:6060"} 1
pipy_outbound_count{instance="01-4",peer="[192.168.66.64]:80"} 2
pipy_outbound_count{instance="01-4",peer="[192.168.66.170]:80"} 1
pipy_outbound_count{instance="01-4",peer="[192.168.66.170]:80"} 1
pipy_outbound_count{instance="01-4",peer="[192.168.66.64]:80"} 1
pipy_outbound_count{instance="01-4",peer="[192.168.66.170]:80"} 1
pipy_outbound_count{instance="01-4",peer="[192.168.66.64]:80"} 1

Impact

Large set of pipy instances and upstream connections, may cause the metric explosion over time.

Expect behavior

There should not be any duplcated metric.

Version info

pipy repo: 0.50.0-88

Pjs object property integer number key will be converted to float string

Bug Report

Pjs object property integer number key will be converted to float string

Pipy Version

Version     : nightly-202211132335
Commit      : 08f33aae4a5d2a943af8b574f0d27e4d4fb3a13e
Commit Date : Mon, 14 Nov 2022 11:04:23 +0800
Host        : Linux-3.10.0-1160.76.1.el7.x86_64 x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : Yes
Samples     : Yes

Steps to reproduce

pipy({
  cfg: { 1: "a", 2: "b" }
})

.task('3s')
  .onStart(() => new Message)
  .handleMessage(() => console.log(cfg))

$ pipy test.js

Output

2022-11-14 22:53:07.327 [INF] [config]
2022-11-14 22:53:07.327 [INF] [config] Module /test.js
2022-11-14 22:53:07.327 [INF] [config] ===============
2022-11-14 22:53:07.327 [INF] [config]
2022-11-14 22:53:07.327 [INF] [config]  [Task #1 (3s)]
2022-11-14 22:53:07.327 [INF] [config]  ----->|
2022-11-14 22:53:07.327 [INF] [config]        |
2022-11-14 22:53:07.327 [INF] [config]       handleMessage -->|
2022-11-14 22:53:07.327 [INF] [config]                        |
2022-11-14 22:53:07.327 [INF] [config]  <---------------------|
2022-11-14 22:53:07.327 [INF] [config]  
2022-11-14 22:53:07.327 [INF] [object Literal] {"1.000000":"a","2.000000":"b"}

Admin console to support providing specific IP

Feature Request

Is your feature request related to a problem? Please describe clearly and concisely what is it.

Sometimes or on some specific platforms it is desired to have admin console listen on specific IP:PORT
Describe the feature you would like, optionally illustrated by examples, and how it will solve the above problem.

Would be better to have command line option to allow passing in the specific IP address for admin console. Similar to what is being provided for --admin-port. So something like --admin-ip or something similar might be beneficial.
Describe considered alternative solutions, and the reasons why you have not proposed them as a solution here.

N/A
Does it break backward compatibility, if yes then what's the migration path?

No

文档资料还是太少了

东西是好东西

Accessing Existing/Non-Existing variables in onEnd not working

Pipy version

Using the latest commit

Version     : 0.50.0-18
Commit      : b80b1bc464f95b3ad6c2f2b38c2e7c0acc065877
Commit Date : Thu, 25 Aug 2022 11:47:51 +0800
Host        : Darwin-21.6.0 arm64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : Yes
Samples     : Yes

Bug Report

Accessing Global/Context variables in onEnd is not working.
Accessing any invalid or non-existing variable in onEnd doesn't generate any error/warning

Problem 1

For example trying to access any valid variable doesn't yield any outcome.

 .onEnd(
  () => console.log('print some global variable', __inbound.remoteAddress)
)

Won't print anything, but logging any string literal works.

Problem 2

For example trying to access any invalid or non-existing variable doesn't yield any error/warning.

 .onEnd(
  () => console.log('print some non existing variable', _not_existing_variable)
)

No error/warning is generated.

Can not access admin port with container

Reproduce steps

git clone [email protected]:flomesh-io/pipy.git
cd pipy
docker run --rm --name pipy -v $PWD/tutorial:/tutorial -p 6060:6060 flomesh/pipy:0.90.0-18  pipy --admin-port=6060 tutorial/02-echo/hello.js

Other

And then access IP:6060, it's not working for me. also tryed args of --admin-port=:6060 and --admin-port=192.168.3.87:6060 .

Pipy Logs

...
2023-03-09 05:50:49.131 [INF] [listener] Listening on TCP port 8080 at 0.0.0.0
2023-03-09 05:50:49.131 [INF] [listener] Listening on TCP port 8081 at 0.0.0.0
2023-03-09 05:50:49.131 [INF] [listener] Listening on TCP port 8082 at 0.0.0.0
2023-03-09 05:50:49.131 [INF] [start] Thread 0 started
2023-03-09 05:50:49.131 [INF] [admin] Starting admin service...
2023-03-09 05:50:49.132 [INF] [listener] Listening on TCP port 6060 at ::

Remind me if i missed something,Thanks.

whether it can be support javascript eval expression

Discussion

a config.json

[
  {
   "k": "abc",
   "v": "e.head.headers['user-agent'] + e.head.headers['host']"
  }
[

eg:

((
  conf= pipy.solve('config.json'),
) => pipy()

.pipeline()
 .handleMessageStart(
  e => (
    (headers = e.head.headers)  =>(
           conf.forEach(one => e.head.headers[one.k] = evel(one.v))
    )
   )()
 )()

result:

headers:
user-agent: chrome
host: 1.1.1.1
abc: chromd1.1.1.1

build.sh failing on Fedora 37

Similar to #136

....
[ 36%] Built target brotlicommon-static
[ 37%] Building CXX object deps/leveldb-1.23/CMakeFiles/leveldb.dir/util/env_posix.cc.o
[ 37%] Building CXX object deps/leveldb-1.23/CMakeFiles/leveldb.dir/util/status.cc.o
[ 37%] Building CXX object deps/leveldb-1.23/CMakeFiles/leveldb.dir/helpers/memenv/memenv.cc.o
[ 37%] Built target zlibstatic
[ 37%] Linking CXX static library libleveldb.a
[ 37%] Built target yajl_s
[ 37%] Built target brotlicommon
[ 37%] Built target yajl
[ 37%] Built target zlib
[ 37%] Built target leveldb
make: *** [Makefile:136: all] Error 2

Versions:

clang version 15.0.7 (Fedora 15.0.7-2.fc37)
cmake version 3.26.1
node v16.14.0
openssl.x86_64     1:3.0.8-1.fc37

Segfaults when context vars are duplicated

Bug Report

Duplicating context variables is causing Pipy to Segv on *nix based systems. Behavior is inconsistent on different Oses as on CentOS it crashes at startup, while on Ubuntu (focal) 20.04 LTS , it crashes at exit.

Mac M1 machine doesn't show this behavior

Pipy Version

Version     : nightly-202210120857
Commit      : 78b04769a3c9144aa42edb3b893bbce203c32e1f
Commit Date : Wed, 12 Oct 2022 16:57:28 +0800
Host        : Linux-5.15.0-1020-azure x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No

Steps to reproduce

$ pipy -e "pipy({_g: {}, _g: {}}).listen(8000)"

Output

$ ./pipy -e "pipy({_g: {}, _g: ''}).listen(8000)"
2022-10-14 14:13:09.638 [INF] [config]
2022-10-14 14:13:09.638 [INF] [config] Module
2022-10-14 14:13:09.638 [INF] [config] =======
2022-10-14 14:13:09.638 [INF] [config]
2022-10-14 14:13:09.638 [INF] [config]  [Listen on 8000 at 0.0.0.0]
2022-10-14 14:13:09.638 [INF] [config]  ----->|
2022-10-14 14:13:09.638 [INF] [config]        |
2022-10-14 14:13:09.638 [INF] [config]
2022-10-14 14:13:09.638 [INF] [config]  <---|
2022-10-14 14:13:09.638 [INF] [config]
2022-10-14 14:13:09.638 [INF] [listener] Listening on TCP port 8000 at 0.0.0.0
^C2022-10-14 14:13:12.648 [INF] [shutdown] Shutting down...
Segmentation fault (core dumped)

metrics of multi-threading version is not compatible with main branch

What happened

Start pipy repo and worker in different version, the repo is unable to get metrics from worker. Reporting error in console:

2023-01-04 14:01:04.162 [ERR] [stats] Invalid JSON structure for metrics

How to reproduce

Use samples/gateway for proxy worker and enable metrics plugin:

{
  "listen": 8000,
  "listenTLS": 8443,
  "plugins": [
    "plugins/router.js",
    "plugins/metrics.js",
    "plugins/jwt.js",
    "plugins/cache.js",
    "plugins/hello.js",
    "plugins/balancer.js",
    "plugins/serve-files.js",
    "plugins/default.js"
  ],
  ...
}

Run pipy repo from main branch, and worker from multi-threading branch, or
Run pipy repo from multi-threading branch, and worker from main branch

Expect behavior

repo could receive the metrics from worker, they should be compatible

Version info

repo

Version     : nightly-202301041356
Commit      : 9bae523b94099360fd415e86cc84ad66e6514ac8
Commit Date : Fri, 30 Dec 2022 14:32:56 +0800
Host        : Linux-5.15.0-39-generic x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No
Samples     : No

worker

Version     : nightly-202301041229
Commit      : 55c9e5d546f1e415688decfee7d823983577dd4c
Commit Date : Wed, 4 Jan 2023 11:37:10 +0800
Host        : Linux-5.15.0-39-generic x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No
Samples     : No

Pipy Repo REST API didn't work

Try REST API in https://flomesh.io/pipy/docs/zh/operating/repo/3-api, the interface behavior is inconsistent with expectations.

Repeat method：

[I] user@mac ~> # a new start pipy
[I] user@mac ~> curl http://localhost:6060/api/v1/repo
[I] user@mac ~> curl -X POST http://localhost:6060/api/v1/repo/hello
[I] user@mac ~> curl http://localhost:6060/api/v1/repo
/hello⏎                                                                         
[I] user@mac ~> curl -s http://localhost:6060/api/v1/repo/hello
{"version":"0","path":"/hello","main":"/main.js","files":["/main.js"],"editFiles":[],"erasedFiles":[],"baseFiles":[],"derived":[],"instances":{}}⏎              
[I] user@mac ~> curl -s http://localhost:6060/api/v1/repo/hello/main.js
((
  // Global variables go here, e.g.:
  // config = pipy.solve('config.js'),

) => pipy({
  // Context variables go here, e.g.:
  // _target: null,

})

  // Pipeline layouts go here, e.g.:
  .listen(80)
  .dump()
  .dummy()

)()
[I] user@mac ~> curl -X POST http://localhost:6060/api/v1/repo/hello/main.js --data """pipy()
                             .listen(8080)
                             .serveHTTP(
                               new Message('Hi!')
                             )"""
[I] user@mac ~> curl -s http://localhost:6060/api/v1/repo/hello/main.js
((
  // Global variables go here, e.g.:
  // config = pipy.solve('config.js'),

) => pipy({
  // Context variables go here, e.g.:
  // _target: null,

})

  // Pipeline layouts go here, e.g.:
  .listen(80)
  .dump()
  .dummy()

)()
[I] user@mac ~> # main.js didn't change
[I] user@mac ~> curl http://localhost:6060/api/v1/repo
/hello
/hello/main.js⏎                                                                 
[I] user@mac ~> curl -s http://localhost:6060/api/v1/repo/hello
{"version":"0","path":"/hello","main":"/main.js","files":["/main.js"],"editFiles":[],"erasedFiles":[],"baseFiles":[],"derived":[],"instances":{}}⏎              
[I] user@mac ~> # didn't have `edit files`
[I] user@mac ~>

The update file interface appears to be recognized as the new repo interface。

Pipy version：

git log -1
commit 893fa2de9b3546c2f53b39bd2ec5f50e2503b597 (HEAD -> main)
Author: pajama-coder <[email protected]>
Date:   Wed Jun 5 13:30:22 2024 +0800

    [api] Use a flush in print() and println()

内置的Dataformat能不能加入gzip 等解压缩工具

Feature Request

1.我们又很多业务场景都是body都是压缩上传，pipy无法解码

pipy crash when visit admin-port

How to reproduce

Start pipy, it may crash when visit http://localhost:6060/metrics, I used wrk

wrk -c 10 -t 4 -d 300s http://localhost:6060/metrics

I'm not sure but try dropping OS cache before testing may be a bit easier to reproduce.

sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

$ pipy --threads=4 --admin-port=6060 -e 'pipy().listen(8000).serveHTTP(()=>new Message("hello\n"))'
2023-01-04 15:33:32.598 [INF] [config]
2023-01-04 15:33:32.598 [INF] [config] Module
2023-01-04 15:33:32.598 [INF] [config] =======
2023-01-04 15:33:32.598 [INF] [config]
2023-01-04 15:33:32.598 [INF] [config]  [Listen on 8000 at 0.0.0.0]
2023-01-04 15:33:32.598 [INF] [config]  ----->|
2023-01-04 15:33:32.598 [INF] [config]        |
2023-01-04 15:33:32.598 [INF] [config]       serveHTTP -->|
2023-01-04 15:33:32.598 [INF] [config]                    |
2023-01-04 15:33:32.598 [INF] [config]  <-----------------|
2023-01-04 15:33:32.598 [INF] [config]
2023-01-04 15:33:32.599 [INF] [listener] Listening on TCP port 8000 at 0.0.0.0
2023-01-04 15:33:32.599 [INF] [admin] Starting admin service...
2023-01-04 15:33:32.600 [INF] [listener] Listening on TCP port 6060 at ::
2023-01-04 15:33:32.600 [INF] Running 4 worker threads...
Segmentation fault (core dumped)

btw, start pipy again soon after it crashed, it may complain the admin-port is being used. However I can't find anything about port 6060 via lsof or ss command. This may be related to the OS but the crash issue is the point.

$ pipy --threads=4 --admin-port=6060 -e 'pipy().listen(8000).serveHTTP(()=>new Message("hello\n"))'
2023-01-04 15:44:13.474 [INF] [config]
2023-01-04 15:44:13.474 [INF] [config] Module
2023-01-04 15:44:13.474 [INF] [config] =======
2023-01-04 15:44:13.474 [INF] [config]
2023-01-04 15:44:13.475 [INF] [config]  [Listen on 8000 at 0.0.0.0]
2023-01-04 15:44:13.475 [INF] [config]  ----->|
2023-01-04 15:44:13.475 [INF] [config]        |
2023-01-04 15:44:13.475 [INF] [config]       serveHTTP -->|
2023-01-04 15:44:13.475 [INF] [config]                    |
2023-01-04 15:44:13.475 [INF] [config]  <-----------------|
2023-01-04 15:44:13.475 [INF] [config]
2023-01-04 15:44:13.475 [INF] [listener] Listening on TCP port 8000 at 0.0.0.0
2023-01-04 15:44:13.475 [INF] [admin] Starting admin service...
[listener] Cannot start listening on TCP port 6060 at ::: bind: Address already in use

Expect behavior

Everything is fine and no crash

Version info

OS:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04 LTS
Release:        22.04
Codename:       jammy

$ uname -r
5.15.0-39-generic

pipy:

$ pipy -v
Version     : nightly-202301041410
Commit      : 55c9e5d546f1e415688decfee7d823983577dd4c
Commit Date : Wed, 4 Jan 2023 11:37:10 +0800
Host        : Linux-5.15.0-39-generic x86_64
OpenSSL     : OpenSSL 1.1.1q  5 Jul 2022
Builtin GUI : No
Samples     : No