Comments (11)
negbie
commented on June 18, 2024
Hi,
thank you for providing a log file. As you can see postgres runs out of shared memory. This could have various reasons. Most obvious one would be that you simply have not enough memory. Next possible reasons are that the max_locks_per_transaction limit and shared buffer setting inside postgres config are to low.
Another reason could be that heplify-server does something wrong with the transaction. Since this part of the code runs highly concurrent it could be something which isn't obvious at the first glance.
Please check the first possible reasons I mentioned first. If you still have this I will dig deeper. Keep in mind that all the code related to homer7 was tested with a very low amount of traffic and in total time mby not more than some minutes because of h7 early alpha stage.
from heplify-server.
rogelio-telnyx
commented on June 18, 2024
Hello, for the DB we're using the default config file from the postgres:10.5-alpine docker image:
#max_locks_per_transaction = 64 # min 10
shared_buffers = 128MB # min 128kB
Upon starting heplify-server it sends approx 250pps of HEP and the mem usage of the postgres container spikes up to 1GB, until heplify-server crashes.
from heplify-server.
negbie
commented on June 18, 2024
Do you still have the "out of shared memory" warnings?
from heplify-server.
rogelio-telnyx
commented on June 18, 2024
Yes I do, I haven't made any changes yet.
from heplify-server.
negbie
commented on June 18, 2024
You should try to fix that first. One week ago I tested it with 1k pps and hadn't those warnings.
from heplify-server.
negbie
commented on June 18, 2024
@rogelio-telnyx please do following. mount a postgresql.conf file like:
volumes:
- ./postgresql.conf:/etc/postgresql.conf
command: 'postgres -c config_file="/etc/postgresql.conf"'
raise inside postgresql.conf max_locks_per_transaction like:
max_locks_per_transaction = 512
Keep an eye on your memory from nothing comes nothing so you need enough for your load.
Please tell me how many cores you machine has.
from heplify-server.
rogelio-telnyx
commented on June 18, 2024
@negbie Alexandr has helped us fine tune our postgres instances, and it seems like the shared mem issues is gone, but now heplify-server is crashing every 2 min with the following errors:
nodeID: 6553600, protoType: 1, version: 2, protocol: 6, length: 347, flow: 10.255.0.1:46677->10.15.11.4:5070
2018/10/16 19:29:14.445200 decoder.go:125: WARN parseStartLine err: received err while parsing start line: parseStartLineRequest err: request line did not split on LWS correctly
"nt/8000\r\na=fmtp:101 0-15\r\na=sendrecv\r\na=maxptime:20\r\n\r\n--sonus-content-delim\r\nContent-Disposition: signal; handling=required\r\nContent-Length: 55\r\nContent-Type: application/isup; version=ansi88; base=ansi88\r\n\r\n\x01\x10`\x10\n\x03\x06\r\x03\a\x03\x10&(\n\a\x03\x13\"% %\x03\"\x05\x05\b\x03\x10\x15pq\x11=\x01\x1c\x033'B\r\n\r\n--sonus-content-delim--\r\n"
nodeID: 6553600, protoType: 1, version: 2, protocol: 6, length: 281, flow: 10.255.0.1:46677->10.15.11.4:5070
2018/10/16 19:29:15.609049 decoder.go:125: WARN parseStartLine err: received err while parsing start line: parseStartLineRequest err: request line did not split on LWS correctly
"ent/8000\r\na=fmtp:101 0-15\r\na=sendrecv\r\na=maxptime:20\r\n\r\n--sonus-content-delim\r\nContent-Disposition: signal; handling=required\r\nContent-Length: 65\r\nContent-Type: application/isup; version=ansi88; base=ansi88\r\n\r\n\x01\x11H\x10\x03\x06\r\x03\a\x03\x10W$R\n\a\x03\x13YtV\x01\a\x03\x10Yt)DQ\b\x03\x10W$bQ=\x01\x18\x03Y$s\x01=\r\n\r\n--sonus-content-delim--\r\n"
nodeID: 6553600, protoType: 1, version: 2, protocol: 6, length: 288, flow: 10.255.0.1:46677->10.15.11.4:5070
2018/10/16 19:29:15.682227 decoder.go:125: WARN parseStartLine err: received err while parsing start line: parseStartLine err: length of s.Val is less than 3. Invalid start line
"\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:18 G729/8000\r\na=rtpmap:101 telephone-event/8000\r\na=fmtp:101 0-15\r\na=sendrecv\r\na=ptime:20\r\n"
nodeID: 6553600, protoType: 1, version: 2, protocol: 6, length: 145, flow: 10.255.0.1:46677->10.15.11.4:5070
2018/10/16 19:29:16.526109 decoder.go:125: WARN parseStartLine err: received err while parsing start line: parseStartLineRequest err: request line did not split on LWS correctly
"MU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:18 G729/8000\r\na=rtpmap:101 telephone-event/8000\r\na=fmtp:101 0-15\r\na=sendrecv\r\na=ptime:20\r\n\r\n--sonus-content-delim\r\nContent-Disposition: signal; handling=required\r\nContent-Length: 68\r\nContent-Type: application/isup; version=ansi88; base=ansi88\r\n\r\n\x01\x10\b\x10\x03\x06\r\x03\a\x03\x1061\x10\n\a\x03\x13\x197\x11\b\x03\x106A\x03\x19\b\x106q\x16Y`\x10 Cell Phone NY\r\n--sonus-content-delim--\r\n"
nodeID: 6553600, protoType: 1, version: 2, protocol: 6, length: 368, flow: 10.255.0.1:46677->10.15.11.4:5070
2018/10/16 19:29:16.748856 decoder.go:125: WARN parseStartLine err: received err while parsing start line: parseStartLineRequest err: request line did not split on LWS correctly
"/8000\r\na=fmtp:101 0-15\r\na=sendrecv\r\na=maxptime:20\r\n\r\n--sonus-content-delim\r\nContent-Disposition: signal; handling=required\r\nContent-Length: 53\r\nContent-Type: application/isup; version=ansi88; base=ansi88\r\n\r\n\x01\x10H\x10\n\x03\x06\r\x03\a\x03\x10\ta3\n\a\x03\x137ag\b\x03\x10\t!!=\x01\x18\x03\x06!\x01>\r\n\r\n--sonus-content-delim--\r\n"
nodeID: 6553600, protoType: 1, version: 2, protocol: 6, length: 275, flow: 10.255.0.1:46677->10.15.11.4:5070
2018/10/16 19:29:18.057238 decoder.go:125: WARN parseStartLine err: received err while parsing start line: parseStartLineRequest err: request line did not split on LWS correctly
"ent/8000\r\na=fmtp:101 0-15\r\na=sendrecv\r\na=maxptime:20\r\n\r\n--sonus-content-delim\r\nContent-Disposition: signal; handling=required\r\nContent-Length: 50\r\nContent-Type: application/isup; version=ansi88; base=ansi88\r\n\r\n\x01\x10 \x10\n\x03\x06\r\x03\a\x03\x10 \n\a\x03\x133`i\x01R\b\x03\x10\x16d1q@=\x01\x11\x03\x164\x10\r\n\r\n--sonus-content-delim--\r\n"
nodeID: 6553600, protoType: 1, version: 2, protocol: 6, length: 279, flow: 10.255.0.1:46677->10.15.11.4:5070
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x812844]
goroutine 491 [running]:
github.com/negbie/heplify-server/database.(*SQLHomer7).insert(0xc421c94090, 0xc42186c0c0)
/go/src/github.com/negbie/heplify-server/database/sqlhomer7.go:123 +0x784
github.com/negbie/heplify-server/database.(*Database).Run.func1(0xc421bc60c0)
/go/src/github.com/negbie/heplify-server/database/database.go:59 +0x3c
created by github.com/negbie/heplify-server/database.(*Database).Run
/go/src/github.com/negbie/heplify-server/database/database.go:58 +0x1ff
from heplify-server.
negbie
commented on June 18, 2024
@rogelio-telnyx ok thats good and thanks to alexandr! Since it crashes on regular time would it be possible to send alexandr a pcap so I can take a look on this tomorrow?
from heplify-server.
negbie
commented on June 18, 2024
I still need to know more about your setup. How many cores does your machine has?
from heplify-server.
negbie
commented on June 18, 2024
@rogelio-telnyx I pushed some changes. Please update and give me an update.
from heplify-server.
rogelio-telnyx
commented on June 18, 2024
@negbie we've deployed the latest version and we haven't had any crashes for the last 3hs, which is very promising. Thanks for the fix, I think we can close this issue.
from heplify-server.
Related Issues (20)
- panic when decoding HOT 1
- heplify-server.toml HOT 3
- Remove sip header via lua or other means? HOT 1
- Missing src_id, dst_id, ID labels HOT 4
- DB Error - could not map dynamic shared memory segment HOT 7
- Duplication with 'Dedup' set to 'true' HOT 4
- Prometheus metrics HOT 5
- [FEATURE REQ] multiple sockets listening at same hep udp port: setting "so_reuseport" in udp.go HOT 1
- Heplify-server não apresenta os dados HOT 31
- Prometheus was not accessed on installation and access is denied http and telnet on port 9090 HOT 1
- heplify-server resets TCP connection HOT 4
- Certificate with only 'Digital Signature' Key Usage not accepted
- Add subnet to PromTargetIP HOT 2
- ISUP multipart encoding issue HOT 4
- correlation id lost when pushing to loki HOT 1
- timestamp of hep packets discarded in loki
- Configure Multiply connections
- Multiple instances cause rotator to get stuck
- image for release v1.59.7 not yet available on ghcr HOT 2
- heplify-server's DB health monitoring HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from heplify-server.