Comments (41)
Tresillo2017
commented on June 2, 2024
The problem with orange cloud is that the file limit is 100mb.
from chibisafe.
antonioraminhos
commented on June 2, 2024
that doesn't matter, chibisafe automatically split the file into chunks
from chibisafe.
Tresillo2017
commented on June 2, 2024
Well, i've tried with orange cloud and I'm getting the same error, I guess is nginx proxy side
Request URL:
https://example.com/api/upload
Status code:
(413) Request Entity Too Large
Headers:
Transfer-Encoding: chunked
Connection: keep-alive
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=iVvFsWjCdyE6GC04OysOG%2Fl5zNPgnnKI6%2Fy6464YhuxNt%2FzyiFsVnrJzcGm97a%2BpZcUrKTJvZDnFV57BQnmpjTFrjhYRS39s%2F5jb40Rjs7UvhTQjh%2F3KC8AAiuRNQOI1NMY1"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Content-Type-Options: nosniff
CF-RAY: 8490e2ba897866ad-MAD
alt-svc: h3=":443"; ma=86400
Content-Type: text/html
Date: Sun, 21 Jan 2024 16:20:38 GMT
Server: cloudflare
Response text:
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>cloudflare</center>
<script>(function(){var js = "window['__CF$cv$params']={r:'8490e2ba897866ad',t:'MTcwNTg1NDAzOC4xNzEwMDA='};_cpo=document.createElement('script');_cpo.nonce='',_cpo.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js',document.getElementsByTagName('head')[0].appendChild(_cpo);";var _0xh = document.createElement('iframe');_0xh.height = 1;_0xh.width = 1;_0xh.style.position = 'absolute';_0xh.style.top = 0;_0xh.style.left = 0;_0xh.style.border = 'none';_0xh.style.visibility = 'hidden';document.body.appendChild(_0xh);function handler() {var _0xi = _0xh.contentDocument || _0xh.contentWindow.document;if (_0xi) {var _0xj = _0xi.createElement('script');_0xj.innerHTML = js;_0xi.getElementsByTagName('head')[0].appendChild(_0xj);}}if (document.readyState !== 'loading') {handler();} else if (window.addEventListener) {document.addEventListener('DOMContentLoaded', handler);} else {var prev = document.onreadystatechange || function () {};document.onreadystatechange = function (e) {prev(e);if (document.readyState !== 'loading') {document.onreadystatechange = prev;handler();}};}})();</script><script defer src="https://static.cloudflareinsights.com/beacon.min.js/v84a3a4012de94ce1a686ba8c167c359c1696973893317" integrity="sha512-euoFGowhlaLqXsPWQ48qSkBSCFs3DPRyiwVu3FjR96cMPx+Fr+gpWRhIafcHwqwCqWS42RZhIudOvEI+Ckf6MA==" data-cf-beacon='{"rayId":"8490e2ba897866ad","r":1,"version":"2024.1.0","token":"190f86ad714a4a4fb50cab9c4f0672b7"}' crossorigin="anonymous"></script>
</body>
</html>
from chibisafe.
pilar6195
commented on June 2, 2024
The chunk splitting only works if you use the website. This is due to the client side custom logic required for it to work. So unfortunately using something like ShareX will upload the file normally without splitting the file and therefore be subject to cloudflare's 100MB limit.
from chibisafe.
Tresillo2017
commented on June 2, 2024
I can upload files from the web but it's take to long for the first part of the file to be uploaded.
from chibisafe.
Tresillo2017
commented on June 2, 2024
Happens If I use share x and the local ip addr.
from chibisafe.
Pitu
commented on June 2, 2024
@Tresillo2017 as @pilar6195 said, this is a limitation between ShareX and our upload logic. ShareX doesn't split by chunks but the website does, ultimately making ShareX hit the 100mb per file upload limit that CloudFlare forces upon you.
Other than that, make sure the max file size in nginx is set to a big number.
from chibisafe.
pilar6195
commented on June 2, 2024
@Tresillo2017 to give a bit more detail:
There is the Chunk Size and Max Size settings.
The Chunk Size is the max size a chunk should be when splitting. The Max Size is what the overall max size a file can be.
If you upload a file using the website, the file will be split into chunks if the file is larger than the Chunk Size. Max Size is used to determine if the overall file size is too large to be uploaded.
If you upload a file outside of the website (like with shareX) then the file gets uploaded normally since chunk uploading would not be possible unless the client supports our implementation. In this case the Max Size is ignored and Chunk Size is used in its place to determine if a file is too large to be uploaded.
This means if you upload a file with shareX, with or without cloudflare, the file cannot be larger than what Chunk Size is set to.
I can upload files from the web but it's take to long for the first part of the file to be uploaded.
As for the slow first chunk upload, that'll be something we'd have to look into at some point. We haven't come across that issue yet.
from chibisafe.
Pitu
commented on June 2, 2024
We haven't come across that issue yet.
Until now
from chibisafe.
Aareksio
commented on June 2, 2024
Isn't limiting the single file upload size to Chunk Size instead of Max Size something that could be easily changed?
@Tresillo2017 uses proper nginx limit (client_max_body_size 6000M;) and does not proxy with CloudFlare. There is no technical limitation that would prevent a larger file to be successfully uploaded. From outsider perspective, only minor conditional changes to file size validation logic are required here and there.
from chibisafe.
Tresillo2017
commented on June 2, 2024
I have the following values set to:
- Chunk size: 81000000
- Max Size: 7000000000
The thing is that it could be cloudflare but when try to upload files using the internal ip, I get the same error and it can't be cloudflare or nginx.
from chibisafe.
Pitu
commented on June 2, 2024
@Tresillo2017 change the chunk size to 7000000000 and try without cloudflare
from chibisafe.
Tresillo2017
commented on June 2, 2024
@Tresillo2017 change the chunk size to 7000000000 and try without cloudflare
Tried without cloudflare proxy and got the same error, via web. (forget the file name)
from chibisafe.
lakema17
commented on June 2, 2024
Can you do me a favor and try setting client_max_body_size 0? It seems like an issue with NPM according to this issue where the fix is just disabling the setting altogether.
from chibisafe.
Pitu
commented on June 2, 2024
@Tresillo2017 That error reeks of nginx problem. What @lakema17 suggested might work.
from chibisafe.
Tresillo2017
commented on June 2, 2024
@Tresillo2017 That error reeks of nginx problem. What @lakema17 suggested might work.
Works on the browser without cloudflare proxy, but takes a while to start the first chunk. (like 5-10s)
If I set the file chunk size to 100mb it takes to long to upload
from chibisafe.
lakema17
commented on June 2, 2024
@Tresillo2017 That error reeks of nginx problem. What @lakema17 suggested might work.
Works on the browser without cloudflare proxy, but takes a while to start the first chunk. (like 5-10s)
This could be due to proxy requests being buffered by Nginx. You can try disabling the setting by specifying proxy_request_buffering off; and testing it, though it is recommended that you keep buffering on for performance and security reasons.
from chibisafe.
Tresillo2017
commented on June 2, 2024
@Tresillo2017 That error reeks of nginx problem. What @lakema17 suggested might work.
Works on the browser without cloudflare proxy, but takes a while to start the first chunk. (like 5-10s)
This could be due to proxy requests being buffered by Nginx. You can try disabling the setting by specifying
proxy_request_buffering off;and testing it, though it is recommended that you keep buffering on for performance and security reasons.
I think it's worse with that parameter
from chibisafe.
lakema17
commented on June 2, 2024
Are you able to monitor your server's hardware metrics (specifically iowait or disk latency) while uploading? I want to make sure it's not your storage backend that's facing issues. And yeah turning request buffering off does degrade performance since you have to write data directly to disk (as far as I'm aware), which isn't too great if you're running on a hard disk.
from chibisafe.
Tresillo2017
commented on June 2, 2024
The disk name is backup
from chibisafe.
lakema17
commented on June 2, 2024
Unfortunately bpytop does not show iowait, which is what I was looking for. You can use top and monitor the wa metric., but it's not as accurate as running benchmarks directly on the disk. You can try changing directory to your disk backup and use this benchmark and disable the other benches except for io curl -sL yabs.sh | bash -s -- -gin
e: Apparently you can enable the iowait metric in the CPU graph. It's somewhere in the settings.
from chibisafe.
Tresillo2017
commented on June 2, 2024
It's an hdd btw
from chibisafe.
lakema17
commented on June 2, 2024
Yeah it's highly likely you're running into a HDD bottleneck. The 5 second wait you're seeing seems to be nginx flushing the chunks of data onto the disk. Depending on how your drive is mounted to the filesystem, the 20MB/s (100MB / 5 seconds) approximation is feasible.
from chibisafe.
Tresillo2017
commented on June 2, 2024
I don't remember how it's mounted but that speeds doesn't seem right for an hdd it normally would be 100mbs or something like that.
-------- Original message --------From: Lake ***@***.***> Date: 1/23/24 10:52 PM (GMT+01:00) To: chibisafe/chibisafe ***@***.***> Cc: Tomás Palma ***@***.***>, Mention ***@***.***> Subject: Re: [chibisafe/chibisafe] [BUG] Nginx proxy (Issue #542)
Yeah it's highly likely you're running into a HDD bottleneck. The 5 second wait you're seeing seems to be nginx flushing the chunks of data onto the disk. Depending on how your drive is mounted to the filesystem, the 20MB/s (100MB / 5 seconds) approximation is feasible.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
from chibisafe.
lakema17
commented on June 2, 2024
If you look at your benchmark results, it highly depends on the block size of the data being written. You can check your mount settings using mount | grep "/dev/sdb1". And yeah your disk doesn't really look like it does 100MB/s like you think it does
from chibisafe.
Tresillo2017
commented on June 2, 2024
If you look at your benchmark results, it highly depends on the block size of the data being written. You can check your mount settings using
mount | grep "/dev/sdb1". And yeah your disk doesn't really look like it does 100MB/s like you think it does
It's mounted like that.
Should I change the drive location into a network mounted nfs drive? (1tb).
from chibisafe.
Tresillo2017
commented on June 2, 2024
If you look at your benchmark results, it highly depends on the block size of the data being written. You can check your mount settings using
mount | grep "/dev/sdb1". And yeah your disk doesn't really look like it does 100MB/s like you think it does
The thing is that download speeds are really good I have 1gb
from chibisafe.
Tresillo2017
commented on June 2, 2024
I can't still upload big files while on local ip. I get payload to large. I have to see what's causing that because locally there's no proxy involved
from chibisafe.
lakema17
commented on June 2, 2024
If you look at your benchmark results, it highly depends on the block size of the data being written. You can check your mount settings using
mount | grep "/dev/sdb1". And yeah your disk doesn't really look like it does 100MB/s like you think it doesThe thing is that download speeds are really good I have 1gb
48MB/s is not 1Gbps (1 byte is 8 bits, so 48MB/s is approx. 384Mbit/s). Anyways, I wouldn't say it's rare to have a gap between read and write speeds, especially for SSDs, but yeah for hard disk they're usually closer. You could try monitoring disk activity (read and write speeds) as the first chunk is being uploaded, timing how long the disk is active for compared to how long chibi waits for the data to be written. It could be a problem with something else, considering how the wait only happens on the first chunk. I'm not seeing that same wait on my system, so the best I can tell you to do is profile as much of your system as possible while doing the upload. You could also try running chibisafe in debug mode by using yarn run dev and see if the logs have anything to give you, but that's a bit more involved and you might want to run a separate, temporary instance for that.
from chibisafe.
Tresillo2017
commented on June 2, 2024
I just tried on a local enviroment with the git repository and not the docker image, and I'm getting the same payload to big error while trying to upload directly from sharex.
Uploading for the web worked fine and fast. I'm thinking the problem is the hdd of the server.
from chibisafe.
lakema17
commented on June 2, 2024
When uploading with ShareX, max file size is limited to your chunk size, if I recall correctly. You could try increasing chunk size or just not use ShareX for large files.
from chibisafe.
Tresillo2017
commented on June 2, 2024
When uploading with ShareX, max file size is limited to your chunk size, if I recall correctly. You could try increasing chunk size or just not use ShareX for large files.
That's right, I uploaded a 63mb video via sharex and worked fast and without any errors.
from chibisafe.
Tresillo2017
commented on June 2, 2024
Looks like the delay is network side (apart from the disk)
Uploading from local lower the wait time to 2s while with the domain it was 13s
from chibisafe.
lakema17
commented on June 2, 2024
Can you try uploading to Chibisafe over WAN? Basically just port forward Chibi and try uploading to it using your public_ip:24424
I have a sneaking suspicion it's a reverse proxy issue, but would like some verification.
Though I guess you could also run the reverse proxy on port 80 and access it over LAN instead
from chibisafe.
Tresillo2017
commented on June 2, 2024
Can you try uploading to Chibisafe over WAN? Basically just port forward Chibi and try uploading to it using your public_ip:24424 I have a sneaking suspicion it's a reverse proxy issue, but would like some verification.
Though I guess you could also run the reverse proxy on port 80 and access it over LAN instead
There's a lot of delay when accessing it over the internet.
from chibisafe.
rakema01
commented on June 2, 2024
That's fine. What's interesting is that we're not seeing the same wait time on the first chunk. I assume you're not using the reverse proxy, so that might be the issue here. Try using NPM again and see if the wait time comes back
from chibisafe.
Tresillo2017
commented on June 2, 2024
That's fine. What's interesting is that we're not seeing the same wait time on the first chunk. I assume you're not using the reverse proxy, so that might be the issue here. Try using NPM again and see if the wait time comes back
This it what happens, it's interesting to see the difference between the first and second chunk
from chibisafe.
rakema01
commented on June 2, 2024
Huh... So the first chunk doesn't have that random 5s wait anymore? That's good I guess, but I'm not sure what happened there.
from chibisafe.
Tresillo2017
commented on June 2, 2024
Seems to work fine with the latest Ubuntu release 23.10. Now I get 900ms on local
from chibisafe.
Tresillo2017
commented on June 2, 2024
I'm using now ARC browser and it's working really fine. 100ms of response but at the last chunk I got 2.1 minutes.
from chibisafe.
Pitu
commented on June 2, 2024
Feel free to open this issue again if you need
from chibisafe.
Related Issues (20)
- [BUG] "Serve Uploads From" banner does not link to the right place HOT 1
- Snippets on homepage HOT 1
- [BUG]Cannot Read Properties HOT 18
- [FEATURE REQUEST] options to disable Preview generation HOT 1
- [BUG] Caddy reports an error, cannot access port 8001 HOT 10
- [FEATURE REQUEST] Store image thumbnails and video previews in S3 bucket
- [FEATURE REQUEST] Message System HOT 1
- [BUG] Trying to download any file results in 404 HOT 3
- [FEATURE REQUEST] Support MinIO Path-style
- [FEATURE REQUEST] 2 Factor Authentication
- [FEATURE REQUEST] Sub-folders (albums within an album) HOT 1
- [FEATURE REQUEST] Image preview Zoom with mousewheel and move with click+cursor HOT 1
- [FEATURE REQUEST] Sorting files on gallery view or list view by their original names
- [FEATURE REQUEST] community support
- [BUG] Uploads Link Error HOT 1
- [FEATURE REQUEST] Bulk Download of files
- How to change port?
- [FEATURE REQUEST] Add usage metrics for files HOT 1
- [FEATURE REQUEST] Remove the requirement to have a 'Generated Filename Length' for fully customizable urls HOT 2
- [BUG] Getting error when trying to build chibisafe on docker HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chibisafe.