Bandwidth rate limting about acd_cli HOT 18 CLOSED

Rate limiting would be great. Even at 100mbit it fully saturates the line.

from acd_cli.

Here is the best workaround I found. It restricts by user, so it is best to create a new user just for this purpose. The user also needs to be able to read the files.

I tried trickle, but it doesn't work with acd_cli. I would use wondershaper, but then that would limit all uploads from the machine. Which is especially bad if there is only one network interface.

adduser acd
rate_limit_by_user eth0 1Gbit 16Mbit acd
su - acd
acd_cli upload foo/ /bar/

Usage: ./rate_limit_by_user interface interface_speed upload_limit username
Example: ./rate_limit_by_user eth0 1Gbit 16Mbit acd

rate_limit_by_user:

#!/bin/bash

INTERFACE=$1
INTERFACE_SPEED=$2
UPLOAD_LIMIT=$3
USER=$4

tc qdisc add dev $INTERFACE root handle 2:0 htb default 99
tc class add dev $INTERFACE parent 2:0 classid 2:1 htb rate $INTERFACE_SPEED ceil $INTERFACE_SPEED
tc class add dev $INTERFACE parent 2:1 classid 2:11 htb rate $UPLOAD_LIMIT ceil $UPLOAD_LIMIT prio 2
tc qdisc add dev $INTERFACE parent 2:11 handle 20: sfq perturb 10

iptables -t mangle -A OUTPUT -p tcp -m owner --uid-owner $USER -j CLASSIFY --set-class 2:11

from acd_cli.

Use trickle, it works fine for this http://man.cx/trickle

from acd_cli.

I tried trickle. It didn't work for me, and I went as far as stracing acd_cli. I don't see acd_cli using socket syscalls, which is how trickle works.

from acd_cli.

Strange, I used it and it limited it to whatever I set. Not saying you are lying though, just that it seems to work when I tried it.

from acd_cli.

My version of tricke is 1.07. What is your version?

from acd_cli.

+1 from my side - makes sense for uploading in the background. Ideal solution would be to listen for keyboard input while uploading is active and allow to temporarily limit the upload (by pressing s or something similar) just like you can enable "slow mode" in Transmission. This way the user could upload at full speed, but slow it down, when bandwidth is needed for other purposes.

from acd_cli.

I'd rather it not be an interactive thing like having to hit s. That makes it difficult to have it happen in cron or some other scheduled job system. A command line option would be the ideal way to control it for those scenarios.

from acd_cli.

Obviously, the interactive part should be optional and exist beside a command-line option ... imagine you started uploading 100 GB of files and figure out that you still wanted to do this important Skype call. Sure, you can stop the upload all together, but an interactive pause / slow-down option sounds much more appealing to me.

from acd_cli.

I will not implement this feature because (1) no relevant justification is given and (2) there is no easy way to do it properly.

A single (i.e. non-parallel) upload will always behave "fair" in networking terms. That is, it will share the available bandwidth with other applications. Drawing a comparison with a torrent client is not very convincing, as a torrent client easily has to handle dozens of simultaneous connections.

If someone really needs this, there are some ways around this limitation like cgroups + blkio, a ssl-capable proxy with bandwidth throttling, etc.

from acd_cli.

I also think this would be a good setting to have available. It is not a difficult thing to do, from what I can read. However, I am not a programmer. I have seen error messages stating that I am getting Throttled by Amazon as well.

What I believe would be a good enhancement is to have a command line switch when running the upload task that sets a rate limit for uploading. This can help prevent the messages from Amazon AND allow me to not have to stop using acd_cli when I want to use my server from outside my local network while acd_cli is doing an upload. Additionally, I cannot do an acd_cli upload when someone wants to do any online gaming since it causes so much traffic that the game play is affected.

Here is an article that might help in accomplishing this task.

How to handle a "Throttling - Maximum sending rate exceeded" error
https://sesblog.amazon.com/post/TxKR75VKOYDS60/

(yes, I know it is for Amazon SES, not ACD, but the links and sample code for rate limiting might help.)

from acd_cli.

That code appears to rely heavily on a java library (developed by Google for Android I assume), whereas this application is being written in Python. Unless you find a similar capable API for python, it would probably be more work than you think to accomplish this. It surely is possible and useful at times, but for now I would recommend to simply use an existing approach.

A simple solutions for this problem could be to use any of the available bandwidth limiters to accomplish what you want (some can limit bandwidth per application or globally for a user): http://superuser.com/questions/126642/throttle-network-bandwidth-per-application-in-mac-os-x

from acd_cli.

@yadayada I would suggest you reconsider this request. Limiting the discussion to single-threaded uploads for a moment....

I would not categorize the network environment as "fair" when you have bandwidth contention on an upstream interface. In an ideal TCP-only environment n sockets uploading will use approximately bw/n for each socket.

The problem is, not all sockets stream at a constant rate, buffering makes the math ugly, and not everything is TCP. A real world situation -- Given a network with a single-threaded acd_cli upload in progress, throw in a SIP phone call with a non-CBR codec. The person on the other end can barely understand you. Even with a CBR codec buffering by the upstream devices (cable modems are notoriously bad here) and that 1980's drive thru window is more intelligible.

Sure, you can do some QoS things on the router/firewall... if it supports it. But if you're on a shared network, be it with other services, or other people, being able to be the "nice" neighbor is important.

As far as multiple threads go, ideally they would all use the same bandwidth "pool" but I can see, and agree, the problems in doing so may not be worthwhile. You can attempt to mimic that behavior somewhat by dividing the total pool amongst the number of threads, which has some potential issues (under utilization of the bandwidth allocated is a possibility for many small files.) Or, easiest overall, copy that pool to each thread. Either way, it's probably easier to code the latter, and spend a few extra minutes on documentation.

from acd_cli.

I think yadayada has to draw the line somewhere, in this case the developer offered to merge in this functionality if someone else added the functionality. This allows yadayada to stay focused on the more urgent bugs, and doesn't bury it in a backlog prioritized so low it never gets addressed. I think it's a good strategy - at some point those that want this functionality bad enough will start to cobble something together, and that will get improveed upon by others in the community. Until this happens there are plenty of alternative solutions, including the one I posted (Launch acd_cli when your media devices are offline)..

from acd_cli.

I would suggest using third party programs too, such as trickle or iptables.

from acd_cli.

If someone really cares enough, you can dig out the old libcurl file transfer and write a new ContentMixin for the ACD client API.

from acd_cli.

The rclone project has recently added ACD support and it's got a --bwlimit=SIZE. Haven't tested it myself, though.

from acd_cli.

Does anyone have suggestions on how to make trickle work with acd_cli? I tried this but it did not limit the upload.

trickle -u 500 acd_cli upload /x /x

from acd_cli.