• populate [files] in package.json. Delete .npmignore.
• automated code linting. #3308
  • dep: eslint-plugin-haraka -> @haraka/eslint-config
  • update 'lint' script in package.json
  • verify 'lint' CI test config
• lint: remove duplicate / stale rules from .eslintrc
• automated code formatting (see also #3308)
• automated CI testing.
  • mostly done, verify that local copy of ci.yml is up-to-date.
• CONTRIBUTORS: see 3309
• consistent naming of "special" files like CHANGELOG.md.
• CHANGELOG: verify links at bottom (due to inconsistent tag naming)
  • latest .release does this, fixes most, and warns of errors it can't fix
• verify GitHub repo About link points to npm package
• convert test suites to mocha "style"
  • works great today and with node --test in v18+

Currently this module inherits the Redis plugin always. Modify it such that it only inherits Redis when the concurrency and rate limits modules (that need Redis) are enabled.

Related to haraka/Haraka#2746

It would be really great, if we could limit users to only send emails from a specified domain.

For example, if a user is specified as following in auth_flat_file.ini:
[email protected]=password

The user is only allowed to send emails from "example.com".

Is your feature request related to a problem? Please describe.
We are having issues with delivering emails to some hosts that limit the number of connection that can be opened.
In the version 2 it was possible to limit the number of connections opened using pool_concurrency_max.

Example error:

421 smtp-19.iol.local smtp-19.iol.local Too many connections, slow down [smtp-19.iol.local; 

Will it be possible to add support for it in v3?

Describe the solution you'd like
Add the generic-pool implementation from version 2, with this added, the number of connections open to a host can be controlled from a plugin as well.

Also implemented a plugin to limit the delivery speed to some domains and that fixed error messages related to Too many messages, slow down error.

But for connections is not possible to control the number of connections open to a host from a plugin (without forking haraka).

Describe alternatives you've considered
Tried to fix the issue using concurrency_max setting, but lowering this value slows the delivery for all hosts and I want to be able to lower only for domains that don't support a higher number of connections.

Additional context
We are using haraka to send email campaigns and when we warmup new ips some email providers impose limitations regarding the number of emails sent and the number of connection opened.

Let me know if this is a good idea, maybe there was a good reason why it was removed.

Hi @msimerson, Happy New Year!

I've observed redis errors with the plugin (using Haraka 2.8.28), looking like that:

err:TypeError: this.db.hIncrBy is not a function, rate_conn:TypeError: this.db.hGetAll is not a function

Describe the bug

When enabling the rate_conn setting, the limit plugin creates/updates a Redis key named rate_conn:<IP> which has a hash as its value. The hash uses timestamps as keys, with a counter updated by HINCRBY.

The TTLS of the rate_conn:<IP> key is being set/reset to the configured window of time times two on every connection, e.g. for a setting of 60m it's 7200, and every new connection from the same IP resets the TTL.

The hash keys, however, have no TTLs on their own, which means, they only expire when the whole key rate_conn:<IP> expires.

This leads to a situation where a client that connects regularly within the configured time window, the hash is only growing all the time. One very prominent and common case of a client that connects regularly is: Monitoring!

Expected behavior

I expected outdated timestamps within the rate_conn:<IP> Redis key to expire by some plugin logic, as Redis doesn't have a TTL per hash key (AFAIK), keeping the dataset current and small.

Observed behavior

No expiration of hash keys is ever happening. The hash grows forever, as long as the same IP address connects again within 7200 seconds.

In our case, our monitoring does a short SMTP check every minute, each one generating a timestamp hash key under the according rate_conn:<IP> key. This is 60 new hash keys per hour, 1.400 a day, 43.200 a month, 511.000 per year. In our case, after about 2,5 years, we discovered about 1.300.000 timestamped hash keys for our monitoring IP in Redis.

This explained why the Haraka processes over time used immense CPU resources and immense memory: Reading 1,3 million hash keys on every monitoring connection into memory, doing 1,3 million quick calculations if the timestamp lies within the configured 60m and hence ignoring basically all of them except the most current 60 - that took a quite a toll on our servers. The moment we stopped monitoring our Haraka ports, cluster load basically halved and Haraka memory usage dropped by about 2 GB - per host.

Steps To Reproduce

1. Assuming an already configured Redis, configure a limit in config/limit.ini, in our case:

[rate_conn]
enabled=true
default=200/60m

2. Regularly make SMTP connections from the same IP, e.g. every minute. In our case we set up a blackbox_exporter on localhost:9115 as we're using Prometheus for monitoring, and do a request every minute:

$ for REQ in `seq 1 1000` ; do curl "http://localhost:9115/probe?target=our.test.host:465&module=smtp_c2s_tls_v4" ; sleep 60 ; done

Any other means of SMTP should work as well, e.g. using Swaks.

3. Check the actual Redis data, replacing 111.222.333.444 with the actual IP you use for testing:

$ redis-cli -s /path/to/your/redis.socket -n 4 hkeys rate_conn:111.222.333.444 | wc -l

Expectation would be to find timestamps no older than the configured 60m, so in this case, about 60. Reality is you'll find timestamps of any age.

System Info

(It's an old CentOS 7 host which is about to be replaced soon, but I don't think the environment is relevant here anyway.)

Since we're getting issues delivering too much mail to the Outlook / Hotmail / Live / MSN services, we're looking into rate limiting the amount of mail that goes to these providers. This plugin seems to provide that functionality, but it doesn't seem to work correctly.

I've been debugging for a while, and found out that the variable hmail.domain in the outbound_increment hook is NULL. After inspecting the hmail object and looking in the documentation, the domain variable can be accessed by using the hmail.todo.domain variable.

The documentation also mentions the always_split configuration option, but this option does not seem to have impact on the contents of the hmail.domain variable.

Is this an issue in our environment, or is it a compatability issue with the Haraka version we're using (the latest - 2.8.13)? I've fixed it for now by writing a custom plugin which hooks into the outbound hooks before the limit plugin and copies the hmail.todo.domain variable to hmail.domain.

Version 1.0.8 of eslint-plugin-haraka just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As eslint-plugin-haraka is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Hello

When trying to install Hiredis, I encountered the following error: Initially, I tried with Python 3.9 (the latest version), but it failed because hiredis only supports up to Python 2.9(link). After switching to Python 2.7, I am still encountering errors because node-gyp does not support versions lower than 3.6. How can I successfully install hiredis?

[root@relay ~]# npm install redis hiredis ipaddr.js
npm ERR! code 1
npm ERR! path /root/node_modules/hiredis
npm ERR! command failed
npm ERR! command sh -c node-gyp rebuild
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | linux | x64
npm ERR! gyp ERR! find Python
npm ERR! gyp ERR! find Python checking Python explicitly set from NODE_GYP_FORCE_PYTHON
npm ERR! gyp ERR! find Python - process.env.NODE_GYP_FORCE_PYTHON is "/usr/local/bin/python2.7"
npm ERR! gyp ERR! find Python - executable path is "/usr/local/bin/python2.7"
npm ERR! gyp ERR! find Python - version is "2.7.18"
npm ERR! gyp ERR! find Python - version is 2.7.18 - should be >=3.6.0
npm ERR! gyp ERR! find Python - THIS VERSION OF PYTHON IS NOT SUPPORTED
npm ERR! gyp ERR! find Python - "/usr/local/bin/python2.7" could not be run
npm ERR! gyp ERR! find Python
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python You need to install the latest version of Python.
npm ERR! gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
npm ERR! gyp ERR! find Python you can try one of the following options:
npm ERR! gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
npm ERR! gyp ERR! find Python (accepted by both node-gyp and npm)
npm ERR! gyp ERR! find Python - Set the environment variable PYTHON
npm ERR! gyp ERR! find Python - Set the npm configuration variable python:
npm ERR! gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
npm ERR! gyp ERR! find Python For more information consult the documentation at:
npm ERR! gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python
npm ERR! gyp ERR! configure error
npm ERR! gyp ERR! stack Error: Could not find any Python installation to use
npm ERR! gyp ERR! stack at PythonFinder.fail (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:306:11)
npm ERR! gyp ERR! stack at PythonFinder.findPython (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:164:17)
npm ERR! gyp ERR! stack at async configure (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:23:18)
npm ERR! gyp ERR! stack at async run (/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js:81:18)
npm ERR! gyp ERR! System Linux 5.14.0-362.24.2.el9_3.x86_64
npm ERR! gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd /root/node_modules/hiredis
npm ERR! gyp ERR! node -v v21.7.3
npm ERR! gyp ERR! node-gyp -v v10.0.1
npm ERR! gyp ERR! not ok

npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-04-16T12_13_06_125Z-debug-0.log
[root@relay ~]#

Hey!

I just started to study your very cool product - Haraka.

I have a few questions. I need to limit/increase the speed of sending letters for each recipient domain individually.
For example, for the domain @gmail.com - 10/min, @hotmail.com - 25/min, @yahoo.com - 50/min

I found the file /usr/lib/node_modules/Haraka/node_modules/haraka-plugin-limit/config/limit.ini
But I don’t understand what recording format should be done?

Please give me a sample of how to correctly write down the speed limit for sending letters for each recipient domain individually?

[rate_rcpt]
what should I write here?

Thanks!

Either use https://github.com/haraka/Haraka/blob/master/plugins/max_unrecognized_commands.js and remove functionality from this plugin, or remove that and tell users to use this plugin.

Hi,
No matter where I looked and how I've configured the limit.ini file I cant limit a single domain for rate limit

example.com = 1/30s
where is that need to be placed? docs not suggesting how to implement it

Thank you.

Hi!
We are trying to implement the limit plugin (version 1.0.2), but it is disconnecting to soon. Maybe as soon as 8 concurrent connections.

Our limit.ini consist only of two rows.
[concurrency]
max=500

What are we doing wrong?
Please advise!
:) Peter

We use Haraka Version: 2.8.13

Attached, limit.ini, plugins and example from our log

limit.ini.txt
plugins.txt
example_from_log.txt

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

• Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

Your Greenkeeper Bot 🌴

The devDependency eslint-plugin-haraka was updated from 1.0.9 to 1.0.10.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-haraka is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

see https://www.npmjs.com/package/haraka-plugin-limit?activeTab=code

The devDependency eslint-plugin-haraka was updated from 1.0.11 to 1.0.12.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-haraka is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

I m running haraka server and it only receives emails. I get 50+ emails/sec on my server. Some emails are being delayed and are received in 10 or 5 min how can I raise this limit using this plugin to 5000 emails/sec.
I m confused if this plugin is for inbound or outbound specific

Hello,

I have integrated Haraka on a virtual server and I ma happy to use it. The host switches the server off if too much emails are sent. They recommend not more than 10 emails per minute.

I do not see how to make tests in this field so I would like to double-check before... is it enough to use the following parameters of the limit.ini file :

[outbound]
enabled=true 
delay=60 
my-domain.com=10

Many thanks

Matthieu

I am sorry for asking here but I didn't find another way

Version 1.0.20 of haraka-test-fixtures just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As haraka-test-fixtures is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

The devDependency eslint was updated from 6.7.1 to 6.7.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

The devDependency eslint was updated from 5.16.0 to 6.0.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

Hi guys, I'm having a problem during the implementation of outbound rate limits, it seems to work just for the first time for the outbound part.

Example :

If I'm using the following Config in limit.ini :

[outbound]
enabled=true
domain=1
delay=10

Then queuing 5 mails.

Expected :
Deliver 1 mail every 10 seconds.

Current behaviour :
Deliver the first mail then delayed the rest '4' mails for 10 seconds (as expected), but after the first 10 seconds the 'send_mail' hook will be called and delayed the '4' mails another time and keep delay the delivery every 10 seconds... without delivering.

NOTE : I think that happen because of the TOTAL that will be incremented every time so the count variable will be always greater than limit and will reach the delivered condition just the first time :

if (count <= limit) return next();

Thanks in advance.

Version 1.0.21 of haraka-test-fixtures just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As haraka-test-fixtures is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

The dependency ipaddr.js was updated from 1.9.0 to 1.9.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

ipaddr.js is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴