tpitale / mail_room Goto Github PK
View Code? Open in Web Editor NEWForward mail from gmail IMAP to a callback URL or job worker, simply.
License: MIT License
Forward mail from gmail IMAP to a callback URL or job worker, simply.
License: MIT License
If someone is running this code as a library (rather than just from the CLI) allow them to load and extend the registry of delivery methods available.
This could maybe be done with gems as plugins for each delivery method? Pure speculation.
Not sure if this is in scope for this awesome gem. Could mail query IMAP for NEW instead of UNSEEN? this would have the upside of catching missed messages and the downside of reading all the messages in a previously used mailbox.
After update to last version 0.6.1 - got an error _cannot load such file -- charlock_holmes_.
It works if I manually add charlock_holmes
gem to my Gemfile
. Anybody know why I lost dependency?
/home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery/sidekiq.rb:4:in `require': cannot load such file -- charlock_holmes (LoadError)
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery/sidekiq.rb:4:in `<top (required)>'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery.rb:4:in `require_relative'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/delivery.rb:4:in `[]'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox.rb:50:in `delivery_klass'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox.rb:58:in `delivery'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox.rb:75:in `deliver'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_handler.rb:19:in `block in process'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_handler.rb:17:in `each'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_handler.rb:17:in `process'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:166:in `process_mailbox'
from /home/alexey/.rvm/gems/ruby-2.2.2@partsable/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:146:in `block in run'
I am using mail_room in combination with the postback
delivery method (hence using Faraday
). I somehow noticed that emails containing a %
got "lost" (they were marked as 'seen' in GMail, but the specified delivery_url
was apparently never called).
I digged a little into it and, by enabling logging for Faraday, confirmed my suspicion that there might be an encoding issue: When an email containing a %
(for example: An email with the message body test %
) gets POSTed through Faraday, the message delivery silently fails:
I, [2015-03-23T14:46:12.456947 #98851] INFO -- : post https://xxxxxxx.xxxxxxx
D, [2015-03-23T14:46:12.457528 #98851] DEBUG -- request: User-Agent: "Faraday v0.9.1"
Authorization: "Token token=\"my_token\""
Content-Type: "application/x-www-form-urlencoded"
I, [2015-03-23T14:46:13.057616 #98851] INFO -- Status: 400
D, [2015-03-23T14:46:13.057934 #98851] DEBUG -- response: transfer-encoding: "chunked"
connection: "close"
status: "400 Bad Request"
That kind of makes sense, because a %
cannot stand on its own using x-www-form-urlencoded. (Try, for example http://stackoverflow.com/test%test).
I seem to be able to successfully POST an email with a %
if I pass URI.encode_www_form_component(message)
instead of message
to Faraday. I am a little confused by that though, because I would have assumed that Faraday automatically handles encoding...?
In the validate!
for a mailbox we should check for required fields like name
. Raise a ConfigurationError
otherwise.
To further help newcomers, we could add an example of a simple endpoint with auth for the postback delivery method.
MailRoom should not crash when a message or delivery raises an error. The mailbox watcher should be properly supervised such that it simply restarts on that mailbox. Let it fail, and make a new watcher.
It may be valuable to know that messages are being fetched/processed/delivered to a remote location in a structured log. Still undecided on this. If using que/sidekiq new jobs will be processed; if using logger all messages are already logged. Only in a remote delivery case (callbacks) would it be especially hard to know if new messages were being fetched/delivered without a log if the remote service is not within the users control.
I've been using this gem for a few days now, and it seems that it works for maybe 30 minutes, but then it seems to stop noticing any new emails coming through. Could it be possible that IDLE isn't working as intended? Restarting mail_room then correctly pulls in emails for another period of time until it stops again.
After seeing that emails weren't coming through, I checked mail_room was still running, it just wasn't noticing any new emails.
I’m quite confused. It uses IMAP, so it probably should work with any IMAP server (e.g. Dovecot), but the readme is full of Gmail… Please clarify this, Google is not the only IMAP server in the world.
Failures:
1) MailRoom::Delivery::Sidekiq#options when sentinel is specified client has same specified sentinel params
Failure/Error: expect(redis.client.instance_variable_get(:@connector)).to be_a Redis::Client::Connector::Sentinel
TypeError:
no implicit conversion of Symbol into Integer
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:346:in `establish_connection'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:106:in `block in connect'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:306:in `with_reconnect'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:105:in `connect'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:381:in `ensure_connected'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:231:in `block in process'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:319:in `logging'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:230:in `process'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:125:in `call'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:222:in `block in client'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `block in synchronize'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `synchronize'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:221:in `client'
# ./spec/lib/delivery/sidekiq_spec.rb:68:in `block (4 levels) in <top (required)>'
2) MailRoom::Arbitration::Redis redis client connection params when sentinel is present client has same specified sentinel params
Failure/Error: expect(client.client.instance_variable_get(:@connector)).to be_a Redis::Client::Connector::Sentinel
TypeError:
no implicit conversion of Symbol into Integer
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:346:in `establish_connection'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:106:in `block in connect'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:306:in `with_reconnect'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:105:in `connect'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:381:in `ensure_connected'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:231:in `block in process'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:319:in `logging'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:230:in `process'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis/client.rb:125:in `call'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:222:in `block in client'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `block in synchronize'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:52:in `synchronize'
# /usr/share/rubygems-integration/all/gems/redis-4.1.2/lib/redis.rb:221:in `client'
# ./spec/lib/arbitration/redis_spec.rb:120:in `block (4 levels) in <top (required)>'
Finished in 0.0533 seconds (files took 0.35497 seconds to load)
51 examples, 2 failures
Failed examples:
rspec ./spec/lib/delivery/sidekiq_spec.rb:67 # MailRoom::Delivery::Sidekiq#options when sentinel is specified client has same specified sentinel params
rspec ./spec/lib/arbitration/redis_spec.rb:119 # MailRoom::Arbitration::Redis redis client connection params when sentinel is present client has same specified sentinel params
Randomized with seed 3049
When I got mail into my inbox I am not able to see anything else into the logs except below..
$ bundle exec foreman start -p 3000
18:27:25 web.1 | started with pid 27546
18:27:25 mail.1 | started with pid 27547
18:27:37 web.1 | [RAILS-DEV-BOOST] Unable to start rails-dev-boost in an asynchronous mode. Are you running on an OS not supported by the `listen` gem?
18:27:37 web.1 | [RAILS-DEV-BOOST] To get rid of this message disable the rails-dev-boost's async mode by putting the following code in a Rails initializer file (these are found in config/initializers directory):
18:27:37 web.1 | [RAILS-DEV-BOOST]
18:27:37 web.1 | [RAILS-DEV-BOOST] RailsDevelopmentBoost.async = false if defined?(RailsDevelopmentBoost)
18:27:37 web.1 | [RAILS-DEV-BOOST]
18:27:38 web.1 | [2014-12-13 18:27:38] INFO WEBrick 1.3.1
18:27:38 web.1 | [2014-12-13 18:27:38] INFO ruby 2.0.0 (2013-02-24) [i686-linux]
18:27:38 web.1 | [2014-12-13 18:27:38] INFO WEBrick::HTTPServer#start: pid=27546 port=3000
18:28:22 web.1 | [deprecated] I18n.enforce_available_locales will default to true in the future. If you really want to skip validation of your locale you can set I18n.enforce_available_locales = false to avoid this message.
If -q
is set, do not raise an error if there is no configuration or configuration is missing the top level mailboxes key.
In the case of GitLab, this means every "Reply by email" message would result in N comments being created.
See https://gitlab.com/gitlab-org/gitlab-ce/issues/2870.
@tpitale What's your take on this? Our (@jacobvosmaer and me) solution has mail_room use Redis to keep track of what messages have been processed, so that only one mail_room ever processes a specific message, but this does add complexity to mail_room that a lot of use cases (like yours, it appears) don't need.
If we want to start a simple postback delivery, we could accept the configuration for it from the CLI directly, without the need for a configuration file.
https://gitlab.com/gitlab-org/gitlab-ce/issues/50842
bundler: failed to load command: mail_room (/home/git/gitlab/vendor/bundle/ruby/2.4.0/bin/mail_room)
Net::IMAP::ByeResponseError: AXIGEN Proxy front1 IMAP4rev1: cannot connect to AXIGEN
/usr/local/lib/ruby/2.4.0/net/imap.rb:1188:in `get_tagged_response'
/usr/local/lib/ruby/2.4.0/net/imap.rb:1248:in `block in send_command'
/usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/usr/local/lib/ruby/2.4.0/net/imap.rb:1230:in `send_command'
/usr/local/lib/ruby/2.4.0/net/imap.rb:434:in `login'
/home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:84:in `log_in'
/home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:68:in `setup'
/home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:54:in `rescue in wait'
/home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/connection.rb:44:in `wait'
/home/git/gitlab/vendor/bundle/ruby/2.4.0/gems/mail_room-0.9.1/lib/mail_room/mailbox_watcher.rb:34:in `block in run'
It may be valuable to know which decision arbitration is making for a given message uid. Though, it is a boolean option … either it's in the list or not. So, undecided there.
We should log this to STDOUT if configuration for arbitration includes logging: true (or something, undecided). Logs should be in a standardized, structure form that is machine readable (research required for that).
For structured logging, we may want to direct output to STDOUT/STDERR and not just a file path.
/Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1150:in `get_tagged_response': end of file reached (EOFError)
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1210:in `block in send_command'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1192:in `send_command'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1345:in `block in search_internal'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:1341:in `search_internal'
from /Users/Mac/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/imap.rb:768:in `search'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_handler.rb:39:in `new_message_ids'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_handler.rb:29:in `new_messages'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_handler.rb:17:in `process'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_watcher.rb:152:in `process_mailbox'
from /Users/Mac/Desktop/mail_room/lib/mail_room/mailbox_watcher.rb:129:in `block in run'
It would be really helpful to have a debug mode that shows what mail_room is doing so that we can diagnose whether there is an issue with the configuration, mail_room, or the IMAP server. It can just be a configuration flag (e.g. -d
or -v
).
/cc: @godfat
I can't find anywhere that mail_room tells the IMAP server to actually delete messages flagged as DELETED; my understanding is that it would be by calling close() or expunge(). I can confirm that our gmail inbox has a large number (>200K) messages, nearly all marked DELETED but never actually expunged.
Am I missing something?
Could you include support for defining a sentinel-enabled connection?
redis-rb already support so, all you have to do is pass "sentinels" array of hashes connection param.
Ideally you would change "redis_url" YAML key to just "redis" and accept every redis possible setting:
Both Que
and Sidekiq
dump to json. Encodings other than properly recognized UTF-8 will trip json up.
From @DouweM:
A reproduction of the specific case GitLab users were having trouble with is this:
message = "é"
message.force_encoding("ASCII-8BIT")
Note that in this case, the message was actually UTF-8-encoded, but for some reason Ruby thought it was ASCII-8BIT, and when trying to do JSON.generate
, it tried to convert this string that was actually UTF-8 encoded from ASCII-8BIT to UTF-8, yielding the "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
error you see in https://gitlab.com/gitlab-org/gitlab-ce/issues/2698.
Reproduce this by doing:
JSON.generate(message) # => error
In this case, simply doing message.force_encoding('UTF-8')
would work, but not in this case, where the original message was ISO-8859-1 rather than UTF-8:
message = "é".encode('ISO-8859-1')
message.force_encoding('ASCII-8BIT')
message.force_encoding("UTF-8")
JSON.generate(message) # => error
@DouweM What ruby versions does gitlab support? If I dropped support from mail_room prior to 2.3 would that be an issue?
Currently, if a disconnect occurs during idle
, net/imap raises the superclass Error
that is used for many other errors https://github.com/ruby/ruby/blob/trunk/lib/net/imap.rb#L3599-L3645 for more specific failures.
We only want to reset the connection if it's specifically a disconnect during idle.
In our GitLab deployment we are running into the following error:
2016-07-26_08:31:54.08717 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/mailbox_watcher.rb:50:in `quit': undefined method `join' for nil:NilClass (NoMethodError)
2016-07-26_08:31:54.08721 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:30:in `each'
2016-07-26_08:31:54.08721 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:30:in `quit'
2016-07-26_08:31:54.08722 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:25:in `ensure in run'
2016-07-26_08:31:54.08722 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/coordinator.rb:25:in `run'
2016-07-26_08:31:54.08722 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/lib/mail_room/cli.rb:52:in `start'
2016-07-26_08:31:54.08723 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.8.0/bin/mail_room:5:in `<top (required)>'
2016-07-26_08:31:54.08723 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2016-07-26_08:31:54.08724 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'
It seems to be caused by some kind of race condition.
Original issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/20273
Delivery errors have so far not been a problem that simply restarting mail_room doesn't fix. But it's possible we would not want those errors on delivery to stop fetching, or to mark messages as SEEN.
When using UNSEEN as the search command, it's impossible to have transactional mail because Google handles marking as SEEN. Whereas NEW/DELETED are controlled by mail-room after delivery.
This stems from a GitLab issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/13357
It seems a number of users are having issues with mail_room getting stuck and not processing any messages. gdb backtraces are available in the notes.
It looks like there may be two issues here with Ruby's Net::IMAP implementation:
You can replicate the first problem by just pointing mail_room to a local server on port 5000 by running nc -l 127.0.0.1 5000. It just sits there.
Regarding the second case, it looks like Ruby 2.3 added a timeout parameter to the idle method perhaps to help avoid this issue. We're using Ruby 2.1 now and won't be at 2.3 for some time.
I can see a couple of solutions that would help:
Thoughts?
https://edgeguides.rubyonrails.org/action_mailbox_basics.html#sendgrid (for example)
https://example.com/rails/action_mailbox/{{service}}/inbound_emails
mail_room
as an ingress configurationLogging works as a delivery method, but postback doesn't seem to send POST request to locally running Rails app (no trace of any POST request in the console/log).
See https://github.com/ppppppachyderm/mail_room_rails_example for attempted implementation and http://stackoverflow.com/questions/21138237/mail-room-with-ruby-on-rails for additional details.
I've been trying the Rails 4.0.2 barebone implementation above, an identical Rails 3.2.16 implementation and implementing mail_room into the users controller of an existing 4.0.2 app. In every case logging works but postback doesn't seem to.
The concurrent-ruby library appears to have reached a reasonable level of stability and is in use by a number of major ruby projects.
It may be worth investigating to see if it could be leveraged to improve the code around threaded behaviors in mail_room. It would be our only required dependency.
Areas that may be improved:
I was running mail_room overnight, to test it, outputting to logger. It worked for awhile, but when I woke up in the am, it hadn't pulled in any of the new messages that had come in. Ctrl+C and I got this error:
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb:185:in `lock': can't be called from trap context (ThreadError)
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb:185:in `mon_enter'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/monitor.rb:209:in `mon_synchronize'
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/net/imap.rb:941:in `idle_done'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/mailbox_watcher.rb:69:in `stop_idling'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/mailbox_watcher.rb:90:in `quit'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:22:in `each'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:22:in `quit'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/cli.rb:45:in `stop'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/cli.rb:34:in `block in start'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:18:in `call'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:18:in `sleep'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/coordinator.rb:18:in `run'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/lib/mail_room/cli.rb:41:in `start'
from /Library/Ruby/Gems/2.0.0/gems/mail_room-0.1.0/bin/mail_room:5:in `<top (required)>'
from /usr/bin/mail_room:23:in `load'
from /usr/bin/mail_room:23:in `<main>'
By default we use Gmail IMAP's UNSEEN. It may be useful to configure this to be NEW in some cases.
Is it possible to have the mailroom.yml config file read from env vars? We'd rather not have credentials and tokens in our source tree.
If the Mailroom can't connect to the mailserver due to an SSL error it will keep attempting to connect to it going so far as to dedicate a CPU core to the task of banging the mailserver.
To be precise, the command /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/bundle exec mail_room -c /var/opt/gitlab/gitlab-rails/etc/mail_room.yml
is using 100% CPU.
My system is Ubuntu Server 14.04 and I am using the omnibus distribution of GitLab.
Here is an excerpt from my mailroom log file:
2015-12-05_16:28:10.65112 /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
2015-12-05_16:28:10.65116 from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `start_tls_session'
2015-12-05_16:28:10.65116 from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
2015-12-05_16:28:10.65117 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `new'
2015-12-05_16:28:10.65117 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `imap'
2015-12-05_16:28:10.65118 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
2015-12-05_16:28:10.65121 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:69:in `setup'
2015-12-05_16:28:10.65122 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:132:in `run'
2015-12-05_16:28:10.65123 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `each'
2015-12-05_16:28:10.65123 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `run'
2015-12-05_16:28:10.65124 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/cli.rb:52:in `start'
2015-12-05_16:28:10.65124 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/bin/mail_room:5:in `<top (required)>'
2015-12-05_16:28:10.65125 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2015-12-05_16:28:10.65125 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'
2015-12-05_16:28:10.82077 Runit: starting mail_room
2015-12-05_16:28:23.33484 /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
2015-12-05_16:28:23.33493 from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `start_tls_session'
2015-12-05_16:28:23.33494 from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
2015-12-05_16:28:23.33496 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `new'
2015-12-05_16:28:23.33498 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `imap'
2015-12-05_16:28:23.33499 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
2015-12-05_16:28:23.33501 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:69:in `setup'
2015-12-05_16:28:23.33502 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:132:in `run'
2015-12-05_16:28:23.33504 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `each'
2015-12-05_16:28:23.33506 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `run'
2015-12-05_16:28:23.33507 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/cli.rb:52:in `start'
2015-12-05_16:28:23.33509 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/bin/mail_room:5:in `<top (required)>'
2015-12-05_16:28:23.33510 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2015-12-05_16:28:23.33512 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'
2015-12-05_16:28:23.51205 Runit: starting mail_room
2015-12-05_16:28:36.27779 /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
2015-12-05_16:28:36.27782 from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1454:in `start_tls_session'
2015-12-05_16:28:36.27783 from /opt/gitlab/embedded/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
2015-12-05_16:28:36.27783 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `new'
2015-12-05_16:28:36.27783 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:20:in `imap'
2015-12-05_16:28:36.27784 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
2015-12-05_16:28:36.27784 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:69:in `setup'
2015-12-05_16:28:36.27785 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/mailbox_watcher.rb:132:in `run'
2015-12-05_16:28:36.27787 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `each'
2015-12-05_16:28:36.27787 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/coordinator.rb:19:in `run'
2015-12-05_16:28:36.27787 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/lib/mail_room/cli.rb:52:in `start'
2015-12-05_16:28:36.27788 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/mail_room-0.6.1/bin/mail_room:5:in `<top (required)>'
2015-12-05_16:28:36.27788 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `load'
2015-12-05_16:28:36.27788 from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/mail_room:23:in `<main>'
2015-12-05_16:28:36.44894 Runit: starting mail_room
Originally reported in https://github.com/gitlabhq/gitlabhq/issues/9891
I'm not a ruby guy so I can't implement this myself (but it seems the change would be primitive). My mail server only supports StartTLS (it doesn't acceppt non-encrypted connections and 993 is closed). So to connect to it, the STARTTLS
upgrade needs to be performed.
From what I gather, Net::IMAP#starttls
is a simple command to execute after connecting so it would be cool to have an option to trigger this once the connection is established.
https://github.com/thoughtbot/griddler accepts POST in a format from sendgrids parse api hooks. When configured, mail_room should be able to send in that format.
If the server disconnects, we get a connection closed Net::IMAP::Error
. Rather than failing out, or silently swallowing it, we should reconnect.
/usr/local/lib/ruby/2.1.0/openssl/ssl.rb:240:in `post_connection_check': hostname "localhost" does not match the server certificate (OpenSSL::SSL::SSLError)
from /usr/local/lib/ruby/2.1.0/net/imap.rb:1456:in `start_tls_session'
from /usr/local/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:20:in `new'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:20:in `imap'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:88:in `log_in'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:69:in `setup'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/mailbox_watcher.rb:132:in `run'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/coordinator.rb:19:in `each'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/coordinator.rb:19:in `run'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/lib/mail_room/cli.rb:52:in `start'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/mail_room-0.5.2/bin/mail_room:5:in `<top (required)>'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/mail_room:23:in `load'
from /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/mail_room:23:in `<main>'
If -q
is used, and no configuration is found or the app is misconfigured, it will not run (no mailboxes). This is better than running and sleeping continuously, but it may just cause a process manager to continually try to restart.
Make a note in the readme of this behavior, and about throttling process restarts.
I think there's a problem with the redis arbitration locking, and we're seeing unread messages get stuck in our inbox as a result.
The 'lock' is taken in deliver? which is called during the search for UNSEEN messages (new_message_ids). If something goes wrong during the actual message fetch, then the message will not be marked SEEN. When a mail_room calls new_message_ids (the restarted one, or another instance) before the key expires (10min by default), we'll call deliver? which will increment the key and reset the TTL, and return false (the value will != 1). If we never go 10 minutes without receiving mail (or with no mail_room processes running), the key will keep on being incremented and the TTL keep on being reset, and the message will never be picked up again.
I don't know what the solution is I'm afraid, but I'm happy to try and help figure it out.
Currently it's difficult to view the performance of the mail_room
gem. If the gem emitted metrics, we can start measuring performance and other relatable data in order to alert Engineers if things are going wrong. This can also be useful as an endpoint could be utilized for monitoring the health of the service in general, whether it be a docker daemon or Kubernetes healthcheck.
Also consider that it may be wise to include metrics of monitoring the mailbox for which mail_room
is configured to look at. This prevents the need for persons to stand up a secondary system such as https://github.com/camptocamp/imap-mailbox-exporter to monitor the mailbox.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.