We are currently using our own trigger framework (very similar to this, and I am looking into switching) We have a LOT of trigger activity. Often, the triggers can get swamped and we can get several hours backlogged into the triggers. This can be a big problem when triggers that update states in the entities or other entities are delayed and users see discrepencies.

So, one item we wanted to thing about, which is related to multi processing approach, is that the triggers are pretty independent per project. Has anyone set this up so that each project can run simultaneously?

I was thinking that either:
1). The network could be configured to run for a project. Right now I do not think this is possible? So, I was thinking of maybe adding the ability have it spawn a process for each project and each process would run its own event loop through the shotgun events dispatching that project's events to that processes copy of the plugins.
2) Setting up the trigger so that it has one event loop. Plugins could register that they are per project, and if so, a process would be created per project and events would be queued to that process from the event loop. One would have to use a queue to avoid one project hanging the loop. This makes recovery a bit trickier though. This would be a more involved modification to this sw

Ideas?

I setup the daemon and used logArgs as a test plugin as suggested. I noticed that nothing was being logged to the plugin.logArgs file (it was created by contained no log message). I found that it was necessary to add this line

    self._logger.setLevel(self._engine.config.getLogLevel())

at the end of Callback.init() to make the logging module write to the log file.

I don't totally understand why since the logger's effective log level is 10 already. But there it is.

We are implementing ShotgunEventDaemon for some systematic task.
It runs perfectly when run in Foreground but I’m unable to launch it in background using the start argument.
Here’s the error I get:

[ws] user ~ >python3 /prod/studio/libs/python/3.6/linux/site-packages/shotgunEvents/src/shotgunEventDaemon.py start
Traceback (most recent call last):
File “/prod/studio/libs/python/3.6/linux/site-packages/shotgunEvents/src/shotgunEventDaemon.py”, line 1418, in
sys.exit(main())
File “/prod/studio/libs/python/3.6/linux/site-packages/shotgunEvents/src/shotgunEventDaemon.py”, line 1382, in main
func()
File “/prod/studio/libs/python/3.6/linux/site-packages/shotgunEvents/src/shotgunEventDaemon.py”, line 1353, in start
super(LinuxDaemon, self).start(daemonize)
File “/prod/studio/libs/python/3.6/linux/site-packages/shotgunEvents/src/daemonizer.py”, line 115, in start
self._daemonize()
File “/prod/studio/libs/python/3.6/linux/site-packages/shotgunEvents/src/daemonizer.py”, line 74, in _daemonize
se = open(self._stderr, “a+”, 0)
ValueError: can’t have unbuffered text I/O

Any clue on what could cause this error? Thank!

The calls when logging exceptions that use format_exc(err) don't work in python3

format_exc only takes a limit and chain arguments
https://docs.python.org/3/library/traceback.html?highlight=format_exc#traceback.format_exc

I'm getting errors like these

  File "Z:\tools\shotgun\shotgunEvents\shotgunEventDaemon.py", line 586, in _saveEventIdData
    traceback.format_exc(err),
  File "C:\Program Files\Python37\lib\traceback.py", line 167, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
  File "C:\Program Files\Python37\lib\traceback.py", line 121, in format_exception
    type(value), value, tb, limit=limit).format(chain=chain))
  File "C:\Program Files\Python37\lib\traceback.py", line 508, in __init__
    capture_locals=capture_locals)
  File "C:\Program Files\Python37\lib\traceback.py", line 337, in extract
    if limit >= 0:
TypeError: '>=' not supported between instances of 'FileNotFoundError' and 'int' 

python 2 doesn't seem to care about this.

I restarted the event daemon after a couple of days of it being down, and I started getting a lot of these messages:

2011-03-29 11:27:06,351 - plugin.turnoverStatusChanged - DEBUG - Event 340298 is too old. Last event processed was (340306).
2011-03-29 11:27:06,351 - plugin.turnoverStatusChanged - DEBUG - Event 340299 is too old. Last event processed was (340306).
2011-03-29 11:27:06,352 - plugin.turnoverStatusChanged - DEBUG - Event 340300 is too old. Last event processed was (340306).
2011-03-29 11:27:06,353 - plugin.turnoverStatusChanged - DEBUG - Event 340301 is too old. Last event processed was (340306).
2011-03-29 11:27:06,354 - plugin.turnoverStatusChanged - DEBUG - Event 340302 is too old. Last event processed was (340306).
2011-03-29 11:27:06,355 - plugin.turnoverStatusChanged - DEBUG - Event 340303 is too old. Last event processed was (340306).
2011-03-29 11:27:06,355 - plugin.turnoverStatusChanged - DEBUG - Event 340304 is too old. Last event processed was (340306).
2011-03-29 11:27:06,356 - plugin.turnoverStatusChanged - DEBUG - Event 340305 is too old. Last event processed was (340306).
2011-03-29 11:27:06,357 - plugin.turnoverStatusChanged - DEBUG - Event 340306 is too old. Last event processed was (340306).

It seemed like it was trying to process from the last processed ID every time - it would print out loads of these every cycle.

My solution was to stop the daemon, remove shotgunEventDaemon.id and then restart it, starting from the current position.

Would it be possible to have the config file parsed earlier, and have lines in there for paths to add to sys.path

In my case, I'm launching shotgunEventDaemon through a script in /etc/init.d - I need to add something to sys.path to ensure that it gets the path with shotgun_api3 - it would be nice if this were in the config file rather than me having to edit shotgunEventDaemon.py manually.

Hi all, when I try to run the command

python shotgunEventDaemon.py start

I have an error because of the actual version of python which is 2.7.15+.
I assume this is because of this specific version formatting with the '+' at the end.

Traceback (most recent call last):
  File "shotgunEventDaemon.py", line 60, in <module>
    CURRENT_PYTHON_VERSION = StrictVersion(sys.version.split()[0])
  File "/usr/lib/python2.7/distutils/version.py", line 40, in __init__
    self.parse(vstring)
  File "/usr/lib/python2.7/distutils/version.py", line 107, in parse
    raise ValueError, "invalid version number '%s'" % vstring
ValueError: invalid version number '2.7.15+'

Is there any workaround to get this working?
Thank you for your answer.

Hi,

I just tried something out here where I tried to restart the daemon while it was attempting to reconnect to Shotgun...

When I tell it to restart, the log starts chucking out loads of:

2011-04-24 09:59:18,420 - engine - WARNING - Unable to connect to Shotgun (attempt 1 of 5): [Errno 4] Interrupted system call
2011-04-24 09:59:18,521 - engine - WARNING - Unable to connect to Shotgun (attempt 2 of 5): [Errno 4] Interrupted system call
2011-04-24 09:59:18,622 - engine - WARNING - Unable to connect to Shotgun (attempt 3 of 5): [Errno 4] Interrupted system call
2011-04-24 09:59:18,722 - engine - WARNING - Unable to connect to Shotgun (attempt 4 of 5): [Errno 4] Interrupted system call
2011-04-24 09:59:18,823 - engine - ERROR - Unable to connect to Shotgun (attempt 5 of 5): [Errno 4] Interrupted system call

What's happening, I believe, is that the exception catcher that's meant to be catching Shotgun errors is also catching the kill that is being sent from daemonizer's stop() function.

This is related to #30

If a plugin fails for some reason - shotguneventDaemon should retry executing the plugin after an amount of time - waiting for a file change is not useful in an automatic environment since there is no easy way to recover.

We have plugins that are requesting a database connection to execute stuff(TM) - if the database is offline I want my plugin to fail so the eventDaemon can reschedule event without reload shotgunEventDaemon.

The shotgun api scriptname/key combination are currently store in each plugin source file as well as the shotgunEventDeamon.conf file. This is not the most secure way to say the least.
Not only are the name/key pairs available in the files themself, they probably end up in someones repository as well.

A more secure way would be to handle the keys using environment variables.
The name/key pair still is stored somewhere on disk in a config file, just in one place.
I would also remove the name/key pair from the config file.

The setup would be something like this.
~/.myscript/.access_key - containing
[shotgun_login]

The Shotgun script name the framework should connect with.

name: myscript

The Shotgun api key the framework should connect with.

key: myapikey

Some where in the startup of the code these values need to be placed in an environment variable.
e.g. SGED_SCRIPTNAME and SGED_ACCESS_KEY

from os import environ
environ ['SGED_SCRIPTNAME'] = name
environ ['SGED_ACCESS_KEY'] = key

Each plugin file would have access to the environment variable name and would only need to import 'environ' from 'os' to pass the name/key in the registration call.

It it would be required for each plugin to be able to have it's own name/key pair a method to setup
within the plugin could be provided.

I'm still reading the code to find the best place and way to add this. For now I'm looking a the Plugin class.

Something I noted is that I can't find a 'general' login for the eventdaemon it seems the name/key pair in the config file is not really used.

Thanks for this really cool tool set

Hi,

I'm keen to know how to launch unit tests for this project. I didn't find any test folder btw. My religion forbids me to bring something to production without any prior testing. Many thanks.

Hi,

as described in more detail in the forum I can no longer start the shotgun event handler service on windows 10.
I thought it might be because I was still using Python2 so I uninstalled it and reinstalled using Python 3 via these steps:

• pulled the latest shotgunEvents framework from github
• installed Python 3.7.7
• installed win32 via " D:\Python37_7\python.exe -m pip install win32"
• installed service in an elevated powershell via: D:\Python37_7\python.exe D:\SG_Event_Daemon\src\shotgunEventDaemon.py install
• tried to start the service via the "Services" window but got this error in the event viewer:
  "The Shotgun Event Handler service terminated with the following service-specific error:
  Incorrect function."

When trying to start the service from the powerline I don't get an error but it won't start anyway
D:\Python37_7\python.exe D:\SG_Event_Daemon\src\shotgunEventDaemon.py start

I tried the same thing with (Active)Python 2.7 (the very same one that had worked for a couple of years on htis machine) but am getting the same error.

EDIT: Running the service in the foreground in a powershell work just fine.

Any ideas how to fix or at least debug this?

Cheers,
frank

We're finding that our instance of shotgun will often skip many event ids in the event log. However, the shotgunEvents daemon currently makes the assumption that a drop just indicates inactivity in the daemon. The result is each time a dropped id exists within a batch of batch_size (e.g. 500), it will infinitely loop over those 500 events until the 5 minute timeout happens. This should be fixed to do some sort of check against shotgun such that older events (beyond some threshold) should not be added to the backlog.

Line 376 - /src/shotgunEventDaemon.py(master) : state[pluginName][0] = latestEventId
latestEventId is an undefined variable, it probably should be lastEventId
The engine crashes whenever a new event is raised.
Got it working by changing the line to state[pluginName] = lastEventId

I'm sure it's not intentional but in _getNewEvents(), there's code after a return call at line 453.

A per plugin log file to separate logging output.

Given a sufficiently large event file, attempting to load the eventID pickle causes a crash. The workaround is to forcibly rotate logfiles (I moved all mine to /var/logs/shotgunEventDaemon/old) and restart.

(Note that I am testing locally and we haven't gone into production yet, so the first traceback is for 'you haven't configured any email alerts' and can be safely disregarded. It's the actual crash I'm interested in. If it helps, I promise to configure email alerts when everything's sorted ;)

Some sort of automated log rotation or other functionality to avoid that on pickle load would be great. Thanks for your attention and support.

mcm004:src jcollier$ sudo ./shotgunEventDaemon.py foreground
Password:
INFO:engine:Using Shotgun version 3.0.17
INFO:engine:Loading plugin at /usr/local/shotgun/shotgunEvents/plugins/logArgs.py
INFO:engine:Loading plugin at /usr/local/shotgun/shotgunEvents/plugins/statusFinalShotOnApproval.py
Traceback (most recent call last):
File "./shotgunEventDaemon.py", line 983, in emit
smtp.connect(self.mailhost, port)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/smtplib.py", line 310, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/smtplib.py", line 285, in _get_socket
return socket.create_connection((host, port), timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno 8] nodename nor servname provided, or not known

Logged from file shotgunEventDaemon.py, line 323
CRITICAL:engine:Crash!!!!! Unexpected error (<type 'exceptions.EOFError'>) in main loop.

Traceback (most recent call last):
File "./shotgunEventDaemon.py", line 316, in start
self._loadEventIdData()
File "./shotgunEventDaemon.py", line 340, in _loadEventIdData
self._eventIdData = pickle.load(fh)
EOFError

So the Python Clock is ticking and 2020 is just around the corner. As they say:

Python 2, thank you for your years of faithful service.
Python 3, your time is now.

Any plans on switching to Python 3 in the nearest future?

since some larger studios have multiple Shotgun servers, would be nice to include support for one daemon to handle plugins for all of them. This would mean the ability to enable/disable plugins per-server instance as well. Needs some spec-ing out.

http://docs.python.org/library/logging.html#smtphandler

The SMTPHandler added support for authentication as of Python v2.6. We should check the python version and catch the exception if credentials are provided on versions < 2.6.

(not sure why the code markup is all franked out here sorry...)

$ ./shotgunEventDaemon.py start
Traceback (most recent call last):
File "./shotgunEventDaemon.py", line 888, in ?
sys.exit(main())
File "./shotgunEventDaemon.py", line 856, in main
daemon = Engine(_getConfigPath())
File "./shotgunEventDaemon.py", line 188, in init
self._log = self._logFactory.getLogger('engine', emails=True)
File "./shotgunEventDaemon.py", line 115, in getLogger
self.addMailHandlerToLogger(logger, self._toAddrs)
File "./shotgunEventDaemon.py", line 160, in addMailHandlerToLogger
mailHandler = CustomSMTPHandler(self._smtpServer, self._fromAddr, toAddrs, self._subject, (self._username, self._password))
TypeError: init() takes exactly 5 arguments (6 given)

Hi,

It's maybe not the best way for me to have written it, but I've got some debug statements in some of my classes (that are a few levels beyond the plugin) that are printing stuff out to STDOUT. It would be great if the eventDaemon could catch STDOUT and include that in the log as debug messages....

Hugh

what and where is the backlog?

having issues with the timeouts on backlog events taking a very long time to process

Hey guys,
Trying to get this setup in our small studio. Ive got a small virtual host setup that Im trying to run it from. I have no issues importing any python modules we have in our docserv such as shotgun_api3
Seems the issue here is 18.04 comes with python2.7.15rc1
When I try to run daemon in foreground to test I am getting this:
user@host:/mnt/docserv/software/scripts/shotgunEvents-master/src$ sudo ./shotgunEventDaemon.py foreground
Traceback (most recent call last):
File "./shotgunEventDaemon.py", line 58, in
CURRENT_PYTHON_VERSION = StrictVersion(sys.version.split()[0])
File "/usr/lib/python2.7/distutils/version.py", line 40, in init
self.parse(vstring)
File "/usr/lib/python2.7/distutils/version.py", line 107, in parse
raise ValueError, "invalid version number '%s'" % vstring
ValueError: invalid version number '2.7.15rc1'

Wondering if anyone has any ideas or workarounds. I feel so close here, yet so far. Setup my Shotgun API keys etc, pointed config to our SG server. I think the issue here is I am just on a release candidate for Python 2.7?

Hi,

It would be great if it could log when a connection is successfully re-established, at the same logging level as the highest level that the connection failed one went out on...

If I've been getting a few emails telling me that it's failed to connect to Shotgun, I'd like the last of them to be one telling me that the issue is resolved. If they were only warnings, then I'd like the log to have one saying that the issue is resolved...

I've actually implemented this at my end - I'm happy to chuck the code over if you'd like...

Hey!
I experience that plugins sometimes fail to reload when overwritten.
It would be great if there was a way to pass a reload argument to the daemon that forces reload of registered plugins in stead of having to restart the whole daemon.

Thanks for a great framework!
-Daniel

An issue we have with our current trigger sw, and I think this has too, is that we have some triggers that based on certain different actions (status changes, task on the object status changes, task assignee changes, etc) go through and recompute a field on the shot. The update is a bit expensive in a few seconds. The problem is, creating a shot, or certain API operations, cause many of these fields to update on the same operations. Our event loop recomputes the field for each field change, even though it was actually one operation. Because it is fairly expensive update operation, this causes our event triggers to fall behind.

I was wondering if anyone has dealt with this. This may actually be more an attribute of the plugin. One idea was to change the plugin to basically queue the events uniquely per entity in a queue, and have a separate process read the queue and do the operations. If multiple items come in, then they will all be handled once.

When placing the plugin path on a shared network resource which has a potentially flaky connection (Gasp, I know but there's a variety of reason a shared drive (even our GPFS) can go offline, even for 1 minute or so), shotgun should attempt to load the plugin, and failing an OSError, should gracefully refuse to reload the plugin instead of crash. This will have implications on startup if no plugins are able to be found (easy to detect).

For example, the code block at 560 -> 569 should be wrapped in a try, except (OSError), and an additional check should be made to loudly whine (or then crash/email, etc), if no plugins are loaded.

Hi,

I've not had a chance to properly look through it, but I'm finding that calling shotgunEventDaemon.py stop doesn't seem to be working entirely properly.

It usually seems to happen after I've had a plugin crash - the stop command just sits there seemingly doing nothing.

Originally posted by @tk421storm in #38 (comment)

We had an unexpected power failure and the id file was corrupted on disk. It looks like this has happened before to us. Also, we have a monitoring service that often loads an incomplete id file.

While it's not 100% guaranteed to be atomic, many tools write to a temp file and rename it in place. Some OSes guarantee atomicity and for the rest it should at least be a significant improvement.

We're currently hardcoding the backlog timeout to 5 minutes. While this is a decent default, it'd be best if this were user-configurable.

A system by which you could have per plugin sections in the config and logging could be one of the initial options.

Trying to run this using Python 3.7 on Windows and I'm getting the following error when trying to start the service:

Faulting application name: PythonService.exe, version: 3.7.300.0, time stamp: 0x5faf5deb
Faulting module name: python37.dll, version: 6.3.9600.19678, time stamp: 0x5e82c88a
Exception code: 0xc0000135
Fault offset: 0x00000000000ecf40
Faulting process id: 0x738
Faulting application start time: 0x01d72fa979da8cde
Faulting application path: C:\Python_Envs\venv3.7\Lib\site-packages\win32\PythonService.exe
Faulting module path: python37.dll
Report Id: b7930609-9b9c-11eb-8108-00155d29880f
Faulting package full name:
Faulting package-relative application ID:

It runs fine in Foreground mode. Any ideas on what could be causing this or how I could get it running?

Hi,

I've just recently moved over to using the latest (gh-perPluginLog) branch of shotgunEvents.

I've got a couple of plugins here - one of which is very quick to run, and another which can take a bit of time... (I'm working on optimising it...!)

I'd love to be able to put the slower one in its own thread, so the quicker plugin can still do its own thing without having to wait for the slower one...

Reload the config when the file changes on disk.

This has major repercussions on the whole logging setup, Shotgun connections, email auth, etc.
It would be a nice to have though especially if the config holds per plugin options.

There needs to be some installation instructions for the uninitiated. First, how to get the source? Either instructions on how to pull it down with git or a link to the .tar.gz like this one.

Next, where to put the python module? Is there an installation or compilation step? I think it just needs to be in $PATH, since it is a standalone executable. Is that correct? I'd also recommend looking at using distutils and creating a setup.py file so that users can just do sudo setup.py install and have it do the right thing for their system.

I think putting that info in the README or the first page of the docs would be a good start.

I've got the code in the following directory structure:

conf/shotgunEventDaemon.conf
scripts/shotgunEventDaemon.py
script/daemonizer.py
plugins/whatever.py

Could I request that _getConfigPath() has the following line instead of the current "paths=" that is has (this is in the format used in gh-perPluginPath)

paths = ['/etc', os.path.join(os.path.dirname(file), "../conf/"), os.path.dirname(file)]

This will cause it to also look for the config file in the same folder as the script, and also with the kind of directory structure that I'm using.

We're in the process of dockerizing shotgunEventDaemon to be used in our kubernetes Cluster.

Because of the nature of a container environment the state of the shotgunEvent.id will be gone if a container is rescheduled. This can of course be compensated with a mount of a static resource inside the container.

[Feature Request]
Better would be to use a distributed key/value store to save state to like etcd or i.e consul.

Is there something like this on the roadmap? Thanks for your time.

i'm trying to install the Daemon but no matter which python version i reinstall with all the PyWin32 and ShotgunAPI installs, it won't find the file to install the Daemon.

I followed the installation documentation and used this cmd
C:\Users\me\AppData\Local\Programs\Python\Python37-32\python.exe shotgunEventDaemon.py install

the shotguneventdaemon.py is in my C:\shotgrid\shotgridEvents\shotgunEvents\src

I want to modify subject email. when i test development project. How to do it ?

Should create some solid scenarios to test for the handling of missed events that commonly occur when doing large-scale imports. Basic testing succeeded but this is more about a thorough test procedure

Support standard python distribution schemes by adding a setup.py script.

Add configuration for plugins to run on a per-project basis. This could be written into the plugin logic itself, but would be better to include something more flexible like the matchEvents parameter to specify the Projects the plugin should (or shouldn't) run on.

Hi,

I'd like to run the shotgun event handler as a systemd service so that when our trigger box/vm gets restarted, the services are also restarted.

Is there an example systemd service unit file I can reference for our local tweaking ?

Cheers