jamesyonan / brenda Goto Github PK

It looks like the AMI that this project references no longer exists. It would nice to have that updated to a working AMI, or update the README to point to an up-to-date fork of this repo. It's hard to navigate which ones are still supported but I eventually ran into the following which works really well:
https://github.com/gwhobbs/brenda (updated to blender 2.80)

I decided to create my own fork from gwhobbs in order to use a newer version of Blender:
https://github.com/msmith93/brenda (Updated to blender 2.83.4 and python3)

Thanks for setting up this project, it works really well!

seems to install fine, but I get a few errors trying to run brenda-work. when running from normal user I get:

$ brenda-work
Traceback (most recent call last):
File "/usr/bin/brenda-work", line 20, in
from brenda import config, work, aws, version
ImportError: cannot import name 'config'

When I run with sudo I get:

$ sudo brenda-work
Traceback (most recent call last):
File "/usr/bin/brenda-work", line 20, in
from brenda import config, work, aws, version
File "/usr/lib/python3.4/site-packages/brenda/work.py", line 79
print task,
^
SyntaxError: Missing parentheses in call to 'print'

It seems as though the syntax error is related to changes in Python 3. Not sure what is causing the import error though.

I've started working on a task script for distributed baking with Brenda, but I don't want to create a pull request until I've worked some of the issues out. The current work-in-progress version of the script is available on github at https://github.com/jbaicoianu/blender-brenda-master/blob/master/templates/brenda/bake and I've made a detailed post about its usage on brendapro.com - http://brendapro.com/forum/viewtopic.php?f=17&t=34

Basically this works by splitting out each object into its own job, which works well if all your objects are of similar size in the final baked image but in real-world usage leads to grossly unbalanced jobs. In my test run (100 samples, 70 objects, baked to 3 separate 4096x4096 lightmaps), the easiest tasks started finishing around 2.5h in, while some of the harder ones took almost 30 hours:

My question is, is it possible to further subdivide these sorts of bake jobs, similar to the single-frame splitting where you can subdivide each texture into smaller tiles?

Hi I'm getting the following error message when trying to create a spot instance request, can someone point me in the right direction? I'm on osx mojave 10.14.6, thanks.

File "/Users/dioncoder/Library/Python/2.7/lib/python/site-packages/boto/connection.py", line 1186, in get_list
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request

InvalidAMIID.NotFoundThe image id 'ami-0529086c' does not exist510a4f7b-a0cf-49d3-8eee-c563bd4d504c

I have a blend scene with linked objects with cloth sim on them. The sim is done in the main scene file and the blend-cache is saved in the main scene file directioy . I packed the libraries tared up the blend file and the cache directory and submitted it to a work queue.
Now as we all know, for whatever stupid reason, blender likes to delete thees kind of cache files next time the scene is opened. The usual workaround is to set proper permissions on the cache directory. However brenda runs blender as root on the instances and the permissions are meaningless, so blender deletes the cache files and breaks the cloth sim. Cloth sim works fine on local sims.. Blender does not wish to delete thees cache files..
Any ideas for a workaround?

Can you please give me the step to get idle spot instances?

So, that I can assign another job into it without creating new spot instance.
If we get this worked, we can use this one spot instance for other jobs until it terminate.

Any idea is more appreciable ...

I just signed up with aws. My spot limit for c1.xlarge seems to be 20, but brenda is getting a MaxSpotInstanceCountExceeded back from amazon.
I have one spot running and it is the only one I've managed to start. Anyone know what could be happening?

trist@bizmuth:~$ brenda-run -P -i c1.xlarge -N 1 -p 0.15 spot
----------------------------
AMI ID: ami-0529086c
Max bid price 0.15
Request type: persistent
Instance type: c1.xlarge
Instance count: 1
Instance store device: /dev/sdb
SSH key name: brenda
Security groups: ('brenda',)
Startup Script:
   #!/bin/bash
   # run Brenda on the EC2 instance store volume
   B="/mnt/brenda"
   if ! [ -d "$B" ]; then
     for f in brenda.pid log task_count task_last DONE ; do
       ln -s "$B/$f" "/root/$f"
     done
   fi
   export BRENDA_WORK_DIR="."
   mkdir -p "$B"
   cd "$B"
   /usr/local/bin/brenda-node --daemon <<EOF
   AWS_ACCESS_KEY=[redacted]
   AWS_SECRET_KEY=[redacted]
   BLENDER_PROJECT=s3://grindtamp-bucket/amazon_grindtamp_render.tar.gz
   WORK_QUEUE=sqs://grindtamp-bucket
   RENDER_OUTPUT=s3://grindtamp-frames
   DONE=shutdown
   EOF
Traceback (most recent call last):
  File "/usr/local/bin/brenda-run", line 169, in <module>
    main()
  File "/usr/local/bin/brenda-run", line 150, in main
    run.spot(opts, conf)
  File "/usr/local/lib/python2.7/dist-packages/brenda/run.py", line 100, in spot
    reservation = ec2.request_spot_instances(**run_args)
  File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1647, in request_spot_instances
    verb='POST')
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1186, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>MaxSpotInstanceCountExceeded</Code><Message>Max spot instance count exceeded</Message></Error></Errors><RequestID>047e4eda-adfc-43ba-bc58-e1115c66a5e8</RequestID></Response>

I recently ran a large lightmap baking job, where some of the individual jobs took around 30 hours. This caused some problems, because Amazon's SQS service only seems to allow you to "hide" an item in the queue for up to 12h, and if you try to extend this time for longer it throws an exception. This causes the brenda-node script to terminate, and the jobs are returned to the queue, even though the Blender process is still happily churning away.

Once Blender finishes, the output file is created but not synced to s3, and the instance is not shut down. I had to do these two steps manually for my bake.

Here's a graph which shows this problem in action:

root@kali:~# brenda-run -i m2.xlarge price
Traceback (most recent call last):
File "/usr/local/bin/brenda-run", line 20, in
from brenda import aws, config, run, version
File "/usr/local/lib/python2.7/dist-packages/brenda/aws.py", line 18, in
import boto, boto.sqs, boto.s3, boto.ec2
ImportError: No module named boto

Would it be possible to implement this on vast.ai?

This is driving me nuts. I've created a custom AMI on AWS to incorporate my own tools / add-ons used with Blender. While configuring the image on AWS, everything works as expected, both as user ubuntu as well as user root. I need to set an environment variable to locate the tools, and I do it like this:

in /etc/environment:
BLENDER_USER_CONFIG=/mypath/dingdong/
BLENDER_USER_SCRIPTS=/mypath/dongding/

I then created the AMI and configured Brenda to use it, which works just fine. With one exception: When Brenda fires off the job, it does not see my custom vars. I ssh into the running instance again, and as before, no matter if I am user ubuntu or root, or even use sudo printenv, all the vars are listed and correct. I can run Blender from the command line in all circumstances, the tools will be loaded. It is only when submitting the job using brenda that the custom dirs are never found.

What can I do? Which key concept do I miss here?

Hey guys,

I have looked everywhere in the documentation, but can't find how to change the intance region from us east to ireland(europe).

Anyone who knows?

Note: I'm new to git, so if I placed this question in the wrong section. Please notify me.

Hi,

Does this also work on the GPU clusters at all?

g2.2xlarge

looking for a fairly quick way of rendering a project of around 250 frames

Hey guys,

According to brenda my bid of $0.05 is too low even though when running the price command. It says that the current price is $0.016.

Am I doing something wrong or?...

I'm following along with the tutorial. I've put my tar.gz on s3 and can see it.

I've set up a template...

[Propeller:Earth]563$ cat frame-template
blender -b *.blend -F PNG -o $OUTDIR/frame_###### -s $START -e $END -j $STEP -t 0 -a

But trying to push work fails due to '"Can only include alphanumeric characters, hyphens, or underscores. 1 to 80 in length".

[Propeller:Earth]564$ brenda-work status
[Propeller:Earth]565$ brenda-work reset
[Propeller:Earth]566$ brenda-work -T frame-template -e 360 push
Traceback (most recent call last):
File "/Users/chuck/anaconda/bin/brenda-work", line 121, in
main()
File "/Users/chuck/anaconda/bin/brenda-work", line 112, in main
work.push(opts, args, conf)
File "/Users/chuck/anaconda/lib/python2.7/site-packages/brenda/work.py", line 75, in push
q = aws.create_sqs_queue(conf)
File "/Users/chuck/anaconda/lib/python2.7/site-packages/brenda/aws.py", line 145, in create_sqs_queue
return conn.create_queue(qname, visibility_timeout=visibility_timeout)
File "/Users/chuck/anaconda/lib/python2.7/site-packages/boto/sqs/connection.py", line 95, in create_queue
return self.get_object('CreateQueue', params, Queue)
File "/Users/chuck/anaconda/lib/python2.7/site-packages/boto/connection.py", line 1188, in get_object
raise self.ResponseError(response.status, response.reason, body)
boto.exception.SQSError: SQSError: 400 Bad Request

Hello,

I'm testing brenda in order to avoid render with my old PC. It worked fine rendering 1 frame for instance, as you explained at the tutorial, but at subframe rendering, the tasks are not generated. The subframe-template is fine, and the work queue script works, apparently, but there are 0 Queued task everytime.

Has you, or anybody, this issue? what am i doing wrong? I miss any script file?

My instructions:

$ brenda-work -T subframe-template -e 1 -X 4 -Y 4 -d push
cat >subframe.py <<EOF
import bpy
bpy.context.scene.render.border_min_x = 0.0
bpy.context.scene.render.border_max_x = 0.25
bpy.context.scene.render.border_min_y = 0.0
bpy.context.scene.render.border_max_y = 0.25
bpy.context.scene.render.use_border = True
EOF
blender -b *.blend -P subframe.py -F PNG -o $OUTDIR/frame_######_X-0.0-0.25-Y-0.0-0.25 -s 1 -e 1 -j 1 -t 0 -a
cat >subframe.py <<EOF
import bpy
bpy.context.scene.render.border_min_x = 0.0
bpy.context.scene.render.border_max_x = 0.25
.
.
. and so on

$ brenda-work status
Queued tasks: 0

Obviously, no .png is generated when i run the instances.

subframe-template file:

"
cat >subframe.py <<EOF
import bpy
bpy.context.scene.render.border_min_x = $SF_MIN_X
bpy.context.scene.render.border_max_x = $SF_MAX_X
bpy.context.scene.render.border_min_y = $SF_MIN_Y
bpy.context.scene.render.border_max_y = $SF_MAX_Y
bpy.context.scene.render.use_border = True
EOF
blender -b *.blend -P subframe.py -F PNG -o $OUTDIR/frame_######_X-$SF_MIN_X-$SF_MAX_X-Y-$SF_MIN_Y-$SF_MAX_Y -s $START -e $END -j $STEP -t 0 -a

"

Thank you in advance

Hi All,

Here is my requirements, please go through it and give me a solution for this.

1. How to track the spot instances?? whether its executing any job or not?
   From this method we can find the Active spot instance with Idle status(which means its not bind with
   any job, so that we can assign new job to this spot instance without shutting down).

2.How to track the** jobs status** of Spot instance?

3.Is there any method to find the jobs assigned to Spot instances(if count more than 1 or 2)??

I have brenda running on an AWS Ec2 instance (Amazon Linux AMI). I executed the command brenda-tool ssh uptime and got the following:

$brenda-tool ssh uptime
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

Exception in thread Thread-6:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

Exception in thread Thread-7:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

Exception in thread Thread-8:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/lib64/python2.6/site-packages/brenda/tool.py", line 61, in worker
output = utils.system_return_output(cmd, capture_stderr=capture_stderr)
File "/usr/lib64/python2.6/site-packages/brenda/utils.py", line 116, in system_return_output
output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
AttributeError: 'module' object has no attribute 'check_output'

If someone messes up the configuration by including folders, misspelling, forgetting to upload the blend files archive or similar mistakes he will get a cryptic paracurl.Exception: (-5, 'PC_ERR_HTTP : bad HTTP status code') error.
Maybe it would help if there would be a more specific error message like "Could not find blend file ...".
The origin of the error is at:
File "/usr/local/lib/python2.7/dist-packages/brenda/node.py", line 382, in get_s3_project file_len, etag = aws.s3_get(conf, s3url, fn, etag=etag)

And thank you for this great tool.

Hi James,

I'm having a challenge locating the log files for brenda on the nodes.

Hi,
I've got Brenda set up and I'm able to render with spot prices. I'm trying to get a performance evaluation using the script that's here in github (in the readme), but brenda-tool perf doesn't output anything. I the instances aren't active it just quits as expected, but if they are active it freezes for several minutes and then quits without error or any message.

Am I doing something wrong or missing something?
Thanks!

Hello again

I have created new private AMI with 2.72b blender on it and everything working fine but I cannot retrieve tail log from servers via brenda-tool I am getting this error:

ec2-54-197-165-4.compute-1.amazonaws.com
Please login as the user "ubuntu" rather than the user "root".

I can log in via putty myself as a root without problems using my private *.pem file (not brenda one). When I log in I can see via the Top command that blender is working in a background and using all CPUs. So only problem is that I cannot get a log via brenda-tool, means I cannot see how fast does it renders.

Thanks

I'm using the Brenda console script (https://github.com/rider-rebooted/win_brenda_console ) to run Brenda. I've successfully run tests with the Brenda console script in the past. I stopped using it several months ago but now am ready to start running it again, this time with a real project.

Unfortunately, I am having difficulty acquiring a reasonably-priced instance because the Brenda console, Brenda itself, or AWS is always picking an expensive availability zone that I don't want, and thus I can never get my render started.

I'm in US-East (Virginia). When I check prices on the Brenda console, I get something that looks like this:

Spot price data for instance c1.xlarge
us-east-1a 2017-07-22T09:28:44.000Z $0.106
us-east-1b 2017-07-22T10:14:18.000Z $0.106
us-east-1c 2017-07-22T13:00:14.000Z $0.1075
us-east-1d 2017-07-22T09:28:44.000Z $5.2
us-east-1e 2017-07-22T09:28:44.000Z $0.1066

price data for instance c3.large
us-east-1a 2017-07-22T12:28:47.000Z $0.0156
us-east-1b 2017-07-22T12:28:47.000Z $0.0158
us-east-1c 2017-07-22T12:28:47.000Z $0.0159
us-east-1d 2017-07-22T12:28:47.000Z $1.05
us-east-1e 2017-07-22T13:31:02.000Z $0.016
us-east-1f 2017-07-22T12:28:47.000Z $1.05

Spot price data for instance c3.xlarge
us-east-1a 2017-07-22T09:00:40.000Z $0.0428
us-east-1b 2017-07-22T13:04:01.000Z $0.0426
us-east-1c 2017-07-22T12:16:19.000Z $0.0427
us-east-1d 2017-07-22T07:12:25.000Z $2.1
us-east-1e 2017-07-22T11:42:37.000Z $0.0436
us-east-1f 2017-07-22T07:12:25.000Z $2.1

Spot price data for instance c3.2xlarge
us-east-1a 2017-07-22T13:39:42.000Z $0.1049
us-east-1b 2017-07-22T13:41:00.000Z $0.1047
us-east-1c 2017-07-22T13:20:00.000Z $0.1047
us-east-1e 2017-07-22T13:41:16.000Z $0.1092

I'll pick one instance of c3.large, for, say, $0.10 or $0.20, which -- based on the prices under c3.large, you'd think it would start up an instance. Instead, in the AWS console I get the status of "capacity-not-available" and there it sits forever because, according to the console, the job is scheduled to run on us-east-1d -- the most expensive availability zone ($1.05) for an instance. I want to run on any availability zone other than us-east-1d but can't figure out how to change it. Even when I choose a different instance type, like c3,xlarge, it still defaults to run on us-east-1d -- the most expensive option and way above my bid price.

I'm starting to pick apart what the Brenda console script is doing, and it looks like it's sending a command like this to Brenda:

python brenda-run -i c3.large -N 1 -p 0.20 spot

I looked through the Brenda documentation but didn't see a way to specify, say, us-east-1a as the availability zone. Is that possible? If so, what would be the syntax to do that?

If that's not possible, is there a way to set up things in AWS so that it defaults to using us-east-1a?

I'm getting this error from some of the instances but not all of them:

    func = lambda : node.run_tasks(opts, args, conf)
  File "/usr/local/lib/python2.7/dist-packages/brenda/node.py", line 333, in run_tasks
    proj_dir = get_project(conf, blender_project)
  File "/usr/local/lib/python2.7/dist-packages/brenda/node.py", line 424, in get_project
    get_s3_project(conf, url, proj_dir)
  File "/usr/local/lib/python2.7/dist-packages/brenda/node.py", line 382, in get_s3_project
    file_len, etag = aws.s3_get(conf, s3url, fn, etag=etag)
  File "/usr/local/lib/python2.7/dist-packages/brenda/aws.py", line 92, in s3_get
    return paracurl.download(dest, url, **paracurl_kw)
paracurl.Exception: (-5, 'PC_ERR_HTTP : bad HTTP status code')

Probably my settings, but I'm having trouble figuring it out.

Any ideas?

I'm utilizing c4.2xlarge instances to render an animation, and at a random point early on in the render, I get this error:
./brenda-go: line 2: 2099 Bus error (core dumped) /usr/local/blender-2.79-a6f750dd414-linux-glibc219-x86_64/blender -b *.blend -F EXR -o /mnt/brenda/brenda-outdir1.tmp/frame_###### -s 240 -e 240 -j 1 -t 0 -a RMTREE /mnt/brenda/brenda-outdir1.tmp ******* RETRY 1/5: fatal error in active task

The process is never able to recover. I checked and the memory being used by the scene is well within the 15GB of memory on the c4.2xlarge instances. I also tried the render with a c5.4xlarge instance (32GB memory) and received the same error.

The instances are running a custom AMI with Blender 2.79 and the latest version of Ubuntu. Other scenes and files have rendered fine on these instances. I even tried rendering a test frame from the file throwing errors, and it renders fine on my desktop system.

Anyone know what this could mean?

If you have other instances with the same AMI, they will be stopped as well. This is very strange and sloppy behavior.

Hi I'm running brenda through a python console in windows, so I'm trying to run this command that stops the brenda-node script on all instances, but not shut down the instances:
$ brenda-tool ssh 'kill $(cat brenda.pid)'

So in my console I have to add python at the beginning of the line like that:
python brenda-tool ssh 'kill $(cat brenda.pid)'

And it should work, but it doesn't. The instances don't stop. I think it has to do with the $ at the beginning, but I don't know to change it. Anyone can tell me about it? Maybe other ways to get the same?

Looks like it hasn't been updated in a while and it doesn't seem to like newer generation EC2 instances. Is there an alternative to this project?

I am requesting spot instances that have an EBS snapshot attached that contains project files to render. I get a bad-parameters failure on the spot request with the following state: failed: Invalid device name /dev/sdf1 The output from the brenda-run command gives me my Snapshot ID, and what appears to be the device mapping:
Project EBS snapshot: [('snap-xxxxxxxx', '/dev/sdf1')]

Would the spot requests be failing because brenda (or boto) is device mapping to something AWS is saying is invalid? brenda is assigning block_device_map to bdm , and in brenda/aws.py, bdm is using boto.ec2.blockdevicemapping.BlockDeviceMapping()

from brenda/aws.py:

def blk_dev_map(opts, conf, itype, snapshots):
    if not int(conf.get('NO_EBS', '0')):
        bdm = boto.ec2.blockdevicemapping.BlockDeviceMapping()
        snap = project_ebs_snapshot(conf)
        snap_id = translate_snapshot_name(conf, snap, snapshots)
        snap_description = []
        if snap_id:
            dev = utils.blkdev(0)
            bdm[dev] = boto.ec2.blockdevicemapping.EBSBlockDeviceType(snapshot_id=snap_id, delete_on_termination=True)
            snap_description.append((snap, snap_id, dev))
        i = 0
        for k in additional_ebs_iterator(conf):
            i += 1
            snap = parse_ebs_url(conf[k].split(',')[0])
            snap_id = translate_snapshot_name(conf, snap, snapshots)
            if snap_id:
                dev = utils.blkdev(i)
                bdm[dev] = boto.ec2.blockdevicemapping.EBSBlockDeviceType(snapshot_id=snap_id, delete_on_termination=True)
                snap_description.append((snap, snap_id, dev))
        istore_dev = add_instance_store(opts, conf, bdm, itype)
        return bdm, snap_description, istore_dev
    else:
        return None, None, None

In the file used to start up the instances, brenda/run.py, bdm is given as the block_device_map

from brenda/run.py:

def spot(opts, conf):
    ami_id = utils.get_opt(opts.ami, conf, 'AMI_ID', default=AMI_ID, must_exist=True)
    price = utils.get_opt(opts.price, conf, 'BID_PRICE', must_exist=True)
    reqtype = 'persistent' if opts.persistent else 'one-time'
    itype = brenda_instance_type(opts, conf)
    snapshots = aws.get_snapshots(conf)
    bdm, snap_description, istore_dev = aws.blk_dev_map(opts, conf, itype, snapshots)
    script = startup_script(opts, conf, istore_dev)
    user_data = None
    if not opts.idle:
        user_data = script
    ssh_key_name = conf.get("SSH_KEY_NAME", "brenda")
    sec_groups = (conf.get("SECURITY_GROUP", "brenda"),)
    run_args = {
        'image_id'      : ami_id,
        'price'         : price,
        'type'          : reqtype,
        'count'         : opts.n_instances,
        'instance_type' : itype,
        'user_data'     : user_data,
        'key_name'      : ssh_key_name,
        'security_groups' : sec_groups,
        'block_device_map' : bdm,
        }

    print "----------------------------"
    print "AMI ID:", ami_id
    print "Max bid price", price
    print "Request type:", reqtype
    print "Instance type:", itype
    print "Instance count:", opts.n_instances
    if snap_description:
        print "Project EBS snapshot:", snap_description
    if istore_dev:
        print "Instance store device:", istore_dev
    print "SSH key name:", ssh_key_name
    print "Security groups:", sec_groups
    print_script(opts, conf, script)
    aws.get_done(opts, conf) # sanity check on DONE var
    if not opts.dry_run:
        ec2 = aws.get_ec2_conn(conf)
        reservation = ec2.request_spot_instances(**run_args)
        print reservation

It seems similar to what's going on in this thread, but I'm not sure how to implement the solution in my case with brenda.

boto version: 2.49.0

Appreciate any help!

hi

I'm getting this error message when I run brenda/setup.py (maxosx10.9). I don't think I have gcc4.0?

gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch ppc -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c paracurl/paracurl.c -o build/temp.macosx-10.5-fat3-2.7/paracurl/paracurl.o
unable to execute gcc-4.0: No such file or directory
error: command 'gcc-4.0' failed with exit status 1

It seems it can create the work queue etc. but nothing will actually render! Any help would be appreciated!
thx

I've been able to run Win-Brenda using the existing recommended AMIs. Now I'm trying to set up my own AMI with the newest version of Blender (2.78c) and I think I'm missing something.

Long story short: when I run an instance of the custom AMI via Win-Brenda, nothing gets rendered. I don't know how to view what's going on in my particular AMI because I get nothing from the "Tail log from instances" feature. The AMI is shown to be running in the AWS console -- I think I don't have something set up correctly.

Here are the steps I take to create my AMI:

1. Go to the EC2 console and click on launch instance. On the left you will see AMI options. Quickstart is highlighted by default. Click on Community AMIs. From the checkboxes on the left, choose Ubuntu, 64-bit, and EBS. Enter "Yakkety Yak" for the search term. ("Yakkety Yak" is the code name for Ubuntu 16.10.) Pick an AMI with Virtualization type: HVM (not paravirtual). I picked this image: "ubuntu/images-testing/hvm-ssd/ubuntu-yakkety-daily-amd64-server-20170716 - ami-01212b17"

2. Choose an Instance Type: I picked "c3.large"

3. Configure Instance Details: I left all the defaults. FYI -- Shutdown Behavior is set to "Stop".

4. Add Storage: Leave all the defaults (8 GB for the root).

5. Add Tags: I left the default options.

6. Configure Security Groups: For now, I'm going with the defaults:

   • Assign a Security Group: Create a NEW security group
   • Type: SSH, Source: Anywhere

7. Select an existing key pair or create a new key pair: I chose an existing key pair that I know that works.

8. If you created a new key in step 7, generate the proper key here: http://blog.powercram.com/2009/07/connecting-to-aws-ec2-instance-linux.html

9. After you launch the PuTTY command line, you'll see the command line say "login as:". Put in "ubuntu" (sans quotes).

10. Now run this:
    sudo perl -p -i.bak -e 's/^disable_root: 1/disable_root: 0/' /etc/cloud/cloud.cfg
    sudo perl -p -i.bak -e 's/.*ssh-rsa/ssh-rsa/' /root/.ssh/authorized_keys
    sudo add-apt-repository -y ppa:irie/blender
    sudo apt-get update
    sudo apt install blender
    sudo apt-get install -y blender python-pip gcc python-dev libcurl4-openssl-dev git unzip
    sudo apt-get install python-pip
    sudo pip install -U boto
    sudo pip install -U s3cmd
    sudo apt-get install git
    sudo apt-get update; sudo apt-get install python-dev -y
    sudo apt-get install libcurl4-openssl-dev

11. Next, download and install Brenda:
    git clone http://github.com/jamesyonan/brenda.git
    cd brenda
    sudo python setup.py install

12. At this point, you still don't have the latest version of Blender. To do that...
    sudo add-apt-repository ppa:thomas-schiex/blender
    sudo apt update
    sudo apt install blender

13. To check which version of Blender you have installed:
    blender -v

I've used Linux a bit, but I do not consider myself an expert. Any ideas on things that I can check to make sure that this image runs correctly?

Recently I used your Brenda for rendering some task on EC2. Its a great tool and thank you very much for making it. Just wanted to give you some feedback and question when you got time.

Feedback: Bug - for some reason I got this error when I launched our first work: BotoClientError: Bucket names cannot contain upper-case characters when using either the sub-domain or virtual hosting calling format.

I fixed it by a changing bucket names to lowercase and it was solved.

Also, seems that node log retrieval command skipping downloading logs from nodes for some reason. Will copy terminal message next time.

My question is: If for some reason happens that node is up but it didn't started to work, is there a some Brenda command or ssh command to restart the work on particular node so I don't have to shut it down. Would reboot on that node help?

Again, thank you very much for such a exceptional tool for Blender and EC2.

Best regards
George

Python2.6.9 ec2 amazon linux free tier
I had to change aws.py to fix a syntax error:

From:
def get_spot_request_dict(conf):
ec2 = get_ec2_conn(conf)
requests = ec2.get_all_spot_instance_requests()
return { sir.id: sir for sir in requests }

To:

def get_spot_request_dict(conf):
ec2 = get_ec2_conn(conf)
sirs = [ sir.id for sir in ec2.get_all_spot_instance_requests() ]
return sirs

I've set up an AMI with blender installed but it seems that brenda doesn't start anything after starting the instances.

Are there any logfiles I can check to figure out what's wrong?

There are 180 jobs in the queue

I've completed a job, once the job was done and I checked the project_bucket (not the actual name of the s3 bucket) the PNG frame were not there. The Ec2 instances were already terminate by the DONE=shutdown parameter in ~/.brenda.conf file. I'm re-running the job and I have commented the parameter so that I can have a look at the EC2 log files.

Is it possible I have done something wrong during the setup of the job which would result in this issue?

Hi James,

I have ended to tranlate you explanations of Brenda in french.
May be would be usefull for francophone's users, not?
https://www.dropbox.com/s/di2r5syrg7j2spi/Brenda_README_Fr.pdf?dl=0

You can send me your answer here: patrick dot depoix arobase gmail dot com
Bye bye
Spirou4D

I'm on a retina Macbook pro running Yosemite and having an issue regarding Amazon certificates and getting Brenda setup.

I've run the s3cmd --configure and it has completed successfully, but I'm still unable to use brenda-work. See the output below for more details. It appears to be a certificate issue. I will say that I'm pretty new to EC2, so I could have the configuration wrong or something. Maybe I'm using the wrong values for the ACCESS and SECRET keys? I pulled these values from the Access Keys (Access Key ID and Secret Access Key) section of Your Security Credentials inside of the Amazon Web Services Console. Any ideas?

New settings:
  Access Key: [MYACCESSKEY]
  Secret Key: [MYSECRET]
  Encryption password:
  Path to GPG program: /usr/local/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] Y

Now when running brenda-run init, I get the following:

Robs-MacBook-Pro:brenda robsawyer$ brenda-run init
Pushing ssh public key '/Users/robsawyer/.ssh/id_rsa.pub' to AWS under 'brenda' key pair.
Error creating ssh key pair [Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Creating AWS security group 'brenda'.
Error creating security group [Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Out of curiousity I tried brenda-work status and the following is happening.

Robs-MacBook-Pro:brenda robsawyer$ brenda-work status
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.6/bin/brenda-work", line 121, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/2.6/bin/brenda-work", line 114, in main
    work.status(opts, args, conf)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/brenda/work.py", line 84, in status
    q = aws.get_sqs_queue(conf)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/brenda/aws.py", line 153, in get_sqs_queue
    return get_sqs_conn_queue(conf)[0]
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/brenda/aws.py", line 150, in get_sqs_conn_queue
    return conn.get_queue(qname), conn
  File "/usr/local/lib/python2.7/site-packages/boto/sqs/connection.py", line 462, in get_queue
    return self.get_object('GetQueueUrl', params, Queue)
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 1192, in get_object
    response = self.make_request(action, params, path, verb)
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 1116, in make_request
    return self._mexe(http_request)
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
    raise ex
ssl.SSLError: [Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Hello
I am testing Brenda with Blender 2.8 for Cylces and all my renders are black frames. Therefore, I am wondering if Brenda is 2.8 friendly.
Thank you

I'm needing to render out frames of an animation as multi-layer .exr files, and have the render output settings in my .blend file set as OpenEXR MultiLayer. If I change "PNG" to "EXR" in my brenda script template, would that be causing issues? I'm able to get a work queue going and spool up instances, but it seems like the instances aren't rendering frames. I'm not getting any errors, but I also get no output when I run brenda-tool ssh tail log (or any of the ssh commands). I did configure the cloud.cfg in my AMI to allow brenda-tool ssh tail log root access.

I'm trying to narrow down possibilities for why the instances aren't processing frames. If it's not the EXR, it might be an issue with my custom AMI, which is running Blender 2.79.

Hi,

I am working on setting up brenda on OS X 10.7.5. When I run:

brenda-run -i c1.xlarge price

Traceback (most recent call last):
File "/Users/Henry/Blender/utils/brenda/brenda-run", line 20, in
from brenda import aws, config, run, version
File "/Users/Henry/Blender/utils/brenda/brenda/aws.py", line 20, in
import paracurl
ImportError: No module named paracurl

I see paracurl/paracurl.c what are the steps to compile?

Hey guys,

After a lot of problems I finnaly got an intance up and running, the only problem is that it doesn't render... at all. When looking at the sqs list it looks like the servers render the frames very quickly, only when monitoring the servers through AWS, the cpu doens't even pop above 10% and de frames aren't in the bucket when "finished"

Any suggestions?