For some reason, only the first two papers are being built right now by the server, even after I tear it down and rebuild it.

@stefanv @katyhuff Any thoughts?

I've switched it back to python2 and it still has this problem. I'm sad to say that I'm a little stumped.

@sarostru and @sbenthall : it would be best if one of you could serve procbuild on zibi from your user account (this way, you can respond to authors needs directly without me in the way). Since people are starting to submit, the server needs to be up and running at all times, to allow submitting authors to see whether their pr is building.

I have it serving right now from my user account on zibi but something is funky (http://zibi.bids.berkeley.edu:5000/) . While it finds the paper PR (through the github webhook), doesn't seem to be trying to build it. ... anyone feel like debugging? @stefanv I don't remember what it means when the brackets are empty...

This is how it went down:

khuff@maria:~ $ ssh [email protected]
Last login: Wed May 25 11:49:50 2016 from 169.229.224.158
zibi:~ huff$ source activate py27
discarding /Users/huff/anaconda3/bin from PATH
prepending /Users/huff/anaconda3/envs/py27/bin to PATH
(py27)zibi:~ huff$ cd procbuild/
(py27)zibi:procbuild huff$ ./update_prs
Fetching page 1 (state=open) from scipy-conference/scipy_proceedings...
/Users/huff/anaconda3/envs/py27/lib/python2.7/site-packages/urllib3/connectionpool.py:790: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Fetching page 2 (state=open) from scipy-conference/scipy_proceedings...
/Users/huff/anaconda3/envs/py27/lib/python2.7/site-packages/urllib3/connectionpool.py:790: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
(py27)zibi:procbuild huff$ python runserver.py &
[1] 98515
(py27)zibi:procbuild huff$ Setting up build queue...
Launching queue monitoring process...
serving on http://0.0.0.0:5000

• Verify that #29 is good to go
• Talk through how to create sockets that respond to multiple channels and do appropriate things.
  • sockets are organised around channels, you can have sockets recv messages and then switch functionality on their target value/channel.
  • this should be useful for adding back the queue_size logic
• Talk through how to setup basic travis stuff.
  • Talk through how to autopush to dockerhub from travis
  • Talk through how to autopush a Docker file to the heroku registry from within travis
• Set up basic server test (so as to have something to test on travis to make sure it works correctly)

Stéfan added:

• What are we doing to keep the server alive?
• Can we preserve state across deployments? (One option is to issue a build all by default with each new update.)

we have a couple of build errors on the server that seem to be caused by unicode characters in the users input documents, you can look at a couple of the failed builds on zibi for examples (http://zibi.bids.berkeley.edu:5000/#).

Is that expected? How should they encode the characters? Or is this a problem with the build server that should be debugged?

@mpacer Would you please add me to the scipyproc organization (https://hub.docker.com/u/scipyproc) on dockerhub? My dockerhub is cbcunc.

Right now it's not straight-forward to debug the server, because the easiest way to actually deposit a payload to a running server is to actually create a new PR that meets the Paper: naming criteria on scipy-conference/scipy_proceedings.

There are a number of things to improve:

• trigger individual builds
• trigger the webhook

GitHub seems to think that the older versions have security vulnerabilities. It probably doesn't matter much for procbuild, but we should still endeavor to do The Right Thing™.

Currently, the timestamp of the last successful build of a paper is presented next to the download button:

presumably, this is to give some indication of how stale or out of date a build is. From the time offset, this looks like it's currently being rendered in UTC, which might make obfuscate the actual age of the build. Would it make sense to pull in the locale in the js layer and modify appropriately? Or marking these in the template specifically as non-local time?

Need the following fixes to the Dockerfile:

1. Should be able to set UID and GID of procbuild user in order to persist paper cache.
2. Ubuntu:17.10 no longer available on DockerHub. Switch to latest.
3. Installing packages on Ubuntu:latest pulls in tzdata dependency which interactively asks for timezone, stopping docker build. Needs ENV DEBIAN_FRONTEND=noninteractive.
4. Ubuntu:latest requires explicit installation of python3-venv.

So... in the buildbot, we have a handful of papers which should successfully build (see Dask), but the procbuild system seems to be building only the example vanderwalt file. Currently we exclude '00_vanderwalt', 'vanderwalt', and 'jane_doe'... however the pdf that is being built is the vanderwalt paper rather than the (for example) Dask paper.

Last bits of error output:

ERROR:waitress:Socket error
Traceback (most recent call last):
File "/Users/mpacer/anaconda3/envs/py36/lib/python3.6/site-packages/waitress/channel.py", line 166, in handle_read
data = self.recv(self.adj.recv_bytes)
File "/Users/mpacer/anaconda3/envs/py36/lib/python3.6/asyncore.py", line 374, in recv
data = self.socket.recv(buffer_size)
TimeoutError: [Errno 60] Operation timed out

Here's the error currently after the server has been running for a while:

n2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/mpacer/anaconda3/envs/py27/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/mpacer/procbuild/procbuild/server.py", line 84, in index
    log("Updating papers...")
  File "/Users/mpacer/procbuild/procbuild/server.py", line 39, in log
    message + '\n')
TypeError: write() argument 1 must be unicode, not str
ERROR:waitress:uncaptured python exception, closing channel <waitress.server.TcpWSGIServer listening 0.0.0.0:5000 at 0x1050b98d0> (<class 'socket.error'>:[Errno 22] Invalid argument [/Users/mpacer/anaconda3/envs/py27/lib/python2.7/asyncore.py|read|83] [/Users/mpacer/anaconda3/envs/py27/lib/python2.7/asyncore.py|handle_read_event|443] [/Users/mpacer/anaconda3/envs/py27/lib/python2.7/site-packages/waitress/server.py|handle_accept|252] [/Users/mpacer/anaconda3/envs/py27/lib/python2.7/site-packages/waitress/server.py|set_socket_options|315] [/Users/mpacer/anaconda3/envs/py27/lib/python2.7/socket.py|meth|228])
WARNING:waitress:unhandled close event

@katyhuff

Not sure what's causing this but here's the traceback

@stefanv any thoughts?

ERROR:waitress:uncaptured python exception, closing channel <waitress.channel.HTTPChannel connected 128.32.30.77:41652 at 0x104f00048> (<class 'TypeError'>:coercing to str: need a bytes-like object, NoneType found [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|read|83] [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|handle_read_event|423] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/channel.py|handle_read|174] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/channel.py|received|191] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/parser.py|received|102] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/parser.py|parse_header|201] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/compat.py|tostr|53])
ERROR:waitress:uncaptured python exception, closing channel <waitress.channel.HTTPChannel connected 128.32.30.77:41656 at 0x104f2a208> (<class 'TypeError'>:coercing to str: need a bytes-like object, NoneType found [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|read|83] [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|handle_read_event|423] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/channel.py|handle_read|174] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/channel.py|received|191] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/parser.py|received|102] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/parser.py|parse_header|201] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/compat.py|tostr|53])
ERROR:waitress:uncaptured python exception, closing channel <waitress.channel.HTTPChannel connected 128.32.30.77:42604 at 0x104f33710> (<class 'TypeError'>:coercing to str: need a bytes-like object, NoneType found [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|read|83] [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|handle_read_event|423] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/channel.py|handle_read|174] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/channel.py|received|191] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/parser.py|received|102] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/parser.py|parse_header|201] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/compat.py|tostr|53])
ERROR:waitress:uncaptured python exception, closing channel <waitress.server.TcpWSGIServer listening 0.0.0.0:7001 at 0x104c2e198> (<class 'OSError'>:[Errno 22] Invalid argument [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|read|83] [/Users/mpacer/anaconda3/lib/python3.5/asyncore.py|handle_read_event|417] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/server.py|handle_accept|252] [/Users/mpacer/anaconda3/lib/python3.5/site-packages/waitress/server.py|set_socket_options|315])

Because of the way indexing is handled, every time there is a new PR all the old indices break.

This is because the new PR is given the 0 index value (rather than the n index value). A simple fix would be to just organise the building in reverse.

A real fix would be to index papers by unique keys related to their identity at the time of their creation that can be trusted not to change (paper titles would not work, but user names may not be unique if more than one submission is submitted by the same author).

Sometimes authors merge the canonical branch into their scipy_proceedings fork's branch. When they do so, builder.BuildManager.build_paper() builds the wrong paper. This is because the BuildManager.paper property manager incorrectly assumes that the author's paper is the only unexcluded paper in the author's fork's branch:

return papers[0].split('/')[-1]

Determine another method for selecting the author's paper from the author's branch and implement it into builder.BuildManager.

#5 introduces a new way of getting the queue length, but it never decreases the queue size, so after 50 papers have been built, the builder gets stuck

I tend to default to https for everything, it’d be nice if we could get an ssl certificate for the site so we could have https access.

When a PR is closed, the code seems to detect this and reindex everything, but then it does not propagate that to what is displayed in the html view.

Not sure why this doesn't work. I had thought it did but after the info changed, in a recent PR (scipy-conference/scipy_proceedings#303) it still has the old info.

Probably need to check the hook checking code & make sure it is triggered on merged PRs.