stevearc / pypicloud Goto Github PK

View Code? Open in Web Editor NEW

502.0 15.0 142.0 2.23 MB

S3-backed pypi server implementation

License: MIT License

Shell 0.09% Python 85.48% CSS 0.18% JavaScript 7.64% HTML 5.19% Jinja 1.40% Dockerfile 0.03%

python pypi

pypicloud's Introduction

PyPI Cloud

WARNING: Pypicloud has transitioned to maintenance mode. For details, see #325

Build: _ _
Documentation: _
PyPI: _

This package is a Pyramid web app that provides a PyPI server where the packages are stored on Amazon's Simple Storage Service (S3), Google's Cloud Storage (GCS) or Azure's Blob Storage.

Quick Start

pip install pypicloud[server]
pypicloud-make-config -t server.ini
pserve server.ini

Go to http://localhost:6543/ to view the web interface.

For more detailed step-by-step instructions, check out the getting started section of the docs.

Docker

There is a docker image if you're into that sort of thing: https://github.com/stevearc/pypicloud-docker

pypicloud's People

Contributors

Stargazers

Watchers

Forkers

alonisser mathom duckworthd raizyr craigbeck zzugg danielbwatson iserko johtso sofiamargariti powellc smokey42 a-tal alexd-siaras avihoo lnguyen jcasner johnswanson meunierd dbaxa yxd-hde lingrlongr philwhln pkariz buildingrobotics ediston prezi ahmet2mir benh57 pierrei patricklucas yelp hexadite hexadite-tomer analogue willdrew scizo chriskuehl bkochendorfer kristi danigosa michaelpalmer1 xiaolushare taylorterry3 tinyclues vkolev carl4 triplepoint dschaller dguaraglia doublerainbo frankjian rasahq kevinpropdata rubikloud yoichi threadsstyling gmykhailiuta rtkmhart stylight amussey adamlamers koleror jpds mrgleeco sanyaade-machine-learning joaovictortr fatal10110 jessflan ahlfors duckty thehesiod-forks aexvir mchalek staaam zarlant brianrower wanderxjtu ramjet-labs imran2140 expobrain haarcuba fchorney ryefccd tomlaube simonstjg tmbo brentdorsey nr7s mollie nshipman-io quovo angry-penguins rtkwlf hdscorpio wycttb noscendo-gmbh josephglanville alexpaulzor postmates

pypicloud's Issues

Fresh install doesn't populate packages from existing S3 bucket

Hi,

I'm having some trouble where, when I install a new copy of PyPiCloud from git, it doesn't find the packages which were stored in S3 by a previous copy.

We're using it inside a Docker container. The Dockerfile looks like

FROM stackbrew/ubuntu:13.04
MAINTAINER Gabe Rosenhouse <[email protected]>

RUN sed -i 's/main$/main universe/' /etc/apt/sources.list && apt-get update
RUN apt-get install -y wget ca-certificates python2.7-dev python-pip git

RUN git clone git://github.com/mathcamp/pypicloud.git
RUN pip install virtualenv
RUN virtualenv /ppe
RUN /bin/bash -c "source /ppe/bin/activate && cd /pypicloud && python setup.py install"
RUN /bin/bash -c "source /ppe/bin/activate && pip install pypicloud waitress"

ADD nonprod.ini /nonprod.ini

EXPOSE 6543
CMD ["/bin/bash", "-c", "cd /pypicloud && source /ppe/bin/activate && pserve /nonprod.ini"]

and the nonprod.ini file was generated by the pypicloud-make-config tool, and then edited to add two extra lines:

aws.prefix = pypicloud-packages/
pypi.prepend_hash = false

If I upload packages to the PyPiCloud endpoint, they are stored in S3, but existing packages in the same location aren't found. In other words, curling the /simple/ endpoint of a fresh copy of the app shows only an empty index page.

Is there a way to force the caching layer to refresh itself from S3?

Thanks!

allow_overwrite doesn't seem to be respected

I'm having trouble getting pypi.allow_overwrite to work as I expect it to; perhaps my expectation isn't right?

I've got a scenario where I have a pipeline execute multiple times, especially in development, creating the same filename multiple times but from different commit shas. For example, nupic-0.2.4.dev0-cp27-none-linux_x86_64.whl.

In my prod.ini file, I have:

<snip>
pypi.default_read = everyone
pypi.default_write = authenticated

pypi.allow_overwrite = True
<snip>

and I'm using the default sqlite3 cache. When a package that has been previously uploaded is built, it tries to upload it, but gets back an HTTP 500 error because of the sqlite constraints on filename. Any thoughts?

Traceback (most recent call last):
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 99, in tm_tween
    reraise(*exc_info)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 87, in tm_tween
    manager.commit()
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/transaction/_manager.py", line 111, in commit
    return self.get().commit()
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/transaction/_transaction.py", line 280, in commit
    reraise(t, v, tb)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/transaction/_transaction.py", line 271, in commit
    self._commitResources()
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/transaction/_transaction.py", line 417, in _commitResources
    reraise(t, v, tb)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/transaction/_transaction.py", line 389, in _commitResources
    rm.tpc_begin(self)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/zope/sqlalchemy/datamanager.py", line 90, in tpc_begin
    self.session.flush()
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
    mapper, table, insert)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 728, in _emit_insert_statements
    execute(statement, multiparams)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/opt/numenta/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: packages.filename [SQL: u'INSERT INTO packages (filename, name, version, last_modified, data) VALUES (?, ?, ?, ?, ?)'] [parameters: (u'nupic-0.2.4.dev0-cp27-none-linux_x86_64.whl', u'nupic', u'0.2.4.dev0', '2015-06-18 11:22:59.368843', '{"url": "https://<S3_BACKEND>/nupic/nupic-0.2.4.dev0-cp27-none-linux_x86_64.whl?Signature=0Xzral9YyFFWCwtODTmWtRT2aEo%3D&Expires=1434712980&AWSAccessKeyId=<AWS_ACCESS_KEY_ID>", "path": "c544/nupic/nupic-0.2.4.dev0-cp27-none-linux_x86_64.whl", "expire": 1434712380.16067}')]
[pid: 7148|app: 0|req: 7477/10404] 10.0.1.42 () {44 vars in 748 bytes} [Thu Jun 18 11:22:58 2015] POST / => generated 0 bytes in 1244 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)

Add ability to rebuild package list without first purging it

I'm trying to upgrade from a simpler PyPI server that was backed by a directory populated from S3. Our workflow involves uploading new packages to an S3 bucket, where they are synced to each PyPI server using S3 ObjectCreate notifications.

My first pass using pypicloud was going to involve just hitting the 'rebuild index' endpoint whenever a new package is uploaded, but it seems that that purges the entire cache first. This would cause downtime which would result in build failures.

One possibility is to change the behavior of /admin/rebuild to be diff-oriented, adding metadata for any new packages and removing metadata for packages that no longer exist, instead of purging the cache and starting over.

Another could be to add an endpoint that would notify pypicloud of a newly-added package file so it can be indexed incrementally.

Regardless, I think having a non-destructive full-refresh option would be an asset—it would be nice to be able to periodically ensure the cache is in sync with the datastore without causing downtime.

Add proxy prefix filter to default paste deploy config

Do you think it would be a good idea to add a proxy prefix filter to the default paste deploy config? It took me quite a while to work out how to get the Host to be https when the relevant "forwarded" headers were set. I'm sure this is a rather common use case, as unless the server is completely internal you'll definitely want to be serving it over SSL..

http://pythonpaste.org/deploy/modules/config.html?highlight=filter#paste.deploy.config.PrefixMiddleware

filter-with = proxy-prefix

[filter:proxy-prefix]
use = egg:PasteDeploy#prefix
prefix = /

Storage backend should support CloudFront

Unable to log in over HTTP and gives no feedback telling you why

I'm aware this most likely is not a bug, but I have a fresh install with one user (config file access control) who is an admin, set up using the ppc-make-config script. When I browse to the web interface and try logging in, upon submission of the username/password form it just returns me back to the main page, but not logged in. However, if I enter a wrong username/password it will tell me it's wrong.

What can I do to try to diagnose this problem?

Thanks

no installation instructions and apparently i could use them

Hi! -

this project seems like exactly what i'm looking for - thanks for sharing it!

however when i did:

$ �wget https://github.com/mathcamp/pypicloud/archive/master.zip
$ unzip master.zip
$ cd pypicloud-master
$ python setup.py install

i got:

Traceback (most recent call last):
File "setup.py", line 96, in
DATA['version'] = get_version()
File "setup.py", line 91, in get_version
with open(VERSION_MODULE, 'r') as version_file:
IOError: [Errno 2] No such file or directory: '/<base_dir>/pypicloud/pypicloud-master/pypicloud/version.py'

so! perhaps this is not the way to install pypicloud?

(i also tried
$ pip install pypicloud
to no avail.)

uploading packages doesn't seem to work

when i use the following to upload a package:
python setup.py sdist upload -r pypicloud

i get receive a server response: 200 OK, but the package doesn't show up on our pypicloud server

pypicloud doesn't seek new versions of public libraries if older versions are cached

If pypi.fallback = cache, it will download a public library the first time said library is requested, but will never bother to check pypi.fallback_url if a non-cached version of the same library is requested later. For example, if I set a pypicloud as my pip's index-url

> pip install 'requests==2.0.0' --index-url="http://localhost:8888/simple"
Downloading/unpacking requests==2.0.0
User for localhost:8888: [email protected]
Password:
  http://localhost:8888/simple/requests/2.0.0 uses an insecure transport scheme (http). Consider using https if localhost:8888 has it available
  http://localhost:8888/simple/requests/ uses an insecure transport scheme (http). Consider using https if localhost:8888 has it available
  Downloading requests-2.0.0-py2.py3-none-any.whl (391kB): 391kB downloaded
Installing collected packages: requests
Successfully installed requests
Cleaning up...

(env)[10:36:08] duckworthd@Daniels-MacBook-Pro:~/Desktop
> pip install 'requests>=2.2.0' --index-url="http://localhost:8888/simple"
Downloading/unpacking requests>=2.2.0
User for localhost:8888: [email protected]
Password:
  http://localhost:8888/simple/requests/ uses an insecure transport scheme (http). Consider using https if localhost:8888 has it available
  Could not find a version that satisfies the requirement requests>=2.2.0 (from versions: 2.0.0)
Cleaning up...
No distributions matching the version for requests>=2.2.0
Storing debug log for failure in /Users/duckworthd/.pip/pip.log

(env)[10:36:17] duckworthd@Daniels-MacBook-Pro:~/Desktop
> pip install 'requests>=2.2.0' --index-url="http://pypi.python.org/simple"
Downloading/unpacking requests>=2.2.0
  http://pypi.python.org/simple/requests/ uses an insecure transport scheme (http). Consider using https if pypi.python.org has it available
  Downloading requests-2.3.0-py2.py3-none-any.whl (452kB): 452kB downloaded
Installing collected packages: requests
  Found existing installation: requests 2.0.0
    Uninstalling requests:
      Successfully uninstalled requests
Successfully installed requests
Cleaning up...

LDAP Support

Wouldn't that be cool.

Python 3 support

Ability to migrate packages between storage backends

Users should be able to migrate their package between two storage backends (e.g. file storage and S3).

Proposed interface:

pypicloud-migrate-packages old_config.ini new_config.ini

Group access control setting for all packages

Hi,

We have pypicloud locally and push all Jenkin's builds to it. We have few users setup:

admin user for managing pypicloud
jenkins user for uploading to pypicloud
installation user that is used when installing packages

Admin user should have read/write access to all packages and should be able to manage groups, this is done by adding him to auth.admins in pypicloud config.
Jenkins user should have read/write access to all packages but should not have administrator access. This could not be achieved, so temporarily we added him to auth.admins.
Installation user should have read only access to all packages, this has been achieved by creating user but not adding him to auth.admins.

I could not find information in docs about granting one user read/write access to every package. It is possible to specify exact packages you wish grant access to but this requires maintaining a list and keeping it up to date. Same applies to granting read only access to all packages in the pypicloud.

Do you have any recommendation on how to approach this?
Have I missed any feature in the docs that would allow creating such configuration?

repository failed to get version higher than current stored version in repo

Hi,

I encountered an issue, when I have for example, application called app with version 0.1 on my pypicloud repository, and I'm asking to download app with version other than 0.1 that does not exists on the pypi repo, instead of fetching it from the world's pypi repo, I get an error saying the package is not found.

Using pypicloud version 0.2.7 with fallback = cache and S3 storage.

Thanks.

Ability to dump/load/migrate access control data

This would make the SQL access control backend more appealing, because it could be backed up to S3 or converted into a config file. Also just makes switching formats easier.

Auto-archive to Glacier

After uploading X versions

Host javascript libraries

Requiring a CDN means that the users have to be connected to the Real Internet.

Update Docker Image

The Docker image you provide has been quite useful. However it seems to be a bit behind compared the releases.

Would you mind updating it?

Or do you recommend people just use it as a base in building their own Dockerfile. If so, I think the docs should be updated to suggest that route.

Support wheels

Right now you can only have one package per name/version pair.

Hanging while Rebuilding cache using Postgresql backend.

Empty Storage backend, tried both s3 and file.
New empty database on an Amazon RDS instance.

Logging with debug on:

*** Starting uWSGI 1.2.3-debian (64bit) on [Mon Jun  9 14:51:45 2014] ***
compiled with version: 4.7.2 on 06 July 2013 12:20:09
detected number of CPU cores: 2
current working directory: /
detected binary path: /usr/bin/uwsgi-core
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uwsgi socket 0 bound to TCP address :3031 fd 3
Python version: 2.7.3 (default, Mar 13 2014, 11:26:58)  [GCC 4.7.2]
Python main interpreter initialized at 0x82fd90
threads support enabled
your server socket listen backlog is limited to 100 connections
*** Operational MODE: preforking ***
Loading paste environment: config:/pypicloud/config.ini
{'session.validate_key': 'VALIDATE_KEY', 'pypi.storage': 'file', 'db.url': 'postgresql://user:pass@RDS_HOST:5432/pypicloud', 'pyramid.debug_routematch': 'false', 'pyramid.debug_authorization': 'false', 'pyramid.default_locale_name': 'en', 'pyramid.reload_templates': 'False', 'storage.dir': '/pypi', 'pypi.auth': 'sql', 'pyramid.debug_notfound': 'false', 'session.encrypt_key': 'ENCRYPT_KEY', 'session.secure': 'True', 'auth.db.url': 'sqlite:////pypicloud/pypicloud.db', 'pypi.fallback': 'cache'}
DEBUG 2014-06-09 14:51:45,258 [passlib.utils.compat] loaded lazy attr 'SafeConfigParser': <class ConfigParser.SafeConfigParser at 0x7f9079ecd7a0>
DEBUG 2014-06-09 14:51:45,259 [passlib.utils.compat] loaded lazy attr 'NativeStringIO': <built-in function StringIO>
DEBUG 2014-06-09 14:51:45,259 [passlib.utils.compat] loaded lazy attr 'BytesIO': <built-in function StringIO>
DEBUG 2014-06-09 14:51:45,572 [txn.140258580006752] new transaction
INFO 2014-06-09 14:51:45,575 [pypicloud.cache.base] Cache is empty. Rebuilding from storage backend...

pip freeze

Warning: cannot find svn location for distribute==0.6.24dev-r0
Beaker==1.6.4
Jinja2==2.7.2
MarkupSafe==0.23
Paste==1.7.5.1
PasteDeploy==1.5.2
PasteScript==1.7.5
SQLAlchemy==0.9.4
WebOb==1.4
argparse==1.2.1
boto==2.29.1
botocore==0.50.0
distlib==0.1.9
## FIXME: could not find svn URL in dependency_links for this package:
distribute==0.6.24dev-r0
dynamo3==0.1.3
flywheel==0.2.0
jmespath==0.4.1
passlib==1.6.2
psycopg2==2.4.5
pycrypto==2.6.1
pypicloud==0.2.4
pyramid==1.5.1
pyramid-beaker==0.8
pyramid-duh==0.1.2
pyramid-jinja2==2.3
pyramid-tm==0.7
python-dateutil==2.2
repoze.lru==0.6
six==1.6.1
transaction==1.4.3
translationstring==1.1
venusian==1.0a8
wsgiref==0.1.2
zope.deprecation==4.1.1
zope.interface==4.1.1
zope.sqlalchemy==0.7.4

Support for OpenStack Swift

I think this can be achieved either through the native Swift Object Storage API or the S3 compatibility API. This would make pypicloud suitable for private clouds.

Ability to back up access control data to S3

Depends on #7.

Store more package metadata

Store and display all the metadata sent up by python setup.py upload

ConditionalCheckFailedException when rebuilding DynamoDB index from datastore

I have a preexisting bucket with a large number of packages (~15000) and I'm trying out pypicloud 0.3.7 using the DynamoDB cache.

When I click 'Rebuild package list' from the admin console, it runs for 30–60 sec before failing with a ConditionalCheckFailedException[1] and stops. Each time it makes a little more progress, though.

Any tips on debugging this further?

[1]

content-type;host;user-agent;x-amz-date;x-amz-security-token;x-amz-target
a668c406474e5a233dc05445439efc3f6887a7381c4142e473f42d86b4f2e304
DEBUG 2016-01-22 10:57:57,385 [botocore.auth] StringToSign:
AWS4-HMAC-SHA256
20160122T185757Z
20160122/us-west-1/dynamodb/aws4_request
f8ecd4f3db43a4aaf8fafa947a684f4042d8b74292c93faf8725212f21f71e9c
DEBUG 2016-01-22 10:57:57,385 [botocore.auth] Signature:
a6900b98116d3f221b8821cbb2b4eeb1efcdd3d01823665d209de4854d437824
DEBUG 2016-01-22 10:57:57,386 [botocore.endpoint] Sending http request: <PreparedRequest [POST]>
DEBUG 2016-01-22 10:57:57,391 [botocore.vendored.requests.packages.urllib3.connectionpool] "POST / HTTP/1.1" 400 120
DEBUG 2016-01-22 10:57:57,392 [botocore.parsers] Response headers: {'x-amzn-requestid': 'O580VB8MLHT0IFGBF9M2OKS5NBVV4KQNSO5AEMVJF66Q9ASUAAJG', 'date': 'Fri, 22 Jan 2016 18:57:56 GMT', 'content-length': '120', 'content-type': 'application/x-amz-json-1.0', 'x-amz-crc32': '396270901'}
DEBUG 2016-01-22 10:57:57,392 [botocore.parsers] Response body:
{"__type":"com.amazonaws.dynamodb.v20120810#ConditionalCheckFailedException","message":"The conditional request failed"}
DEBUG 2016-01-22 10:57:57,392 [botocore.hooks] Event needs-retry.dynamodb.UpdateItem: calling handler <botocore.retryhandler.RetryHandler object at 0x7ffbd4a74f50>
DEBUG 2016-01-22 10:57:57,392 [botocore.retryhandler] No retry needed.
DEBUG 2016-01-22 10:57:57,393 [txn.140719567050496] abort
ERROR 2016-01-22 10:57:57,394 [waitress] Exception when serving /admin/rebuild
Traceback (most recent call last):
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/waitress/channel.py", line 336, in service
    task.service()
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/waitress/task.py", line 169, in service
    self.execute()
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/waitress/task.py", line 388, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 223, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 198, in invoke_subrequest
    response = handle_request(request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/tweens.py", line 20, in excview_tween
    response = handler(request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 101, in tm_tween
    reraise(*exc_info)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 83, in tm_tween
    response = handler(request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 145, in handle_request
    view_name
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/view.py", line 541, in _call_view
    response = view_callable(context, request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 328, in attr_view
    return view(context, request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 304, in predicate_wrapper
    return view(context, request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 244, in _secured_view
    return view(context, request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 353, in rendered_view
    result = view(context, request)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 483, in _class_requestonly_view
    response = getattr(inst, attr)()
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pypicloud/views/admin.py", line 26, in rebuild_package_list
    self.request.db.reload_from_storage()
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pypicloud/cache/base.py", line 73, in reload_from_storage
    self.save(pkg)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/pypicloud/cache/dynamo.py", line 184, in save
    self.engine.sync(summary)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/flywheel/engine.py", line 618, in sync
    updates, returns=ALL_NEW, **keywords)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/dynamo3/connection.py", line 869, in update_item
    **keywords)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/dynamo3/connection.py", line 232, in call
    exc.re_raise()
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/dynamo3/exception.py", line 22, in re_raise
    six.reraise(type(self), self, self.exc_info[2])
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/dynamo3/connection.py", line 222, in call
    data = op(**kwargs)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/share/python/pypicloud/local/lib/python2.7/site-packages/botocore/client.py", line 407, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ConditionalCheckFailedException: ConditionalCheckFailedException: The conditional request failed

no such file develop.ini - only in OS X

Hey man,

first thanks for your prompt response on the issue i filed yesterday - much appreciated (and unexpected in the middle of a holiday week)!

here's a weird one:
i ran your install script - pretty cool, i haven't seen that "unbox" script before! - then i activated the virtual env, edited my develop.ini, and ran pserve -- but it says it can't find develop.ini!!
i've tried with absolute and relative paths, and.. no dice!

this was all in OS X -- so i started up my trusty vagrant box, repeated the procedure, and hey presto! it works in linux!

I suspect this is an issue with pyramid/paste and not your project, but here's the stack trace i was getting (followed by a "head" on the file that pserve says doesn't exist)

(pypicloud_env)16:26:07 pypicloud pserve --reload develop.ini
Starting subprocess with file monitor
Traceback (most recent call last):
File "/Users/joe/pypicloud/pypicloud_env/bin/pserve", line 9, in
load_entry_point('pyramid==1.4', 'console_scripts', 'pserve')()
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 50, in main
return command.run()
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 301, in run
relative_to=base, global_conf=vars)
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 332, in loadserver
server_spec, name=name, relative_to=relative_to, *_kw)
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 255, in loadserver
return loadobj(SERVER, uri, name=name, *_kw)
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
global_conf=global_conf)
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 317, in _loadconfig
loader = ConfigLoader(path)
File "/Users/joe/pypicloud/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 393, in init
with open(filename) as f:
IOError: [Errno 2] No such file or directory: '/Users/joe/pypicloud/develop.ini'

(pypicloud_env)16:41:25 pypicloud head /Users/joe/pypicloud/development.ini

app configuration

http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/environment.html

[app:main]
use = egg:pypicloud

pyramid.reload_templates = true
pyramid.debug_authorization = false

socket.error when using s3 storage

I receive socket.error: [Errno 104] Connection reset by peer when I try to use the s3 storage option. File storage seems to work fine.
I've set up a prod.ini file using pypicloud-make-config specifying my keys and the bucket.
Is there a way for me to debug this further or am I simply missing something?

This is happening using the package from PyPI (0.1), both using uWSGI as specified in the deploy documentation and waitress.

Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/router.py", line 251, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/router.py", line 227, in invoke_subrequest
    response = handle_request(request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 82, in tm_tween
    reraise(*exc_info)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 63, in tm_tween
    response = handler(request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/router.py", line 161, in handle_request
    response = view_callable(context, request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/config/views.py", line 588, in __call__
    return view(context, request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/config/views.py", line 321, in attr_view
    return view(context, request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/config/views.py", line 297, in predicate_wrapper
    return view(context, request)        
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/config/views.py", line 237, in _secured_view
    return view(context, request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid/config/views.py", line 347, in rendered_view
    result = view(context, request)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pyramid_duh/params.py", line 299, in param_twiddler
    return fxn(**scope)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pypicloud/views/simple.py", line 23, in upload
    content.file)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pypicloud/cache.py", line 99, in upload
    path = self.storage.upload(name, version, filename, data)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/pypicloud/storage.py", line 182, in upload
    key.set_contents_from_file(data)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1246, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 725, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 914, in _send_file_internal
    query_args=query_args
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 633, in make_request
    retry_handler=retry_handler
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in make_request
    retry_handler=retry_handler)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/boto/connection.py", line 907, in _mexe
    request.body, request.headers)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 815, in sender
    http_conn.send(chunk)
  File "/usr/lib/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/ssl.py", line 293, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python2.7/ssl.py", line 262, in send
    v = self._sslobj.write(data)
socket.error: [Errno 104] Connection reset by peer

Non-standard names are not shown on index page

During testing, I noticed that if a project uses non-pep440 compliant version names, they don't appear on the index page. pip does the right thing still as far as determining the correct newest package, but it'd be nice if the index had the same info.

Crop of main index page:

project file listing:

Errors using DynamoDB caching backend

I am trying to setup a pypicloud server using AWS S3 for package storage and DynamoDB as caching backend.

The goal is to have pypi servers on different subnets, sharing a common package repository, so that packages pushed to one are visible on the others. Servers do not have connectivity to each other, but can all reach AWS.

When I startup the server, I get exceptions thrown.

Here are steps to reproduce:

Setup virtualenv based on Python 2.7.8: virtualenv pypicloud_env
Activate virtualenv: . pypicloud_env/bin/activate
Install pypicloud from pypi: pip install pypicloud waitress
Create server.ini: `ppc-make-config -t server.ini'

[1] s3
[2] filesystem
Where do you want to store your packages? 1
AWS access key id? <redacted>
AWS secret access key? <redacted>
S3 bucket name? <redacted>
Admin username? admin
Password: 
Password: 
Config file written to 'server.ini'

Edit server.ini. Comment db.url, add following entries under [app:main]

pypi.db = dynamo
db.region = us-east-1
db.access_key = <redacted>
db.secret_key = <redacted>
db.namespace = pypi

Install flywheel: pip install flywheel
Start server: pserve server.ini

At this point, it throws exceptions after running for a bit:

INFO 2014-10-30 13:32:15,758 [botocore.vendored.requests.packages.urllib3.connectionpool] Starting new HTTPS connection (1): dynamodb.us-east-1.amazonaws.com
INFO 2014-10-30 13:32:20,208 [pypicloud.cache.base] Cache is empty. Rebuilding from storage backend...
Traceback (most recent call last):
  File "/Users/zvanwijk/scratch/pypicloud_env/bin/pserve", line 9, in <module>
    load_entry_point('pyramid==1.5.1', 'console_scripts', 'pserve')()
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 51, in main
    return command.run()
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 316, in run
    global_conf=vars)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 340, in loadapp
    return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pypicloud/__init__.py", line 95, in main
    config.include('pypicloud')
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pyramid/config/__init__.py", line 755, in include
    c(configurator)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pypicloud/__init__.py", line 46, in includeme
    config.include('pypicloud.cache')
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pyramid/config/__init__.py", line 755, in include
    c(configurator)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pypicloud/cache/__init__.py", line 24, in includeme
    cache.reload_if_needed()
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pypicloud/cache/base.py", line 37, in reload_if_needed
    self.reload_from_storage()
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pypicloud/cache/base.py", line 73, in reload_from_storage
    self.clear_all()
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/pypicloud/cache/dynamo.py", line 164, in clear_all
    self.engine.delete_schema()
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/flywheel/engine.py", line 228, in delete_schema
    namespace=self.namespace)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/flywheel/model_meta.py", line 498, in delete_dynamo_schema
    desc = connection.describe_table(tablename)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/dynamo3/connection.py", line 255, in describe_table
    return Table.from_response(response)
  File "/Users/zvanwijk/scratch/pypicloud_env/lib/python2.7/site-packages/dynamo3/fields.py", line 283, in from_response
    in response['AttributeDefinitions']))
KeyError: 'AttributeDefinitions'

"python-logstash-0.4.5" package shows up as "python" with "logstash-0.4.5" as the version

I cannot upload this package from the public PyPI to my pypicloud instance:
https://pypi.python.org/simple/python-logstash/

Problem migrating packages from files to S3

pypicloud==0.2.7

Migrating Package(foo-0.1.26.tar.gz)
Traceback (most recent call last):
  File "venv/bin/ppc-migrate", line 9, in <module>
    load_entry_point('pypicloud==0.2.7', 'console_scripts', 'ppc-migrate')()
  File "/apps/cheeseshop/venv/local/lib/python2.7/site-packages/pypicloud/scripts.py", line 204, in migrate_packages
    del package['path']
TypeError: 'Package' object does not support item deletion

URL changes result in 401 for easy_install

It looks like there was a recent change which caused the URLs for https://<pypicloud>/pypi/<package>/ from pointing directly at the destination (in this case S3) to a subpath within https://<pypicloud>.

Unfortunately, this breaks installation via easy_install as the authentication specified in the index-url will be lost when following the redirect. This ends up presenting as a 401 during a setup.py install when fetching dependencies.

Mirror mode: only the first URL for each package is presented

When mirror mode is enabled, the get_fallback_packages function only presents the first URL for each of the packages available upstream (via this break statement).

This can cause problems, because sometimes the first URL is a "wheel" package. Then, if the client uses --no-use-wheel, the package cannot be installed.

I don't have all the context required to make the call, but could that break line simply be removed?

S3 link regeneration

Hey,

I am experimenting with S3 link regeneration options:

storage.expire_after = 3600
storage.buffer_time = 60

Does not seem to work as I would expect.
I have configured ElasticCache Redis for cache backend, and when I restart a service, ppc repopulates the redis with keys. I expect it to repopulate and regenerate the S3 urls every minute. So i done flushall on redis, and in 1 minute it is still empty.
Also URLs does not expire after 10 minutes, as my settings suggest.

What am I miss-understanding ?

Access control backend for Dynamo

Support Wheels

When I try to submit a wheel to pypicloud I get a 400 error.

More on Wheels: http://pythonwheels.com/

pip install fails when caching with s3 on the first time

Hi,

There is an error of 404 not found, when trying to pip install package that is not yet cached on the s3 repository, and after the 404 error, the package is being downloaded to the repository and then being available to download via pip install.

Problem is that package foo with dependency boo will fail twice, one for foo and second time with boo.

Some one can look into it?

Thanks!

Caching database using Dynamo

Anonymous Auth

When configured to use ldap auth, anonymous uses browsing the pypi site cannot see any packages. Upon logging in with ldap auth, you can see the packages.

If you disabled ldap auth in the config file, you can see the packages as an anonymous user.

I have this set in the config:

    pypi.default_read =
        everyone
    pypi.default_write =
        authenticated

Am I missing a configuration?

S3 download links stop working early with IAM roles

I have pypicloud set up using the default local SQLite database and with S3 package storage. Rather than putting the AWS credentials in plain text in the pypicloud.ini file, the system running the server has an IAM role that allows it to access the S3 bucket with my packages.

I'm frequently seeing a problem where 'pip install' reports a 400 HTTP error. If I manually try to fetch the redirect error, I get back an Amazon error saying "The provided token has expired." This is with the default settings (expire_after=86400 seconds, buffer_time=600 seconds), and both the URL and the expiry data stored in SQLite suggest the URL should be valid for another 17 hours or so.

I have a vague suspicion that the IAM role is introducing an unfortunate level of indirection, where the credentials boto comes up with actually expire after a relatively short period of time, but I haven't been able to prove this yet.

pypicloud==0.2.7 (with boto==2.34.0, if that's relevant) on Python 2.7.6 in an Ubuntu 14.04 Docker container on an Ubuntu 14.04 host.

Support for S3 regions deployed after January 2014, like eu-central-1

Wanted to use a s3-bucket in "eu-central-1"/Frankfurt, but it fails with

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

Unfortunaltely, S3 regions deployed after January 2014, like eu-central-1, will only support "AWS Signature Version 4", according to http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

[Update] It seems boto would support this out-of-the-box, but a hostname (and appropriate region?) should be given at connection init.

S3 authentication without explicitly passing AWS Access Key & Secret

Hello,

I'm working with @joekimmel. We're pretty excited about this project. One feature I'd really like is for pypicloud to be able access S3 without us needing to explicitly supply AWS credentials. The simplifies configuration and helps keep sensitive info out of our source control.

By way of background (apologies if you already know this), Boto supports a few different implicit authentication methods. If one calls boto.connect_s3() or creates a new S3Connection but leaves the key & secret fields as None then Boto will search for, among other things

environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
values in ~/.boto
rotating credentials, available via the metadata endpoint when running on an EC2 instance assigned to an IAM role, e.g. http://169.254.169.254/latest/meta-data/iam/security-credentials/<IAM_role_name>/

It appears pypicloud is calling connect_s3 with explicit arguments here. One way to let Boto do its implicit-auth thing, with minimal code changes, might be to have the string-parsing for the settings dictionary convert empty strings into None for the key & secret fields. That way we can just leave the values blank in our production.ini file and Boto will take care of the rest.

I'm happy to dig into this more and submit some code changes via a pull-req. But I figured I'd start by opening an issue.

Thanks!

Hooks for using celery

Allow user to run celery in conjunction with the server so the urls can be regenerated by a task. This would turn the url cache into a true write-through cache and eliminate the periodic latency that comes with regenerating the urls.

Uploading arbitrary files

My use case: along with eggs that contain our cross-project code, we need to ship a requirements.txt that lists exact dependencies for these eggs. Currently we handle this use case by uploading requirements.txt directly into S3 and giving it a public-read permission. This approach is sub-optimal due to needing an additional manual step after a release that is different than the main step of upload and egg, and because the versions file is then world-readable.

So I'd like to add an ability to upload arbitrary files into folders containing eggs on pypicloud. I just want to see if there are any comments or blockers before I start implementation ...

no such table (packages)

Hello again!

it looks like there's some DB initialization that's not happening? the first thing i tried to do with my running pypicloud server was to upload a local fork of gunicorn... setup/python gets a 500 back, and the trace looks like:

DEBUG:txn.140352580830976:abort
DEBUG:txn.140352580830976:new transaction
ERROR:waitress:Exception when serving /
Traceback (most recent call last):
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/waitress/channel.py", line 337, in service
task.service()
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/waitress/task.py", line 173, in service
self.execute()
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/waitress/task.py", line 392, in execute
app_iter = self.channel.server.application(env, start_response)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/router.py", line 251, in call
response = self.invoke_subrequest(request, use_tweens=True)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/router.py", line 227, in invoke_subrequest
response = handle_request(request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid_tm/init.py", line 82, in tm_tween
reraise(*exc_info)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid_tm/init.py", line 63, in tm_tween
response = handler(request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/router.py", line 161, in handle_request
response = view_callable(context, request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/config/views.py", line 586, in call
return view(context, request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/config/views.py", line 319, in attr_view
return view(context, request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/config/views.py", line 295, in predicate_wrapper
return view(context, request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/config/views.py", line 237, in _secured_view
return view(context, request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/config/views.py", line 375, in viewresult_to_response
result = view(context, request)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/pyramid/config/views.py", line 491, in _requestonly_view
response = view(request)
File "/home/vagrant/pypicloud/pypicloud/views.py", line 50, in update
request.db.merge(pkg)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1519, in merge
load=load, _recursive=_recursive)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1558, in merge
merged = self.query(mapper.class).get(key[1])
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 820, in get
return loading.load_on_ident(self, key)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 226, in load_on_ident
return q.one()
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2310, in one
ret = list(self)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2353, in iter
return self._execute_and_instances(context)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in execute
params)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
context)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
exc_info
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
context)
File "/home/vagrant/pypicloud/pypicloud_env/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 324, in do_execute
cursor.execute(statement, parameters)
OperationalError: (OperationalError) no such table: packages u'SELECT packages.url AS packages_url, packages.expire AS packages_expire, packages.name AS packages_name, packages.version AS packages_version, packages.path AS packages_path \nFROM packages \nWHERE packages.name = ? AND packages.version = ?' (u'gunicorn', u'18.2')

Cache parts of pypi

It should be easy to clone and cache parts of pypi. It should also be easy to update the cached versions when necessary.

pypi.fallback = cache is raising an exception

Just installed localy
changed the config to pypi.fallback = cache

pypi.default_read = everyone
pypi.default_write = authenticated
pypi.cache_update = everyone
pypi.fallback_url = http://pypi.python.org/simple
pypi.fallback = cache
pypi.storage = file
storage.dir = %(here)s/packages
db.url = sqlite:///%(here)s/db.sqlite

Install Django:

pip install -i http://localhost:6543/pypi/ Django
Collecting Django
  DEPRECATION: Failed to find 'Django' at http://localhost:6543/pypi/django/. It is suggested to upgrade your index to support normalized names as the name in /simple/{name}.
  Could not find any downloads that satisfy the requirement Django
  No distributions at all found for Django

Error log on the server:

ERROR 2015-01-22 14:07:50,619 [waitress] Exception when serving /pypi/django/
Traceback (most recent call last):
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/waitress/channel.py", line 337, in service
    task.service()
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/waitress/task.py", line 173, in service
    self.execute()
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 95, in tm_tween
    reraise(*exc_info)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 76, in tm_tween
    response = handler(request)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 329, in attr_view
    return view(context, request)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 305, in predicate_wrapper
    return view(context, request)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 355, in rendered_view
    result = view(context, request)
  File "/home/benoit/.virtualenvs/pypicloud/local/lib/python2.7/site-packages/pyramid_duh/view.py", line 181, in slash_redirect
    return fxn(*args)
  File "/home/benoit/git/pypicloud/pypicloud/views/simple.py", line 74, in package_versions
    filename = posixpath.basename(dist.source_url)
AttributeError: 'dict' object has no attribute 'source_url'

switching back to pypi.fallback = redirect is working but

pip install -i http://localhost:6543/pypi/ Django
Collecting Django
  Using cached Django-1.7.3-py2.py3-none-any.whl
Installing collected packages: Django

Successfully installed Django-1.7.3

Another question, is there anyway to have 2 fallback urls? for example pypi and gemfury ?

Logging on production - the recommended way

Hey,

So I deployed pypicloud to production with uWSGI. I know there is some options in uwsgi config to get all uwsgi output to some log files, but in generated prod config I see a number of log handlers, but I assume they are for pyramid not for uwsgi.

What is the recommended way of logging INFO, DEBUG and etc ... on production ?

bug in mirror mode on private packages

Hi,

I found a bug in the new mirror mode, and i have a fix that works out for me.

Scenario:
I have private package i created, which is not in the fallback url. I uploaded it to my pypicloud, but when pip install it (or even go to http://pypicloud/pypi/my-package)
I get the following error:

Traceback (most recent call last):
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 242, in call
response = self.invoke_subrequest(request, use_tweens=True)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 217, in invoke_subrequest
response = handle_request(request)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid_tm/init.py", line 94, in tm_tween
reraise(_exc_info)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid_tm/init.py", line 75, in tm_tween
response = handler(request)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid/router.py", line 163, in handle_request
response = view_callable(context, request)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 329, in attr_view
return view(context, request)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 305, in predicate_wrapper
return view(context, request)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid/config/views.py", line 355, in rendered_view
result = view(context, request)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pyramid_duh/view.py", line 181, in slash_redirect
return fxn(_args)
File "/opt/pypicloud/local/lib/python2.7/site-packages/pypicloud/views/simple.py", line 71, in package_versions
pkgs[package.filename] = package.get_url(request)
TypeError: 'NoneType' object does not support item assignment

Why is that?
because in file: views/simple.py line 67 i have:
pkgs = get_fallback_packages(request, context.name)

but the context.name is not in the fallback url because it's a private package, so the pkgs returns with None and then line 71 is illegal:
pkgs[package.filename] = package.get_url(request)
because it's assignment to NoneType

The workaround i found is replace this line (67):
if can_update_cache:
pkgs = get_fallback_packages(request, context.name)

With this line:
if can_update_cache:
pkgs = get_fallback_packages(request, context.name) or {}

and then, if we get no result from the fallback url, instead of returning None I'll get the {} like I expected to, and then it works!

Added pull request: #49

Bucket names cannot contain '.'

http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
shows that the bucket name could/should actually contain "."

S3 bucket name? test.backet.com
Bucket names cannot contain '.'

when rebuild sql cache, 0.2.5 crashes

we have an existing s3 bucket storage and we want to serve pypicloud from a different VPC, which the sql cache is completely empty. When serving version 0.2.4, pypicloud is smart enough to rebuild the cache, but when serving version 0.2.5, it will crash if there is item in the storage that is not exist in the cache. here's the tracestack:

~$ pserve config.ini
INFO 2014-07-25 11:27:52,049 [pypicloud.cache.base] Cache is empty. Rebuilding from storage backend...
INFO 2014-07-25 11:27:52,680 [pypicloud.cache.base] Cache repopulated
Traceback (most recent call last):
  File "/Users/leonmax/.virtualenvs/pypicloud/bin/pserve", line 9, in <module>
    load_entry_point('pyramid==1.5.1', 'console_scripts', 'pserve')()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 51, in main
    return command.run()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 316, in run
    global_conf=vars)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 340, in loadapp
    return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pypicloud/__init__.py", line 95, in main
    config.include('pypicloud')
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pyramid/config/__init__.py", line 755, in include
    c(configurator)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pypicloud/__init__.py", line 46, in includeme
    config.include('pypicloud.cache')
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pyramid/config/__init__.py", line 755, in include
    c(configurator)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pypicloud/cache/__init__.py", line 24, in includeme
    cache.reload_if_needed()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/pypicloud/cache/sql.py", line 139, in reload_if_needed
    transaction.commit()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/transaction/_manager.py", line 111, in commit
    return self.get().commit()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/transaction/_transaction.py", line 280, in commit
    reraise(t, v, tb)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/transaction/_transaction.py", line 271, in commit
    self._commitResources()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/transaction/_transaction.py", line 417, in _commitResources
    reraise(t, v, tb)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/transaction/_transaction.py", line 394, in _commitResources
    rm.tpc_vote(self)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/zope/sqlalchemy/datamanager.py", line 103, in tpc_vote
    self.tx.commit()
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 375, in commit
    self._assert_active(prepared_ok=True)
  File "/Users/leonmax/.virtualenvs/pypicloud/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 223, in _assert_active
    raise sa_exc.ResourceClosedError(closed_msg)
sqlalchemy.exc.ResourceClosedError: This transaction is closed

Ideally the server is smart enough to look up the storage if something is not in the cache, that's how you define a cache, isn't it ;)

We really like pypicloud, if one can serve multiple pypicloud server that sharing the same s3 bucket, that's got to be even cooler!

stevearc / pypicloud Goto Github PK

pypicloud's Introduction

PyPI Cloud

Quick Start

Docker

pypicloud's People

Contributors

Stargazers

Watchers

Forkers

pypicloud's Issues

app configuration

Recommend Projects

Recommend Topics

Recommend Org