Comments (6)
Thanks for pointing this out. Looks like the instance is running out of memory (running on a t2.micro). I'm guessing the linehaul issue was quite significant, and now with less data loss the google bigquery daily tables are 10-12 GB now. I'll have to upgrade the instance; I'll get it backfilled asap.
from pypistats.org.
In hindsight I should probably be running the ingestion job in a Lambda rather than alongside the server...
I have another 8 months within the AWS free tier. The actual costs would be about $50/month. The aggregate data takes up only about 8 GB so far in a postgres db (20 GB cap), so altogether it's pretty lightweight.
If it makes sense to migrate to PSF at some point I would be happy to do so. I haven't touched this project in a while other than to regularly extend the retention time, but if there are any other metrics that might be interesting for you or other developers to see I can add them to my TODOs.
from pypistats.org.
The linehaul issue was extremely significant, yeah. I think it'll be a week or two until we know for sure what the new tables look like, since the new linehaul was deployed just before the weekend, which is the low-traffic part of the week... the table for 7-27 is actually 18 GB.
If free-tier resource limits are being an issue, we could ask @ewdurbin and @dstufft about whether it would make sense for pypistats.org to move onto PSF infrastructure. I don't think the PyPI maintainers have spare energy to build or maintain a site like this, but it definitely fills an important need, and it sounds like the needed infrastructure is pretty modest.
from pypistats.org.
Let me just say, the site seems like a super useful tool and I'm glad it exists. If you ever do decide you want it to live elsewhere, or that you might even be interested in integrating the functionality into Warehouse, let me know and we'll see what we can do.
from pypistats.org.
Glad to hear that you find it useful. I think that eventually integrating into PSF/Warehouse is probably a good idea for the long term. I could work on it myself to absolve the maintainers of the workload. Who would be the primary point of contact with respect to implementation details?
In the meantime I'll try to get familiar with the Warehouse codebase.
from pypistats.org.
There are some Issues for these:
pypi/warehouse#699
pypi/warehouse#787
from pypistats.org.
Related Issues (20)
- Wheel statistics
- [feature requeset] downloads per user
- Website is not functioning HOT 2
- CORS Headers Issue HOT 1
- Intermittent 429 RATE LIMIT EXCEEDED HOT 4
- API: 404 is returned for some endpoints but not others
- Download stats dropped close to 0 since 2011-11-24? HOT 5
- broken links to Google BigQuery
- [feature request] new endpoint: `/api/top`
- Sort python versions in natural ordering
- [feature request] Toggle last 30/60/90 days for all the graphs
- List dependent packages
- Feature request: get most recent data including mirrors
- optional dependencies are incorrect HOT 2
- Add `<package name>` on page's title
- No Download Statistics pre-2016 on Google BigQuery HOT 1
- Ignore Metadata Files From Download Stats
- [Bug] Downloads stats for all packages got significant drop. HOT 3
- PyPI Infra Incident Impacting BigQuery dataset HOT 4
- Provide wheel platform tag stats HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pypistats.org.