Code Monkey home page Code Monkey logo

archweb's Introduction

Archweb README

Build Status

To get a pretty version of this document, run

$ markdown README > README.html

License

See LICENSE file.

Authors

See AUTHORS file.

Dependencies

  • python
  • rsync (optional for mirrorcheck with rsync mirrors)

Python dependencies

More detail in requirements.txt and requirements_prod.txt; it is best to use virtualenv and pip to handle these. But if you insist on (Arch Linux) packages, you will probably want the following:

  • python-django
  • python-psycopg2
  • python-markdown
  • python-memcached

Testing Installation

  1. Run python -m venv env.

     cd /path/to/archweb && python -m venv ./env/
    
  2. Activate the virtualenv.

     source ./env/bin/activate
    
  3. Install dependencies through pip.

     pip install -r requirements.txt
    
  4. Copy local_settings.py.example to local_settings.py and modify. Make sure to uncomment the appropriate database section (either sqlite or PostgreSQL).

  5. Migrate changes.

     ./manage.py migrate
    
  6. Load the fixtures to pre populate some data. If you don't want some of the provided data, adjust the file glob accordingly.

     ./manage.py loaddata main/fixtures/*.json
     ./manage.py loaddata devel/fixtures/*.json
     ./manage.py loaddata mirrors/fixtures/*.json
     ./manage.py loaddata releng/fixtures/*.json
    
  7. Use the following commands to start a service instance

     ./manage.py runserver
    
  8. To optionally populate the database with real data:

     wget http://mirrors.kernel.org/archlinux/core/os/x86_64/core.db.tar.gz
     ./manage.py reporead x86_64 core.db.tar.gz
     # Package file listing
     wget http://mirrors.kernel.org/archlinux/core/os/x86_64/core.files.tar.gz
     ./manage.py reporead --filesonly x86_64 core.files.tar.gz
    

Alter architecture and repo to get x86_64 and packages from other repos if needed.

  1. Database Updates for Added/Removed packages

     sqlite3 archweb.db < packages/sql/update.sqlite3.sql
    

For PostgreSQL use packages/sql/update.postgresql_psycopg2.sql

Testing SMTP server

To be able to create an account on your test environment an SMTP server is required. A simple debugging SMTP server can be setup using Python and aiosmtpd.

Install aiosmtpd

    pip install aiosmtpd

Run the server

    python -m aiosmtpd -n -l localhost:1025

In local_settings.py add entries to set EMAIL_HOST to 'localhost' and EMAIL_PORT to 1025.

Running tests and coverage

Install the test dependencies:

    pip install -r requirements_test.txt

To the unittests execute the following commands:

    make collectstatic
    make test

Running coverage:

    make coverage
    make open-coverage

Django Debug toolbar

To use the Django Debug toolbar install django-debug-toolbar and in local_settings.py set DEBUG_TOOLBAR to True.

Management commands

Archweb provides multiple management commands for importing various sorts of data. An overview of commands:

  • generate_keyring - Assemble a GPG keyring with all known developer keys.
  • pgp_import - Import keys and signatures from a given GPG keyring.
  • read_rebuilderd_status - Import rebuilderd status into Archweb.
  • rematch_developers - Rematch flag requests and packages where user_id/packager_id is NULL to a Developer.
  • reporead - Parses a repo.db.tar.gz, repo.files.tar.gz file and updates the Arch database with the relevant changes.
  • reporead_inotify - Watches a templated patch for updates of *.files.tar.gz to update Arch databases with.
  • donor_import - Import a single donator from a mail passed to stdin
  • mirrorcheck - Poll every active mirror URLs to store the lastsnyc time and record network timing details.
  • mirrorresolv - Poll every active mirror URLs and determine wheteher they have IP4 and/or IPv6 addresses.
  • populate_signoffs - retrieves the latest commit message of a signoff-eligible package.
  • update_planet - Import all feeds for users who have a valid website and website_rss in their user profile.
  • read_links - Reads a repo.links.db.tar.gz file and updates the Soname model.
  • read_links_inotify - Watches a templated patch for updates of *.links.tar.gz to update Arch databases with.

Updating iPXE images

The binaries required for iPXE based netboot are updated by copying them from the ipxe package to the static content directory (with the run_ipxe script the binaries may be tested beforehand):

cp -v /usr/share/ipxe/x86_64/ipxe-arch.efi /usr/share/ipxe/ipxe-arch.{ipxe,lkrn} sitestatic/releng

Afterwards a detached PGP signature using a valid WKD enabled packager key is created for each file:

for artifact in sitestatic/netboot/*.{efi,pxe,lkrn}; do
  gpg --sender "User Name <[email protected]>" --detach-sign "$artifact"
done

Production Installation

Arch Linux has an Ansible role for Archweb in their infrastructure repo.

vim: set syntax=markdown et:

archweb's People

Contributors

abhisheksrikanth avatar akstrfn avatar angvp avatar barthalion avatar bluewind avatar brain0 avatar dependabot[bot] avatar dieterbe avatar dragoonaethis avatar dvzrv avatar eworm-de avatar f09f909b avatar felixonmars avatar foutrelis avatar grazzolini avatar gregoiredx avatar jelly avatar just1602 avatar kpcyrd avatar kyrias avatar lahwaacz avatar nl6720 avatar pierres avatar ryuslash avatar santiagotorres avatar simoleone avatar toofishes avatar torxed avatar twelveeighty avatar void avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

archweb's Issues

Inline script blocked by CSP

See packages/package_details.html:33

Maybe just remove the inline script and add target="_blank" to the <a>?

EDIT:
Similar thing happens in the django admin. When changing user {permissions, groups}, SelectBox.js tries to apply an inline style. This causes the change to the user to fail.

Fix HTML validation errors

Resolve the "Error: Bad value screen, projection for attribute media on element link: Deprecated media type projection. For guidance, see the Media Types section in the current Media Queries specification." error for all our css includes.

Allow automated sending of mirror issue notification

Talking about this on IRC lead to the conclusion that it would be easier to just integrate the mail sending directly into archweb. Thus the changes above are not needed, but there needs to be some service that monitors the mirror check data and sends notifications like this:

  • notify about "out of sync" if the lastsync time is older than 72hours
  • notify about "unreachable" if the lastsync file could not be fetched for 24hours
  • notify if the lastupdate file is 72 hours older than the lastupdate file on our master mirror

A potential improvement is to be able to set the timeouts for each mirror on their own (some mirrors sync every minute so 1 hour delay might be more appropriate than 72).

If a tier 1 mirror is unreachable or out of sync, tier 2 mirrors that use this mirror should NOT receive emails about being out of sync. There is a field in the database that records the source mirror for tier 2 mirrors, but this field is often not correct due to mirrors changing their upstream without notifying us.

Templates for the email texts can be found here: https://git.server-speed.net/users/flo/arch-mirror-tools/tree/bin/generate-mirror-mail.pl

Original ticket https://bugs.archlinux.org/task/45556

Figure out if modify_attributes in JavaScript can be removed

The modify_attributes function sets attributes for html elements since jQuery doesn't allow us to change the "type" in IE. Since IE is no longer supported and the relevant commit acf252f has been made in 2012 check if this can be rewritten.

Ideally the login page would not need jQuery anymore at all since all it does is set focus on $id_username. The same goes for the developer profile view.

Python 3 issues: mirrorcheck

Traceback (most recent call last):
File "/srv/http/archweb/mirrors/management/commands/mirrorcheck.py", line 125, in check_mirror_url
result = urllib.request.urlopen(req, timeout=timeout)
File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib64/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib64/python3.7/urllib/request.py", line 1345, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/lib64/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/srv/http/archweb/mirrors/management/commands/mirrorcheck.py", line 238, in mirror_url_worker
log = check_mirror_url(url, location, timeout)
File "/srv/http/archweb/mirrors/management/commands/mirrorcheck.py", line 143, in check_mirror_url
if isinstance(e.reason, types.StringTypes) and
AttributeError: module 'types' has no attribute 'StringTypes'

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/lib64/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/usr/lib64/python3.7/http/client.py", line 956, in send
self.connect()
File "/usr/lib64/python3.7/http/client.py", line 928, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib64/python3.7/socket.py", line 727, in create_connection
raise err
File "/usr/lib64/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

Add missing makedepends notification

Display missing (make) depends on either /devel/ or mail when they occur:

Some work has been done in this following commit. This takes about ~ 11 minutes to run on our staging server.

Document showing deprecation warnings for our code

The following command shows all deprecation warnings for our code but also for virtualenv's it would be useful to figure out how to show it just for our code and run that in our CI

PYTHONWARNINGS=always ./manage.py test

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Unbreak django-toolbar support

Document how to get django-toolbar working and include in urls.py

diff --git a/urls.py b/urls.py
index ee899a4d..0a40b4b8 100644
--- a/urls.py
+++ b/urls.py
@@ -1,7 +1,9 @@
 from django.conf.urls import include, url
+from django.urls import include, path
 from django.contrib import admin
 from django.contrib.sitemaps import views as sitemap_views
 from django.contrib.auth import views as auth_views
+from django.conf import settings
 
 from django.views.decorators.cache import cache_page
 from django.views.generic import TemplateView
@@ -104,6 +106,11 @@
     url(r'^logout/$', auth_views.LogoutView.as_view(template_name='registration/logout.html'), name='logout'),
 ])
 
+if settings.DEBUG:
+    import debug_toolbar
+    urlpatterns = [
+        path('__debug__/', include(debug_toolbar.urls)),
+    ] + urlpatterns
 
 # displays all archweb urls
 def show_urls(urllist=urlpatterns, depth=0):  # pragma: no cover

Enhance CSP headers

An initial CSP header has been added to Archweb but it is currently lacking a few options which makes it safer.

  • Clickjacking protection, using frame-ancestors
  • Deny by default, using default-src 'none'
  • Restricts use of the tag by using base-uri 'none' or 'self'
  • Restricts where contents may be submitted using form-action 'self'.

Add rss/atom feed of recently orphaned packages

It would be really useful to have an RSS feed available of recently orphaned packages so people can subscribe and adopt packages they know/use. It would make orphans more visible and hopefully decrease the number of orphaned packages.

Display possible maintainers for orphan packages

This query returns a list of orphan packages and finds possible maintainers for those by looking at all packages that directly depend on the orphan.Please make this a report page in the developer area of archweb.

SELECT distinct sub.depname as "(orphan) child", sub.pkgbase as "parent", coalesce(user.username, "orphan") as "parent maintainer", p.packager_str as "last child packager" FROM ( SELECT distinct p.pkgbase, dep.depname FROM package_depends as dep JOIN packages as p on p.id = dep.pkg_id WHERE dep.depname in ( SELECT distinct p.pkgbase FROM packages as p LEFT JOIN packages_packagerelation as rel on rel.pkgbase = p.pkgbase WHERE rel.id is null) ) as sub LEFT JOIN packages_packagerelation as rel on rel.pkgbase = sub.pkgbase LEFT JOIN auth_user as user on user.id = rel.user_id LEFT JOIN packages as p on sub.depname = p.pkgname ORDER BY sub.depname, sub.pkgbase;

The query is somewhat slow (~1sec execution time) and probably needs to be optimized.

A more optimised version:
SELECT DISTINCT pp.pkgbase, ppr.user_id, child.pkgname FROM package_depends ppd JOIN packages pp ON ppd.pkg_id = pp.id JOIN packages_packagerelation ppr ON pp.pkgbase = ppr.pkgbase JOIN (SELECT DISTINCT cp.pkgname FROM packages cp LEFT JOIN packages_packagerelation pr ON cp.pkgbase = pr.pkgbase WHERE pr.id IS NULL) child ON ppd.depname = child.pkgname ORDER BY child.pkgname;

This feature should only be shown for logged in users because of the possible performance penalty.

Upgrade jQuery 1.8.3 to 3.3.x

jQuery 1.8.3 is long dead, it's unhealthy to run it in production. It's not possible to drop jQuery yet since the tablesorter plugin and typeahead still relies on it.

  • Find a replacement for the typeahead on the main page ("Package Search")
  • Update tablesorter to a version which jQuery 3.3.x supports.
  • Check is visualizations still work (d3.js)

The PR https://github.com/archlinux/archweb/pull/49/commits has some commits which updates jQuery and the tablesorter.

The django.db.backends.postgresql_psycopg2 module is deprecated

In the Django 2.0 release notes the following is mentioned:

The django.db.backends.postgresql_psycopg2 module is deprecated in favor of django.db.backends.postgresql. It’s been an alias since Django 1.9. 
This only affects code that imports from the module directly. 
The DATABASES setting can still use 'django.db.backends.postgresql_psycopg2', though you can simplify that by using the 'django.db.backends.postgresql' name added in Django 1.9.

Add webp format support

Consider adding, not replacing webp images since they seem to be at least twice as small when converted to webp.

For example using

<picture>
	<source srcset="opera.webp" type="image/webp">
	<img src="opera.jpg" alt="The Oslo Opera House">
</picture>

Note: not really high prio, since Firefox just supports it. https://caniuse.com/#search=webp

Clean up signoffs when package is moved

When a package is no longer in [testing] clean up the related signoffs for this package or periodically prune old signoffs.

Both the packages_signoffspecification and packages_signoff tables

Update django to 2.1

Update Django to 2.1 on the python3 branch since Django 1.11 does not support 3.7.

A few things have changed in 2.1, notably the on_delete parameter needs to be added for foreinkeys

Note 2.1 is supported till end 2019. Then we need to switch to 2.2 which is LTS.

Create a real REST API for accessing package data

Remove old flagrequest when manually unflagging

When manually unflagging the old flagrequest is not removed while it should be.

Number of flag requests can be obtained by:

>>> from packages.models import FlagRequest
>>> FlagRequest.objects.count()

Steps:

  • Check the number of flagged packages
  • Flag a package
  • Unflag the same package
  • Number of flagged packages should be the same.

Implement 2 Factor authentication

Add 2FA to archweb, making it possible for a user to sign in with a second factor. Most simple implementation should support freeotp/google authenticator.

  • Enforce 2FA for admin logins. Admins should always login using 2FA
  • Add optional 2FA for normal TU/Dev's if they have 2FA configured then it will be enforced on logon.

Document archweb API

Currently documented on the Arch Wiki, this might be incomplete or not up to date.

Display new packages somewhere in the website

It would be nice to see which packages get promoted from AUR to community or core/extra, or even the new packages in the distro. I think it would be something interesting to have because promoted packages are probably new good software that would be interesting to hear about.This could simply be implemented as a "new" icon next to new packages in the "recently updated packages" box, similarly to what the AUR home page has.

Relevant forum discussion: http://bbs.archlinux.org/viewtopic.php?id=66071

Automate PGP keyring issues

When a Trusted User or Developer steps down, his key may be left in the archlinux-keyring package. There is no automation to revoke this key or update the keyring package, so there should be some automation in place to handle this :)

Some scenario's which can be automated:

  • When a key is in the keyring but there is no archweb entry, report an issue.
  • When a fingerprint is registered in archweb but not in the keyring, report an issue
  • When a key is almost expired report an issue.The issues can be shown in archweb or mailed to a mailing list.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.