wikimovimentobrasil / deadlinkchecker Goto Github PK

3.0 0.0 1.0 147 KB

License: GNU General Public License v3.0

Python 45.86% HTML 4.71% JavaScript 49.11% Procfile 0.32%

deadlinkchecker's Introduction

Dead link checker

An app that checks for deadlinks on Wikipedia. its is currently deployed at https://deadlinkchecker.toolforge.org/
The documentation of the tool is available on Meta at https://meta.wikimedia.org/wiki/Dead_Link_Checker

Getting started:

Clone the repository

git clone https://github.com/WikiMovimentoBrasil/deadlinkchecker.git

Enter the project directory

cd deadlinkchecker

Create a virtual environment

python -m venv venv

Activate the virtual environmeny

.\venv\Scripts\activate

Install project dependencies

pip install -r requirements.txt

create a .env file in the root of your project and add to it the following variables

SOCIAL_AUTH_MEDIAWIKI_KEY= Your oauth key from the oauth consumer registration
SOCIAL_AUTH_MEDIAWIKI_SECRET= Your oauth secret from the oauth consumer registration
SOCIAL_AUTH_MEDIAWIKI_URL=https://meta.wikimedia.org/w/index.php
SESSION_SECRET= a randomly generated secret value
TOOL_TOOLSDB_USER= Tools database user
TOOL_TOOLSDB_PASSWORD= Tools database secret
REDIS_URL= Redis URL
REDIS_PREFIX= Redis key prefix

Start the app in development

uvicorn app:app --reload

deadlinkchecker's People

Contributors

Stargazers

Forkers

alwoch

deadlinkchecker's Issues

Implement REDIS

We need to finish the cache' implementation with REDIS

Script throws an error on pages with 100 or more links

The script splits the links into batches of 10 / 15 (This can be changed) when there are more than 15 links on a page. Notably, its only able to return responses for the first 2 when the batch size is set to 10 even though it sends the 3rd batch to the server. when given a batch size of 15, it only returns the first batch and the second batch fails with the response sent

Sometimes it returns 500

Traceback (most recent call last):
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in __call__
    with collapse_excgroups():
  File "/layers/heroku_python/python/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/_utils.py", line 91, in collapse_excgroups
    raise exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/base.py", line 191, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/app.py", line 63, in session_middleware
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/base.py", line 165, in call_next
    raise app_exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/base.py", line 151, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/cors.py", line 91, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/cors.py", line 146, in simple_response
    await self.app(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/sessions.py", line 83, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/routing.py", line 762, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/routing.py", line 782, in app
    await route.handle(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/fastapi/routing.py", line 299, in app
    raise e
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/routes/linkchecker.py", line 102, in check_links
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/routes/linkchecker.py", line 36, in make_request
    response = await client.head(url[1])
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpx/_client.py", line 1844, in head
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpx/_client.py", line 1559, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpx/_client.py", line 1646, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpx/_client.py", line 1674, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpx/_client.py", line 1748, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
    raise exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpcore/_async/connection.py", line 156, in _connect
    stream = await stream.start_tls(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 78, in start_tls
    raise exc
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 69, in start_tls
    ssl_stream = await anyio.streams.tls.TLSStream.wrap(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/anyio/streams/tls.py", line 132, in wrap
    await wrapper._call_sslobject_method(ssl_object.do_handshake)
  File "/layers/heroku_python/dependencies/lib/python3.12/site-packages/anyio/streams/tls.py", line 172, in _call_sslobject_method
    raise EndOfStream from None
anyio.EndOfStream

Refactor server-side code to make it asynchronous

The current set up is synchronous and uses Threading to make requests to the urls. It will be changed to make it asynchronous by using the aiohttp package for making asynchronous requests with hopes of making the code execute faster.

security: Ensure requests to the tool come only from the Wikimedia domains

Create a frontend page

We need a public page for the website, so when someone opens the simple URL (without get parameters) they can understand its functionality. It is important that it has some information:

Instructions for installation and use
It is a fork of deadlinkfinder (it is important to give credit to the original author)
It is a request from the Lusophone technological wishlist
It was developed within the scope of the Outreachy project
It was developed by Wiki Movimento Brasil

Set script to get data from this project

We have to change the script in this line in order to work with this new project. The address is already set at deadlinkchecker.toolforge.org

deadlinkchecker/script.js

Line 340 in 029fa3c

    
           var linkText = 'https://deadlinkfinder.toolforge.org/headerproxy/get.php?'+flags+'link='+link.href;

Add caching using sqlite3 database

Description

A cache to be set up using sqlite3 database such that when a link is provided in the query params, Its queried in the database and the result is returned.

Tasks:

Write a function that checks the database for the existence of the link in the database and if it exists, return the result in the database
Add the link to the database if it does not exist
Dynamically clear links that have lasted more than 24 hours in the database

Solve bug with event listener on results cancel button

DLC defect

Incorrect, on this article of the German Wikipedia, an OK was given, but a deadlink existed

Create a server for the script

Frame-work: Flask
create a server to accept a query string a parameter and run a script to make a request to the link to obtain the status code of the link

script does not connect to links on the web archive

Links that go the web archive such as this one return a connection error

Remove "static" from the script address

At this moment, the javascript is reachable by this address:

https://deadlinkchecker.toolforge.org/static/script.js

Would be ideal to be reachable by:

https://deadlinkchecker.toolforge.org/script.js

create fall back template for when a query string is not passed

Description

Currently, When a query string is not passed to the url, a 400 error is returned. a fall back template is required to prevent throwing this error when the server loads

Set a GitHub auto-updater

Should be as simple as:

@app.route("/update-server", methods=["POST"])
def webhook():
    if request.method == "POST":
        subprocess.check_output(["git", "pull", "origin", "main"])
        return "Updated Toolforge project successfully", 200
    else:
        return "Wrong event type", 400

Source: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Auto-update_a_tool_from_GitHub/GitLab#Python_tool_hosted_in_$HOME/www/python/src

Define app.py as an entry point

The default uwsgi configuration for the uwsgi webservice backend expects to find the uwsgi entry point as the variable app loaded from the $HOME/www/python/src/app.py module. If your application has another entry point, the easiest thing to do is create a $HOME/www/python/src/app.py module, import your entry point, and expose it as app.

Source: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python#Using_a_uWSGI_app_with_a_default_entry_point_that_is_not_app.py

Copy script-languages.js to this repo

This file are still in the original repo, but it needs to be pointed to this repo instead

deadlinkchecker/script.js

Line 78 in 029fa3c

mw.loader.load( "https://deadlinkfinder.toolforge.org//script-languages.js",);

Make the frontend capable to deal 500 responses from the backend

deadlinkchecker/static/script.js

Line 265 in db247c3

//TODO: Deal with 500 responses

Internal Server Error

Comes when I'm supposed to authorise myself

Multiple statuses are indicated on a single link with more than one instance

Description

Where there is more than a single instance of a particular link on a page, there's a tendency to append the status codes of all its occurrences to the first instance for example in the picture below, link 14 is the same as link 17

TODO

pick the index of the link and append its status to it

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.