lrnselfreliance / wrolpi Goto Github PK

Config is saved to the WROLPi's system in /opt/wrolpi. This causes a problem if the WROLPi has to be recreated.

All configs should be moved to {MEDIA_DIRECTORY}/configs

Archive module should automatically archive articles from an RSS feed.

The WROLPi hotspot needs to be toggle-able in the UI. It should always turn on when the WROLPi reboots (in case the user cannot access a computer to log into the SSH interface).

This must be handled without WROLPi running as the root user!

The characters †(or some variation) are present in some Readability documents. This appears to be an encoding problem.

Videos nav menu has multiple links, requiring at least three clicks to navigate to a new section. Could this be moved into tabs?

A server status page is needed. The page should have the following data:

• uptime
• cpu temperature
• cpu frequency
• drives mounted
• drive space usage
• drive health (smart)

First iteration will not have any historical data. Only current data generated in-the-moment will be displayed.

Support as many sites as possible, preferably the same amount Youtube-DL supports.

When a video is downloaded, it would be helpful if the user could click a link on the video's page to navigate to the source domain and video.

When videos are played, the cannot be resumed at the last watch location.

Refreshing the videos collection can take a very long time and consume a lot of power. It would be better to write a wrolpi.yaml file to each channel's directory summarizing all videos in that directory.

• Videos will be grouped by their ID
• Video size is stored
• This file will be updated after download and refresh.
• Videos in YAML file will be compared to the directory's contents. Adding/removing as necessary to match the files in the directory.

• select map layers such as topographical or street
• print a map
• save a pin

The current events handling in the Python API is terrible. Convert it to some other events system like https://github.com/inn0kenty/sanic_sse

Downloading a video reports this error. Modify yt-dlp download params to use --no-cache-dir to disable caching.

api_1      | [2022-02-07 19:24:00,131] [modules.videos.downloader.youtube-dl] [WARNING] Writing cache to '/.cache/yt-dlp/youtube-nsig/0cd11746.json' failed: Traceback (most recent call last):
api_1      |   File "/usr/local/lib/python3.8/site-packages/yt_dlp/cache.py", line 49, in store
api_1      |     os.makedirs(os.path.dirname(fn))
api_1      |   File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
api_1      |     makedirs(head, exist_ok=exist_ok)
api_1      |   File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
api_1      |     makedirs(head, exist_ok=exist_ok)
api_1      |   File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
api_1      |     mkdir(name, mode)
api_1      | PermissionError: [Errno 13] Permission denied: '/.cache'

Videos can only be downloaded from a channel/playlist. The user needs a way to download only a single video. The video files should be sorted by domain and channel.

A new feature should be added to WROL Mode, a lower power mode. This should slow the CPU frequency to something usable, but lower wattage. Maybe this should be toggle-able separately by the user?

The install.sh script fails with this:

warning semantic-ui-react > react-popper > [email protected]: You can find the new Popper v2 at @popperjs/core, this package is dedicated to the legacy v1
[2/4] Fetching packages...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "https://registry.yarnpkg.com/date-fns/-/date-fns-2.28.0.tgz: ESOCKETTIMEDOUT".
info If you think this is a bug, please open a bug report with the information provided in "/opt/wrolpi/app/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...

This is an issue with yarn pulling packages.

Work Around

Run the installation script again. The install script can be run as many times as necessary.

Favoriting a video does not work in the config when the video does not have a source id.

Alter the saving/importing of the channel config so that it supports the filename as well as the source_id.

Videos are downloaded as fast as possible, this can cause throttling problems. We need a download limiter. The limiter should do the following:

• Throttle download speed for all modules
• Limit downloads per day
• Schedule downloads within a specific window (download only at night?)
• Randomize downloads

A timezone can be set via the Timezone dropdown in Admin/Settings. But, the timezone is not displayed on refresh. Timezone should be displayed. The timezone should be changeable.

The Archive module has no search functionality.

The download textarea in the home page sends then entire textarea contents as a single string to the Downloader.

• Split the textarea into individual URLs
• Send the array of URLs the API, change the API to accept these.

The Videos module needs to be converted to use MediaPath

Video Channels should be returned as results in the Global Search. Only the title of the Channel should be searched.

Dark theme would be useful to protect night vision. Could this be toggle in the settings of the server?

A channel's name can contain whitespace on the front or back, strip that out automatically on save.

Inventory (sub)categories can have whitespace around them. Strip whitespace from around all string values on save.

What languages does PostgreSQL 12 support? Can the video table handle multiple languages? Do we need to require the maintainer to select only one language?

Many websites display a pay/subscribe full-page banner. Many websites display large amounts of ads which increases file size.

Can an ad-blocker be installed on a locally-run web browser (not in docker)?

If so, this needs to be installed in the install.sh script and used by default for all archiving.

Inventory Summary page displays oddly in portrait mode. Could the entire row be displayed on a single line?

Downloading fails on a new system with this error:

api_1  | Traceback (most recent call last):
api_1  |   File "/opt/wrolpi/api/videos/downloader.py", line 116, in update_channels
api_1  |     update_channel(channel)
api_1  |   File "/opt/wrolpi/api/videos/downloader.py", line 59, in update_channel
api_1  |     channel['next_download'] = today() + timedelta(seconds=download_frequency)
api_1  | TypeError: unsupported type for timedelta seconds component: NoneType

WROLPi uses an insecure password during development. Before release the maintainer should be able to change the wrolpi user's password, as well as set the WiFi Hotspot password.

Inventory expiration date should be a human-readable timestamp. The user should be able to sort by expiration date.

When installing WROLPi on a fresh Ubuntu 20.04 install with the latest packages and upgrades installed, the install.sh script fails.

The first error is:

You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: libnode64 : Conflicts: nodejs-legacy nodejs : Conflicts: nodejs-doc yarn : Conflicts: cmdtest but 0.32.14.gcdfe14e-1 is to be installed

Removing Yarn and NodeJS from the install.sh script seems to at least bypass this problem.

The following error is this one:

`error: cannot perform the following tasks:

• Setup snap "chromium" (1899) security profiles (cannot setup udev for snap "chromium": cannot reload udev rules: exit status 1
  udev output:
  Failed to send reload request: No such file or directory
  )
• Setup snap "chromium" (1899) security profiles (cannot reload udev rules: exit status 1
  udev output:
  Failed to send reload request: No such file or directory
  )
• Setup snap "chromium" (1899) security profiles for auto-connections (cannot reload udev rules: exit status 1
  udev output:
  Failed to send reload request: No such file or directory
  )
  dpkg: error processing archive /var/cache/apt/archives/chromium-browser_1%3a85.0.4183.83-0ubuntu0.20.04.2_amd64.deb (--unpack):
  new chromium-browser package pre-installation script subprocess returned error exit status 1
  Errors were encountered while processing:
  /var/cache/apt/archives/chromium-browser_1%3a85.0.4183.83-0ubuntu0.20.04.2_amd64.deb
  E: Sub-process /usr/bin/dpkg returned an error code (1)`

Since I am pretty new at Linux I have no idea how to fix these issues. Maybe a script update is needed.

get_sanic_url returns 127.0.0.1 when it should return the rpi's address.

It would be helpful if a QR code could be printed by the maintainer so users can join the Hotspot easily.

#55 should be completed first so the password can be included in the QR code.

In main.py it attempts to turn on the hotspot, this is not supported in docker containers and should be ignored.

    if config.hotspot_on_startup:
        admin.enable_hotspot()
    if config.throttle_on_startup:
        admin.throttle_cpu_on()

We need to be able to only download videos matching a specific language.

Downloading of all channels will fail if only one of them fails. The download should log the error, then continue with the rest of the channels.

When joining the hotspot, a new user may not know where the WROLPi interface is located. Can we redirect them to it when they browse to any URL?

It may be that the maintainer does not want the users to be able to access their local network (LAN port) when connected to the hotspot. The hotspot should not have access to the LAN network.

The Map module needs to be restored.

Requirements

• user can view any part of the map while online, whatever is rendered can be viewed again while offline
• maintainer can import packaged maps (pbf files)
• supports viewing on mobile device

Video downloads can hang when attempting to download a livestream.

• Detect a live video, try it again later.

WROLPi currently uses captions from the file "near" the video file. We need to also extract captions from a video file itself.

• Detect videos with embedded captions
• Extract captions from these videos when they do not have a caption file

Video subtitles are on the left of the screen. Can they be moved to the center? Could the user customize the location?

Many sites require a sign-in to access videos. WROLPi must support storing of encrypted passwords for the maintainer.

• Passwords are only written to disk when encrypted by a master password
• The master password is then required to unlock the passwords
• Decrypted passwords are only in memory when needed
• Passwords are never in logs

A new page should be created which displays hundreds of videos and allows easy deletion. This could be a table with checkboxes near each video. Each video should have a much smaller thumbnail.

The Files module can be search for the React Keyed File Browser, but they don't support WROLPi's universal search.

Research: Can we generate thumbnails dynamically? We don't want to leave thumbnails for PDFs, Images, etc all over the media directory.

• Create a File card in React which can display a file
• Display a thumbnail for each file
• Modify files search to support Cards as well as File Browser

A Wiki module needs to be created.

Requirements

• Download Wikipedia dumps
• Search articles
• Tag articles

When a Video refresh isn't yet complete, the statistics fails with:

Nov 02 18:47:44 ubuntu python[5551]: Traceback (most recent call last):
Nov 02 18:47:44 ubuntu python[5551]:   File "/opt/wrolpi/venv/lib/python3.8/site-packages/sanic/app.py", line 909, in handle_request
Nov 02 18:47:44 ubuntu python[5551]:     response = await response
Nov 02 18:47:44 ubuntu python[5551]:   File "/opt/wrolpi/wrolpi/schema.py", line 100, in wrapped
Nov 02 18:47:44 ubuntu python[5551]:     result = await result
Nov 02 18:47:44 ubuntu python[5551]:   File "/opt/wrolpi/modules/videos/api.py", line 159, in statistics
Nov 02 18:47:44 ubuntu python[5551]:     ret = await get_statistics()
Nov 02 18:47:44 ubuntu python[5551]:   File "/opt/wrolpi/modules/videos/lib.py", line 298, in get_statistics
Nov 02 18:47:44 ubuntu python[5551]:     historical_stats['average_size'] = (sum(i['size'] for i in monthly_videos) // len(monthly_videos)) \
Nov 02 18:47:44 ubuntu python[5551]: TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'