File downloading should take into account whether it's currently possible to download a file.

It should be possible to delete an already downloaded file.

Use a local S3-compatible S3 instance to test.

When a file finishes downloading, a message is sent to frontend which removes it from the "currently downloading" component.
However, the "download queue" and "processed files" table are not updated.

1. When the "download start" message is received, the file should be removed from the "download queue".
2. When the "download finished" message is received, the file should be added to the "processed files" list.

The application should ensure there is enough free space on disk before downloading a file.

If there is not enough free space, no other downloads should be made, unless triggered manually.

The "last run" date is currently a placeholder. It should show the actual last run date.

Currently the progress bar only shows when a progress update event is sent via WebSockets.
Frontend should be able to ask the backend for the progress during initial loading.

File downloads should be linked with downloaded files, so that it's easier to manage them and also so that missing files can be recognized in the future.

Currently, it's possible to click "Discover new files" if not authenticated.
The button should be disabled with a big warning label if the user is not authenticated with any source.

Found in logs:

2023-02-26 15:42:58.691  INFO 3632 --- [MessageBroker-6] d.c.b.c.f.d.d.services.FileDownloader    : Downloading game file 1714 (url=/downloads/virtuaverse/en1installer0)...
2023-02-26 15:43:07.844  INFO 3632 --- [MessageBroker-8] o.s.w.s.c.WebSocketMessageBrokerStats    : WebSocketSession[1 current WS(1)-HttpStream(0)-HttpPoll(0), 1 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(1)-CONNECTED(1)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 32, active threads = 0, queued tasks = 0, completed tasks = 48], outboundChannel[pool size = 32, active threads = 0, queued tasks = 0, completed tasks = 2141], sockJsScheduler[pool size = 16, active threads = 2, queued tasks = 3, completed tasks = 24]
2023-02-26 15:43:24.507 ERROR 3632 --- [ctor-http-nio-4] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
	io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2248)
	io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1337)
	io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
	io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:833)

There should be a possibility to redownload already processed files.

A file that is being redownloaded should REPLACE the old file (no suffix should be added to its name).

The code is a bit of a mess now, with, e.g., Hibernate annotations in domain code.

The codebase should be thoroughly cleaned up and made more easily understandable.

• Domain should be isolated from Hibernate/JPA representations
• Domain entities should protect their invariants (non-null properties etc.) - it should not be possible to create aggregates in an invalid state.
• Controllers and domain should use their own PaginationJson/PageJson objects.
• Remove use of INSTANCE for mappers.
• The domain should not care if the files are stored on an HDD or on S3. "File management" should just be another adapter and it should be easy to write a custom one.
• Controller pagination variables should be called paginationHttpDto.
• Add Facade for GameFileDetails.
• Add Facade to backup/discovery
• FileBackupProgress should have its own HttpDto.
• Move @IncludeInDocumentation to adapters.driving.http.
• Consider if GameFileDetailsHttpDto needs to be shared between adapters.
• Backend Sonarcloud analysis should not fail
• All controller and repository tests should resuse contexts
• All JPA repositories should have @Transactional(readOnly=true) on top of the class, instead of on the methods.
• Pitest-strict should work (to confirm)
• Pitest-strict should not fail
• Create a LogCreatedWsMessage for LogCreatedMessage.
• Move IncludeInDocumentation out of domain?
• SourceFileBackupService is returning a String. Does it have to? If so, make it return a value object.
• Rename GameFileDetails to just FileDetails?
• Entire API should only use HttpDto objects (write ArchUnit test)
• SonarQube mutation test reporting should work correctly
• Rename *Message to *Event
  • On backend
  • On frontend
• Frontend Sonarcloud analysis should not fail
• Entire API should be RESTful/resource-based.
• Consider taking some services out of domain and into application

https://netbasal.com/diy-keyboard-shortcuts-in-your-angular-application-4704734547a2

E.g. FileDownloader updates GameFileVersion with tempFilePath, but FileDownloadQueueProcessor updates it with the final file path. Those things should ideally happen in the same place.

File path creation should go straight into FileManager, as an implementation detail.

Perhaps ask ChatGPT how to simplify the situation (with simplified versions of the affected classes).

FileDownloader::downloadToDisk uses SourceDownloader which renames the file after download to whatever it chooses. Then FileDownloader passes the new path to gameFileVersion.markAsDownloaded.

Need to investigate whether final renaming could be left to FileDownloader.

At regular intervals, Backity should send e-mails with information about discovered, downloaded, ignored and failed files.

It should be possible to rename a file that has already been downloaded.

Using something like Cypress

Discovered files, especially, should be grouped by game so that it's easier to pick which files the user wants to download.

Instead, use @configuration to define @beans.

1. A new Game File Version might need to replace an existing older version. This should be handled.
2. Backity should be able to notice that there is a newer version of a file available and inform the user about it.
3. There should be a choice whether the file should be replaced or not.

When enqueueing a file for download, it should be possible to specify a custom file name for it.

Currently, filesize is stored as string, which requires some additional magic to handle. It should be stored as Long.

Perhaps filesize should also be taken from the URL header, instead of JSON field?

Currently files are being downloaded like patch_virtuaverse_v._1.29_%2841922%29_to_1.29a_%2842043%29.exe.
File names should be URL decoded.

There should be the ability to ignore files instead of enqueuing them.

They should show up in the "processed files" list as "ignored".

Currently, it only counts from entering the page, instead of asking the backend for the total.

E.g. FileDownloaderTest::shouldDownloadGameFile tests several things at once.

The application should find all GameFileVersions marked as DOWNLOAD_IN_PROGRESS and delete their files if they exist, as well as either marking them as FAILED or ENQUEUED_FOR_DOWNLOAD.

Perhaps each file should have a counter in the database of how many attempts were made to redownload and if less than 3 then retry, but more than that is a fail?

Currently, filenames are retrieved after downloading. They should be retrieved prior to downloading, then updated when starting downloading.

If the filename has changed between discovery and download, then the file should be downloaded using the original filename (and so also replace any existing files), then renamed. If a file with the new name already exists, it should be removed (probably want a Spring property to adjust this behavior).

The application might look better if the bg-dark color is applied to the body and the cards (making it so that the navbar is the same color as the rest of the page).

Generating client code per the instructions in README.md breaks the build - npm can't build the app because the generated code doesn't exist yet.
The only fix is to temporarily disable the npm build plugin in the pom.

Must find a better way.

It should be possible to download files concurrently.

The number of concurrent downloads should be configurable.

The "Discover new files for GOG" button functionality should be implemented.

The "Cancel" button should remove the file from the queue.

The download speed (mbps) should be shown next to the remaining time when downloading a file.

If a new version of an installer is released, Backity should download it automatically.

Depends on #18

File discovery has been slow since introducing an additional check for file name.

It should be possible to cancel the file currently being downloaded.

It should be possible to pause and resume downloading files.

There should be a view showing a paginated list of all downloaded files, i.e. the newest versions of Game Files.

A SELECT query could look like this:

SELECT gf.*
FROM GameFileVersion gf
JOIN (
  SELECT url, MAX(date_modified) AS max_date_modified
  FROM GameFileVersion
  GROUP BY url
) latest ON gf.url = latest.url AND gf.date_modified = latest.max_date_modified

Recommended indexes:

CREATE INDEX idx_gamefiles_url ON GameFiles (url);
CREATE INDEX idx_gamefiles_date_modified ON GameFiles (date_modified);

A Spring JPA query might look like this (maybe):

    @Query("SELECT gf FROM GameFileVersion gf "
           + "JOIN (SELECT url, MAX(dateModified) AS maxDateModified "
           + "FROM GameFileVersion GROUP BY url) latest "
           + "ON gf.url = latest.url AND gf.dateModified = latest.maxDateModified")
    Page<GameFileVersion> findNewestVersionsOfGameFiles(Pageable pageable);
```java