cm-core-team / wol-api Goto Github PK

Just quickly add some CI and dependabot configs

Add the ability to get the number of chapters in a bible book. This can help with scraping

For some reason after I refactored getting the html into it's own function bent does not work anymore and I get a weird issue which completely breaks the server somehow....

I have not yet tried doing it as I was doing it before but I am just putting this issue out there to see for your opinion @JudeDavis1

TASKS:

• Check if it still works the previous method -> Me :)

Of course can't forget the logs!

[nodemon] restarting due to changes...
[nodemon] restarting due to changes...
[nodemon] restarting due to changes...
[nodemon] restarting due to changes...
[nodemon] restarting due to changes...
[nodemon] starting node ./build/server.js
Server running on http://localhost:3000
StatusError: Not Found
at ClientRequest. (H:\wol-api\src\backend\node_modules\bent\src\nodejs.js:133:23)
at Object.onceWrapper (node:events:628:26)
at ClientRequest.emit (node:events:513:28)
at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)
at TLSSocket.socketOnData (node:_http_client:534:22)
at TLSSocket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10) {
statusCode: 404,
json: [AsyncFunction (anonymous)],
text: [Function (anonymous)],
arrayBuffer: [Function (anonymous)],
headers: {
'content-type': 'text/html; charset=utf-8',
'content-length': '7101',
'x-frame-options': 'SAMEORIGIN',
'x-content-type-options': 'nosniff',
'x-xss-protection': '1;mode=block',
'last-modified': 'Wed, 02 Nov 2022 04:07:08 GMT',
etag: 'W/"bbc36ba44946a224148bb31b890bea59"',
'content-encoding': 'gzip',
'cache-control': 'public, max-age=21600, s-maxage=21600',
expires: 'Sun, 13 Nov 2022 02:35:55 GMT',
date: 'Sat, 12 Nov 2022 20:35:55 GMT',
connection: 'close',
vary: 'Accept-Encoding',
'strict-transport-security': 'max-age=31536000'
}
}
StatusError: Not Found
at ClientRequest. (H:\wol-api\src\backend\node_modules\bent\src\nodejs.js:133:23)
at Object.onceWrapper (node:events:628:26)
at ClientRequest.emit (node:events:513:28)
at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)
at TLSSocket.socketOnData (node:_http_client:534:22)
at TLSSocket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10) {
statusCode: 404,
json: [AsyncFunction (anonymous)],
text: [Function (anonymous)],
arrayBuffer: [Function (anonymous)],
headers: {
'content-type': 'text/html; charset=utf-8',
'content-length': '7101',
'x-frame-options': 'SAMEORIGIN',
'x-content-type-options': 'nosniff',
'x-xss-protection': '1;mode=block',
'last-modified': 'Wed, 02 Nov 2022 04:07:08 GMT',
etag: 'W/"bbc36ba44946a224148bb31b890bea59"',
'content-encoding': 'gzip',
'cache-control': 'public, max-age=21600, s-maxage=21600',
expires: 'Sun, 13 Nov 2022 02:35:55 GMT',
date: 'Sat, 12 Nov 2022 20:35:55 GMT',
connection: 'close',
vary: 'Accept-Encoding',
'strict-transport-security': 'max-age=31536000'
}
}
StatusError: Not Found
at ClientRequest. (H:\wol-api\src\backend\node_modules\bent\src\nodejs.js:133:23)
at Object.onceWrapper (node:events:628:26)
at ClientRequest.emit (node:events:513:28)
at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)
at TLSSocket.socketOnData (node:_http_client:534:22)
at TLSSocket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10)

If we could add an endpoint to scrape all verses in a single chapter (all at once) and return it, that would speed up my transformer process.

The request package we are using seems to be heavily deprecated and I was wondering if we could implement an alternative.

I have found one which I haven't yet tested.
It's called got

Let me know what you think.
I think however that although it is deprecated it seems to be fast and if there is not suitable alternative which is just as fast or even faster we should just stick with request for now.

For the backend we shall use the rocket framework

Is your feature request related to a problem? Please describe.
I was hoping that the URLs we're using to make requests to wol-api would only require the locale to change, however other parts of the url also change...

This is the url for english

The url for another language (this is polish as an example)

As you can see, other than the locale things like the "lp-e" and the "r1" tag have changed to "lp-p" and "r12"

Describe the solution you'd like
I'm guessing some kind of index which has all this other "metadata" which can be plugged into the req url

Anything to do with the script discussion will go here

To Do

Completed

• Migrate to Java

• #226

• Find a new way to get the verse from the html, current solution doesn't even work 😭🤦‍♂️ -> Jonathan

• Make endpoint to get all verses in a single chapter #99

• Fix tests

• Create React app for the front end... -> Jonathan

• Create route for getting the verse, and exporting previously made function into its own module in a controllers folder -> Jonathan

• Create route generator -> Jude

• #15

• Test other http request packages -> Jonathan/Jude

• Frontend from TS -> JS

The route generator should add a handler in the controller folder.

When creating a new route, should this happen regularly?

If so, I'll work on this.

#2 @JudeDavis1
Small thing but it should probably also check for undefined when passing in the route name, otherwise it just creates a route "undefined". I can work on this, so don't worry about it.