Introduce Safe mode (on by default), which will limit the subrequests in order to play nice with the free tier of Cloudflare Workers.

Then setting safe mode off (to be paired with a Cloudflare Worker paid plan), will remove the safeguards and allow feeds to make as many subrequests as Cloudflare Workers allows.

Through my testing I've found there's no way to catch an exception when one exceeds the Worker limits, which means the endpoint returns a 500 and it's a bad experience for everyone.

Duplicate posts are showing up in feeds using multiple search terms

Pull from a user list?

Hi folks, I'm getting the following error form Cloudflare when deploying

Is this something I did wrong?

At least couple of feeds derived from Contrails stopped updating a week ago, about when Bluesky went public:

• https://bsky.app/profile/did:plc:pkzt5mk76dc3fnd7w7l7veyr/feed/webassembly
• https://bsky.app/profile/did:plc:tb5q5hwakt3fczz4smodkqhj/feed/ffxiv

There are no errors in Cloudflare dashboard - it still seems to be fetching search results as expected, but the feeds are stuck. Redeploying doesn't help either.

In my FFXIV feed, it picks up a lot of posts using a lot of search terms and most are fine, but there's a specific chunk of the community who include common search terms and use them with something else nsfw.

I'd love to have the ability to exclude a post that contains a search term that I add to a list.

For example, FFXIV would be okay in a post but if the post contains both FFXIV and NSFW, I'd like to be able to have Contrails detect the good search term with the bad one and exclude that post from the feed.

It'll be a useful tool to clean up posts that don't entirely match what the feed creator desires to show up.

tl;dr - getting error "Process completed with exit code 1.", is there a solution to that? Code below

`Run cd feed-generator/

added 147 packages, and audited 148 packages in 10s

13 packages are looking for funding
run npm fund for details

found 0 vulnerabilities
FEEDGEN_HOSTNAME=bluesky-worker-hockey.rockerdude80.workers.dev
yarn run v1.22.19
$ ts-node publishFeedGenerators.ts
/home/runner/work/Bluesky-Contrails-Hockey/Bluesky-Contrails-Hockey/feed-generator/node_modules/@atproto/xrpc/src/client.ts:125
throw new XRPCError(resCode, res.body.error, res.body.message)
^
XRPCError: Invalid identifier or password
at ServiceClient.call (/home/runner/work/Bluesky-Contrails-Hockey/Bluesky-Contrails-Hockey/feed-generator/node_modules/@atproto/xrpc/src/client.ts:125:15)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at async _AtpAgent.login (/home/runner/work/Bluesky-Contrails-Hockey/Bluesky-Contrails-Hockey/feed-generator/node_modules/@atproto/api/src/agent.ts:120:19)
at async publishAll (/home/runner/work/Bluesky-Contrails-Hockey/Bluesky-Contrails-Hockey/feed-generator/publishFeedGenerators.ts:13:3) {
status: 401,
error: 'AuthenticationRequired',
success: false
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 1.`

First off, Jerry, thank you for putting your time and effort into Contrails, really hoping the issue I'm running into though is a quick solve.

I came across Contrails thanks to @lazylazyjane.bksy.social, and I was looking to set up a hockey thread agnostic of teams and just start adding all the hockey related folks I could to it for my own viewing. Again, thanks for setting this up, hope to hear from you soon!

It is very important to have a denilyst but there could be some kind of warning before you find yourself on that list. Miners are often listed on the denylist with only an outdated location. it is necessary to improve

Having the ability to have a denylist to exclude skeets by specific users would be excellent. If possible, the ability to reference a mute list may also be great so this list could be public.

https://bsky.app/profile/readyedgemont.com/post/3k7eh2d2scx2n

Following up on the above thread @jcsalterego it looks like adding the searchTerm that is a link created on SkyFeed does not update the search terms affecting the thread. Genuinely hoping this is user error on my part, but incase its not, kicking an issue your way!

Right now, Contrails cannot find the repost URI to include in a SkeletonItem.reason:

type ReasonRepost = {
  $type: 'app.bsky.feed.defs#skeletonReasonRepost'
  repost: string // repost URI
}

I think this means there's no "reposted by X" in the app.

This is due to the fact that getAuthorFeed doesn't supply the repost URI.

Right now there are two known ways to support multiple feeds:

1. create another fork (including all the secrets & variables setup), or
2. create another branch, update CLOUDFLARE_WORKER_NAME, update CONFIG.md, deploy & publish, and make these changes to switch between branches/feeds

For the future, multiple feeds per worker are possible, if:

• Support multiple CONFIG.md files
• render-config.py renders multiple CONFIG.md files into a shared CONFIGS data structure
• worker.js parses the feed query parameter and routes accordingly

When I try to deploy I get:

⛅️ wrangler 3.20.0

✘ [ERROR] Unterminated string

/github/workspace/wrangler.toml:1:31:
  1 │ name = "worker-small-night-fd13

Any idea why?

Hi @jcsalterego, recently I upgraded from an older version of Contrails and it seems to have happily layered your changes over my minor config file changes, no complaints from Git about any conflicts, so that seemed fine.

Now that I've made some changes to my feeds (their search terms, specifically) I'm trying to publish a new version and I'm receiving an error back after running the Publish Feed Generators github workflow:

Error: The bluesky server is not ready to accept published custom feeds yet
at publishAll (/home/runner/work/Contrails/Contrails/feed-generator/publishFeedGenerators.ts:18:11)
at processTicksAndRejections (node:internal/process/task_queues:95:5)

It doesn't seem to matter what I change, I always receive this error. I updated my Bluesky app password and handle, just in case, but it doesn't complain about those. I'm trying to figure out where the error's generated during the process and what I should be checking.

Pagination support will require keeping track of which subrequests need to be advanced, all in the cursor.

For example, given the search terms:

• dogs
• cats
• hamsters

If cats take up the whole first page, we have to try the same first page of dogs and hamsters on subsequent requests. If we simply request the next cursor/page across the board, we'll miss the first page of dogs and hamster.

Perfect pagination is pretty difficult especially if new results show up for dogs, cats, and hamsters, so this will be best effort.

Right now results from getActorFeed filter out replies and reposts. Add a mechanism such that +reposts and +replies will include said types of posts.

We are planning to tear down the legacy search.bsky.social service immanently. As noted in #26, that index has not been getting updates for about a week.

This migration was hinted at a while back (bluesky-social/atproto#1711), and there is a migration path to switching over to specified search endpoints on the appview (api.bsky.app). The cached public version of the appview (public.api.bsky.app) would probably be a good fit for this use case. It might require tweaks to reduce the query/refresh rate to stay within rate-limits, but a lot more hydrated metadata is now returned by the endpoint.

I think @jcsalterego has been pinged about this recently, but commenting here for visibility.

The Bluesky search service, previously named "thecloud", got merged in to our public golang git repo (indigo), and renamed to "palomar".

Source and a README are here: https://github.com/bluesky-social/indigo/tree/main/cmd/palomar

Contrails looks neat!

The feed I have created does not seem to be sorted correctly. I have set keywords in both Japanese and English, but the sorting seems to be split between Japanese and English posts.
Below is the order in which the posts are displayed. Please let me know how to solve this problem.

2024-02-02 21:58
2024-02-02 16:04
2024-02-02 16:02
2024-02-02 07:25
2024-02-02 07:15
2024-01-31 18:22
2024-01-30 11:47
2024-01-30 10:48
2023-12-31 00:35 en
2023-12-04 02:47 en
2023-11-25 17:52 en
2023-11-05 05:10 en
2023-10-29 09:20 en/ja
2024-01-31 22:00
2023-09-22 22:30 en
2023-09-19 14:57 en/ja
2023-09-14 19:42 en/ja
2023-09-11 23:39 en
2023-08-04 23:05 en
2023-08-03 21:25 en
2023-08-01 22:29 en
2024-01-29 18:53
2024-01-28 23:20
2024-01-28 18:57
2024-01-28 08:45
2024-01-27 21:55
2024-01-25 23:18
2024-01-25 20:05
2024-01-25 07:40
2024-01-24 22:46
2024-01-24 20:36
2024-01-24 14:52
2024-01-24 13:32
2024-01-24 09:04
2024-01-24 05:17
2024-01-22 19:42
2024-01-22 15:23
2024-01-22 01:21
2024-01-30 11:47

*en…English / ja(or no mark)…Japanese

render-config.py should ignore blockquotes in CONFIG.md so we can add comments:

recordName

A unique identifier, max 24 lowercase characters, no spaces.

floppy-fish

Reference config: https://github.com/bawanyadam/Contrails-LoveIsland/blob/main/CONFIG.md

+Noticed 3-4 of the profiles not being loaded into the feed on Bsky at all. Not sure how it determines to pick who shows and who doesn't lol. The order of the profiles in CONFIG.md have no effect on that.