quirrel-dev / owl Goto Github PK
View Code? Open in Web Editor NEWA high-performance, Redis-backed job queueing library originally built for Quirrel. Has an in-memory mode for development use cases.
License: MIT License
A high-performance, Redis-backed job queueing library originally built for Quirrel. Has an in-memory mode for development use cases.
License: MIT License
would you be interested in upgrading? not sure I could, but I could give it a try.
Is there a way to store the response data from the job?
Hey there,
I am building a system using Owl that is inspired how Quirrel uses Owl and Owl's scheduled executions on the backend but I am running into an issue where sometimes only some of my cron jobs are executed, or none are executed at all. Sometimes the cron jobs that were missed are executed late by several schedules and sometimes not. Overall it is very odd and I am at a loss as to what could be causing it. Here is a simple version of my cron system code, which is heavily inspired by Quirrel's:
import Owl from "@quirrel/owl";
import cronParser from "cron-parser";
import { redisFactory } from "./utils/redis";
import Sentry from "./utils/sentry";
export function parseTimezonedCron(
cronExpression: string,
): [cron: string, tz: string] {
return [cronExpression, "Etc/UTC"] as [cron: string, tz: string];
}
export function cronSchedule(last: Date, cron: string): Date {
return cronParser
.parseExpression(cron, {
currentDate: last,
tz: "Etc/UTC",
})
.next()
.toDate();
}
export const owl = new Owl({
redisFactory,
scheduleMap: {
cron: cronSchedule,
},
onError: (error) => {
console.log(error);
},
});
type CronJob = () => Promise<void>;
type CronJobs = {
name: string;
cron: string;
handler: CronJob;
};
export const createCronJobs = async (cronJobs: CronJobs[]) => {
const producer = owl.createProducer();
const redis = redisFactory();
const worker = await owl.createWorker(async (job, ackDescriptor) => {
if (job.id !== "@cron") {
return;
}
try {
const cronJob = cronJobs.find(
(cronJob) => cronJob.name === job.queue.split("/")[1],
);
if (!cronJob) {
throw new Error(`No cron job found for queue ${job.queue}`);
}
console.log(`Running cron job ${cronJob.name}`);
await producer.acknowledger.acknowledge(ackDescriptor);
await Sentry.withMonitor(
cronJob.name,
async () => {
await cronJob.handler();
},
{
schedule: {
type: "crontab",
value: cronJob.cron,
},
checkinMargin: 2,
maxRuntime: 7,
},
);
console.log(ackDescriptor);
} catch (error) {
await producer.acknowledger.reportFailure(ackDescriptor, job, error);
console.log(error, "ERROR");
}
return;
});
const existingQueues = await redis.smembers(`queues:cron`);
const queuesThatShouldPersist = cronJobs.map((cronJob) => cronJob.name);
const deleted = new Set<string>();
await Promise.all(
existingQueues.map(async (queue) => {
if (!queuesThatShouldPersist.includes(queue)) {
await producer.delete(`cron/${queue}`, "@cron");
deleted.add(queue);
// await redis.srem(`queues:cron`, queue);
}
}),
).then(() => {
console.log(`Deleted ${deleted.size} cron jobs`);
});
try {
await Promise.all(
Object.entries(cronJobs).map(async ([_, cronJob]) => {
await redis.sadd(`queues:cron`, cronJob.name);
return producer.enqueue({
id: "@cron",
queue: `cron/${cronJob.name}`,
override: true,
payload: "null",
schedule: {
type: "cron",
meta: cronJob.cron,
},
runAt: cronParser.parseExpression(cronJob.cron).next().toDate(),
});
}),
);
console.log("Cron jobs started");
console.table(cronJobs.map(({ name, cron }) => ({ name, cron })));
} catch (error) {
await worker.close();
await producer.close();
throw error;
}
return {
close: async () => {
await worker.close();
await producer.close();
redis.disconnect();
},
};
};
Any clues as to why this might be? As far as I can tell this is no different from how Quirrel does it. Thank you in advance.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Warning
These dependencies are deprecated:
Datasource | Name | Replacement PR? |
---|---|---|
npm | @types/pino |
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
mocha
, @types/mocha
)node
, @types/node
)These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
chai
, @types/chai
)mocha
, @types/mocha
).github/workflows/CI.yml
actions/checkout v2
node 12
.github/workflows/codeql-analysis.yml
actions/checkout v2
github/codeql-action v1
github/codeql-action v1
github/codeql-action v1
package.json
ioredis ^5.2.3
ioredis-mock ^8.2.2
minimatch ^3.0.4
opentracing ^0.14.5
pino ^6.11.3
@istanbuljs/nyc-config-typescript 1.0.1
@types/chai 4.2.22
@types/debug 4.1.7
@types/minimatch ^3.0.5
@types/mocha 9.0.0
@types/node 16.6.1
@types/pino 6.3.11
chai 4.3.4
delay 5.0.0
mocha 9.1.2
nyc 15.1.0
ts-node 10.2.1
typescript 4.4.3
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.