New Issue

Better handling of background jobs #622

Open

opened by elegaanz · 3 comments

elegaanz commented (Migrated from github.com)

Is your feature request related to a problem? Please describe.

For the moment, there is no way to retry a failed job. And if the app is stopped, all queued jobs are lost.

Describe the solution you'd like

Having a jobs table that stores queued jobs, and use a worker pool to regularly pull them and run them. Rustodon has a good implementation of background jobs that does that IIRC, and we could probably get inspiration from here (maybe it was even published as a crate that we could just pull?).

Describe alternatives you've considered

Storing these jobs elsewhere like in Redis or RabbitMQ as it is more usually done (and there are probably crates to handle most of the basic logic for that), but I don't really like adding another dependency, so going for a database table seems better (and also, installing Redis when you want to run a very small instance is… meh).

~~Additional context~~

**Is your feature request related to a problem? Please describe.** For the moment, there is no way to retry a failed job. And if the app is stopped, all queued jobs are lost. **Describe the solution you'd like** Having a `jobs` table that stores queued jobs, and use a worker pool to regularly pull them and run them. Rustodon has a good implementation of background jobs that does that IIRC, and we could probably get inspiration from here (maybe it was even published as a crate that we could just pull?). **Describe alternatives you've considered** Storing these jobs elsewhere like in Redis or RabbitMQ as it is more usually done (and there are probably crates to handle most of the basic logic for that), but I don't really like adding another dependency, so going for a database table seems better (and also, installing Redis when you want to run a very small instance is… meh). **~~Additional context~~**

epsilon-phase commented (Migrated from github.com)

What particular jobs would be useful to support?

elegaanz commented (Migrated from github.com)

For the moment, I can think of:

media processing jobs
federation jobs
anything else that we currently do in background threads

For the moment, I can think of: - media processing jobs - federation jobs - anything else that we currently do in background threads

trinity-1686a commented

Owner

at the moment, background threads are used for federation (broadcast and fetch), and key rotation upon deleting comments

👍 2

No Milestone

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Loading…

There is no content yet.

Labels Milestones

Better handling of background jobs #622