Repository navigation
After requeueing a job, first run often fails to update date_done with concurrent error #858
Description
Module
queue_job
Describe the bug
I see in my log many cases of:
# good case
2025-12-09 07:22:43,904 2229881 WARNING ? odoo.addons.queue_job.jobrunner.runner: Re-queued dead job with uuid: 76c33f7c-2a1f-4564-9545-2daccf99026d
2025-12-09 07:22:43,908 2229881 INFO ? odoo.addons.queue_job.jobrunner.runner: asking Odoo to run job 76c33f7c-2a1f-4564-9545-2daccf99026d on db ....
# bad case
2025-12-09 07:22:43,922 2229881 WARNING ? odoo.addons.queue_job.jobrunner.runner: Re-queued dead job with uuid: b8b82330-6c0d-44a6-b12c-1fde5bf8ee42
2025-12-09 07:22:43,926 2229881 INFO ? odoo.addons.queue_job.jobrunner.runner: asking Odoo to run job b8b82330-6c0d-44a6-b12c-1fde5bf8ee42 on db ....
2025-12-09 07:22:43,954 2820876 ERROR .... odoo.sql_db: bad query: UPDATE "queue_job" SET "date_done" = '2025-12-09 07:22:43.951733', "dependencies" = '{"depends_on": [], "reverse_depends_on": []}', "exec_time" = 0.034894, "retry" = 1, "state" = 'done' WHERE id IN (22671925)
ERROR: could not serialize access due to concurrent update
To Reproduce
Affected versions: 16.0
Steps to reproduce the behavior:
- Have a job start
- Kill the worker to make the job "dead"
- Job gets enqueued
- Job starts
- Concurrent error; second retry goes through
Expected behavior
First run directly goes through
Additional context
I don't know at the moment what's locking the queue job record - if it is the previous "dead" job (unlikely, because jobrunner itself already wrote to that record), or the jobrunner itself locking the record (more likely, but also strange since the cursor should be committed and closed at that point)
Activity
thomaspaulb commentedon Dec 9, 2025
More details after running through one example, there seems to be a chain of events leading up to this:
So timeline of events here is:
Jobrunner again asks Odoo to do the job
thomaspaulb commentedon Dec 10, 2025
Okay I found the cause - this happens whenever all of Odoo's workers are occupied for longer than 10 seconds. Even though the job is already 'enqueued' in the sense that, the HTTP request started and we are waiting for Odoo to pick it up; the jobrunner proactively requeues the job even though it's still in the HTTP pipeline so to say.
Proposing to solve this by having a second lock in the job runner that marks whether a job is enqueued; aside from the current one that marks when a job is started.
sbidoul commentedon Dec 10, 2025
Hmm that case should be cared for by the code that checks that the job is well in enqueued state when starting it.
thomaspaulb commentedon Dec 10, 2025
sbidoul commentedon Dec 10, 2025
Yes but I don't quite understand where the concurrent update error comes from.
In your log above the job seems to run twice?
thomaspaulb commentedon Dec 10, 2025
You're right, it repeats because of the concurrent update. I haven't yet figured out where that comes from - there must be something that touches the queue_job record while the job is running.
That is supposed to be prevented by this "for update" lock, but that lock is released again after this commit call. Maybe
job.lock()should also lock the actualqueue_jobrecord again, after asserting at that point the job state is "started", ... that could be another safeguard but still does not explain why it happens.