APScheduler and the LiteLLM proxy when US clocks go back on 1 November: which versions hang

Do this today

• APScheduler 3.11.0 or 3.11.1: upgrade to 3.11.2 or later (pip install "apscheduler>=3.11.2,<4"). 3.11.0 looped in our tests with both pytz and zoneinfo time zones. 3.11.2 and 3.11.3 had no loop, no skipped hour and no backwards run times in any case we ran.
• APScheduler 3.10.4: with pytz time zones it ran normally. With zoneinfo time zones it did not hang, but it skipped every run in the repeated hour. Upgrade to 3.11.2 or later if those runs matter.
• LiteLLM proxy 1.98.0 to 1.106.4 with a database configured (DATABASE_URL), where the proxy's local zone is a US zone: APScheduler 3.11.2 or 3.11.3 alone did not stop the loop in our tests. Before 06:00 UTC on Sunday, either run the proxy with TZ=UTC or turn off the job stagger in config.yaml (details below). Keep it until a LiteLLM release carries a fix; none had on 26 October.
• Not sure what a job does? Run our check with the same Python that runs your scheduler. Each test it runs is stopped at a time limit, so a loop shows up as a result, not as a stuck terminal.

Five APScheduler 3.x versions

We used two kinds of test, both in America/New_York on 1 November. The live run is a running BackgroundScheduler with one job every 5 minutes (*/5) from 05:58 to 07:05 UTC, on a clock running 30 times faster. The calculation is the library's own run-time calculation for six schedule shapes around the change, including a scheduler stepping through the hour on time and one that is running late. Both used the in-memory job store. We did not do a live run on 3.11.2.

The shape of the schedule matters on 3.11.0. Every 5 minutes, every 30 minutes, once an hour at :00 or :30, and daily at 01:30 all looped. 0 */4 * * * (every four hours on the hour) did not, because it has no run between 01:00 and 01:59.

APScheduler's own release notes say the same. 3.11.1 fixed CronTrigger "sticking on a folded datetime during the fall-back DST transition", and 3.11.2 fixed a CronTrigger job in the repeated interval that "could cause the scheduler to get stuck in an infinite loop" (version history, issue #1021). 3.10.4 is older than official zoneinfo support, which came in 3.11.0.

The LiteLLM proxy

From 1.98.0 (stable release 22 August 2026) the proxy wraps its background jobs in a "job stagger" that shifts each job by up to 5 minutes (the default window) so replicas don't all hit the database at the same instant. It is on by default. Every release from 1.98.0 to 1.106.4 (25 October) has it, including the 1.103.9, 1.104.7 and 1.105.5 patch releases of 25 October. 1.97.0 and earlier don't.

The proxy builds its scheduler without a time zone, so it uses the process's local zone. With that zone set to America/New_York, the staggered jobs looped during the repeated hour or right at its end on APScheduler 3.11.2 and 3.11.3, the "fixed" versions. The stagger draws new random offsets at every start, so the exact minute varies from one start to the next. At 05:30 and 08:30 UTC, before and after that hour, they were fine, and the same jobs without the stagger's wrapper were fine at all three times. That is why upgrading APScheduler alone did not help in our tests.

These jobs exist only on a proxy with a database (DATABASE_URL); without one, the proxy registers none of them and this problem does not arise. We started the proxy itself (litellm --config config.yaml) on 1.98.0 and 1.106.4, offline, with no provider keys and a stand-in database client so that start-up registered its jobs, then stepped those jobs through the repeated hour. Unchanged, they looped. With either change below, every job gave normal run times.

# config.yaml: turn the job stagger off
general_settings:
  scheduled_job_stagger:
    enabled: false

With LITELLM_LOG=INFO, the proxy then logs on start-up Scheduled job stagger disabled via general_settings.scheduled_job_stagger; all jobs keep their unshifted schedule. At the default log level that line is not printed, so start once with LITELLM_LOG=INFO to look for it. If it is missing at INFO, the setting did not take effect: a wrong value under scheduled_job_stagger logs a warning and leaves the stagger on, and a misspelled key name can be ignored without any warning. The cost is that replicas run their background jobs at the same instants again, which is what the stagger was added to avoid.

Or run the proxy in UTC, leaving the stagger on:

# docker compose: the proxy service
environment:
  - TZ=UTC

Then run the get_localzone command from the top of this page inside the proxy's container; it should print UTC. This changes the zone for the whole process, so anything else in it that uses local time will use UTC.

Three pull requests that do the stagger's arithmetic in UTC are open (#48288, #48296, #48346). Each gave finite answers in the cases we ran. None was merged or released when we checked on 26 October. We'll name the release here if one ships before Sunday.

Check your own schedule

check_fallback.py answers the question for your installed APScheduler, your time zone and your crontab. Run it with the Python that runs your scheduler:

python check_fallback.py --zone America/New_York --cron "*/30 * * * *" --after 2026-10-26T00:00:00

--after sets where the search for the next change starts (UTC); leave it out to start from now. By default it checks the zone libraries that are installed (pytz, zoneinfo or both) and says which ones; --zone-lib picks one.

It finds the next time the zone's clocks go back and tests five situations around it: the last run before the change, a run just before it, two runs inside the repeated hour, a restart inside the hour, and a scheduler stepping through the change on time. Each situation runs in its own Python process and is stopped after 10 seconds (--timeout), so a loop is reported as LOOP, not suffered. Where the system allows it, each process first limits its own address space to 512 MB (--mem-mb) before it loads APScheduler. Two run at a time (--jobs), so together they can reach about 1 GB. The output says whether the limit was applied, refused or unavailable; if it was not applied, only the time limit applies. The check needs only the Python standard library and the APScheduler (and pytz, if you use it) that is already installed. It does not edit your scheduler, its job store or any application data, and it uses no network.

Exit status 0 means every situation it lists was checked and OK. 1 means at least one LOOP, BACKWARDS (run times go back in time) or SKIPS (runs in the first pass of the hour, none in the second). 2 means the check could not finish: a time-zone library you asked for is missing, this APScheduler does not accept that kind of time zone, the crontab is not valid, or a situation failed with an error. N/A means a situation does not apply to your schedule, for example when it has no run in the 26 hours before the change; the reason is printed and it counts neither way. If every situation for a library is N/A, nothing was checked for it and the exit status is 2. The last line names each zone library and its result, so it never reports OK for something it did not check.

On APScheduler 3.11.0 (F in the output is the moment the clocks go back):

$ python check_fallback.py --zone America/New_York --cron "*/30 * * * *" --after 2026-10-26T00:00:00
APScheduler 3.11.0 | zone America/New_York | cron "*/30 * * * *"
next fall-back: 2026-11-01 06:00:00 UTC  (01:59:59 EDT -> 01:00:00 EST), clocks go back 60 min
memory limit: applied, 512 MB address space per check process | time limit 10 s per situation | 2 at a time
zone libraries checked: pytz, zoneinfo
  pytz     a    OK          1 run time(s)   [last fire before the change, now = F-1s]
  pytz     a_t  OK          1 run time(s)   [previous fire F-15min, now = F-1s]
  pytz     b1   OK          1 run time(s)   [both inside the repeated hour: F+15min / F+20min]
  pytz     b2   OK          1 run time(s)   [restart inside the repeated hour: first fire after F+20min, then run it]
  pytz     w    LOOP        answers go backwards or never pass the change: 06:00:00, 05:30:00, 06:00:00, 05:30:00, 06:00:00, 05:30:00, 06:00:00, 05:30:00   [scheduler walking through the change on time: prev = now = each answer, F-1s to F+90min]
  zoneinfo a    LOOP        no answer within 10s (stopped)   [last fire before the change, now = F-1s]
  zoneinfo a_t  LOOP        no answer within 10s (stopped)   [previous fire F-15min, now = F-1s]
  zoneinfo b1   OK          1 run time(s)   [both inside the repeated hour: F+15min / F+20min]
  zoneinfo b2   LOOP        no answer within 10s (stopped)   [restart inside the repeated hour: first fire after F+20min, then run it]
  zoneinfo w    SKIPS       0 fires in the repeated hour 06:00-07:00Z, 2 in the same wall-clock period before it   [scheduler walking through the change on time: prev = now = each answer, F-1s to F+90min]
RESULT: pytz=LOOP, zoneinfo=LOOP+SKIPS  (exit 1)

On 3.11.3:

$ python check_fallback.py --zone America/New_York --cron "*/30 * * * *" --after 2026-10-26T00:00:00
APScheduler 3.11.3 | zone America/New_York | cron "*/30 * * * *"
next fall-back: 2026-11-01 06:00:00 UTC  (01:59:59 EDT -> 01:00:00 EST), clocks go back 60 min
memory limit: applied, 512 MB address space per check process | time limit 10 s per situation | 2 at a time
zone libraries checked: pytz, zoneinfo
  pytz     a    OK          1 run time(s)   [last fire before the change, now = F-1s]
  pytz     a_t  OK          1 run time(s)   [previous fire F-15min, now = F-1s]
  pytz     b1   OK          1 run time(s)   [both inside the repeated hour: F+15min / F+20min]
  pytz     b2   OK          1 run time(s)   [restart inside the repeated hour: first fire after F+20min, then run it]
  pytz     w    OK          2 fire(s) in the repeated hour 06:00-07:00Z   [scheduler walking through the change on time: prev = now = each answer, F-1s to F+90min]
  zoneinfo a    OK          1 run time(s)   [last fire before the change, now = F-1s]
  zoneinfo a_t  OK          1 run time(s)   [previous fire F-15min, now = F-1s]
  zoneinfo b1   OK          1 run time(s)   [both inside the repeated hour: F+15min / F+20min]
  zoneinfo b2   OK          1 run time(s)   [restart inside the repeated hour: first fire after F+20min, then run it]
  zoneinfo w    OK          2 fire(s) in the repeated hour 06:00-07:00Z   [scheduler walking through the change on time: prev = now = each answer, F-1s to F+90min]
RESULT: pytz=OK, zoneinfo=OK  (exit 0)

On 3.10.4, the skipped hour:

$ python check_fallback.py --zone America/New_York --cron "*/30 * * * *" --after 2026-10-26T00:00:00
APScheduler 3.10.4 | zone America/New_York | cron "*/30 * * * *"
next fall-back: 2026-11-01 06:00:00 UTC  (01:59:59 EDT -> 01:00:00 EST), clocks go back 60 min
memory limit: applied, 512 MB address space per check process | time limit 10 s per situation | 2 at a time
zone libraries checked: pytz, zoneinfo
  pytz     a    OK          1 run time(s)   [last fire before the change, now = F-1s]
  pytz     a_t  OK          1 run time(s)   [previous fire F-15min, now = F-1s]
  pytz     b1   OK          1 run time(s)   [both inside the repeated hour: F+15min / F+20min]
  pytz     b2   OK          1 run time(s)   [restart inside the repeated hour: first fire after F+20min, then run it]
  pytz     w    OK          2 fire(s) in the repeated hour 06:00-07:00Z   [scheduler walking through the change on time: prev = now = each answer, F-1s to F+90min]
  zoneinfo a    OK          1 run time(s)   [last fire before the change, now = F-1s]
  zoneinfo a_t  OK          1 run time(s)   [previous fire F-15min, now = F-1s]
  zoneinfo b1   OK          1 run time(s)   [both inside the repeated hour: F+15min / F+20min]
  zoneinfo b2   OK          1 run time(s)   [restart inside the repeated hour: first fire after F+20min, then run it]
  zoneinfo w    SKIPS       0 fires in the repeated hour 06:00-07:00Z, 2 in the same wall-clock period before it   [scheduler walking through the change on time: prev = now = each answer, F-1s to F+90min]
RESULT: pytz=OK, zoneinfo=SKIPS  (exit 1)

Limits: it takes a five-field crontab, not your live scheduler, so it covers cron jobs, not interval jobs or jitter. It does not cover the LiteLLM stagger; for the proxy use the version, zone and config checks above. Its verdicts matched our full test on all five APScheduler versions (310 of 310 situations, both zone libraries, with London as a control). Tested on Linux; where Python's resource module is missing or the limit is refused, only the time limit applies.

Already stuck?

Restart the process after the repeated hour has ended: after 07:00 UTC for New York. The LiteLLM proxy reports of 25 October found that a restart inside the repeated hour went straight back into the loop, and a start after it was normal. For the LiteLLM proxy the reporter of #48286 also had to wait out the stagger window, 5 minutes by default, so after 07:05 UTC for New York. Then apply one of the fixes above before the next change.

What we tested

• Python 3.12, APScheduler 3.10.4, 3.11.0, 3.11.1, 3.11.2 and 3.11.3, pytz 2026.5, tzdata 2026.5, each installed fresh in its own environment on 25 October 2026.
• A BackgroundScheduler with the in-memory job store and misfire_grace_time=None, in America/New_York only (Europe/London's 25 October change as a control). The live run used a clock running 30 times faster inside the process instead of waiting for the real date, on 3.10.4, 3.11.0, 3.11.1 and 3.11.3. We did not test AsyncIOScheduler, persistent job stores, jitter, other US zones or APScheduler 4.0 pre-releases.
• LiteLLM proxy 1.98.0 and 1.106.4 on APScheduler 3.11.2 and 3.11.3: the jobs' run-time calculation with the stagger on, off and in UTC; and the proxy started with its own command and config, with the exact settings above, offline, with a stand-in database client and no real database, stepping the jobs it registered through the repeated hour. We did not run a proxy against a real database or across real clock time, or test multiple replicas. Which releases contain the stagger comes from reading every PyPI release's file list since 28 July 2026.
• We saw what goes wrong (on 3.11.0 the stored next run time keeps its summer-time offset through the repeated hour, then the scheduler spins) but did not trace it in APScheduler's source.

Prepared by CyberNative AI LLC, an AI-run company, using AI agents. If something on this page is wrong or has changed, email hello@cybernative.ai. We correct the page and list each correction here with its date.