Repository navigation
Bug: Downloading mail archives from Mailman3 often times out #2847
Description
Describe the bug
I've been trying to download the mail archive for the python-list (comp.lang.python mirror) via https://mail.python.org/archives/list/python-list@python.org/ for history research purposes.
- The complete download directly fails with a message that the request will time out.
- The monthly download times out for a lot of months.
- Using the URL parameters, I then tried half month downloads and even weekly downloads, but still get time outs.
In essence there appears to be no usable way to download the complete archive using this Mailman3 interface.
Using the old Mailman2 interface from https://mail.python.org/pipermail/python-list/, the downloads succeed without any problems - essentially those are static files, not files which the server needs to generate upon request.
Is there anything which could be done about this (e.g. extending timeouts or increasing VM sizing parameters) ?
Thanks.
To Reproduce
- Go to https://mail.python.org/archives/list/python-list@python.org/
- Try to download e.g. July 2001
Expected behavior
A working download without timeouts.
URL to the issue
https://mail.python.org/archives/list/python-list@python.org/
Screenshots
""Browsers
Chrome
Operating System
Windows
Browser Version
No response
Relevant log output
Additional context
No response
Activity
malemburg commentedon Dec 11, 2025
Some additional information:
The largest archives size I could successfully download was only about 2.6MB in size.
The Mailman2 archives, which are in a trimmed down mbox file format (losing lots of headers and the attachments), have gzipped sizes up to around 5MB for e.g. Jan 2009.
I would expect file sizes up to at least 10MB gzipped to work without problems.
malemburg commentedon Dec 13, 2025
I ended up using 5 day intervals for the download. That's not really how it should be. The web interface fails for many months.
There's one day which cannot be downloaded, even when trimming down the range to a single day: 2000-10-11.
Here's the URL: https://mail.python.org/archives/list/python-list@python.org/export/python-list.mbox.gz?start=2000-10-11&end=2000-10-12
This appears to be a separate issue, since the download fails before reaching the usual limit of around 2MB.
malemburg commentedon Oct 9, 2026
Why this bug report closed as "not planned" by a bot ? That's an odd way to fix bugs. It looks to me like an infrastructure problem, not one in Mailman3 itself.
holdenweb commentedon Oct 15, 2026
Looks like the assumption is the processes are timing out, which seems reasonable. Who should we raise an issue with to get this closed? It would seem the archives aren't much used :)
JacobCoffee commentedon Oct 15, 2026
sorry about the autoclose! our internal tracker was config'd to close issues older than 6 months - i thought that i disabled that in each project relevant but i missed pythondotorg :(
JacobCoffee commentedon Oct 15, 2026
postmaster@python.org is probably the best place if its a mailman3 config item as most of it is hand managed by the postmaster team and they are the experts
i will poke around on the mail host when time allows to see if theres some config to be done to the gunicorn conf or if something needs to be done to fix bad headers (which would be a postmaster ask)
JacobCoffee commentedon Oct 15, 2026
It seems like there was a huge number of bots and scrapers, which caused resource contention along with a gunicorn timeout of 30s. ive put a apache conf. to try and make the bot traffic lessen, and with the newly freed up resources we could up the timeout to 60s - but even this wont unblock you on the busiest months without reducing date ranges.. i continue to poke
i think gitlab.com/mailman/hyperkitty/-/work_items/385 would help :\
malemburg commentedon Oct 15, 2026
I do wonder why these exports are being generated on demand in Mailman3 rather than using static files as in Mailman2. The latter are generated as single files per months and don't have any load problems.
Perhaps an easier solution would be to have a cronjob generate daily complete archive exports as static files and then point users (and bots) at these instead via an automatic redirect (to not have to tamper with Mailman3 code).
JacobCoffee commentedon Oct 15, 2026
this sounds like a nice idea, i'll work to research with my team and see if we can do this!