Depend on psycopg2-binary instead of psycopg2.

[torotil]

This makes it possible to install the Python venv without the need to compile anything.

fixes #323

[dracos]

I understand why this would be nice for development/ease of setup; however psycopg2 says, at https://www.psycopg.org/docs/install.html#install-from-source "If you are the maintainer of a published package depending on psycopg2 you shouldn’t use ‘psycopg2-binary’ as a module dependency. For production use you are advised to use the source distribution."
If a user of mapit wishes to use psycopg2-binary they can of course specify it themselves in their requirements and it will be used.

[torotil]

Thanks I didn’t know that. Is there a way of specifying it only as dev-dependency then?

[dracos]

Python packaging, as far as I know, does not have such a concept. If you wish to use the binary package, you can install it in your venv first, manually, or (I guess) by specifying it first in your own requirements.txt

Looking at psycopg/psycopg2#674, I don't want to get involved with that :)

[RossLYoung]

This reliance massively increases build times. When developing locally in containers or in CI, the build time for psycopg is 5+ minutes. Can it be a decision made with an environment variable?

[dracos]

Hi @RossLYoung - that sounds odd, https://github.com/mysociety/mapit/actions/runs/21130062639/job/60758899547 only takes 25 seconds to install all the python packages in that Action, including psycopg2 as part of it. What is your install doing in all that time?
psycopg/psycopg2#1335 appears to be the same issue; as the links above say, we can't depend on the binary, and I'm not sure how to alter it based on an environment variable.
The best solution is probably to upgrade mapit to use psycopg3, which sounds like it might fix this issue by having the binary as an extra, rather than a separate package (though I haven't tried it).

[RossLYoung]

Hi @dracos - simply running 'docker compose build' has my beefy CPU at 100% for at least 5 minutes whenever I update any dependencies and need a full build. Is your CI caching the pip install layer?

I've submitted #440 which removes the psycopg2 dependency if you're using mapit as a library, and includes both psycopg versions if your are running it standalone.

I think the only changes required would be to update the docs to say something like
pip install -e ".[psycopg2]" or pip install -e ".[psycopg3]" and if you want to specify versions in setup.py

[dracos]

Is your CI caching the pip install layer?

No, there is no caching of which I'm aware. You can see the action at https://github.com/mysociety/mapit/blob/master/.github/workflows/default.yml

Your screenshot doesn't provide any extra information as to what it's doing; a plain install of psycopg2 (and django-mapit is the same) from scratch on a normal server takes the same time as our CI:

~/temp$ python3 -m venv venv
~/temp$ source venv/bin/activate
(venv) ~/temp$ pip install wheel
Collecting wheel
  Downloading wheel-0.46.3-py3-none-any.whl.metadata (2.4 kB)
Collecting packaging>=24.0 (from wheel)
  Downloading packaging-26.0-py3-none-any.whl.metadata (3.3 kB)
Downloading wheel-0.46.3-py3-none-any.whl (30 kB)
Downloading packaging-26.0-py3-none-any.whl (74 kB)
Installing collected packages: packaging, wheel
Successfully installed packaging-26.0 wheel-0.46.3
(venv) ~/temp$ time pip install psycopg2
Collecting psycopg2
  Downloading psycopg2-2.9.11.tar.gz (379 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (pyproject.toml) ... done
  Created wheel for psycopg2: filename=psycopg2-2.9.11-cp313-cp313-linux_x86_64.whl size=509125 sha256=eb02ae238cdf9083f81c7e4bced4700e3c659830fc3c05f398f9a0e6053f3e7c
  Stored in directory: /home/matthew/.cache/pip/wheels/63/ed/1a/7f7f58e98cbe6623951e4308d81a93c8087d1ac9804513a056
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.9.11

real	0m15.401s
user	0m12.623s
sys	0m1.940s

Does your build provide any logging or anything to say what it is doing during the time?

Anyway, wherever the issue lies, thanks for the PR; why can't we just switch it to depend upon psycopg? That's as the docs say to do at https://www.psycopg.org/psycopg3/docs/basic/install.html#handling-dependencies and then an end user can choose to install psycopg[binary] or psycopg[c] if they want on top, with no conflicts. That feels like the best solution to me, unless I've missed something.

psycopg and psycopg[binary] are pretty instant here; I see psycopg[c] takes 45s rather than 15s for psycopg2, who knows why, but still, nothing like 5 minutes.

[RossLYoung]

I'm not sure what is going on with my build times. I'm using UV, Python 3.14, in an Ubuntu 24 docker container.
I did notice that I was installing an old version (3.0) as my uv.lock hadn't updated since I specified a mapit git commit in my pyproject.toml

why can't we just switch it to depend upon psycopg?

I was suggesting that we don't depend on it at all.
Psycopg/2 is only a dependency if you run a standalone mapit so can be added to extras_require in setup.py.
If you use mapit as a Django app, the the parent Django project will already have a postgres library installed.
My PR has replaced all psycopg imports with Django InternalError/IntegrityError. There were also some SQL issues moving to psycopg3.

The only change would be updating the install docs to pip install -e .[psycopg]. Where is the code for https://code.mapit.mysociety.org ?

We could also update the advice to "pip install django-mapit" as it will install v3.0 from Nov 2020.

[dracos]

I've opened #442 which uses changes from your PR, but does keep psycopg as a dependency. This is as I understand from https://www.psycopg.org/psycopg3/docs/basic/install.html#handling-dependencies what we should do, and should work consistently.
People using MapIt as a Django app can choose to use psycopg[c] or psycopg[binary] (or nothing) as they desire.
We'll also do a new PyPi release with a new major version. That should mean the install documentation still works the same.
Hope that makes sense, thanks for your contributions.