I have been struggling hard doing proper :python: #Python development on :nixos: #NixOS.
pip install'ed binary packages (numpy et. al) don't work (i.e. don't find system libraries like libz libstdc++ etc.), making scientific data analysis completely impossible. The workarounds (using the nixpkgs versions or setting LD_LIBRARY_PATH) are not viable.
With this #poetry2nix flake template (the clue is preferWheels=true) it seems I can finally work properly:
@nobodyinperson Oh, and another aspect: I still need to figure out how I'm supposed to call such a Python script from within shell scripts that don't know anything about poetry/nix dev/...
So far, all the information I've seen refers to interactively executing Python scripts by hand not not invocation from tools like Thunar, crond, shellscripts, ...
@publicvoit If you don't want poetry/nix etc as dependency within Thunar e.g., then you need to install said script into a common location, e.g. with 'pip[x] install' and the like. If you allow poetry as a dep, you could have your Thunar script 'git clone' the project, then have Thunar run 'poetry -C path/to/project install;poetry -C path/to/project run -- progname arg1 arg2 ...`. Same with nix. Depends on what you want to do.
@publicvoit @nobodyinperson
Not sure what level of separation you want, but if the assumption is still on a nix system, the answer is generally packaging at each layer. At the shell layer you'd probably use a nix-shell shebang, makeWrapper, or resholve to depend on the packaged python script/executable.
@abathur@nobodyinperson Ah. So it's true that I need to maintain two wrapper scripts for my stuff: one for Nix systems and another one for my Debian systems (and other people using my tools). 🫣
This solution to do Python development on Nix forces you to use poetry for your Python projects. I found however that poetry is a very good tool and that it makes sense to use it anyway, so I switched all my current projects over to it. The age of not having to run a command to enter your dev environment (e.g. poetry shell) is over, so running nix develop instead is not too much of a problem. And non-nix people can still just continue with their normal workflow. 👍
@nobodyinperson The bad part is, that this does not just refer to Python development. It also refers to executing Python scripts as well. 😔
This way, somebody who is not familiar with all this poetry stuff (the majority) is not able to simply invoke a small Python tool, e.g., from a colleague that does some minor reformatting, calculation, ...
I'd say that #NIxOS is not compatible with #Python in general for this reason.
(Please do correct me if I'm not seeing the whole picture here.)
@publicvoit Oh for executing I'd argue that once you as a developer have a project in poetry, non-NixOS users can just pipx install git+https://... your script and happily use it. For NixOS users you would make a flake that people can nix run ..., so everybody should be happy. 🙂
@publicvoit@nobodyinperson Honestly, as a developer who uses #nixos on several of my daily devices, I utilize an integrated container environment for my larger projects that require testing out or building components with the newest libraries.
It's seemed a bit redundant to me at first, but it's the most awesome thing ever having a bleeding edge Archlinux environment running in tandem with my NixOS system thanks to #distrobox...I can literally have the best of both! 🤘 :nixos: :arch:
@dedsyn4ps3@publicvoit I hear many people use this #distrobox thing. I really avoid containers where possible, although I like #Apptainer's simplicity. My main use case is #reproducibleResearch, for which you need to store the entire container image for every little change you make. But yeah, use cases...
@nobodyinperson@publicvoit That's understandable. I was a bit undecided for awhile, but then after creating an Arch dev container and realizing that apps installed there were also available from my main GNOME applications list, I was very much intrigued.
It provides the opportunity for a seemless integration of separate systems without much hassle at all. Definitely something worth considering...or at least testing out! 😎
@publicvoit@nobodyinperson It's been a while since I tried, but poetry doesn't have a good solution for "libraries" -- it's intended for end-user applications. That is, I can't use poetry to package my library -- so then how do I "do library dev" on NixOS?
(My experience watching people make Python run on NixOS has also been ... poor).
I do like the idea of NixOS and there's some cool stuff there for sure, but..
@meejah@publicvoit I don't really see how that's true. poetry is just a tool helping you to package any Python project, libraries and user-facing programs alike. It's basically a frontent to pyproject.toml to replace the old setup.py/setup.cfg mechanism of packaging. It works perfectly fine for library development, see e.g. my PARMESAN (https://gitlab.com/tue-umphy/software/parmesan), a library without executables that I use poetry for.
Not all Python packages are in nixpkgs. Sure, self-packaging is an option, but seriously: no. The same ease of use for Python development on other platforms must be transferrable to NixOS, otherwise it's too much friction (for me).
You don't have control over nixpkgs' Python versions. Sure, you can pin nixpkgs, but ultimately you want Python's own pkg version pinning, not nixpkgs'.
Nixpkgs doesn't have all versions of Python packages.
...
@nobodyinperson Interesting, thanks for this perspective! I went all in with Nix for my R development needs, and am quite lucky that every CRAN and Bioconductor package is available via nixpkgs, and for quite some time too.
@brodriguesco It's definitely possible to use a distro's #Python package versions and I did that for a long time with :manjaro: #Manjaro for example. But ultimately, you will run into version conflicts and are forced to use a per-project dependency management system, especially if you value #reproducibleResearch. #nix can do that, sure, but not satisfyingly for my above-mentioned points.
@nobodyinperson yes, I’ve been exploring pyenv+pipenv to address exactly this and have per project development environments. I will be looking how Nix handles Python in the future, but it seems not so well!
@brodriguesco If you're on nix, I would advise against pyenv and pipenv. pyenv is not necessary, nix can replace it completely IIUC. pipenv is slower and has less features than poetry, plus, poetry2nix integrates nicely with nix.
@brodriguesco I was reluctant at first as well and went all-in on pipenv, making myself a shell.nix headaching myself on a path to enter a python venv in a nix-ish way that is compatible with non-nix people. I should have gone directly to poetry.
Add comment