Improve performance and output of pyenv virtualenvs#502
Improve performance and output of pyenv virtualenvs#502samdoran wants to merge 17 commits intopyenv:masterfrom
pyenv virtualenvs#502Conversation
|
I could use some help getting the tests correct. I made a few changes but I don't understand why the |
|
Seems like only two tests are failing 😄 I just randomly checked into virtualenv. I recently did some perf improvements on the main I am not sure if this applies to your situation as well, haven't looked much into it. The main idea I have is:
|
|
The tests do need some adjusting since I've changed the output a bit. I'm just having a hard time figuring out what exactly |
|
Hehe, and the test file hasn't been touched in 10 years 😆 I did not encounter |
|
The code itself that was the source of the performance issue is 10-12 years old. Not that old code is bad, but there were some inefficiencies in there. |
|
No it's not that the old code is bad. It was probably good at the time it was written. It's more of a problem that the code has design principles which made sense back then, but fast forward 10 years and it's not as intuitive anymore. And if you're unlucky, you're facing tests which were quick-fix set up 10 years ago, and suddenly you need to fix the problem of ... It's a challenge for sure 😄 |
|
Thank you for the feedback @native-api. I'll address this later in the week. |
|
I had a crazy week and a full weekend away from home. I will work on this during the week. |
|
Thanks for all the feedback. I didn't get to fixing up the tests. I need to get some sleep for now. |
|
Did this get dropped? Would love a speedup on virtualenvs 👍 |
|
I have not forgotten about this. My day job has consumed all my capacity lately so I haven't circled back to address the test failures. I'm kinda cheating because I started using the changes in this branch locally so haven't felt the pain in a while. |
|
The prompt hook performance issue (#259) is fixed in v1.3.0 via #523, which caches the active version to skip redundant |
|
I'll try out the fixes in v1.3.0 and see how they affect the issue I was seeing. Thanks for the improvements you made. |
|
So it does seem like this helps with the prompt completion, but |
samdoran
force-pushed
the
improve-performance-and-output
branch
2 times, most recently
from
bdcfb08 to
cbaa29a
Compare
|
Sorry this took so long, but I finally circled back and got the tests in order. I fixed some existing issues and added more tests cases. Please review when you have a chance. Thank you! |
|
Thanks for fixing those things up. |
Use the code from pyenv-versions for efficiency and consistent output. The main performance problem was in the call to pyenv-virtualenv-prefix, which called pyenv-prefix, which then enumerated every virtual environment. This was done inside a loop, compounding the problem. Simply the virtual environment listing so that it does not have to call pyenv-virtualenv-prefix anymore.
This is useful for listing the “frinedly” virtualenv names instead of the short path which includes the Python version.
…vironments Having the current virtual environments listed as options in the competion is noisy since only bare Python versions, such as 3.11.1, make sense as suggested completions for `pyenv virtualenv [version]`.
This makes the suggested completetions cleaner.
This was removed by accident.
For consistency, use conditional expressions instead of arithmetic evaluation when comparing bash versions.
The new implementation is not using realpath.
Update the existing tests based on new behavior/output format. Add additional tests to cover scenarios that were not previously tested.
While it looks a bit funny, it is valid to use an existing virtual environment as the [version] when creating a new one.
samdoran
force-pushed
the
improve-performance-and-output
branch
from
894a488 to
64b8644
Compare
| venv27 (created from /usr) | ||
| * venv33 (created from /usr) |
There was a problem hiding this comment.
Base environment is critical information for an env. Unless you have rationale for it, it should stay.
There was a problem hiding this comment.
Calling pyenv-virtualenv-prefix in order to get that information is a huge source of the performance problem. I'll see if I can find a more efficient way of getting that information.
There was a problem hiding this comment.
Actually, I think that is the source of the performance problems. With 160 virtual environments, calling pyenv-virtualenv-prefix to get the base environment increases the total output time from 665 ms to 12 s.
> pyenv virtualenvs --skip-aliases | wc -l
160
> time pyenv virtualenvs > /dev/null 2>&1
________________________________________________________
Executed in 12.73 secs fish external
usr time 5.99 secs 0.20 millis 5.99 secs
sys time 6.11 secs 1.14 millis 6.11 secs
# omitting the base environment lookup with pyenv-virtualenv-prefix
> time pyenv virtualenvs > /dev/null 2>&1
________________________________________________________
Executed in 664.70 millis fish external
usr time 168.56 millis 189.00 micros 168.37 millis
sys time 195.23 millis 889.00 micros 194.34 millis
There was a problem hiding this comment.
I see that pyenv-virtualenv-prefix accepts multiple arguments. Would calling it just once help?
5 participants
The biggest change is reusing most of the code used for
pyenv versionsinpyenv virtualenvs. This dramatically improves performance while making the output consistent.Related to #490. Maybe fixes the problem?
With 128 virtual environments, it currently takes about 13 seconds to run
pyenv virtualenvs --bare.Output
With the changes in this branch it is 236ms:
Output
Output formatting changes
This PR also includes changes to the output format to make it more consistent with
pyenv versions.The completions for
pyenv virtualenvnow omit envs.