How to improve memory usage of a rector run

[Flyingmana]

Hi, I run some refactorings on a big project.

I do run the project with a single rector rule(ParanthesizeNestedTernary), autoloading for all needed codepaths, and also have the main directories all set as Paths(although via command line I limit to a single path of them)

now the the run takes a while, thats expected(21minutes in total, with a single process), but the memory usage is continually increasing reaching 10GB.

This makes me believe, there is no cleanup of loaded code happening. Is there a Setting I missed, Or any workaround?
Or any theoretical approach I could build a PR for to improve this?
Like, moving some parts from the memory onto the disk, then I could work with more processes in parallel.

[staabm]

If you have a small reproducer I can look into it

[Flyingmana]

I will provide one when I have better Internet again, but I would assume any big php project with a few thousand files would be enough

[yguedidi]

This in progress PR may help here rectorphp/rector-src#3563

[staabm]

rector-src main-branch contains now a few improvements I submitted over the last few days. people report it already made it faster and use less memory

I wrote a blog post about it https://staabm.github.io/2023/05/01/diff-speeding.html

the changes are not yet part of a release though

[Flyingmana]

I tested them with a bigger OpenSource project and the 1.6 release which I think includes them, and while they indeed improve memory usage, they are not enough to resolve the issue that memory usage scales with number of total files instead of processes.

Test setup and result

Setup:
WSL2, Linux, php 8.1
cpu: physical cores 6, logical 12
Ram: 32 GB (25GB allowed for WSL2)

used config:

return static function (RectorConfig $rectorConfig): void {
    $rectorConfig->paths([
        __DIR__ . '/src',
        __DIR__ . '/../symfony',
    ]);

    // usual config
    //$rectorConfig->parallel(120*30,12,200);
    // config for yguedidi
    $rectorConfig->parallel(120*30,12,500, true);

    $rectorConfig->rule(\Rector\Php74\Rector\Ternary\ParenthesizeNestedTernaryRector::class);
};

On a current symfony/symfony checkout, 9629 files
last release: rector 1.6.0

-	Run A	Run B	Run C	Run D	Run E	Run F	Run G	Run H
version	last release	last release	last release	last release	pr-3563	pr-3563	pr-3722	pr-3722
special commands			--no-diffs --no-progress-bar
parallelization	16	6	6	12	12	12	12	12
job size	20	20	20	500	500	200	20	10
files per restart	X	X	X	x	x	x	200	100
Time Real	02m 28s	02m 04s	02m 06s	01m 40s	01m 54s	02m 12s	01m 55s	02m 21s
Time User	26m 49s	11m 48s	12m 02s	14m 44s	17m 28s	22m 39s	21m 24s	26m 12s
process memory	2G	2G ~ 4G	2G ~ 4G	0.6G ~ 3G	0.3G ~ 3G	0.2G ~ 2.8G	0.2G ~ 1.8G	0.2G - 0.7G
totalmemory max	~24G	~18G	~18G	~14G	~9G	5GB	5G	4G

[TomasVotruba]

Avoiding node connector like parent and next/previous might help a lot. PHPStan reported great improvements :)

[samsonasik]

This should already resolved at PR:

• Implement a max jobs per worker budget rector-src#4965