Skip to content

Analyze memory performance #250

@vidiecan

Description

@vidiecan

During big import of tens of thousands of files, the RAM usage increases to 10+GB, inspect where

2026-02-17 16:35:50,010:INFO: =====
2026-02-17 16:35:50,010:INFO: Importing   [   4] workspaceitems

  0%|          | 0/4 [00:00<?, ?it/s]
100%|██████████| 4/4 [00:01<00:00,  2.60it/s]
2026-02-17 16:35:51,550:INFO: OK Imported [   4] workspaceitems
2026-02-17 16:35:53,141:INFO: =====
2026-02-17 16:35:53,141:INFO: Importing   [   0] workflowitems
2026-02-17 16:35:53,142:INFO: OK Imported [   0] workflowitems
2026-02-17 16:35:54,797:INFO: =====
2026-02-17 16:35:54,798:INFO: Importing   [70434] items

  0%|          | 0/70434 [00:00<?, ?it/s]
  0%|          | 32/70434 [00:25<15:35:19,  1.25it/s]
  0%|          | 32/70434 [02:00<15:35:19,  1.25it/s]
  0%|          | 36/70434 [02:04<86:20:02,  4.41s/it]
  0%|          | 38/70434 [02:51<123:03:30,  6.29s/it]
  0%|          | 40/70434 [04:41<239:13:46, 12.23s/it]
  0%|          | 42/70434 [05:33<280:08:15, 14.33s/it]
  0%|          | 43/70434 [05:53<290:32:17, 14.86s/it]
  0%|          | 82/70434 [06:03<49:56:02,  2.56s/it] 
  0%|          | 106/70434 [06:29<37:45:15,  1.93s/it]
  0%|          | 106/70434 [06:57<37:45:15,  1.93s/it]
  0%|          | 108/70434 [06:59<49:40:28,  2.54s/it]
  0%|          | 114/70434 [07:09<46:57:11,  2.40s/it]
  0%|          | 130/70434 [07:22<34:53:25,  1.79s/it]
  0%|          | 151/70434 [07:33<24:09:02,  1.24s/it]
  0%|          | 151/70434 [08:00<24:09:02,  1.24s/it]
  0%|          | 160/70434 [08:01<31:42:39,  1.62s/it]
  0%|          | 160/70434 [08:55<31:42:39,  1.62s/it]
  0%|          | 164/70434 [08:57<59:47:56,  3.06s/it]
  0%|          | 165/70434 [09:10<67:31:36,  3.46s/it]
  0%|          | 165/70434 [09:37<67:31:36,  3.46s/it]
  0%|          | 169/70434 [09:37<79:27:25,  4.07s/it]
  0%|          | 173/70434 [09:51<77:26:04,  3.97s/it]
  0%|          | 182/70434 [10:05<58:26:26,  2.99s/it]
  0%|          | 207/70434 [10:16<27:47:56,  1.43s/it]
  0%|          | 234/70434 [10:27<17:52:40,  1.09it/s]
  0%|          | 251/70434 [10:37<16:07:02,  1.21it/s]
  0%|          | 270/70434 [10:47<14:17:51,  1.36it/s]
  0%|          | 270/70434 [11:10<14:17:51,  1.36it/s]
  0%|          | 288/70434 [11:16<19:31:52,  1.00s/it]
  0%|          | 288/70434 [12:31<19:31:52,  1.00s/it]
  0%|          | 299/70434 [12:37<45:23:59,  2.33s/it]
  0%|          | 305/70434 [12:47<43:59:00,  2.26s/it]
  0%|          | 323/70434 [12:58<31:33:47,  1.62s/it]
  0%|          | 340/70434 [13:08<24:48:26,  1.27s/it]
  1%|          | 370/70434 [13:18<16:25:23,  1.19it/s]
  1%|          | 398/70434 [13:28<12:51:42,  1.51it/s]
  1%|          | 421/70434 [13:39<11:40:28,  1.67it/s]
  1%|          | 443/70434 [13:51<11:30:11,  1.69it/s]
  1%|          | 464/70434 [14:11<13:21:22,  1.46it/s]
  1%|          | 482/70434 [14:32<15:44:47,  1.23it/s]
  1%|          | 493/70434 [14:42<16:11:04,  1.20it/s]
  1%|          | 509/70434 [14:57<16:44:13,  1.16it/s]
  1%|          | 524/70434 [15:09<16:27:36,  1.18it/s]
  1%|          | 524/70434 [15:36<16:27:36,  1.18it/s]
  1%|          | 537/70434 [15:37<22:47:57,  1.17s/it]
  1%|          | 537/70434 [15:58<22:47:57,  1.17s/it]
  1%|          | 540/70434 [16:01<32:49:40,  1.69s/it]
  1%|          | 542/70434 [16:14<39:28:00,  2.03s/it]
  1%|          | 552/70434 [16:34<38:55:55,  2.01s/it]
  1%|          | 552/70434 [16:59<38:55:55,  2.01s/it]
  1%|          | 557/70434 [17:00<50:57:30,  2.63s/it]
  1%|          | 559/70434 [17:14<59:08:08,  3.05s/it]
  1%|          | 559/70434 [18:00<59:08:08,  3.05s/it]
  1%|          | 562/70434 [18:22<125:03:05,  6.44s/it]
  1%|          | 563/70434 [22:54<517:28:50, 26.66s/it]
  1%|          | 564/70434 [23:14<503:59:10, 25.97s/it]
  1%|          | 564/70434 [23:54<503:59:10, 25.97s/it]Terminated

and only after 564 items it used the 10GB RAM

What to do:

  • investigate what can takes so much
  • double verify
  • address the performance bottleneck, precisely describe what and why changed
  • create a PR with the suggested changes

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions