Why provisionner is slow?

[benoit74]

I'm flashing the 2025-08 preppers premium image (453G) on a "standard" hotspot with PNY CS1030 500G disk installed inside the hotspot with Argon PCIe <=> NVMe board. Image is on external NVMe reader (Sabrent reader, noname 500G NVMe disk inside)

When I provisioned the device last Friday, it took about 1h10m, or an average 115MB/s. This was kinda unexpected / quite slow.

I ran some more tests today with same device / setup.

When I "copy" same things directly with dd (not saying it is a recommendable solution), average speed starts around 350MB/s for about 30s and then almost inevitably decreases. It is already at 150MB/s after 480s and 74GB copied. I stopped it at 132MB/s and 104GB copied after 780s. There is small increase of average speed of 1MB/s at times, but otherwise the average speed only keeps decreasing.

Restarting dd immediately after stopping it gives same "speed profile".

Looks like rpi-imager and dd have consistent results and we do have a kind of issue / unexpected behavior.

I have no idea what the "culprit" could be, but it looks like it might be interesting to try reproduce and perform more tests:

• what if the same source disk is installed on USB port with another NVMe reader? (could the Sabrent NVMe reader be part of the issue)
• what if we use another source disk on same USB port? (could the source disk be part of the issue)
• what if the same target disk is installed on USB port with an NVMe reader? (could the internal board be part of the issue)
• what if we run with same source and target disks on another desktop machine? (could the Pi be part of the issue)
• what if we use another PNY CS1030 of same capacity as target disk? (could current device be the issue)
• what if we use another PNY CS1030 of bigger capacity as target disk? (could disk capacity be part of the issue)
• what if we use another model (than PNY CS1030) of target disk? (could current disk model be the issue)

All these questions are going to be quite time consuming (probably 1 or 2 days effort), but I feel like solving this might make disk provisioning significantly faster ; I'm not sure which priority we should give to this issue, since about 1h30 for the whole premium provisioning is not that a big deal especially since process is anyway very scalable (we can quite easily provision multiple devices at once should the need arise)

[rgaudin]

Two things:

• Provisioning this way is not about performance (to some extent).
• We need to know the provisioner performance/expected durations.

Why could me many things but hypothesis have little value here. We'll need to get to the bottom of this but it's not a blocker IMO.

My device is busy ATM but I'll run a 256GB one (I don't have larger disk at hand) to cross the single disk test.

[rgaudin]

That's the largest I can do with my 256GB disk

[rgaudin]

One thing that's important to acknowledge is that SSD and NVME have very large caches so writing profiles usually look like a very fast speed (that would reveal the limit of the rest of our stack) for a fixed (per drive model) number of GB then a immediate drop to the actual, nominal writing speed.

This matches your experience. Given your values, your disk would have a ~10GB cache.

Disks using those caches is not something we can do much about nor should care.
In the tool, we display a general writing speed that's an average over the whole writing process. It's incorrect in that it cannot be linearly applied to larger disk but that's simple and good enough for basic same-size comparisons.

I've tested my SAMSUNG MZALQ256HBJD-00BL2 (65€) disk and it appears to have a 32GB cache.

On the testbench I have an “INTEGRAL” INSSD256GM2242G3 (35€) that I bought because it was cheap but revealed to be very good (speed wise)

Write those 100GB at a constant, very fast speed. Those tests were on a 10Gbps USB bus so it might be the limiting factor.

I have thus tested to provision it and got indeed better results

375MB/s is the max we can actually get from my prior tests. This should be our target.

---

Regarding your disk (PNY CS1030 500G) ; either it's faulty or this model is just too cheap.

Given this test shows it should at least stay constant over 200GB, I would think this one is defective or worn out (it's not one of those used drives by chance?)

Please test it on your computer (use that HD Tune Pro trial).

• Good results? ➡️ incompatibility with our solution. We should source different models in the future.
• Poor results? Test another one (same model, same capacity)
  • Good ➡️ defective unit. replace it.
  • Poor ➡️ model is too low-end. We should source different models in the future.

⚠️ Same model of different capacity are actually different things. We shouldn't assume performance for 1TB based on 500GB ones for instance.

[kelson42]

@rgaudin Thank you very much for this analysis.

Just an idea: would be helpful to display average writing speed with an indicator having one of the following values: bad, pretty bad, average, pretty good, excellent? That way the user could have a hint if something is normal or not?

[benoit74]

Very nice knowledge acquired! Well done!

Please test it on your computer (use that HD Tune Pro trial).

I will need to find a decent Windows machine first (one with 10Gbps USB-C, or at least the 5Gbps of the Raspberry Pi). I will search for it.

Regarding your disk (PNY CS1030 500G) ; either it's faulty or this model is just too cheap.

Given this test shows it should at least stay constant over 200GB, I would think this one is defective or worn out (it's not one of those used drives by chance?)

This is a brand new disk.

[benoit74]

I did the 100G file benchmark on my PNY CS1030 brand new 500G (model is M280CS1030-500 mostly like ssd-tests but withtout the RB suffix).

Machine used is running Windows 10 and has USB3 SUPERSPEED ports, i.e. it is supposed to max at 5GB/s on the bus, meaning around 400 / 450 MB/s of real data transfer from what I found on the internet. Just like the Raspberry Pi 5.

At first, with a freshly formatted exFAT partition it looked like there was about 78G cache

I did a new test and here performance started to drop way faster, looking like there was only about 5 G cache.

I did a new test and got the same "bad 5g-cache performance".

I formatted the disk and got the "good 78G-cache performance" back.

I did a new test and got the "bad 5g-cache performance".

I switched to another external NVMe reader (same USB3 cable, same machine, same disk) and got the "bad 5g-cache performance".

I formatted the disk and got the "good 78G-cache performance" back.

I did a new test and got the "bad 5g-cache performance".

I switched to a 2T Samsung 990 PRO, also formatted in exFat, since quite long ago, and got consistent performance over the whole 100G file, with consistent read and write speeds of 436MB/s for both read and write (i.e. the USB SUPERSPEED is the limiting factor, as expected).

I did a 400G file benchmark test on this disk and got same consistent speeds over the whole test.

All this confirms that problem is the PNY CS1030 500G disk.

I will now need to go to Lausanne to test other PNY CS1030 500G disks to know if the model is problematic or if only my current disk is faulty. I will also test other PNY CS1030 capacities we have (200G and 1000G) and I will probably benchmark other disks we have in stock (even if they are not operating correctly once inside the Argon enclosure, would be interesting to know)

[rgaudin]

Thank you !

[kelson42]

Not sure about the status here? Is that our PNY disks are not trustworthy because they don't always behave the same?

[benoit74]

Status is still same than in my last paragraph in this issue.

I hoped to do tests today but forgot some "parts" needed to run my Windows machine to tests these disks so I will grab some disks with me to test them at home.

From first observations while provisioning it looks like all PNY CS1030 500G disks suffer the same slowness issue. And I'm now almost sure it has nothing to do with provisioner itself, it is just that there was no estimation of time needed to flash these disks before (with old provisioning process) and so it was not shocking (at least it did not worried me) to need time to complete.

[benoit74]

I just provisioned successfully the DACABQ device with provisioner 1.0.0 with preppers premium 2025-08 which is now 485.98G.

It took about 1h17 mins to complete, quite consistent with original timing reported in this issue given size increase.

It confirms the PNY CS1030 500G is a problem in terms of writing speed.

I will move on to PNY CS1030 250G and 1T to check.

[rgaudin]

@kelson42 FYI, when we say “is a problem in terms of writing speed” we mean for bulk writing as is the case for provisioning. For Hotspot usage, it's not an issue (123.13MB/s)