Players occasionally stuck between PlayerEnterConfigurationEvent and PlayerEnteredConfigurationEvent

[mfnalex]

Expected Behavior

Players should be able to switch servers without timing out

Actual Behavior

Sometimes players are randomly stuck inbetween the PlayerEnterConfigurationEvent and PlayerEnteredConfigurationEvent when switching between servers.

They do enter the "Reconfiguring"-phase clientwise, but Velocity seems to never actually put them into the reconfig phase, leading to them timing out. See below for what I tried for debugging.

Steps to Reproduce

This happens randomly for some (but not all) players, both vanilla users and people with mods. It doesn't happen everytime, just rarely.

For debugging, I have listened on Velocity to all the config events, and the PlayerEnterConfigurationEvent is fired. I then dug into client's NMS and added logging there too (primarily in (mojang mapped) ClientPacketListener#handleLogin and ClientPacketListener#handleConfigurationStart):

Client log when the timeout happened: https://mclo.gs/kU6CX7m
Proxy log when the timeout happened: https://mclo.gs/sDcebap

Client log of the same player, a few minutes earlier, when everything went fine: https://mclo.gs/DCRbJiY
Proxy log when it went fine: https://mclo.gs/blpS1hN

As you can see in the first two logs, the client enters the configuration phase as expected, and then sends the ServerboundConfigurationAcknowledgedPacket - this should result in Velocity firing the PlayerEnteredConfigurationEvent (if I'm not mistaken), which did not happen.

Plugin List

Velocity:

[09:16:35 INFO]: Plugins: velocity, floodgate, geyserutils, litebans, luckperms, nuvotifier, openaudiomc, primecore, primeappplugin, primediscord, primelaby, primesessions, spark, tebex, velocity-prometheus-exporter, voicechat

Backend Origin server:

[09:34:25 INFO]: ℹ Server Plugins (61):
[09:34:25 INFO]: Paper Plugins (34):
[09:34:25 INFO]:  - GeyserModelEngine, HuskSync, PlugManX, PrimeAffiliate, PrimeArcheology, PrimeArmorStands, PrimeAuction, PrimeBooster, PrimeCrateKeys, PrimeDisguise
[09:34:25 INFO]:  PrimeEnchants, PrimeEssentials, PrimeFarm, PrimeFishing, PrimeFriends, PrimeFurnace, PrimeHide, PrimeItems, PrimeJobs, PrimeKeyAll
[09:34:25 INFO]:  PrimeMarriage, PrimePerks, PrimePlotAdditions, PrimeQuestAPI, PrimeQuests, PrimeRedstone, PrimeShop, PrimeSlayerClient, PrimeStorage, PrimeTeleports
[09:34:25 INFO]:  PrimeTicketEvents, PrimeTutorial, PrimeWarps, PrimeWifi
[09:34:25 INFO]: Bukkit Plugins (27):
[09:34:25 INFO]:  - Chunky, CoreProtect, DecentHolograms, FastAsyncWorldEdit, floodgate, GeyserUtils, LibsDisguises, LiteBans, LuckPerms, ModelEngine
[09:34:25 INFO]:  MythicMobs, NBTAPI, OpenAudioMc, packetevents, PrimeAntiCrash, PrimeClans, PrimeCore, PrimeDebug, PrimeHomes, PrimeProfileProtect
[09:34:25 INFO]:  PrimeSettings, PrimeTitles, PrometheusExporter, ProtocolLib, ViaBackwards, ViaVersion, voicechat

Backend Destination server:

[09:16:54 INFO]: ℹ Server Plugins (75):
[09:16:54 INFO]: Paper Plugins (44):
[09:16:54 INFO]:  - GeyserModelEngine, HuskSync, Images, PlugManX, PrimeAffiliate, PrimeArcheology, PrimeArmorStands, PrimeAuction, PrimeBooster, PrimeCrateKeys
[09:16:54 INFO]:  PrimeCrates, PrimeDisguise, PrimeEnchants, PrimeEssentials, PrimeFarm, PrimeFishing, PrimeFriends, PrimeFurnace, PrimeGSBewerten, PrimeHide
[09:16:54 INFO]:  PrimeItems, PrimeItemValueCollector, PrimeJobs, PrimeKeyAll, PrimeLagFinder, PrimeMarriage, PrimeMinions, PrimeNPCs, PrimePerks, PrimePlotAdditions
[09:16:54 INFO]:  PrimeQuestAPI, PrimeQuests, PrimeRedstone, PrimeRewards, PrimeShop, PrimeSlayerClient, PrimeSpawner, PrimeStorage, PrimeTeleports, PrimeTicketEvents
[09:16:54 INFO]:  PrimeTrade, PrimeTutorial, PrimeWarps, PrimeWifi
[09:16:54 INFO]: Bukkit Plugins (31):
[09:16:54 INFO]:  - ArmorStandTools, Citizens, CoreProtect, DecentHolograms, FastAsyncWorldEdit, floodgate, GeyserUtils, LibsDisguises, LiteBans, LuckPerms
[09:16:54 INFO]:  ModelEngine, MythicMobs, NBTAPI, OpenAudioMc, packetevents, PlotSquared, PrimeAntiCrash, PrimeClans, PrimeCore, PrimeDebug
[09:16:54 INFO]:  PrimeField, PrimeHomes, PrimeProfileProtect, PrimeSettings, PrimeTitles, PrometheusExporter, ProtocolLib, Shopkeepers, ViaBackwards, ViaVersion
[09:16:54 INFO]:  voicechat

Velocity Version

Velocity 3.5.0-SNAPSHOT (git-e0db2566-b577)
(Also happened on latest 3.4.0)

All backend servers running on Paper version 1.21.11-126-main@3f5728e

Additional Information

This really only happens quite rarely, players can sometimes play just fine for two hours and switch servers, and then suddenly it happens on the next server switch. Not everyone is affected, and other people can also switch servers at the same time without problems.

All third-party plugins are on their latest versions, and our own plugins do not listen to any ConfigurationEvents (besides for logging/debugging atm), except for one, which listens on Velocity on PlayerConfigurationEvent - however I doubt that's the issue, as that event isn't even fired during the times when this issue happens.

Also none of the affected players have timeout issues while normal gameplay - they're all only affected when switching servers.

EDIT: We're using HAProxy protocol, if that's relevant. I can of course share configs / further information if needed.

Any help would be much appreciated, we're out of ideas

[RootException]

We are experiencing the same issue on our network and can confirm the behavior described here.

Setup
Velocity 3.4.0
Backend servers: PaperMC 1.21.7
Minecraft client versions: mostly 1.21.7-1.21.10
Proxy network with multiple backend servers (lobby, gameplay servers)
Resource pack handled via server plugins (e.g. ItemsAdder)

Observed behavior
Players sometimes get stuck in the "Reconfiguring..." phase when switching between servers through the proxy. From the client perspective they clearly enter the configuration phase, but they never actually complete it and eventually time out.

More details:

• This happens randomly and not for every player
• Both vanilla clients and modded clients (e.g. LabyMod) are affected
• For us it only happens during server switches, never during normal gameplay
• Players can switch servers many times successfully and then suddenly hit this issue on the next switch
• Other players may switch servers at the same time without any problem
• When this happens, the client shows "Reconfiguring..." and then disconnects with a timeout.
• From the client logs we can see the configuration/resource pack phase being triggered (resource manager reload including the server resource pack), but the connection never progresses further.

This strongly looks like the same situation described in this issue

Additional notes

• Our network also uses a server resource pack
• Backend servers use various plugins but nothing that directly manipulates the configuration phase

Because this only happens occasionally and randomly, it is extremely difficult to reproduce reliably, but we have multiple player reports confirming the same behavior.

[Teslicek]

#1251

[RaynLegends]

We are also having the exact same issue, we have 1.21.11 and 1.8.9 backend servers but have no server resource packs.

[abstractabuser]

Same, backend servers ranging from 1.8.8 - 1.21.11 with 2-3 resource packs.

[electronicboy]

as mentioned, there is already an issue open for this, issue si that I cannot reproduce it, generally need packet flow information from the proxy on what is being sent between the proxy and the server, and the proxy and the client; all I can guess at this point is that some communication is getting stuck somewhere, but it's hard to know where; knowing that some events do fire and some don't kinda helps, need to get back in tune with the logical flow here, however

[mfnalex]

as mentioned, there is already an issue open for this, issue si that I cannot reproduce it, generally need packet flow information from the proxy on what is being sent between the proxy and the server, and the proxy and the client; all I can guess at this point is that some communication is getting stuck somewhere, but it's hard to know where; knowing that some events do fire and some don't kinda helps, need to get back in tune with the logical flow here, however

Is there any setting we could enable to h elp with debugging? I have of course already tried to log packets but obviously it resulted in a gigabyte of "PlayerMovement" and "Particle" packets etc lol

[electronicboy]

There are no settings for diagnosing this. That's part of the headache, is that the only real thing that would be helpful at this point is packet logs to be able to validate what is/isn't being sent, PlayerEnteredConfigurationEvent is odd, because that is fired somewhat directly in response to the client sending a configuration_acknowledged packet

general flow here is that the target server sends the login success, we then start the jump over,

clear server state > PlayerEnterConfigurationEvent > send start_configuration; At this point the client should send us back a configuration_acknowledged which should trigger a few things, almost immediately, PlayerEnteredConfigurationEvent; what is the players active session handler at the point that the connection dies?

[mfnalex]

The issue seems to have been caused by our Anti-DDoS system, sorry for the trouble!