Odd packet reordering on proxy at network level.

  • aldie
  • aldie's Avatar Topic Author
  • Offline
  • Junior Member
More
29 Jul 2019 13:17 #1 by aldie
Hi, recently I got adventurous and stress tested my pimped ANSI seq interpreter with some complex coloured ascii graphics:

In about 10% of the cases the screen output is garbled, some text chunks just display before other ones:

Consider a large screen of about 9291 bytes (including TCP/IP & ethernet overhead), the mud sends TCP/telnet packets ABCDEFG, where B to F are of a
max allowed packet size of 1516 bytes, A is 1089 bytes and the final packet G is 622 bytes.

However, they are displayed as AGBCDEF (this is just one example, there are other reorderings when a different numbers of packets are send, but having the last packet being second seems to be a pattern). At first I assumed a buffering / sync issue within my ANSI interpreter, however I could track this down into Socket.js were onMessage is called with packets in order A, G, B, C, D, E, F.

I cannot totally exclude inflate.js or zlib somehow manage to reorder the contents by mixing up code tables, I didn't debug into the inflate code itself.

As I did not inflate by hand, I cannot totally ensure the WebSocket data reaches my browser in order A-G or AGBCDEF, but checking packet sizes this seems to be the case - or to be more precise - due to compression the ASCII packets AGBCDEF are compressed into two WebSocket packets (one with only A, and then GBCDEF) - at least this is what I read out of the debug messages I added/checked.

What really confuses/surprise me, is that the packet boundaries where the mixup occures do exactly match the TCP packet boundaries! As I do not
believe in random coincidences that smells more like a TCP / IP issue to me than one with zlib.

Looking at the network layers (as seen from the mud) all TCP packets are orderly sequenced, A is acknowledged first when A-G have already been send,
later (after the mud followed it up with G) packets B-G are also acknowledged - in order, no retransmissions... just a normal TCP windowing behaviour.

This TCP Acknowledgement handling is the same regardless if A-G are displayed in order or not (remember only something like 10% of mud printouts fail)

As I said, doing packet captures on the mud, I can ensure packets had been send in order w/o any retransmission; The issue also persists regardless which browser (tried current firefox, chrome, edge) I use. I cannot explain it from the proxy.js source you published on github.

And: When I run the proxy.js on a lab machine here (different node version, different zlib version, likely different os version - of course - totally different network delays and jitter), I cannot reproduce the issue regardless how hard I try.

Hence I claim there is a very odd bug in your node.js executables, network stack of the os on the proxy, or in a load balancer, firewall etc involved at the network layer.

I think it is almost impossible to debug/fix (esp. from here) with the current set of data. But I thought you may want to know nevertheless.
Maybe it can also be reproduced with some other mud sending 9-10k text output in one chunk. I'd assume it is not related to colours at all, but w/o complex MXP or ANSI codes involved you are not likely to get more than 2-3k of text output per player entered command.

I saw the effect also with less output, but not as often. I feel latency and packet reorderings at the IP layer are somehow involved.

Regards,
Aldie.

Please Log in or Create an account to join the conversation.