Odd packet reordering on proxy at network level.

  • aldie
  • aldie's Avatar Topic Author
  • Offline
  • Junior Member
More
29 Jul 2019 13:17 #1 by aldie
Hi, recently I got adventurous and stress tested my pimped ANSI seq interpreter with some complex coloured ascii graphics:

In about 10% of the cases the screen output is garbled, some text chunks just display before other ones:

Consider a large screen of about 9291 bytes (including TCP/IP & ethernet overhead), the mud sends TCP/telnet packets ABCDEFG, where B to F are of a
max allowed packet size of 1516 bytes, A is 1089 bytes and the final packet G is 622 bytes.

However, they are displayed as AGBCDEF (this is just one example, there are other reorderings when a different numbers of packets are send, but having the last packet being second seems to be a pattern). At first I assumed a buffering / sync issue within my ANSI interpreter, however I could track this down into Socket.js were onMessage is called with packets in order A, G, B, C, D, E, F.

I cannot totally exclude inflate.js or zlib somehow manage to reorder the contents by mixing up code tables, I didn't debug into the inflate code itself.

As I did not inflate by hand, I cannot totally ensure the WebSocket data reaches my browser in order A-G or AGBCDEF, but checking packet sizes this seems to be the case - or to be more precise - due to compression the ASCII packets AGBCDEF are compressed into two WebSocket packets (one with only A, and then GBCDEF) - at least this is what I read out of the debug messages I added/checked.

What really confuses/surprise me, is that the packet boundaries where the mixup occures do exactly match the TCP packet boundaries! As I do not
believe in random coincidences that smells more like a TCP / IP issue to me than one with zlib.

Looking at the network layers (as seen from the mud) all TCP packets are orderly sequenced, A is acknowledged first when A-G have already been send,
later (after the mud followed it up with G) packets B-G are also acknowledged - in order, no retransmissions... just a normal TCP windowing behaviour.

This TCP Acknowledgement handling is the same regardless if A-G are displayed in order or not (remember only something like 10% of mud printouts fail)

As I said, doing packet captures on the mud, I can ensure packets had been send in order w/o any retransmission; The issue also persists regardless which browser (tried current firefox, chrome, edge) I use. I cannot explain it from the proxy.js source you published on github.

And: When I run the proxy.js on a lab machine here (different node version, different zlib version, likely different os version - of course - totally different network delays and jitter), I cannot reproduce the issue regardless how hard I try.

Hence I claim there is a very odd bug in your node.js executables, network stack of the os on the proxy, or in a load balancer, firewall etc involved at the network layer.

I think it is almost impossible to debug/fix (esp. from here) with the current set of data. But I thought you may want to know nevertheless.
Maybe it can also be reproduced with some other mud sending 9-10k text output in one chunk. I'd assume it is not related to colours at all, but w/o complex MXP or ANSI codes involved you are not likely to get more than 2-3k of text output per player entered command.

I saw the effect also with less output, but not as often. I feel latency and packet reorderings at the IP layer are somehow involved.

Regards,
Aldie.

Please Log in or Create an account to join the conversation.

  • aldie
  • aldie's Avatar Topic Author
  • Offline
  • Junior Member
More
19 Sep 2020 22:38 #2 by aldie
The issue is still persistent. It does not occur when I run my own node wsproxy.js. It must be related to mudportals OS, node version, or the
AWS infrastructure.

See my post on the local wsproxy, if you want to run your own proxy, but cannot or don't want to run node.js.

Please Log in or Create an account to join the conversation.