Yeah, it's a known issue, and one high on my ToDo list. It's mainly noticeable when scrolling the System Menu or Plugin deep edit screens, right?
It has more to do with the LCD display than the encoder. And more software than hardware. Hardware-wise, the display and ADC share a SPI bus. The baud rate for the bus is set lower than ideal for the sake of the ADC. We could explore increasing that, but as it is, that will affect the ADC resolution and doesn't seem to improve the latency much.
From a software perspective, it could probably be made to be more responsive by improving code inefficiencies.
The display area is broken in to 7 horizontal "zones" so only a zone with changed data is refreshed. That does make it a bit more responsive that it was when refreshing the whole screen. It's likely possible to further target display refreshes.
There is also a sort operation which happens every time you twist the encoder on the System Menu or plugin deep edit screens. That sort list could be cached. There are similar other calculations which could probably be done once and cached.
Another improvement could be made to the encoder handler. Currently, with each detent (30 degree rotation), a handler is called and must complete before the next is called. For large/quicker rotations, what you'd like it to do, is skip ahead without handling each in-between selection. That would make it feel much more responsive.