Overview

You have an appliance that has been upgraded to 9.4.2 patch 1 and you noticed that everything is very slow with a lot of dropped packets. The appliance feels non-responsive and you may experience random intermittent lock-ups in the software appliances or hardware boxes (NG1xx, NG3xx, NG5xx) which require a power-cycle to be overcome.

The Generic Receive Offload article has a great explanation of what GRO is and how it works.

Solution

Version 9.4.2 patch 1

Software appliance running in any Hypervisor

Ensure that you are using Intel E1000 or VMXnet3 drivers for the adapters. If that doesn't fix the issue, please proceed with these steps:

This solution works until an upgrade or a downgrade since the ifupdown script gets rewritten.

1. Access Kerio Control's Shell Using SSH. Please copy-paste all the code of each step as a single line.
2. Verify the current state of the generic-receive-offload by running this line:
   hwinfo --network --short | tail -n +2 | awk '{print $1}' | xargs -i sh -c 'echo {}; ethtool -k {} | grep generic-receive'
3. Execute this line in SSH It disables GRO (generic receive offload) on all the ethernet interfaces (if you skip this step, make sure to reboot the appliance after the last step):
   hwinfo --network --short | tail -n +2 | awk '{print $1}' | xargs -I {} ethtool -K {} gro off
4. If that fixes the issue, to make the solution permanent, type the following line (and hit Enter) to allow making modifications to the filesystem and add the GRO reset upon all interface changes, and then make the filesystem read-only again:
   mount -o rw,remount /; printf "#/bin/bash\n\nhwinfo --network --short | tail -n +2 | awk '{print \$1}' | xargs -I {} ethtool -K {} gro off" > /usr/bin/gro_fix; chmod 0755 /usr/bin/gro_fix; if ! grep -q "gro_fix" "/usr/bin/ifupdown"; then printf "\n\nsetsid /usr/bin/gro_fix &2>1 $\n" >> /usr/bin/ifupdown; fi; mount -o ro,remount /
5. End result: GRO is disabled for all interfaces.

Hardware appliance:

This solution works until an upgrade or a downgrade since the ifupdown script gets rewritten.

First, reboot the device and don't run a SpeedTest as that makes the issue appear. Then follow the steps below to potentially resolve the problem:

1. Access Kerio Control's Shell Using SSH. Please copy-paste all the code of each step as a single line.
2. Verify the current state of the generic-receive-offload by running this line:
   hwinfo --network --short | tail -n +2 | awk '{print $1}' | xargs -i sh -c 'echo {}; ethtool -k {} | grep generic-receive'
3. Execute this line in SSH It enables GRO (generic receive offload) on all the ethernet interfaces and disables GRO for the software loopback, kvnet and VLAN interfaces (if you skip this step, make sure to reboot the appliance after the last step):
   hwinfo --network --short | tail -n +2 | awk '{print $1}' | grep -v '\.' | xargs -I {} ethtool -K {} gro on; hwinfo --network --short | tail -n +2 | awk '{print $1}' | grep '\.' | xargs -I {} ethtool -K {} gro off; ethtool -K lo gro off; ethtool -K kvnet gro off
4. If that fixes the issue, to make the solution permanent, type the following line (and hit Enter) to allow making modifications to the filesystem and add the GRO reset upon all interface changes, and then make the filesystem read-only again:
   mount -o rw,remount /; printf "#/bin/bash\n\nhwinfo --network --short | tail -n +2 | awk '{print $1}' | grep -v '\.' | xargs -I {} ethtool -K {} gro on\nhwinfo --network --short | tail -n +2 | awk '{print $1}' | grep '\.' | xargs -I {} ethtool -K {} gro off\nethtool -K lo gro off\nethtool -K kvnet gro off\n" > /usr/bin/gro_fix; chmod 0755 /usr/bin/gro_fix; if ! grep -q "gro_fix" "/usr/bin/ifupdown"; then printf "\n\nsetsid /usr/bin/gro_fix &2>1 $\n" >> /usr/bin/ifupdown; fi; mount -o ro,remount /
5. End result: GRO is disabled for lo, kvnet, and VLAN interfaces, enabled for all the hardware interfaces.

If you use VLANs a lot, please use these two lines instead of the ones in steps 3 and 4:

hwinfo --network --short | tail -n +2 | awk '{print $1}' | xargs -I {} ethtool -K {} gro on; ethtool -K lo gro off; ethtool -K kvnet gro off 

Permanent solution:

mount -o rw,remount /; printf "#/bin/bash\n\nhwinfo --network --short | tail -n +2 | awk '{print $1}' | xargs -I {} ethtool -K {} gro on\nethtool -K lo gro off\nethtool -K kvnet gro off\n" > /usr/bin/gro_fix; chmod 0755 /usr/bin/gro_fix; if ! grep -q "gro_fix" "/usr/bin/ifupdown"; then printf "\n\nsetsid /usr/bin/gro_fix &2>1 $\n" >> /usr/bin/ifupdown; fi; mount -o ro,remount /

End result: GRO is enabled for all but lo and kvnet interfaces.

Version 9.4.3 Patch 2 and later

From this version on, Kerio Control includes GRO switches for internet and LAN interfaces in the web admin UI. You can view more details in the Changing the GRO Settings guide.

However, the GRO values can also be tuned directly in the Kerio Control backend, via the GroMode parameter from each interface table in winroute.cfg. GroMode parameter accepts the following values, corresponding to each option in the UI

• 0 = Auto
• 1 = On
• 2 = Off

Please take extreme care with these adjustments, and always make sure to take a backup before proceeding with the below steps (Exporting and Importing configuration backups):

1. Proceed with Accessing Kerio Control's Shell Using SSH

2. Run the below command to see the GRO status of each interface:
   hwinfo --network --short | tail -n +2 | awk '{print $1}' | xargs -i sh -c 'echo {}; ethtool -k {} | grep generic-receive'

3. Afterwards, you can proceed with either of the below, depending on your preference:

   1. Disable GRO on a singular interface; for example, the VPN interface does not have a web admin UI switch for GRO, and it can be manipulated via the backend:
      /opt/kerio/winroute/tinydbclient "update Interfaces_v2 set GroMode=2 where Name="kvnet""

   2. Disable GRO system-wide:
      /opt/kerio/winroute/tinydbclient "update Interfaces_v2 set GroMode=2"

4. Run once again the command in step 2, which should show the status of GRO (which also persists a restart)