Are you backing up your AIX systems over Virtual Ethernet adapters? Of course you are, who isn’t right? Are your backup server and clients on the same physical POWER system? You are most likely backing up over Virtual Ethernet to another AIX LPAR that is running your enterprise backup software, such as TSM or Legato Networker for example. And you probably have a dedicated private virtual network (and adapters) on both the clients and the server to handle the traffic for the nightly backups. The next question is, have you tuned your Virtual Ethernet adapters?

 

There are several tips available for tuning your Virtual Ethernet adapters for better performance on AIX. These tips include changing settings such as MTU size, TCP window sizes, enabling largesend, etc. I highly recommend the following blog posts from Anthony English and Nigel Griffiths on this subject:

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/AIXDownUnder/entry/turbo_charge_inter_lpar_communications_using_jumbo_frames1?lang=en

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/powervm_virtual_ethernet_speed_is_often_confused_with_vios_sea_ive_hea_speed?lang=en

 

OK, so you got everything humming along nicely, your backups are flying over the virtual network (across the POWER hypervisor) and everybody is happy. After a period of time, you notice that the backups have started to “slow down”. They are taking longer to finish. The overall throughput of a backup drops. Some backups start in the evening around 9pm and are still running the next morning at 7am! In some cases you need to kill the backups or even reboot the backup server LPAR for things to return to normal.

 

“What is going on!?” You cry.

 

Well, there are a number of reasons why this could be happening. For example, your shared processor pool may be overwhelmed during the backup window. As we know, Virtual Ethernet adapters require CPU to do their work. If the CPU pool is running low on available CPU resources, this could contribute to the problem. And of course there could be tuning issues with the Virtual Ethernet adapters or the AIX OS in general. Or there may be issues with other pieces of the infrastructure, like network and SAN switches, adapters, etc. Perhaps there’s an issue with the applications and/or databases on the AIX systems? They often have their own mechanisms/tools for backing up their data to your enterprise backup software. Is the backup server sized to cope with the load i.e. CPU, memory, disk layout and I/O, sufficient tape drives, disk storage pools, etc?

 

So assuming you’ve checked all of the above (and more), then perhaps you’ve hit a problem that I encountered recently. In my particular case, backups “over the hypervisor” were slowing down, without any discernible cause. Initially the backups would be “very fast” but after a month or so, things would start to slow down dramatically.

 

We noticed that there were very large (and increasing) values for “Packets Dropped”, “Hypervisor Send/Receive Failures” and “No Resource Errors” in the output from the netstat –v command.                              

                                                                       

ETHERNET STATISTICS (ent1) :                                           

Device Type: Virtual I/O Ethernet Adapter (l-lan)                      

Hardware Address: 41:ba:13:e7:25:0b                                    

Elapsed Time: 42 days 4 hours 3 minutes 34 seconds                    

                                                                       

Transmit Statistics:       Receive Statistics:                         

--------------------       -------------------                         

Packets: 5978589961        Packets: 26139832411                        

Bytes: 779465989202        Bytes: 711051516630458                      

Interrupts: 0              Interrupts: 6804561727                      

Transmit Errors: 0         Receive Errors: 0                           

Packets Dropped: 0         Packets Dropped: 86012309               

...                                                                                             

Max Collision Errors: 0    No Resource Errors: 46113807           

...                                                                       

Hypervisor Send Failures: 0                                            

  Receiver Failures: 0                                                  

  Send Errors: 0                                                       

Hypervisor Receive Failures: 46113807                             

                                                                       

After some discussion with IBM AIX support, we discovered that would should increase some of the buffer sizes for our Virtual Ethernet adapter (the entX device). This would alleviate the no resource issues we’d been experiencing. Looking at the output from the netstat -v command, we also noticed that the Medium, Large and Huge buffers had all reached their maximum values in the past.                                                           

                                                                       

...

Receive Information                                                    

  Receive Buffers                                                      

    Buffer Type              Tiny    Small   Medium    Large     Huge  

    Min Buffers               512      512      128       24       24  

    Max Buffers              2048     2048      256       64       64  

    Allocated                 513      535      148       28       64  

    Registered                512      510      127       24       13  

                                                                       

    History                                                            

      Max Allocated           576      951      256       64       64  

      Lowest Registered       502      502       64       12       11  

...

                                                                   

The advice from IBM support was to increase these buffers using the chdev command (they also advised that we should reboot for the changes to take effect): 

                                                                       

# chdev -l ent1 -a min_buf_medium=512 -a max_buf_medium=1024 –a min_buf_large=96 -a max_buf_large=256 -a min_buf_huge=96 –a max_buf_huge=128 -P             

                                       

# shutdown -Fr

 

Since implementing this tuning change (to the adapter on the backup server), we have not had a repeat of the problem. We will continue to monitor the performance and I’ll be sure to let everyone know if we have further issues.

Tags:  networker ethernet min_buf_huge=96 netstat chris_gibson max_buf_large=256 chdev virtual slow backup -l min_buf_large=96 –a min_buf_medium=512 adapter max_buf_huge=128 hypervisor tsm max_buf_medium=1024 -a ent1 aix