MTU, what difference does it make ?

Packet size, often referred to as MTU (Maximum Transmission Unit) is the greatest amount of data that can be transferred in one physical frame on the network. For Ethernet, the MTU is 1500 bytes, for PPPoE 1492, dial-up connections often use 576.
Each transmission unit consists of header and actual data. The actual data is referred to as MSS (Maximum Segment Size), which defines the largest segment of TCP data that can be transmitted. Essentially,
MTU=MSS + TCP & IP headers.
For the purposes of this article and all comparisons I'll make the following assumptions:

• MSS=MTU-40 <-- a standard 40 byte header (20 byte IP and 20 byte TCP)
• packets are not being fragmented
• no packet loss
• no router congestion

  
Packet size vs. latency
Let's examine a transfer of 1,500,000 bytes of data using different packet size over a T1 line (T1=1,544,000 bits/sec) using the following formula:
  ( MSS + header ) * 8 bits/byte---------------------------------- = latency (per hop)          1,544,000 bits/sec.
Then, using different MTU values, we can calculate the relevance of packet size to latency.
If MTU = 1500, then: (1460+40) * 8 / 1,544,000 = 7.772 ms delay per hop
If MTU = 576, then: (536+40) * 8 / 1,544,000 = 2.924 ms delay
Assuming a transfer over 10 hops, the 1500 MTU would wield 77.72 ms delay, while a 576 MTU would take 29.24 ms to transfer over a T1 line.
It is obvious then that smaller packets will be transmitted faster, simply because of the throughput limitation of the line. It's however somewhat moot point in large transfers, all that's illustrated above is the fact that transferring more data takes more time :)
 
Throughput vs. packet size
Using the same formula from our previous example, let's assume we need to transfer 1 MByte file over the same T1 line.
1MByte = 1024 KB = 1,048,576 bytes.
If MTU = 1500, then: (1460+40) * 8 / 1,544,000 = 7.772 ms delay per hop1 MByte / MSS = 1,048,576 bytes / 1460 = 718.2, or effectively 719 packets to transfer 1 MByte.Then, to transfer 1Mbyte: 719 packets * 7.772 ms delay per hop = 5588.068 ms, or 5.588 seconds per hop.If we are transferring our 1 MByte file over 10 hops, it will ideally take us:(1st packet * 10 hops * 7.772ms delay) + 718 * 7.772 = 5.658 seconds.

If MTU = 576, then: (536+40) * 8 / 1,544,000 = 2.924 ms delay per hop.1 MByte / MSS = 1,048,576 bytes / 536 = 1956.3, or effectively 1957 packets to transfer 1 MByte.Then, to transfer 1 MByte: 1957 packets * 2.924 ms delay per hop = 5722.268 ms, or 5.722 seconds per hop.If we are transferring our 1 MByte file over the same 10 hops:(1st packet *10 hops * 2.924ms delay) + 1956 * 2.924 = 5.748 sec.

The difference comes from the fact that when using larger packets the overhead is smaller. To transfer 1 MByte, if using MTU of 1500 there are 719 * 40 = 28,760 bytes of overhead, while if using MTU of 576 1957 * 40 = 78,280 bytes, additional 49,520 bytes of headers transferred each MByte. For our 10-hop transfer, the additional overhead accounts for just a fraction of a second, however it makes a difference if you consider large transfers over long periods of time. This difference is a also higher in practice, considering TCP options and the fact that modern TCP/IP implementations tend to use larger headers (additional 12 bytes header space for Timestamps for example). Also, the above example uses best case scenario, not considering higher latency and congestion.
 
Final Thoughts
Generally, it's logical to assume larger packets are better, because of all the following factors:

• network - reduced number of headers, as illustrated above
• routers - less routing decisions
• clients - less protocol processing and device interrupts

However, if pure throughput is not the ultimate goal, smaller packets might be more "responsive" since they take less time to travel throughout the network. That effect might be preferred in some applications and online gaming, at the expense of throughput.
Ultimately, packet size should be decided based on the type of the desired result, considering the underlying network as well, to avoid negative factors such as fragmentation of packets. Still one has to realize the fact that larger packets will still transmit more useful data than smaller packets, and that there is no single "best" solution for all applications.
Regards,
Philip Filipov

http://www.speedguide.net/articles/mtu-what-difference-does-it-make--111