Why UDP has a TOTAL LENGTH field but TCP does not!?

[Christian Seberino]

I noticed that UDP has a total length field (bytes 5 & 6)
but TCP does NOT! (TCP has a header length field only
which is half of byte 13)

I tried to find the answer to this...the only thing I got
is that sometimes people pad out UDP payloads to make
it to the word boundary.

Why wouldn't this happen on TCP packets too? Must
UDP packets be aligned with word boundaries but NOT TCP
packets?

Chris


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 481 bytes
Desc: This is a digitally signed message part
Url : http://www.kernel-panic.org/pipermail/kplug-list/attachments/20051118/c0d259ea/attachment.pgp

[Michael O'Keefe]

Post by Christian Seberino
I noticed that UDP has a total length field (bytes 5 & 6)
but TCP does NOT! (TCP has a header length field only
which is half of byte 13)

a UDP packet is the entire message. Each UDP packet is self contained.
TCP packets are part of a stream, a fragment of the whole stream. So
nobody knows how long the stream is.

[Christian Seberino]

Yes but wouldn't you need to know the size of *each* TCP packet?....


I assume you don't need an explicit TCP length field because
you can calculate the TCP packet length by substracting the TCP header
length from the *IP packet* length. (IP packets *do* have length
field.)

Since you can do the same for UDP it still appears that the UDP length
field is redundant and unnecessary.

Chris

Post by Michael O'Keefe

Post by Christian Seberino
I noticed that UDP has a total length field (bytes 5 & 6)
but TCP does NOT! (TCP has a header length field only
which is half of byte 13)

a UDP packet is the entire message. Each UDP packet is self contained.
TCP packets are part of a stream, a fragment of the whole stream. So
nobody knows how long the stream is.
--
I like less more or less less than |Work:[phone] / |
more. UNIX-live it,love it,fork() it |Fax :[phone] /_p_|
My views are MINE ALONE, blah, blah, |Home:[phone] \`O'|
blah, yackety yack - don't come back |Fax :+1 858 _/_\|_,

--
_______________________________________

Christian Seberino, Ph.D.
SPAWAR Systems Center San Diego
Code 2872
49258 Mills Street, Room 158
San Diego, CA 92152-5385
U.S.A.

Phone: [phone]
Fax : [phone]
Email: ***@spawar.navy.mil
_______________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 481 bytes
Desc: This is a digitally signed message part
Url : http://www.kernel-panic.org/pipermail/kplug-list/attachments/20051118/bb75063c/attachment.pgp

[Michael O'Keefe]

Post by Christian Seberino
Yes but wouldn't you need to know the size of *each* TCP packet?....

No. An entire IP packet (after defragmentation) is given to to the TCP
stack. The stack then looks at the header to find out which 'socket' to
put the payload into, and then pushes the payload onto the end of the
socket buffer.

Post by Christian Seberino
I assume you don't need an explicit TCP length field because
you can calculate the TCP packet length by substracting the TCP header
length from the *IP packet* length. (IP packets *do* have length
field.)

Only the IP stack needs the length so it can generate a buffer big
enough to give the 'data' to the TCP stack. Once this data gets to the
TCP stack, the TCP stack only cares about everything after the header in
the buffer. It doesn't care how big it is. UDP on the other hand does
care, becoz each packet put into the socket buffer is self contained,
and it's all or nothing with UDP.

Post by Christian Seberino
Since you can do the same for UDP it still appears that the UDP length
field is redundant and unnecessary.

No, that would assume that UDP has to run over IP, which it doesn't

[Christian Seberino]

Michael

Thanks!

Post by Michael O'Keefe
No. An entire IP packet (after defragmentation) is given to to the TCP
stack. The stack then looks at the header to find out which 'socket' to
put the payload into, and then pushes the payload onto the end of the
socket buffer.

Doesn't the IP stack peel off it's own IP header before giving
the IP packet payload to the TCP stack? Otherwise, TCP stack
would have to know about IP details which violates encapsulation right?

In short, every layer of network stack peel off *it's own* header as
packet travels up to application layer IIRC.

Post by Michael O'Keefe
Once this data gets to the
TCP stack, the TCP stack only cares about everything after the header in
the buffer. It doesn't care how big it is.

I'm still not clear how come TCP stack doesn't need to care how big each
TCP segment is. Sure it ultimately only cares about joining all
segments in correct order but how can it do *that* without knowing where
each segment begins and ends!?!?!

Post by Michael O'Keefe
it still appears that the UDP length

Post by Christian Seberino
field is redundant and unnecessary.

No, that would assume that UDP has to run over IP, which it doesn't

What about TCP and UDP's usage of a "pseudo IP header" when calculating
their checksums? Those "pseudo IP headers" contain the sender and
receiver's IP addresses!? The TCP and UDP checksum fields
therefore seems to lock-in TCP and UDP to use only IP!?!? (Very
Microsoftish :)

Chris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 481 bytes
Desc: This is a digitally signed message part
Url : http://www.kernel-panic.org/pipermail/kplug-list/attachments/20051118/ed82b569/attachment.pgp

[Tom Gal]

NO WAY!

Post by Michael O'Keefe
Michael
Thanks!

Post by Michael O'Keefe
No. An entire IP packet (after defragmentation) is given to to the TCP
stack. The stack then looks at the header to find out which 'socket' to
put the payload into, and then pushes the payload onto the end of the
socket buffer.

Doesn't the IP stack peel off it's own IP header before giving
the IP packet payload to the TCP stack? Otherwise, TCP stack
would have to know about IP details which violates encapsulation right?

Then what do you have after you remove the TCP headers at your
destination? NOTHING? Just because a protocol doesn't know the
internals of another protocol doesn't mean it would do anything about
it.

BTW encapsulation and naive protocols are quite passe. The most
optimized performance comes from smart implementations that break just
that rule. Have you heard of RTP....there was a great paper that
Henning Schulzerene of wrote that prefaced that style of prototocol
out there some where.

-Tom

[seberino at spawar.navy.mil ()]

Post by Tom Gal
Then what do you have after you remove the TCP headers at your
destination? NOTHING? Just because a protocol doesn't know the
internals of another protocol doesn't mean it would do anything about
it.

You seem to think headers are the only things in packets. After you
remove a header you still have the payload.

Post by Tom Gal
BTW encapsulation and naive protocols are quite passe.

Tell that to all OOP programmers in the world. They are all
striving for encapsulation by using objects.

Post by Tom Gal
The most
optimized performance comes from smart implementations that break just
that rule.

I agree 100%. Sure you can *implement* something with optimizations
that break encapsulation
but the TCP/IP specs strive for simplicity with encapsulation.

Chris

[Stewart Stremler]

begin quoting ***@spawar.navy.mil as of Fri, Nov 18, 2005 at 12:07:54PM -0800:
[attribution criminally removed prior to this reply]

Post by seberino at spawar.navy.mil ()

Post by Tom Gal
BTW encapsulation and naive protocols are quite passe.

Tell that to all OOP programmers in the world. They are all
striving for encapsulation by using objects.

Any decent programmer, not just those who practice OOP. Encapsulation,
data-hiding, and modularization are not new concepts. (OO isn't even
that new, come to think of it.)

-Stewart "Remember the 8 fallacies of network programming!" Stremler

[Tom Gal]

Post by seberino at spawar.navy.mil ()

Post by Tom Gal
Then what do you have after you remove the TCP headers at your
destination? NOTHING? Just because a protocol doesn't know the
internals of another protocol doesn't mean it would do anything about
it.

You seem to think headers are the only things in packets. After you
remove a header you still have the payload.

right....I think I may have read your statement backwards and thought
you were stripping that info out before it could be used......Sorry.

Post by seberino at spawar.navy.mil ()

Post by Tom Gal
BTW encapsulation and naive protocols are quite passe.

Tell that to all OOP programmers in the world. They are all
striving for encapsulation by using objects.

That has nothing to do with network protocol design. Makes sense for
programming clearly.

Post by seberino at spawar.navy.mil ()

Post by Tom Gal
The most
optimized performance comes from smart implementations that break just
that rule.

I agree 100%. Sure you can *implement* something with optimizations
that break encapsulation
but the TCP/IP specs strive for simplicity with encapsulation.

http://nms.csail.mit.edu/6829-papers/alf.pdf

is the paper I was referencing.
We've been using these ideals as core tenents in designing IETF
protocol for a while now. Of course everyone can disagree with
anything but I like the points they make.

-Tom

[Andrew Lentvorski]

Post by Tom Gal
http://nms.csail.mit.edu/6829-papers/alf.pdf
is the paper I was referencing.
We've been using these ideals as core tenents in designing IETF
protocol for a while now. Of course everyone can disagree with
anything but I like the points they make.

Hmmm, interesting paper. However, I was hoping for something a little
more helpful than "application framing is the most efficient" and
"minimize presentation processing".

While this advice is probably true, it is a bit of a platitude. We
accept slight inefficiencies in many engineering fields in return for
"decomposability". Unfortunately, application framing is almost always
minimally decomposable.

The paper really does not provide much valuable advice for how to
balance the pull between application framing and protocol
decomposability. I was somewhat disappointed.

-a

[Michael O'Keefe]

Post by Christian Seberino

Post by Michael O'Keefe
No. An entire IP packet (after defragmentation) is given to to the TCP
stack. The stack then looks at the header to find out which 'socket' to
put the payload into, and then pushes the payload onto the end of the
socket buffer.

Doesn't the IP stack peel off it's own IP header before giving
the IP packet payload to the TCP stack? Otherwise, TCP stack
would have to know about IP details which violates encapsulation right?
In short, every layer of network stack peel off *it's own* header as
packet travels up to application layer IIRC.

You're right, I meant to say ... an entire IP packet payload is given to
the TCP stack...

Post by Christian Seberino

Post by Michael O'Keefe
Once this data gets to the
TCP stack, the TCP stack only cares about everything after the header in
the buffer. It doesn't care how big it is.

I'm still not clear how come TCP stack doesn't need to care how big each
TCP segment is. Sure it ultimately only cares about joining all
segments in correct order but how can it do *that* without knowing where
each segment begins and ends!?!?!

the TCP stack just does a sizeof(buf) to find out how much data it has
to play with with this one packet (or the fucntion call to enter the
stack is tcp(buf,count), I don't know without looking)

Post by Christian Seberino

Post by Michael O'Keefe

it still appears that the UDP length field is redundant and unnecessary.

No, that would assume that UDP has to run over IP, which it doesn't

What about TCP and UDP's usage of a "pseudo IP header" when calculating
their checksums? Those "pseudo IP headers" contain the sender and
receiver's IP addresses!? The TCP and UDP checksum fields
therefore seems to lock-in TCP and UDP to use only IP!?!? (Very
Microsoftish :)

[seberino at spawar.navy.mil ()]

Post by Michael O'Keefe
the TCP stack just does a sizeof(buf) to find out how much data it has
to play with with this one packet (or the fucntion call to enter the
stack is tcp(buf,count), I don't know without looking)

If it was that easy then how come UDP couldn't do the same thing?
How come UDP needs an explicit field then?

Post by Michael O'Keefe

Post by Christian Seberino

Post by Michael O'Keefe

it still appears that the UDP length field is redundant and unnecessary.

No, that would assume that UDP has to run over IP, which it doesn't

What about TCP and UDP's usage of a "pseudo IP header" when calculating
their checksums? Those "pseudo IP headers" contain the sender and
receiver's IP addresses!? The TCP and UDP checksum fields
therefore seems to lock-in TCP and UDP to use only IP!?!? (Very
Microsoftish :)

Did you read this part? I've been wondering about this for a while.
Seems IP pseudoheader breaks encapsulation.

cs

[Stewart Stremler]

Post by seberino at spawar.navy.mil ()

Post by Michael O'Keefe
the TCP stack just does a sizeof(buf) to find out how much data it has
to play with with this one packet (or the fucntion call to enter the
stack is tcp(buf,count), I don't know without looking)

If it was that easy then how come UDP couldn't do the same thing?
How come UDP needs an explicit field then?

It probably seemed like a good idea at the time. :)

You're looking for a rationale document. I haven't found one yet.

It probably has to do with the next layer up -- with UDP, the size of
the datagram would be important. With TCP, it's meaningless (to that
layer).

[snip]

Post by seberino at spawar.navy.mil ()
Did you read this part? I've been wondering about this for a while.
Seems IP pseudoheader breaks encapsulation.

It's allowed to have coupling[1] between layers. It's just a bad idea to
couple *across* layers.

-Stewart "{TCP/UDP}/IP is not a *clean* system -- it just works" Stremler

[1] Which should still be minimized, nevertheless.

[Andrew Lentvorski]

Post by Stewart Stremler
-Stewart "{TCP/UDP}/IP is not a *clean* system -- it just works" Stremler

I disagree. TCP/IP is really pretty close to both minimal and clean
given the problem it is solving (reliable transport across unreliable
links). Anyone who starts out designing a protocol which is "simpler"
than TCP eventually realizes that they wound up designing 90% of
TCP--generally badly.

The fact that TCPv6 (which had pretty much a blank slate) does not throw
out much of the fundamental architecture of TCPv4 shows that TCPv4 made
pretty much the right decisions.

-a

[Stewart Stremler]

Post by Andrew Lentvorski

Post by Stewart Stremler
-Stewart "{TCP/UDP}/IP is not a *clean* system -- it just works" Stremler

I disagree. TCP/IP is really pretty close to both minimal and clean
given the problem it is solving (reliable transport across unreliable
links). Anyone who starts out designing a protocol which is "simpler"
than TCP eventually realizes that they wound up designing 90% of
TCP--generally badly.

That's not quite how I meant it. By clean, I was trying to refer to
conceptual encapsulation -- the OSI model is clean, in that it's all
nicely layered and labelled... but it's TCP/IP that gets *used*.

The slightly messy abstractions in the IP protocols are more than made up
for by their practical simplicity.

Post by Andrew Lentvorski
The fact that TCPv6 (which had pretty much a blank slate) does not throw
out much of the fundamental architecture of TCPv4 shows that TCPv4 made
pretty much the right decisions.

I think we're in violent agreement.

At least, I'm not at all interested in a pissing contest over _this_ topic.

-Stewart "Internyms are space constrained and not meant for serious co" Stremler

[Michael O'Keefe]

Post by seberino at spawar.navy.mil ()

Post by Michael O'Keefe
the TCP stack just does a sizeof(buf) to find out how much data it has
to play with with this one packet (or the fucntion call to enter the
stack is tcp(buf,count), I don't know without looking)

If it was that easy then how come UDP couldn't do the same thing?
How come UDP needs an explicit field then?

Becoz a UDP packet is not "complete" unless it has all the payload.
If a UDP packet arrives, and the length says 10-octets, but there were
only 9, throw the packet away

[Andrew Lentvorski]

Post by Michael O'Keefe
Becoz a UDP packet is not "complete" unless it has all the payload.
If a UDP packet arrives, and the length says 10-octets, but there were
only 9, throw the packet away

No. UDP does not *require* that the packet be thrown away if it does
not match checksum or length. Lots of implementations do that, but it
is not required.

In fact, it would sometimes be better if UDP did *not* necessarily throw
the packet out. Streaming audio and video have their own error
correction that could make use of partial information.

TCP as used for things like streaming is an absolutely *wretched*
protocol match. TCP requires perfection when the overarching
application may not need it.

-a

[Michael O'Keefe]

Post by Andrew Lentvorski

Post by Michael O'Keefe
Becoz a UDP packet is not "complete" unless it has all the payload.
If a UDP packet arrives, and the length says 10-octets, but there were
only 9, throw the packet away

No. UDP does not *require* that the packet be thrown away if it does
not match checksum or length. Lots of implementations do that, but it
is not required.

That's true. For checksum, 0x0000 is reserved for no-checksum-required,
but most (all ?) stacks don't pass on partial packets.

[seberino at spawar.navy.mil ()]

Stewart posted a snippet from an RFC on this thread which seems
to imply that IP protocol provides mechanism to determine TCP length.

Chris

[seberino at spawar.navy.mil ()]

Post by Michael O'Keefe

Post by seberino at spawar.navy.mil ()
How come UDP needs an explicit field then?

Becoz a UDP packet is not "complete" unless it has all the payload.
If a UDP packet arrives, and the length says 10-octets, but there were
only 9, throw the packet away

UDP verifys integrity and correct length of a UDP packet
by the checksum. Length field not required to make
sure "complete".

Still seems like UDP length field serves no purpose that
can't be done some other way.

Chris

[Andrew Lentvorski]

Post by seberino at spawar.navy.mil ()
UDP verifys integrity and correct length of a UDP packet
by the checksum. Length field not required to make
sure "complete".
Still seems like UDP length field serves no purpose that
can't be done some other way.

You assume that UDP must go over IP. That is not true.

UDP must be self-sufficient irrespective of any underlying protocol
(like IP).

Requiring a checksum computation on each packet simply to verify length
integrity would be computationally painful. Most UDP packets that I see
in Ethereal never bother with checksums.

-a

[Stewart Stremler]

begin quoting Christian Seberino as of Fri, Nov 18, 2005 at 11:39:15AM -0800:
[snip]

Post by Christian Seberino
In short, every layer of network stack peel off *it's own* header as
packet travels up to application layer IIRC.

It's its, not it's.

Post by Christian Seberino
I'm still not clear how come TCP stack doesn't need to care how big each
TCP segment is. Sure it ultimately only cares about joining all
segments in correct order but how can it do *that* without knowing where
each segment begins and ends!?!?!

[snip]

Post by Christian Seberino
What about TCP and UDP's usage of a "pseudo IP header" when calculating
their checksums? Those "pseudo IP headers" contain the sender and
receiver's IP addresses!?

Ultimate addresses, yes. And the psuedo-header also provides a size.

Post by Christian Seberino
The TCP and UDP checksum fields
therefore seems to lock-in TCP and UDP to use only IP!?!? (Very
Microsoftish :)

According to RFC 793, page 51:

Any lower level protocol will have to provide the source address,
destination address, and protocol fields, and some way to
determine the "TCP length", both to provide the functional
equivlent service of IP and to be used in the TCP checksum.

The TCP protocol requires certain features of the protocol it runs over.
You're not locked into IP -- an even if you were "locked in", you can
route IP over other protocols, and many folks have done so.

-Stewart "We normally talk about TCP/IP as an indivisible tuple" Stremler

[Michael O'Keefe]

Post by Christian Seberino

Post by Michael O'Keefe
No. An entire IP packet (after defragmentation) is given to to the TCP
stack. The stack then looks at the header to find out which 'socket' to
put the payload into, and then pushes the payload onto the end of the
socket buffer.

Doesn't the IP stack peel off it's own IP header before giving
the IP packet payload to the TCP stack? Otherwise, TCP stack
would have to know about IP details which violates encapsulation right?
In short, every layer of network stack peel off *it's own* header as
packet travels up to application layer IIRC.

You're right, I meant to say ... an entire IP packet payload is given to
the TCP stack...

Post by Christian Seberino

Post by Michael O'Keefe
Once this data gets to the
TCP stack, the TCP stack only cares about everything after the header in
the buffer. It doesn't care how big it is.

I'm still not clear how come TCP stack doesn't need to care how big each
TCP segment is. Sure it ultimately only cares about joining all
segments in correct order but how can it do *that* without knowing where
each segment begins and ends!?!?!

the TCP stack just does a sizeof(buf) to find out how much data it has
to play with with this one packet (or the fucntion call to enter the
stack is tcp(buf,count), I don't know without looking)

Post by Christian Seberino

Post by Michael O'Keefe

it still appears that the UDP length field is redundant and unnecessary.

No, that would assume that UDP has to run over IP, which it doesn't

What about TCP and UDP's usage of a "pseudo IP header" when calculating
their checksums? Those "pseudo IP headers" contain the sender and
receiver's IP addresses!? The TCP and UDP checksum fields
therefore seems to lock-in TCP and UDP to use only IP!?!? (Very
Microsoftish :)

That's really a function of TCP/IP and UDP/IP becoz the IP header can
get mangled (but still have a valid IP checksum) and the packet can wind
up on another machine. But by including the addresses in 2 check
checksums, it might pass the IP verification , but it won't pass the TCP
verification.
But if you send TCP/UDP over another transport, you don't have to use IP
addresses in the checksum calculation. You can really provide whatever
"prefix" you like (or none at all) to the checksum calculation, so long
as the receiver on the other end knows to do the same thing.

[Andrew Lentvorski]

Post by Michael O'Keefe
That's really a function of TCP/IP and UDP/IP becoz the IP header can
get mangled (but still have a valid IP checksum) and the packet can wind
up on another machine. But by including the addresses in 2 check
checksums, it might pass the IP verification , but it won't pass the TCP
verification.
But if you send TCP/UDP over another transport, you don't have to use IP
addresses in the checksum calculation. You can really provide whatever
"prefix" you like (or none at all) to the checksum calculation, so long
as the receiver on the other end knows to do the same thing.

For a full rationale about decisions concerning TCP stacks, I recommend
reading:

"The Design and Implementation of the FreeBSD Operating System" by by
Marshall Kirk McKusick and George V. Neville-Neil

This discusses how you implement a stack, what gets ignored, and what
gets optimized.

In addition, TCP/IP Illustrated Vol 1 and 2 should touch on these kinds
of issues as well.

My standard response to TCP/IP questions tends to be "Go read." There
are so many dark corners, special cases, and active optimizations that
it is very hard to answer just about any TCP/IP question briefly.

-a

[Gabriel Sechan]

Post by seberino at spawar.navy.mil ()
UDP verifys integrity and correct length of a UDP packet
by the checksum. Length field not required to make
sure "complete".
Still seems like UDP length field serves no purpose that
can't be done some other way.

Part of the reason is to kill space. If you use a 2 byte checksum and no
length, the data starts on a non-multiple of 4, which is inconvenient for
some hardware architectures with 32 bit buses.

Gabe