Marco Tech Blog

April 19, 2007

A Tale of Indy Sockets and a Two Characters Terminator

I've had a very bad experience with a 2 char separator in a stream-based socket. Here is the story.

      function TIdTCPConnection.ReadLn(ATerminator: string = LF; 

      const ATimeout: Integer = IdTimeoutDefault; AMaxLineLength: Integer = -1): string;

LTermPos := MemoryPos(ATerminator, PChar(InputBuffer.Memory) + LSize, LInputBufferSize - LSize);

posted by marcocantu @ 2:51AM | 7 Comments [0 Pending]

A Tale of Indy Sockets and a Two Characters Terminator 

So no matter what terminator you choose, you might run
into problems later. So why use a terminator? Send the
length of the string first. (I guess you may end up
having to buffer some data, though).

Comment by Nicholas Sherlock [http://www.sherlocksoftware.org] on April 19, 15:00

A Tale of Indy Sockets and a Two Characters Terminator 

.."And it is not used by itself, but it can be the
second character of an escaped UTF-8 sequence."

??? I thought UTF-8 bytes in multi-byte sequences
always had the top bit set!

Also, is your #27 dec, hex or octal?

- Roddy

Comment by Roddy Pratt [] on April 19, 15:03

A Tale of Indy Sockets and a Two Characters Terminator 

 
Using terminators is not reliable for sure.
The best solution should be

 [PREFIX] [SIZE]  [DATA...DATA]

Comment by Petar H [http://bigspeed.net] on April 19, 17:58

A Tale of Indy Sockets and a Two Characters Terminator 

 Marco,

A possible solution to that problem is to read an
extra character at the end of buffer. 
More generally, if the buffer is B bytes long and the
terminating string is T bytes long, then always read B
+ (T - 1) bytes at a time, then search for the whole
string.
The next buffer you read will be read starting at the
previous address + B, meaning you will actually read
(T-1) bytes twice for each buffer you read.

But this will never miss a terminating string, and the
logic is quite simple to implement and maintain...

Philippe

Comment by filofel on April 20, 12:33

A Tale of Indy Sockets and a Two Characters Terminator 

Meh.

This is TCP newb goof #1 (the Packet Fallacy), also 
memorialized in The Lame List as #20:

http://tangentsoft.net/wskfaq/articles/lame-list.html

You are *required* to assemble the received stream 
before parsing it.  Always has been, always will be.  
This is just as true when using a leading message 
length header, which can easily arrive fragmented.

Using a delimited message isn't all bad but multi-
octet delimiters solve nothing.  When using a message 
delimiter protocol you are supposed to escape any 
occurrences of the delimiter octet value that may 
occur in the payload.

Character encoding is an irrelevant abstraction at 
the message framing level.  The TCP stream is defined 
as a stream of octets and you impose a structure upon 
it at the application level.

These are simply the facts of life.

Comment by VBer and proud of it on April 22, 16:56

A Tale of Indy Sockets and a Two Characters Terminator 

I encountered a similar problem with Indy 9, which 
solved encoding the text that is transfered using 
MIME64 ... as this makes the string longer, first I 
compress it using Zlib.

So ...

1. First, compress the string with Zlib.

2. Encode the compressed string with MIME64.

The encoded string ( Zlib+MIME64 ) is usually not 
longer that the original string ( in my particular 
case, very often is shorter ), and you can use a 
single character terminator whitout worry.

Comment by Lluis Olle on August 18, 22:12

A Tale of Indy Sockets and a Two Characters Terminator 

 There are often very good reasons for wanting to
parse a data stream 'on the fly' and finding line
terminators is one example.  Another might be doing on
the fly character replcament (perhaps recognising and
tagging URLs, perhaps for additional analytics or
perhaps for use of a local caching proxy).  One
solution we have used is the development of a TStream
filter.  The idea is that the filter class (which is
derived from TStream) is used to pass all of the data
through and internally it recognises what it is
looking for and takes the appropriate action.  This
can not only do line buffering (storing all data until
the sequence and then supplying all of it together)
but also arbitrary length replacement of data (so only
buffering for the (maximum) size of the pattern looked
for.  Given that TStreams only need to cater for the
Write methods this is a simple approach and very
powerful, especially when you chain the filters together.

We've used this for clalcualting signature hashes,
text replacements, line buffering and so on.  All made
possible because of the simple and common TStream
interface.

Of course if an application layer does embed data
sizing or verification information into the data
stream changing the data could lead to a problem so
make sure that the system will work with your changes.

Comment by Rob Lambden on October 27, 15:18