how to find start and stop commands in a serial stream

[johnking]

My previous message did solve one problem (thanks !) but gave me another.

The instrument sends out a serial stream with 15..17 packets every second, always starting with STX (decimal 2 / hex 02) and ending with ETX (decimal 3/ hex 03).
The number of packets are not always the same, but the STX  / ETX are always send at the begin / start

So I thought to read the serial string, break it up, find STX enad ETX and then use everything inbetween. 

The way to read the serial for this instrument is :



open "com1:19200,n,8,1,ds0,cs0,rs" for random as #com1

print #com1, "S";                                           'start command

print #com1, "P";                                           'command to start sending data every second

dataread$ = input$(#com1, lof(#com1))          ‘read data


and then break up the datareadstring in parts using midstring and find STX and ETX. But as STX and ETX are hidden characters and the string contains more decimal 2 and 3 numbers this approach is not working.

So I guess my approach is not the correct way to do this.

I have several instruments from which I read the serial port, but they do not send out data every second, only send data after finishing a measurement, so I don't have a problem there.

I studied the documents from Rod and Alyce, but could not find a solution (or overlooked). Can anyone help ?

[Rod]

I think the solution is covered in the article on the LBPE. You need to double buffer, store the incoming string and parse out the blocks you need using the start and or end sentinel.

If you can supply a real message stream we can show you how.

[johnking]

Thanks Rod, I spend a great part of the week trying your method, but no succes.
The instrument sends out a datastring every second. The information is encrypted within this string as hex, binary, BCD and ascii code. I have no problem decoding the information, that is all sorted out.
The goal is to read the datastring once it is available on the serial port, that is every second.

The datastream has 17 packets. The first packet is always STX (=decimal/hex code 02). Packet 16 is always ETX (=decimal/hex code 03). Packet 17 is a LRC (line redundancy check) which I do not know yet how it is calculated. I do not use it.
So I thought to search for the STX (02) and ETX (03) characters and use everything in between.

In binary one typical datastream would be:

00000010 01000011 00000000 00000001 11110100 00000010 11100000 00000010
01011100 00000000 00011101 00000010 01100000 00000011 11000110 00000011
10110001

In hex this would be:

02 43 00 02 65 02 65 02 4E 00 2C 02 65 00 01 03 44

But I think because other values in the stream can (and will) be 02 or 03 as well, this is not the correct approach.
So far, using my method above, and using your double buffer approach gives me the string, but not every second.
I receive second 1, 2, 4, 5, 7 etc. So I 'miss' seconds inbetween.
The next time it can be second 1, 3, 5, 6 etc., so it is also not consistent.

[johnking]

I have a feeling it is somehow related to the problem in the 'read-write-serial-port-arduino' post, i just do not see it (yet).

[Carl Gundel]

Nov 1, 2020 9:38:43 GMT -5 johnking said:

I have a feeling it is somehow related to the problem in the 'read-write-serial-port-arduino' post, i just do not see it (yet).

Are all the characters coming through?  If they are then it's just a parsing problem.

[johnking]

yes, all packets and thus characters are comming through, but I am missing 'seconds'

[Carl Gundel]

Nov 1, 2020 10:01:05 GMT -5 johnking said:

yes, all packets and thus characters are comming through, but I am missing 'seconds'

You're missing seconds?  You're looking to handle timeouts?  That's something you define yourself, I imagine.

[Rod]

The start and end sentinel are rarely one byte and they would certainly not be repeated within the string. So we are not understanding the message yet. The start or end sentinel is likely a pair of characters or perhaps hidden characters are involved, like chr$(13)

Is there a support page for the device you are using?

[johnking]

No supportpage, the stream every second is as follows:

packet 1 STX
packet 2-15 measurement values
packet 16 ETX
packet 17 LRC

so in the stream -shown as hex- the values 02 and 03 are always there: 02 43 00 02 65 02 65 02 4E 00 2C 02 65 00 01 03 44
I understand that in the string I must not search for 02 or 03, but for the CHR$ equivalent chr$(02) and chr$(03).

I am using now:

open "com1:19200,n,8,1,ds0,cs0,rs" for random as #com1 'poort open

print #com1, "S" 'startcommand

delimiter$=chr$(02)


timer 500, [bufferread]
wait

[bufferread]
if lof(#com1)>0 then
dataread$ = input$(#com1, lof(#com1))

[doublebuffer]
buffer$=buffer$+dataRead$
delimit=instr(buffer$,delimiter$,1)

[loop]
if delimit>0 then
dataread$=left$(buffer$,delimit)
buffer$=right$(buffer$,len(buffer$) - delimit -1)
print dataread$
end if
delimit=instr(buffer$,delimiter$,1)
if delimit>0 then [loop]
end if
'wait for next [bufferread]
wait

[Rod]

Just a little bit more about the message format. You say 2-15 values. So each value is one byte? Does it always send 2 or does it always send 15? How is that set up? What I am getting at is whether the message is actually a fixed length?

Start and end sentinels should be unique and easy to find, yours are not.

Not at my pc right now but will code something. Understanding the check digit may help it’s probably all of the characters anded together.

Will play around and report back.

[Rod]

This is how it would seem the check digit is calculated. Not sure it helps very much but more playing around to do.


for n=1 to 16       'read in all the characters up to the check digit
   read d$         'read hex value
   d=hexdec(d$)    'convert to decimal
   b=b xor d       'xor all the values in the message
   print d$,d,b    'note that the final xor value is decimal 68 hex 44
                   'so we can calculate the checkdigit
next
wait



data 02, 43, 00, 02, 65, 02, 65, 02, 4E, 00, 2C, 02, 65, 00, 01, 03, 44


[Rod]

So, we have a start sentinel of "02" and an end sentinel of "03" followed by a an xor check digit.
But, we can have start and end sentinels within the variable part of the message!

So, this is not a message meant to be parsed by start and end sentinel. It is fixed length, there may be message packets telling you what the length is, if the length can vary.

We might be able to look for each start sentinel "02" and then every end sentinel "03", and check that the xor sums to the following character but this is way too much of a cludge to be the answer.

Are you sure there is no online info? what instrument is it?

[Carl Gundel]

Nov 2, 2020 9:13:13 GMT -5 Rod said:

So, we have a start sentinel of "02" and an end sentinel of "03" followed by a an xor check digit.
But, we can have start and end sentinels within the variable part of the message!

So, this is not a message meant to be parsed by start and end sentinel. It is fixed length, there may be message packets telling you what the length is, if the length can vary.

We might be able to look for each start sentinel "02" and then every end sentinel "03", and check that the xor sums to the following character but this is way too much of a cludge to be the answer.

Are you sure there is no online info? what instrument is it?

It would be interesting and perhaps useful to create a simulator for this.  Make a user interface (a window) that inserts characters into a stream (a string) and have the parser read characters off the beginning of the stream (removing them as it goes).  The UI would let you insert characters one at a time, or as a complete unit, and each time the parser reads a complete sequence or detects a problem it would print about it to the main window or to a a textbox in the UI.

The parser can be driven by a timer.

[Rod]

Here is the problem, the message gets transmitted every time you click send, so you can send once a second or several times a second.

The software checks every 500ms and parses out each individual message. It works with the delimiter set to 03 but that is only because there is no other 03 in that particular message. Try it with the delimiter set to 02, no chance of parsing the message.

Sentinels need to be unique, often something as simple as chr$(13)+chr$(10) is enough. But here one character that is often included in the message content cannot act as an end sentinel. So this message was meant to be interpreted another way. Perhaps the end sentinel comes later in the stream, perhaps handshaking lines were used. If it is a timed stream perhaps the stream is fixed length packets.

nomainwin
statictext #c.st1, "Enter your hex string to send (00 FF 13 10)", 20, 10, 260, 25
textbox #c.txt1, 20, 40, 300, 25
texteditor #c.txt2, 20, 70, 300, 250
button #c.start, "Send", [send], UL, 20, 350
WindowWidth = 350 : WindowHeight = 600
open "Simple Com Stream Tester" for window as #c
#c "trapclose [quit]"

#c.txt1 "02 43 00 02 65 02 65 02 4E 00 2C 02 65 00 01 03 44"
'delimiter$=chr$(hexdec("02"))
delimiter$=chr$(hexdec("03"))

timer 500, [bufferread]
wait

[bufferread]
   'if lof(#com1)>0 then
   'dataread$ = input$(#com1, lof(#com1))
   if dataread$<>"" then

       [doublebuffer]
       buffer$=buffer$+dataread$
       delimit=instr(buffer$,delimiter$,1)

       [loop]
       if delimit>0 then
           dataread$=left$(buffer$,delimit)
           buffer$=right$(buffer$,len(buffer$) - delimit -1)
           for n= 1 to len(dataread$)
               #c.txt2 right$("00"+dechex$(asc(mid$(dataread$,n,1))),2);" ";
           next
           #c.txt2 ""
       end if
       delimit=instr(buffer$,delimiter$,1)
       if delimit>0 then [loop]
   end if
   dataread$=""
   'wait for next [bufferread]
   wait


[send]
   #c.txt1 "!contents? message$"
   dataread$=""
   for n= 1 to len(message$) step 3
       dataread$=dataread$+chr$(hexdec(mid$(message$,n,2)))
   next
   wait


[quit]
   timer 0
   close #c
   end





[johnking]

So, we have a start sentinel of "02" and an end sentinel of "03" followed by a an xor check digit.
But, we can have start and end sentinels within the variable part of the message!

No, we cannot. If I give the command to start sending data every second in RealTerm, RealTerm displays the data nicely every second, so I think nicely waits until a new datastring is available. The end sentinel is unique, it is always ETX in packet 16, followed by the line redundancy check in packet 17. Also the length of each datastream is fixed, it is always 17 packets. So what would be a good coding to read such a datastream ? I tried different approaches, but like I said, after a few seconds I start to miss 'seconds'.