RS485 Traffic Decoder Analyzer

[msteffes]

Hello All,



Total transparency, I am a programming tinkerer and my periodic adventures in code are likely quite clunky to the code gurus out there. I feel better already.



I am looking for input on a project I have inflicted upon myself. As a field Tech I am hoping to have a diagnostic tool for analyzing RS485 traffic capture files but I am not overly confident about how to tackle this effort.

The traffic is simple poll and response. The capture files are csv and can be huge. A recent capture I collected was a 350Meg file holding about 4 million records. Initially I am starting with a more modest file size of about 100k records.

My plan was to load this into an array which after a bit of fumbling I have working. It is rather slow about 9 seconds after parsing. I would like this to be faster but I have bigger binary fish to fry.


I would like the first version of this project to show a window on the left side of the screen displaying a section of records from the array, say 20-30 line. The data could scroll up or down to reveal fresh data into the screen. Also a highlighted line would serve to represent one record that would have detail data on the right side broken down into several Hex byte fields. This would show changing bit status.

I would like two modes

1) The highlight can be walked up and down through the records by the arrow keys or mouse wheel to slowly update the detail on the right side.

2) An animated mode where a modest speed range can be adjusted to advance through the records while the right side continues to show the bit changes.



I was considering using radio buttons for my bit indicators but do not want get to far ahead and box myself into a slow interface. But I will to defer to the experience out there to steer me in the right direction on the following.



I assume huge files are best chunked to make it more appetizing to Liberty basic?

How to load the data more quickly to the array or some other approach.

How best to accomplish the scrolling data window.

And how do efficiently convert and display the bit data.



I would also like a Golden chicken that lays talking eggs.



Here is an example of some data. .





="18:04:25.455",PIM status idle response to PIM command,="0A",="FF",="31",="00",="D0"

="18:04:25.487",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:25.503",PIM status idle response to PIM command,="0A",="FF",="31",="00",="D0"

="18:04:25.535",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:25.550",PIM status idle response to PIM command,="0A",="FF",="31",="00",="D0"

="18:04:25.582",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:25.598",PIM status idle response to PIM command,="0A",="FF",="31",="00",="D0"

="18:04:25.630",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:25.662",PIM status carddata response from WAPM 7 to PIM command,="0A",="FF",="31",="0A",="07",="00",="00",="14",="00",="1A",="D0",="CD",="44",="40",="70"

="18:04:25.726",WAPM 7 timed unlock command,="0A",="07",="56",="02",="03",="00",="9E"

="18:04:25.742",WAPM response to WAPM command,="0A",="FF",="30",="03",="00",="00",="14",="BA"

="18:04:25.823",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:25.823",PIM status idle response to PIM command,="0A",="FF",="31",="00",="D0"

="18:04:25.902",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:25.919",PIM status idle response to PIM command,="0A",="FF",="31",="00",="D0"

="18:04:25.999",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:26.031",PIM status change response from WAPM 7 to PIM command,="0A",="FF",="31",="05",="07",="00",="00",="94",="00",="30"

="18:04:26.111",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

="18:04:26.111",PIM status idle response to PIM command,="0A",="FF",="31",="00",="D0"

="18:04:26.175",PIM 0 poll (checksum) command,="0A",="00",="74",="00",="8C"

[Rod]

This is pretty crude but I am out of time. There is a technique which may or may not work for you. You can instantly load the data to a string. Then use mid$ to navigate the records. So it will load fast but probably process slower. But because you are eyeballing the data it needn't run that fast. So its perhaps something to discuss rather than a solution.

I have not catered for the variable hex width you just need to add more parsing for that. The CRLF pairs get in the way but once understood don't really cause any problem. The binary display is crude too.

nomainwin

'record pointer array, pointer only, data stays in t$
dim rp(100000) 'record pointer

open "data.txt" for input as #1

'get entire dataset
t$="xxxx"
t$=t$+input$(#1,lof(#1))
close #1

'find how many records
eol$=chr$(13)+chr$(10)
pos=1
rec=1
while pos<len(t$) and pos<>0
   rp(rec)=pos
   pos=instr(t$,eol$,pos+3)
   rec=rec+1
wend
maxrec=rec-1

'list records to check
'for rec= 1 to maxrec
'    l$=mid$(t$,rp(rec)+4,rp(rec+1)-rp(rec)-4)
'    for n= 1 to len(l$)
'        print mid$(l$,n,1);asc(mid$(l$,n,1))
'    next
'    print mid$(t$,rp(rec)+4,rp(rec+1)-rp(rec)-4)
'next

WindowWidth=800
WindowHeight=600
texteditor #w.te 10, 10, 300, 200
textbox #w.bin 320,100,400,20
button #w.first, "First", [first], UL, 10, 230
button #w.PageDn, "PageDn", [pageDn], Ul, 80, 230
button #w.Annimate, "Annimate", [annimate], Ul, 150, 230
button #w.PageUp, "PageUp", [pageUp], Ul, 220, 230
button #w.PageDn, "Last", [last], Ul, 290, 230

open "rs485" for window as #w
#w "font consolas 8"
rec=1
gosub [display]
wait

[display]
#w.te "!cls"
for n=rec to rec+9
   'get the record without the crlf pairs
   l$=mid$(t$,rp(n)+4,rp(n+1)-rp(n)-4)
   pl$=mid$(l$,3,12)+" "
   'get the hex
   ll$=right$(l$,len(l$)-instr(l$,"command")-7)
   lll$=mid$(ll$,3,2)+" "+mid$(ll$,9,2)+" "+mid$(ll$,15,2)+" "+mid$(ll$,21,2)+" "+mid$(ll$,27,2)'etc
   pl$=pl$+lll$
   #w.te pl$
   if n=rec+5 then
       bb$=""
       for h=1 to 13 step 3
           v=hexdec(mid$(lll$,h,2))
           b$=""
           for b=0 to 7
               if v and 2^b then b$="1"+b$ else b$="0"+b$
           next
           bb$=b$+bb$
       next
       #w.bin bb$
   end if

next
return

[first]
timer 0
rec=1
gosub [display]
wait

[pageDn]
timer 0
rec=rec+10
if rec>maxrec-10 then rec=maxrec-10
gosub [display]
wait

[annimate]
timer 56,[go]
wait
[go]
rec=rec+1
if rec>maxrec-10 then timer 0 : rec=maxrec-10
gosub [display]
wait

[pageUp]
timer 0
rec=rec-10
if rec<0 then rec=1
gosub [display]
wait

[last]
timer 0
rec=maxrec-10
gosub [display]
wait

[quit]
timer 0
close #w
end





[Rod]

Even though this pulls in the whole dataset you are very likely going to have to chop it in chunks. And eyeballing 350meg file isn't going to work is it? You will need a search facility.

[badbug]

Why not simply create a DB. You can easily hold all your data.
You'll be able to add,change, and delete info.
Scrolling through the data would be easy.

A DB similar to this may work. Then run it through lgGen to generate the Liberty Basic code to maintain, lookup, and search..


# --------------------------------------------------------
# Table structure for: rs485  
# --------------------------------------------------------
CREATE TABLE rs485 (
rsDate DATE,
rsTime TIME,
pimStatus CHAR(1),
pimCmd VARCHAR(20))
;
CREATE UNIQUE INDEX rs485_dtetim ON rs485( rsDate, rsTime);
CREATE INDEX rs485_st ON rs485( pimStatus, rsDate, rsTime);


[tenochtitlanuk]

I often read files to a string variable and search with 'word$()'. BUT LB has a limit of 2M on strings.

I recently got caught by this- it isn't error-trapped and I was getting some very weird results from a Scrabble words file until I realised! Wrote quick code to split the SOWPODS file into 26 bits, starting with each alphabet letter. Now it was small enough to behave as expected...

[Rod]

Not sure that can be true, think I have handled larger strings before. Will need to test. Edit : Think I am creating 100Mb string. Liberty has new memory limit of 1Gb. Sure you wont get near that and sure it will be awfully slow processing anything. Keeping chunks to a manageable 1Mb might be best.

'create 1Kb string

a$=space$(1020)+"9999"

print len(a$)/1024 , "Kb"



'create 1Mb string

for n=1 to 1000

b$=b$+a$

next

print len(b$)/1024, "Kb"



'create 100Mb string

for n= 1 to 100

c$=c$+b$

next

print len(c$)/1024, "Kb"



c$=c$+"888"

print right$(c$,4)



[tenochtitlanuk]

Manual says

In Liberty BASIC, string variables can be huge, containing as many as 2 million characters.

My programs were accepting longer strings, but things likwe mid$() were giving incorrect results.
Mind you I work under Linux and don't have access here at present to MS to check if it's a Wine fault.
Will post again if I learn more.

[tenochtitlanuk]

It's word$() which is misbehaving for me. The following code hangs on trying to use word$(), and in my original problem code it took 8 seconds to fin=d the first word in a dictionary string 0f some 120,000 words.
Does it behave under MicroSoft??

a$ =""

for n = 1 to 1000
   if int( 10 *rnd( 1)) =6 then a$ =a$ +"," else a$ =a$ +chr$( 97 + int( 26 *rnd( 1)))
   scan
next n

for n =1 to 1000
   c$ =c$ +a$
next n

print "Million byte string of random csv 'words'."
print
print len( c$); " bytes."
print
print "Left$()  "
print left$(  c$, 30)
print
print "Right$() "
print right$( c$, 30)
print
print "Mid$()   "
print mid$(   c$, 30, 30)
print
print "Word$()  "
print word$(  c$, 5, ",")
print "Ended."

end