fwm
Full Member
Posts: 105
|
Post by fwm on May 30, 2019 23:11:39 GMT -5
I'm back yet again I have discovered a(nother) strange issue... If I download a .js file, say jquery from here: ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.jsand then modify the .bas file to load this file and serve it instead of the test.html file (in other words, if I try to serve a compressed js file) I get a crash and LB shuts down. I tried a small PNG file and that worked fine. Is the wrapper not liking something about the js file or am I doing something stupid? I tried a few different scripts but no luck. EDIT: I should add that I haven't had any problem serving up the same .js files using a non-TLS server routine, so it only seems to happen when I introduce the TLS code into the routine. Keith.
|
|
|
Post by Chris Iverson on May 30, 2019 23:36:09 GMT -5
It's not just that specific JS file; it's anything of a significant size.
I tried with the output.html from the previous site download test, and get the same thing. Either LB hangs, or it crashes. Need to look into this to see what's going on, but I suspect there's a maximum amount of data I can encrypt at once, and I'm hitting that limit. Either that, or I screwed up the buffer handling, which is quite possible.
EDIT: Yup, I'm having an issue somewhere with data of large size. If I chunk the file to send into smaller chunks and send them individually, it works fine. Definitely need to see what's going on in EncryptSend().
Haven't uploaded a new file to GitHub, but if you take the existing test-https.bas file, and make the following modifications:
Replace this line:
response$ = responseStatus$ + crlf$ + responseHeaders$ + crlf$ + content$
With the following code:
if lenContent < 1024 then response$ = responseStatus$ + crlf$ + responseHeaders$ + crlf$ + content$ else response$ = responseStatus$ + crlf$ + responseHeaders$ + crlf$ end if
After the following code:
if TLSActive then ret = EncryptSend(hTLS, response$, lenResponse) else ret = Send(hConn, response$, lenResponse) end if
Add the following code:
if lenContent >= 1024 then for x = 1 to lenContent step 1024 contentPart$ = mid$(content$, x, 1024) if TLSActive then ret = EncryptSend(hTLS, contentPart$, len(contentPart$)) else ret = Send(hConn, contentPart$, len(contentPart$)) end if next x end if
And you should have an example file that matches mine, and works successfully to send the data.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 30, 2019 23:48:18 GMT -5
Hmmm... I see that now. Just tried a smaller js file and some other smaller files and they serve no problem so looks like the size might be a factor.
Keith.
EDIT: I will give that a try right now! You're the man! (Do people even say that now?)
EDIT: Works perfectly!
|
|
|
Post by Chris Iverson on May 31, 2019 22:11:00 GMT -5
Haven't had a chance to fix it yet, but I definitely found the issue.
In EncryptSend(), I just blindly use the passed-in buffer size as the amount of data to copy, ignoring the fact that it might be bigger than the buffer I prepared to receive it. This is exactly what you DON'T do with buffers. This is exactly why the size of the buffer is passed in in the first place. Oops.
If it is bigger, I get a buffer overflow, and memory corruption everywhere. Hooray!
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jun 5, 2019 1:06:09 GMT -5
Hi Chris, I am sometimes (but only rarely) getting this error when trying to negotiate TLS on some of my servers: PerformClientHandshake() failed. - -2146893052 - Error: 2775 80090304 This is when I am creating a client and connecting to a server. It usually appears after the program stops responding for a minute or so. According to this vaguely MSDN-esque helpful hint: windows-hexerror.linestarve.com/0x80090304the error is something to do with the Local Security Authority, or at least I think it is (not sure if that means an issue with a certificate's CA). I could be totally wrong here - wouldn't be the first time! I am testing the robustness of some of my routines when a user tries to connect via TLS on a port that is not correctly set up to negotiate, and wonder if there ought to be a timeout mechanism on the PerformClientHandshake() function that we can specify so that, in the event of the handshake loop not completing within a specified time (say because the server isn't responding or isn't configured correctly), an error is returned to the program rather than a hang-up? Keith. EDIT: If you need an example of this, open the test-https-client.bas program and change the connectServer$ value to ns1.cloudflare.com, and the port to 53 (DNS). I know this program was written to download HTTP but I am using it purely to illustrate this connection/handshake issue. Without any TLS, attempting to make a TCP connection to a DNS server on port 53 will allow a connection to be made. What I would like is, when a user of my program wants to make this connection but accidentally activates my "use TLS" option, they get an error rather than a hang. The hang, admittedly, doesn't seem to happen with the CloudFlare test but it does with my own nameservers, possibly to do with DNSSEC and possibly because CloudFlare have DNSSEC configured correctly but I don't...
|
|
|
Post by Chris Iverson on Jun 5, 2019 9:56:49 GMT -5
Definitely something I can take a look at. I think I know exactly where the issue is, too; I'm pretty sure the default recv() I issue during RunHandshakeLoop() I don't set up to have a time limit. If the server doesn't respond, it will hang until the system aborts the connection.
Think I just need to rework the code I use during Connect() or somewhere to set up a time limit. Probably something like 10 seconds as a default, and maybe edit the PerformClientHandshake() function to accept a custom time limit. (Not sure a custom time limit would be needed on a server-side handshake, since it's always the client that initiates those. I'd probably leave that at a default of 10 seconds. If a client initiates a connection, but then fails to process a response to their own handshake within 10 seconds, something went wrong with the client or the network.)
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jun 5, 2019 10:18:30 GMT -5
Yeah, makes sense. I would hope there wouldn't be too many instances of negotiation taking in excess of 10 seconds Keith
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jun 8, 2019 9:47:22 GMT -5
Hi Chris, Hope you are well and hope the last couple of fixes aren't turning out to be too much of a headache! I have a theoretical question - is it possible to get the IP address of where an incoming request is coming from? Maybe a GetRemoteIP() function or similar. I ask because I am testing the web server part of one of my programs that I have written based around the DLL you've created, and it occurred to me that it would probably be good to build in some sort of auto-ban mechanism if unscrupulous requests are detected coming from certain IPs - and then they can optionally be ignored or dealt with in whatever way we feel is appropriate. As always, would be good to hear your thoughts - when you get a moment Keith
|
|
|
Post by Chris Iverson on Jun 8, 2019 21:49:23 GMT -5
Shouldn't be a problem - the data is actually already available, I just need to provide a method to expose it. Honestly, a way to do this was always planned, I just haven't done it yet.
It is going to require rearchitecting a good chunk of the library, though, to be able to provide a space to store the remote IP. (It's made available by both the connect() and accept() API calls, but right now, my DLL just discards the info). Pretty much anything that receives or returns a SOCKET in my DLL is going to have to be modified.
The reason I'd have to modify everything is because I return and use the raw SOCKET from the API call directly - the SOCKET that gets returned by my DLL could actually be used by you to make calls against the Winsock API yourself, if you wanted to, for some reason.
While it made creating a wrapper layer easier, it also means there's nowhere for me to attach connection-specific info that isn't already there.
I would need to create a custom struct wrapper to add the SOCKET to, and add my custom data to it as well.
You'll note that I actually do this with the TLS data - I needed to be able to keep track of more stuff besides just the raw security context from the Windows API, so my TLS functions return and deal with a custom struct I created. It does the unwrapping of that struct itself to work with the data when it needs to. This is all kept invisible from LB.
I can't use that same struct with the SOCKETs, though, because by the time those structs get created by an LB program, the data needed(the IP info) is already lost.
So the functions that accept and return SOCKETS would have to be modified.
Actually, maybe not, now that I think about it. I would only need to do that if I wanted to provide a GetRemoteIP() function that could be called at any time.
Another thing I could do is modify the Connect() and AcceptConnection() calls to return the IP address info as well as everything else. This makes sense to me by keeping info in the same areas as it's retrieved from the system(with connect()/accept()), and it's something the system has to be set up to do anyway. May as well make a copy of the data at the same time.
Plus, when it comes to filtering, you'd be doing the filtering at the time the connection is completed, anyway, since that's the first point when IP address info becomes available. Really, it'd be saving a DLL call.
Haven't had much actual free time to work on the DLL, but progress will be made!
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jun 8, 2019 23:08:39 GMT -5
Yes, if you're choosing whether to go forward with a connection or drop it silently, I agree that providing the IP details at the connect() stage would be ideal. The "GetRemoteIP()" was just a theoretical arbitrary concept but I think it makes perfect sense to receive the data where you said.
Even if a developer then wanted to go ahead and continue the connection and later analyse the IP info, it could simply be kept stored in a string.
Keith
|
|
|
Post by Chris Iverson on Jun 10, 2019 16:42:59 GMT -5
So, I've made changes to EncryptSend() to seemingly fix the buffer overflow, but I'm running into a new issue that I honestly can't tell if it's a bug in my code, or a bug in LB.
It crashes once in a while, but the exact behavior of it is just bizarre.
It's some sort of memory issue, as it happens faster if you send larger buffers, but there doesn't seem to be a buffer overflow anywhere, now.
Additionally, arbitrarily expanding the buffers used by EncryptSend() doesn't fix the issue, which it would if it were a buffer overflow. Even adding a kilobyte of buffer space to each memory allocation in EncryptSend() doesn't result in a fix.
Nothing gets copied back to LB's buffers, so it shouldn't be overflowing anything on LB's side.
Also, the crash timing is just weird.
If you go to the test-https.bas file I have, comment out the line timer 1000, [asdffdsa] and the wait statement right afterward, and use output.html from the previous programs, it will crash once every few runs of the program.
Here's the weird thing: it doesn't crash in my function, or around it. It doesn't even crash when the DLL gets unloaded at the CLOSE statement. It crashes when LB program execution ends!
If you double the size of the output.html file, the crash will happen far more often.
(To double the size, I just copied it twice to another file using the LB code below.)
open "output.html" for input as #file a$ = input$(#file, lof(#file)) close #file
open "output-4.html" for output as #file print #file, a$; print #file, a$; close #file
Even weirder, if you add that timer/wait statement back, it won't crash anymore. At all.
That made me think that maybe the DLL cleanup is doing something that LB's termination winds up interfering with if you let it end right away, but that led to the even weirder fact:
If you use the test-http-https-dual.bas file, specifying even the double-size output.html to return, you can connect to the server as many times as you want, and it will never crash. Even when you add the ?command=END to the URL to end the program.
UNLESS
it's the first time you've connected to the server. If you connect to it at least once before sending the command to end the program, it will end gracefully, even if you immediately end LB after closing the DLL.
HOWEVER, if you don't connect at least once before, and you send the END command the first time you connect after starting the server, it will show the same behavior as above: (possibly) crashing unless you add a wait.
I absolutely cannot explain this. If there was memory corruption happening as a result of my DLL, calling my function multiple times should not result in proper execution!
I'm going to try to investigate this further, possibly later(after adding the handshake timeout and IP features), because to do so, I'm going to need to use my DLL from a not-LB language.
I'll most likely rewrite one of my example usage programs in C, and then see if I get the same kind of issues.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jun 10, 2019 17:53:19 GMT -5
I'm sure you'll find the problem - although I'm happy to limit the length to 1024 bytes and cycle through sending the data until it's all sent, as you suggested a few days ago. It works and works well I was curious as to whether your wrapper would be useable with different languages as well as LB, but I guess you've answered that question above! Keith EDIT: I can confirm the same behaviour (on Windows 7, at least).
|
|
|
Post by Rod on Jun 11, 2019 3:08:21 GMT -5
No web skills whatsoever, but his sort of crashing is not unusual if you create a queue of unhandled events within Liberty. Typically with graphics if the loop cannot be completed before the next cycle is called Liberty builds a hidden queue and will crash as you describe. The timer statement is prone to this. Free running ie timer 100 [task] will repeatedly call [task] without caring if the last task is complete. Switching the timer off when the task is called and on when the task is complete usually fixes it.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jun 11, 2019 14:19:49 GMT -5
Ok, two things.
The first - Rod may be onto something. If you replace all of the timer calls with a CPU sleep followed by a goto command instead, I cannot get it to crash:
call OpenTLSDLL
input "press ENTER to begin.";a
hServSock = CreateListenSocket("27016") if IsSocketInvalid(hServSock) then print "CreateListenSocket() failed. - ";GetError() goto [doEnd] end if
print "CreateListenSocket() successful."
[awaitLoop] print "Checking if connection is available..." ret = IsReadAvailable(hServSock, 0) if ret = 0 then print "No connections yet. Waiting..." calldll #kernel32, "Sleep", 1 as ulong, r as void goto [awaitLoop]
end if
if ret = -1 then print "Error with IsReadAvailable(). - ";GetError() goto [doSockEnd] end if
Print "Attempting to accept connection..." hConn = AcceptConnection(hServSock) if hConn = -1 then print "AcceptConnection() failed. - ";GetError() goto [doSockEnd] end if
[initReadLoop] ret = IsReadAvailable(hConn, 0) if ret = 0 then
calldll #kernel32, "Sleep", 1 as ulong, r as void goto [initReadLoop]
end if
bufLen = 128 buf$ = space$(bufLen) ret = Receive(hConn, buf$, bufLen)
if ret = 0 or ret = -1 then print "Receive() failed. - ";ret;" - ";GetError() a = CloseSocket(hConn) goto [doSockEnd] end if
if asc(mid$(buf$, 1, 1)) = hexdec("16") then print "ClientHello detected, passing to TLS handshake..." byteCount = ret goto [beginTLS] end if
num = ret goto [firstInputSkip]
[beginTLS]
print "Creating TLS context..." hTLS = CreateTLSContext()
print "Acquiring TLS credentials..." ret = BeginTLSServer(hTLS, "localhost") if ret <> 0 then print "BeginTLSServer() failed. ret - ";ret;" -- Error - ";GetError() Print dechex$( (abs(ret) XOR hexdec("FFFFFFFF")) + 1) a = DestroyTLSContext(hTLS) goto [doSockEnd] end if
Print "Finishing connection..."
[handshakeLoop] TLSActive = 1
a = SetTLSSocket(hTLS, hConn)
ret = PerformServerHandshake(hTLS, 0, buf$, byteCount) if ret <> 0 then print "PerformServerHandshake() failed. - ";ret; " - Error: ";dechex$(GetError()) Print dechex$( (abs(ret) XOR hexdec("FFFFFFFF")) + 1) a = CloseSocket(hConn) a = DestroyTLSContext(hTLS) goto [doSockEnd] end if
[bufLoop] if TLSActive then ret = IsTLSReadAvailable(hTLS, 0) else ret = IsReadAvailable(hConn, 0) end if
if ret = 0 then 'No data waiting. Stop and wait. calldll #kernel32, "Sleep", 1 as ulong, r as void goto [bufLoop] end if
bufLen = 512 buf$ = space$(bufLen) if TLSActive then num = DecryptReceive(hTLS, buf$, bufLen) else num = Receive(hConn, buf$, bufLen) end if
If num = -1 or (TLSActive = 0 AND num = 0) then Print "Socket error occurred. - ";GetError() a = CloseSocket(hConn) a = DestroyTLSContext(hTLS) goto [awaitLoop] End if
[firstInputSkip]
crlf$ = chr$(13) + chr$(10) lf$ = chr$(10)
cmdBuf$ = leftOver$ + left$(buf$, num)
[lineLoop] lineComplete = instr(cmdBuf$, crlf$) if lineComplete = 0 then lineComplete = instr(cmdBuf$, lf$) if lineComplete = 0 then leftOver$ = cmdBuf$ goto [bufLoop] end if CR = 0 else CR = 1 end if
cmd$ = trim$(left$(cmdBuf$, lineComplete - 1))
Print "< ";cmd$
if cmdBuf$ <> crlf$ and cmdBuf$ <> lf$ then
cmdBuf$ = right$(cmdBuf$, len(cmdBuf$) - lineComplete - CR) goto [lineLoop] end if
responseStatus$ = "HTTP/1.0 200 OK" responseHeaders$ = "Server: LB Test" + crlf$ responseHeaders$ = responseHeaders$ + "Content-Language: en" + crlf$ responseHeaders$ = responseHeaders$ + "Content-Type: text/html; charset=utf8" + crlf$ responseHeaders$ = responseHeaders$ + "Connection: close" + crlf$
if TLSActive then secure$ = "yes" else secure$ = "no" end if
responseHeaders$ = responseHeaders$ + "X-Request-Secure: " + secure$ + crlf$
open "output-4.html" for input as #file content$ = input$(#file, lof(#file)) close #file
lenContent = len(content$)
responseHeaders$ = responseHeaders$ + "Content-Length: " + str$(lenContent) + crlf$
response$ = responseStatus$ + crlf$ + responseHeaders$ + crlf$ + content$
lenResponse = len(response$)
if TLSActive then ret = EncryptSend(hTLS, response$, lenResponse) else ret = Send(hConn, response$, lenResponse) end if print print response$
print "Closing data socket..." a = CloseSocket(hConn)
print "Destroying TLS context..." a = DestroyTLSContext(hTLS)
[doSockEnd] print "Closing server socket..." a = CloseSocket(hServSock)
[doEnd] print "Closing TLS DLL..." call CloseTLSDLL
print "Performing wait..."
' calldll #kernel32, "Sleep", 1000 as ulong, r as void goto [asdffdsa]
[asdffdsa]
print "Ending program..."
Function randNum(min, max) randNum = int(rnd(1) * max) + min End Function
'==================== '==Helper Functions== '==================== Sub OpenTLSDLL open "Debug\LB-Schannel-Wrapper.dll" for DLL as #LBSchannelWrapper a = InitSockets() End Sub
Sub CloseTLSDLL a = EndSockets() close #LBSchannelWrapper End Sub
Function InitSockets() CallDLL #LBSchannelWrapper, "InitSockets",_ InitSockets as long End Function
Function EndSockets() CallDLL #LBSchannelWrapper, "EndSockets",_ EndSockets as long End Function
Function CreateTLSContext() CallDLL #LBSchannelWrapper, "CreateTLSContext",_ CreateTLSContext as ulong End Function
Function DestroyTLSContext(hTLS) CallDLL #LBSchannelWrapper, "DestroyTLSContext",_ hTLS as ulong,_ DestroyTLSContext as long End Function
Function BeginTLSClientNoValidation(hTLS) CallDLL #LBSchannelWrapper, "BeginTLSClientNoValidation",_ hTLS as ulong,_ BeginTLSClientNoValidation as long End Function
Function BeginTLSClient(hTLS) CallDLL #LBSchannelWrapper, "BeginTLSClient",_ hTLS as ulong,_ BeginTLSClient as long End Function
Function IsSocketInvalid(sock) CallDLL #LBSchannelWrapper, "IsSocketInvalid",_ sock as ulong,_ IsSocketInvalid as long End Function
Function BeginTLSServer(hTLS, serverName$) CallDLL #LBSchannelWrapper, "BeginTLSServer",_ hTLS as ulong,_ serverName$ as ptr,_ BeginTLSServer as long End Function
Function SetTLSSocket(hTLS, sock) CallDLL #LBSchannelWrapper, "SetTLSSocket",_ hTLS as ulong,_ sock as long,_ SetTLSSock as long End Function
Function PerformClientHandshake(hTLS, servernName$) CallDLL #LBSchannelWrapper, "PerformClientHandshake",_ hTLS as ulong,_ serverName$ as ptr,_ PerformClientHandshake as long End Function
Function PerformServerHandshake(hTLS, doInitialRead, initBuf$, initBufSize) CallDLL #LBSchannelWrapper, "PerformServerHandshake",_ hTLS as ulong,_ doInitialRead as long,_ initBuf$ as ptr,_ initBufSize as long,_ PerformServerHandshake as long End Function
Function CreateListenSocket(pService$) CallDLL #LBSchannelWrapper, "CreateListenSocket",_ pService$ as ptr,_ CreateListenSocket as ulong End Function
Function AcceptConnection(ServerSocket) CallDLL #LBSchannelWrapper, "AcceptConnection",_ ServerSocket as ulong,_ AcceptConnection as ulong End Function
Function IsReadAvailable(socket, msTimeout) CallDLL #LBSchannelWrapper, "IsReadAvailable",_ socket as ulong,_ msTimeout as long,_ IsReadAvailable as long End Function
Function IsTLSReadAvailable(hTLS, msTimeout) CallDLL #LBSchannelWrapper, "IsTLSReadAvailable",_ hTLS as ulong,_ msTimeout as long,_ IsTLSReadAvailable as long End Function
Function PingHost(host$, packetSize, byref status, byref msResponse, msTimeout) struct a, b as long struct c, d as long
a.b.struct = status c.d.struct = msResponse
CallDLL #LBSchannelWrapper, "PingHost",_ host$ as ptr,_ packetSize as long,_ a as struct,_ c as struct,_ msTimeout as long,_ PingHost as long
status = a.b.struct msResponse = c.d.struct End Function
Function Connect(host$, srv$, msTimeout) CallDLL #LBSchannelWrapper, "Connect",_ host$ as ptr,_ srv$ as ptr,_ msTimeout as long,_ Connect as long End Function
Function CloseSocket(sock) CallDLL #LBSchannelWrapper, "CloseSocket",_ sock as long,_ CloseSocket as long End Function
Function GetError() CallDLL #LBSchannelWrapper, "GetError",_ GetError as long
if GetError < 0 then GetError = (abs(GetError) XOR hexdec("FFFFFFFF")) + 1 end if End Function
Function Send(sock, msg$, msgLen) CallDLL #LBSchannelWrapper, "Send",_ sock as long,_ msg$ as ptr,_ msgLen as long,_ Send as long End Function
Function EncryptSend(hTLS, msg$, msgLen) CallDLL #LBSchannelWrapper, "EncryptSend",_ hTLS as ulong,_ msg$ as ptr,_ msgLen as long,_ EncryptSend as long End Function
Function Receive(sock, byref buf$, bufLen) CallDLL #LBSchannelWrapper, "Receive",_ sock as long,_ buf$ as ptr,_ bufLen as long,_ Receive as long End Function
Function DecryptReceive(hTLS, byref buf$, bufLen) CallDLL #LBSchannelWrapper, "DecryptReceive",_ hTLS as ulong,_ buf$ as ptr,_ bufLen as long,_ DecryptReceive as long End Function
Function EndTLSClientSession(hTLS) CallDLL #LBSchannelWrapper, "EndTLSClientSession",_ hTLS as ulong,_ EndTLSClientSession as long End Function
The second thing - if you run the original test-https.bas, go past the "press ENTER to begin." prompt so the program is waiting for a connection, but then close the window (instead of ending the program properly) and answer the "Please Confirm" prompt telling you that #LBSchannelWrapper will be closed, and then try to run the program again without restarting LB, it often won't be able to create a listen socket on the second attempt:
CreateListenSocket() failed. - 10022
Almost as if something from the DLL is "left over" and hasn't been terminated correctly. Not sure if that is linked to this issue or if it's something else.
Keith.
|
|
|
Post by Chris Iverson on Jun 11, 2019 14:48:00 GMT -5
Ah, thank you for the timer info. I hadn't considered that, because I turn off all timers before proceeding in the program. Looks like something's still hanging around.
As for the second thing, that's actually to be expected, if you're running from the LB IDE. The reason is, when you're running from the IDE, programs don't run as separate processes. They run as part of the IDE. If you halt the executing LB program before it has a chance to clean up resources, then:
1) The DLL is still open. 2) Any TLS credential(and potential certificate) handles you had are still open. 3) Any sockets you had are still open. 4) Any server sockets you had are still open and listening for connections! 5) Because you no longer have references to any of the above, all of those handles(and the memory associated with them) are leaked, until the LB IDE process is terminated.
After all, the executing program never got a chance to close any of that!
You get an error message if you try to run it again, because it's trying to open and bind to a port that's still active!
If you're running as a compiled program under the runtime engine, this is fine, because killing the runtime engine ends the process, and Windows will clean up the other resources as part of process teardown.
However, if it's running under the LB IDE, Windows doesn't step in to do that, because the process is still running!
(This is also why a big enough bug in a written program will kill the IDE.)
|
|