fwm
Full Member
Posts: 105
|
Post by fwm on Sept 12, 2018 16:55:02 GMT -5
It looks like it might be an issue when running the DLL and program on Windows XP or Windows 7, so I will continue investigating it and see what I can find... I'm sure Chris would know what's going on right away - but it will take me longer to work it out! Keith
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Sept 27, 2018 0:52:53 GMT -5
Hi, I have created an example program that (I hope) shows some of the capabilities of being able to communicate over TCP using secure connections, and also use LB to ping servers etc. At this time the ping function only works for IPv4 IP addresses and hostnames, but I will see if I can add IPv6 into it at some point. There are lots of error-checking lines, and programmatically it is not well written but it is purely to show this wonderful DLL working. I'm sure if and when Chris is able to get back to this forum he will be able to provide any info I have missed and fill in the gaps etc. but right now this program shows a few examples of how it works. The example program includes very basic SMTP and HTTP examples, but this can also be used for FTP, IMAP, POP3, bog-standard TCP connections, and so on. You need to know how to have the conversations once the connections are made, but that's for you to figure out! Hope you find it interesting. TLS Examples (LB Program Code)SChannel Wrapper DLLOK, I thought it was time to post an update to this. I have rewritten a couple of things in the .bas code, and the .dll file has been updated very slightly, so you should download those both again using the same links (above or in my previous post in this thread). I also came to realise that the DLL will not function without the C++ redistributable installed (I had this on my system already so took me a while to work out that others would need to install it). I compiled it using Visual Studio 2017, so it makes sense to install the Microsoft Visual C++ 2017 Redistributable, which you can either source from the official Microsoft site or from my download link here to save you some time - you only need the 32-bit version as far as I can tell as the DLL is compiled for 32-bit systems, due to LB being 32-bit only: Microsoft Visual C++ 2017 RedistributableHowever, I have not been able to get the program to run successfully with the DLL on systems other than Windows 10 (fully updated, of course). I have tried on Windows 7 and Windows 8.1, and have spent hours updating them to ensure they were up-to-date, but still found communication errors. I have tweaked ciphers thinking that it has something to do with available SSL/TLS cipher suites on different versions, and although this did allow some communication when certain ciphers were forced, it was a very limited improvement and I think something more universal is at the heart of this, not the cipher suites. I feel I have reached the limit of what I can try and what I understand. I've gone through the C++ code of the DLL and compared it to some examples of SSL/TLS programming in C++, then tried rewriting sections of the DLL but nothing changed. So for now all I can say is that you need to install the redistributable as discussed above, and use Windows 10 for this to work correctly. I am hopeful that Chris will return to these forums at some point and be able to shine a light on the reasons for it not working correctly on earlier versions of Windows - it might be something really simple but I just cannot work out what! Maybe someone else with an understanding of using SChannel might be able to help out? You can find a link to the original C++ code written by Chris in the .bas example program code (this version does not include the additional Ping function that I added). Until then I hope others find this project useful at least on Windows 10, and hopefully at some point we will be able to make this backward-compatible with at least Windows 8.1 and Windows 7. Keith. EDIT: Chris has now got this issue fixed and it seems to work just fine on Windows 7 and Windows 10 from my testing... so I would assume Windows 8.1 etc. Not sure about XP - does XP even matter any more? I guess it might but haven't tested it yet! I have updated the DLL file I made available so download it again and give it a go! EDIT: I have removed the links as Chris has now released an official version on GitHub.
|
|
|
Post by Chris Iverson on Oct 1, 2018 15:20:11 GMT -5
Sorry I've been gone so long, just been so busy at work, I've not really had time to focus on anything else.
I think things are slowing down a bit now, I should be able to start looking at things again soon.
I'll take a look at how the DLL works on Windows 7. I didn't think I'd done anything that specifically requires Windows 10, but I could be wrong.
EDIT: Just for a quick test, I took your DLL and the sample code in your previous post, and both are able to function just fine on my Windows 7 system. What issues are you seeing?
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Oct 1, 2018 17:51:49 GMT -5
Hey Chris - great to hear from you!!
Hmmm... the main issue I was seeing is that the secure connections are being cut off early - so either the SMTP conversation doesn't finish, or it looks like it finishes but it doesn't actually receive the final "goodbye" message because the connection has been cut. The "disconnect" seems to happen at different places depending on if I force my client to use a particular cipher suite or TLS protocol, and sometimes even changing the name I give in the EHLO message makes a difference... When I used it to connect to my own mail servers to see what the logs stated, I was seeing an error in the mail log:
SSL3_GET_RECORD:wrong version number This is a server running Postfix.
So on Windows 10, running, let's say, example 3 of the program I wrote, I get:
Initialising TLS... InitTLS() = 8743920 BeginTLSClientWithValidation() = 0 Opening connection to smtp.gmail.com on port 465... Connection = 1012 SetTLSSocket() = 0 PerformClientHandshake() = 0 220 smtp.gmail.com ESMTP d2-v6sm12260941wrm.71 - gsmtp EHLO example.mydomain.com 250-smtp.gmail.com at your service, [83.67.174.78] 250-SIZE 35882577 250-8BITMIME 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-CHUNKING 250 SMTPUTF8 QUIT 221 2.0.0 closing connection d2-v6sm12260941wrm.71 - gsmtp Closing connection. Completed successfully. Liberty Basic can now do SSL/TLS! I even checked the certificate I was provided to make sure it matched smtp.gmail.com! but on Windows 7 I get:
Initialising TLS... InitTLS() = 7934824 BeginTLSClientWithValidation() = 0 Opening connection to smtp.gmail.com on port 465... Connection = 488 SetTLSSocket() = 0 PerformClientHandshake() = 0 220 smtp.gmail.com ESMTP a9-v6sm10235192wmf.28 - gsmtp EHLO example.mydomain.com Did not receive 250 reply. Closing connection. If I change the EHLO on Windows 7 to, say TEST123, I get a bit further before the cut-off:
Initialising TLS... InitTLS() = 7936912 BeginTLSClientWithValidation() = 0 Opening connection to smtp.gmail.com on port 465... Connection = 492 SetTLSSocket() = 0 PerformClientHandshake() = 0 220 smtp.gmail.com ESMTP k7-v6sm12275384wmf.41 - gsmtp EHLO TEST123 250-smtp.gmail.com at your service, [83.67.174.78] 250-SIZE 35882577 250-8BITMIME 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-CHUNKING 250 SMTPUTF8 QUIT Did not receive 221 reply. Closing connection. See how I get further but I still lose connection and the final QUIT command doesn't receive a reply like it should.
It got me really confused and frustrated as I cannot work it out!
Keith
|
|
|
Post by Chris Iverson on Oct 1, 2018 23:09:21 GMT -5
Ah, I see.
There's definitely something going on. I don't think it's a protocol or cipher suite issue; if it were one of those, the connection would fail to be established in the first place.
I think it's a buffer issue. The full contents aren't being returned somehow, and additionally, I sometimes get a Protection Violation, which means data's moving somewhere it shouldn't.
It's extremely weird that it only happens on Windows < 10, but I'll need to do deeper debugging to figure this out.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Oct 2, 2018 1:43:05 GMT -5
Glad you managed to make sense of my explanation...! Yes it's odd. I tried using WireShark and noticed there was an "ignored unknown record" error generated whenever there was a problem with the SMTP connection, which didn't appear when the connection was good to the end. I think the "ignored unknown record" seemed to be in the outgoing data rather than the incoming, so not sure if SChannel (or another component) is adding something, or changing something, before it sends? I don't know, I am probably completely wrong and decided the only person who would be able to figure it out would be you...! No pressure... But seriously I hope you can spot something as this DLL is a killer add-on to LB. Keith
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Oct 14, 2018 1:36:39 GMT -5
Hi Chris,
Have you managed to get anywhere with this strange issue yet?
Keith
|
|
|
Post by Chris Iverson on Oct 17, 2018 10:18:50 GMT -5
Not yet. I've got a Windows 7 VM set up that I've used to verify the issue, but I'm getting it set up as a remote debug machine for Visual Studio so I don't have to install the entire VS IDE on a VM.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Oct 17, 2018 10:51:24 GMT -5
Ah OK. Not that I would be much help, but if there's anything I can help with, just say... Keith
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Dec 14, 2018 13:01:52 GMT -5
Chris,
Any closer to finding what's going on with this?
Keith.
|
|
|
Post by Chris Iverson on Jan 3, 2019 18:53:53 GMT -5
Sorry I've not been looking at this. I haven't had much time to dig through this.
I've started looking at it again. Running on Windows 7, the DecryptMessage() API call returns SEC_E_UNSUPPORTED_FUNCTION, which bubbles out through the rest of the DecryptReceive() code, and aborts it early.
The problem is, I'm not sure what's causing it to return UNSUPPORTED_FUNCTION, because that's not actually a documented return code from DecryptMessage(). I'm going through my buffer code to see if I've missed something. I would expect UNSUPPORTED_FUNCTION if I was, for example, asking for new encryption ciphers that Windows 7 doesn't support, but such a case would error out far earlier than that(in fact, I wouldn't be able to get a full security context in the first place if I was asking for unsupported stuff).
Additionally, since I'm just asking to use the OS's defaults, that really shouldn't be affecting anything.
EDIT: Well, I believe I've fixed it. A new version has been committed to the GitHub site.
The issue was a misunderstanding of how Windows handled extraneous data that didn't get decrypted during a pass through InitializeSecurityContext(). It didn't happen in Windows 10, I expect, due to it changing the way it handled messages, and therefore not having any extraneous data to deal with.
...And now that I mention that, I just realized that there's probably another part of the code I have to fix, in the DecryptReceive() function. Oh well, I'll take a look at that tomorrow.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jan 4, 2019 16:21:22 GMT -5
Great to hear from you Chris.
I will wait for you to look at the DecryptReceive() function tomorrow before I put it to the test and I'll let you know how it goes.
Many thanks once again.
Keith.
|
|
|
Post by Chris Iverson on Jan 4, 2019 17:57:09 GMT -5
Okay, so, it turns out, I hadn't fixed the UNSUPPORTED_FUNCTION error. The program would work on some sites, but not others. I did fix the crash, but the unsupported function error was being caused somewhere else. I wound up going as far as to do Wireshark captures of my network, trying to figure out what was going on. I noticed that I would get a TLS alert back from the server when I got the DecryptReceive() error. I checked the program's buffers, and sure enough, it seemed to be choking on the alert. So, apparently DecryptMessage() will return error codes other than the ones documented on it's MSDN page if it processes a TLS protocol error. Because I never got the raw, unencrypted data, I wasn't sure what alert was being returned. According to this MSDN page, UNSUPPORTED_FUNCTION is returned when a TLS "Protocol Version" alert is processed. (I was able to later figure out that DecryptMessage() does decrypt the buffer before failing, and the first two bytes in the returned buffer will be the TLS alert level and description code. Using these, I was able to confirm that my server was returning a "Protocol Version" alert.) Then, I noticed something else weird: after the server sent the alert, it sends another encrypted message. By including another DecryptReceive() call in the LB code, I was able to receive that one, too, and lo and behold, it was the HTTP data I was expecting. Yay! Problem remains, though, of why the alert was being sent in the first place. So I went through the capture, and compared it to other captures of, for example, the browser on the same system. Then, I noticed something: as well as a blob of "Application Data", the first message I sent to the server after the handshake(therefore, using EncryptSend()) had an additional tag on it: "Ignored Unknown Record". Checking it, I realized that for some reason, I was sending a bunch of null bytes at the end of the message. I figured that might be what was tripping my server up: unexpected amounts of incoming data. So, I went through the EncryptSend() function, and realized that the EncryptMessage() API function will adjust the sizes of the various buffers you pass in, as well as encrypting the data in place, and adding protocol headers/trailers. Because of this, you need to actually use the size of the buffers returned from the function, not just the size of the buffers you created in the first place. (I create them at the max size for each buffer; the API call sizes them down to keep it all contiguous when being sent as a single blob.) Because I was still using the max sizes, I was telling the send() API call to read more data from my buffer than EncryptMessage() intended to pass on. After fixing this, the code worked just fine against my server. As for why it worked on some servers(such as google.com), but not my own, I suspect various different pieces of server software tend to be either more or less strict when it comes to processing incoming data. My server read through the headers, found that my client sent more data through the TCP stream than was used by the TLS protocol, and while it sent a proper response, it also sent an alert back, saying "Hey, quit screwing around." Google's servers, meanwhile, saw the difference between the TCP header and the TLS header, dealt with the TLS data, saw the rest of the TCP data was just nothing(null bytes), and promptly stopped caring. I picture this as the old, stubborn guy that's done the same job for decades, and just sighs when he sees the crap forms coming in that haven't been filled out properly, or the jobs not done properly. He just does the work while muttering under his breath about how no one seems to be able to do things the right way anymore, but it's been happening for so long that he can't muster up the effort to bother anyone about it. As for why it worked in Windows 10, I suspect that the internal behavior of how it filled the buffers changed, so there was no longer a difference in the size of the data that got sent.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jan 4, 2019 19:06:47 GMT -5
Wow, sounds like you got to the bottom of it! I can honestly say that I understood most of what you said. A few months ago when this thread began I would not really have had a clue what you were talking about - because of my own lack of understanding, not because of anything else - but in the intervening months I have learnt an awful lot from what you have shared and coded. I saw a lot of familiar discoveries in your message... I too had witnessed the "Ignored Unknown Record" in Wireshark and I think I mentioned it a while back, but I have not quite come far enough yet to be able to diagnose what was causing this and to modify the code to fix it... I'm about to rebuild and test from the latest code commit, but once again your efforts have dazzled me and I cannot express in words how grateful I am to you for putting the time, knowledge and experience into this project. I'll let you know how I get on Keith.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on Jan 4, 2019 19:44:05 GMT -5
Chris, Looks like it works perfectly now! Bravo! Look forward to seeing any future development you do with this DLL Keith
|
|