fwm
Full Member
Posts: 105
|
Post by fwm on May 18, 2019 0:50:48 GMT -5
Ah! I did as you suggested and specified the wildcard common name as the subject, and it worked perfectly.
So maybe wildcards are not going to be such a problem after all?
I don't know why I didn't realise it until you pointed it out but it makes total sense, if the common name is "www.yahoo.com", specify that in LB... if the common name is "*.google.com", specify EXACTLY THAT in LB and let Chrome/SChannel do the rest...
Can you explain what you mean and how this is implemented? Do we just leave that field blank in the BeginTLSServerWithPFX values?
Keith.
|
|
|
Post by Chris Iverson on May 19, 2019 0:36:07 GMT -5
Glad to hear it worked!
As for your question, potentially yes. It's a reference to an idea of additional logic I could add to the DLL.
I'd probably still include the logic of searching for the cert by common name, as it'd be a bit faster, but if nothing's found by name, I'd fall back on enumerating all certificates in the PFX file, and checking for one that's NOT a CA certificate(as any included intermediate certs would be). Hmm, I'd want to make sure I only do that with PFX-loaded certs, though, and not bother searching the user's personal store if they're pulling from installed certs.
It'd be too easy to grab the wrong cert if just pulling a random cert from the personal cert store, but for a PFX file explicitly loaded by the program? Most likely, the only end-entity cert in the file is going to be the one intended for the program to use.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 19, 2019 12:17:48 GMT -5
Sounds good Are you any further forward with being able to use separate PEM and key files, or do you think it will be a requirement to have a PFX file for use with the library? Keith.
|
|
|
Post by Chris Iverson on May 19, 2019 20:11:46 GMT -5
No, not any further with that yet. PFX I was able to do because it's supported by the Win32 API directly. While the API does seem to allow for certificates from files, getting it to accept the private key is going to be a whole separate other thing. Not sure how it works yet. EDIT: With the latest changes I've made, if you're loading from a PFX file, you can now pass an empty string as the subject name(""), and it will auto-select a certificate from the PFX file that has an available private key. My initial choice was going to be to iterate through all the certs in the provided PFX file, and find the first cert that didn't have a CA flag(like any intermediate certs would). Then I found at that the CertFindCertificateInStore() API call has a CERT_FIND_HAS_PRIVATE_KEY option - in other words, it searches the provided cert store(the PFX file, in this case) for any cert that has a matching private key. The only cert that should have a private key in a provided PFX file is the cert intended to be used by the server. Made my job quite a bit simpler I still have had no luck with intermediate certificates, though. So far, all of my research is showing that SChannel will only send intermediate certificates if they are loaded into the server's intermediate cert store. It ignores any other certificates in the PFX file. I'm starting to think the only way to get this to work is to auto-load any intermediate certs from the PFX file into the intermediate cert store, which has the problem of making a semi-permanent change to the machine. (Although an expected one; the same change would happen if you browsed to any website that used the same intermediate cert in Internet Explorer, which auto-adds all intermediate certs it finds to the local intermediate store.)
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 20, 2019 15:03:29 GMT -5
Just tried the latest commit and thought you might like to know my results - or maybe not if you're getting sick of me The crash/freeze seems to have gone, but with a wildcard cert if I don't specify anything in ret = BeginTLSServerWithPFX(hTLS, "", fileName$, "") I get: Checking if connection is available... Attempting to accept connection... Creating TLS context... Acquiring TLS credentials... BeginTLSServerWithPFX return - -2146893042 BeginTLSServer() failed. ret - -2146893042 -- Error - 0 8009030E If I explicitly state the wilcard ret = BeginTLSServerWithPFX(hTLS, "*.mydomain.co.uk", fileName$, "") it works. Keith
|
|
|
Post by Chris Iverson on May 20, 2019 15:51:21 GMT -5
Hmm, now that is strange. The wildcard cert shouldn't make a difference there.
Could I have you try something? I've added additional code to the cert selection logic, that outputs additional data to the wrapperdebug.log file that's in the same folder as the LB test .bas files.
Delete the existing wrapperdebug.log file(if there is one), compile the new code(make sure it's in Debug configuration, not Release, or the debugging code I've added will be stripped from the final file), and try running the test again.
Then, check the contents of the wrapperdebug.log file.
You should see something like this:
taa-chain.pfx Loading from file, grabbing first from private key pCertContext not null Cert selected: thearcaneanomaly.net All prep complete, calling BeginTLSServerInternal()... Entered RunHandshakeLoop Received handshake data - processing message InitializeSecurityContext() called.
The first line should be the path to the PFX file that you pass in to the BeginTLSServerWithPFX() function.
The second line should say if it's searching based on the private key, or by name.
The third line should say not null. (If it's null, then it couldn't find the cert.)
The fourth line should have the subject name of the proper cert. (Yours should be *.mydomain.co.uk, I believe.)
The rest is just normal handshake stuff.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 20, 2019 20:35:22 GMT -5
Ok I have tried that. The output when I do not specify a cert is:
CA-test\certificate.pfx Loading from file, grabbing first from private key pCertContext is null
If I specify *.mydomain.co.uk I get:
CA-test\certificate.pfx Loading cert by name pCertContext not null Cert selected: *.mydomain.co.uk All prep complete, calling BeginTLSServerInternal()... Entered RunHandshakeLoop Received handshake data - processing message InitializeSecurityContext() called.
Could it be the certificate I have generated that is at fault?
Keith
|
|
|
Post by Chris Iverson on May 20, 2019 21:55:44 GMT -5
I don't think so; if it were, I don't think it would work when explicitly specified.
In hindsight, I should've added a call to GetLastError() to get logged out if pCertContext came back null, to give an idea as to why.
I've updated the code to do exactly that, if I could have you do another test. In the meantime, I'm going to be checking on another possible idea I had as to the problem. (Basically, finding a Windows 7 machine to run similar tests on.)
Run the test again, and you should get a new line in the wrapperdebug.log file, saying "GetLastError() return value - <value in hex>"
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 20, 2019 22:14:40 GMT -5
Well that's a separate issue right there... I tried running it on my Windows 10 machine but a debug compile never seems to work for me on Win 10, I have to compile it as release to get it working on Win10, losing the debug info. Not sure why, maybe the version of Visual Studio or the redistributable I am using? (VS runs on, and I am compiling on, my Win7 machine, not my Win10 machine)
However, compiling as release (on Win 7) and running on Windows 10 with no explicit common name does work, whereas it won't work on Windows 7... so I am only experiencing this anomaly on Windows 7 (haven't tried others apart from 7 and 10). Sorry, this post is probably very confusing up to this point.
Going back to Windows 7 to test the latest commit with debug, as requested, the result is now:
CA-test\certificate.pfx Loading from file, grabbing first from private key pCertContext is null GetLastError() return value - 0x80092004 So in summary, everything works as expected in Windows 10, but not explicitly defining the wildcard cert name (and maybe a non-wildcard cert too) in Windows 7 does not seem to be working.
HTH.
Keith.
|
|
|
Post by Chris Iverson on May 20, 2019 22:34:25 GMT -5
Huh, don't know why it wouldn't work on Windows 10, unless somehow the dev environment didn't set up properly.
Unless you mean that you compile it on your Windows 7 machine, and run it on your Windows 10 machine?
If that's the case, it's likely missing the debug version of the main runtime DLL, which only gets installed with Visual Studio.
You can find and copy that DLL, and paste it into the LB test file folder(main project folder) on the Windows 10 machine.
The current one is (likely) located at C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\debug_nonredist\x86\Microsoft.VC141.DebugCRT. There's multiple files there; I think the only one that would need to be copied is vcruntime140d.dll.
As for it not working in Windows 7, it's possible I'm making an incorrect API call; I've just realized that the way I'm invoking the API in the private key search instance is technically against MSDN documentation. I'm checking if the issue is there.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 20, 2019 22:49:17 GMT -5
Unless you mean that you compile it on your Windows 7 machine, and run it on your Windows 10 machine? Yes, in my round-about way, that's what I was trying to say... I will look out for the runtime DLL the next time I try running the debug compile on my Win10 PC and see if it solves the problem. I admire your dedication to this Chris! 10 pages into the thread and 12 months after my first question, you're still chipping away at it! Keith
|
|
|
Post by Chris Iverson on May 21, 2019 0:05:02 GMT -5
WOW, I'm an idiot. Big paragraph, with a bolded section at the end, from this MSDN page: docs.microsoft.com/en-us/windows/desktop/api/wincrypt/nf-wincrypt-certfindcertificateinstoreIt's not working because that flag literally doesn't exist on Windows 7. Derp. (This isn't the "technically incorrect" thing I mentioned earlier; this is just flat-out wrong.) Oh, and I missed a DLL. There's another DLL you need to copy besides vcruntime140.dll; however, it's not in the Visual Studio install folder. Instead, the place you get it from is different depending on if your compiling machine is 64-bit or 32-bit. The file is named ucrtbased.dll, and it's in C:\Windows\SysWOW64 on 64-bit systems, and it's in C:\Windows\system32 on 32-bit systems. (Make sure to grab it from the right place; if you take it from the wrong place on a 64-bit system, you're likely grabbing a 64-bit version of the DLL, which won't work for 32-bit applications like LB.) ANYWAY, since I already have the logic in place for selecting certs based on the end-entity certificate, it shouldn't be hard to switch to it. I just figured that flag would be faster, before I actually read the whole thing. Ooops EDIT: Okay, I *think* I've got it working on Windows 7 now. I'd appreciate a test to confirm from your end, because I'm seeing a weird intermittent hanging issue, and I think it's caused by the VM I'm using for Windows 7. Also, I kind of missed at least like six months there EDIT2: Needed to make more changes to fix the hanging issue. It was caused by Windows 7 splitting up the data transfer in a different way, which both part of the DLL and the LB sample code needed to be rewritten to handle. (Actually, the DLL could handle it already; it just needed the ability to tell the LB code that extra data was available without needing to wait.) This does come with the new caveat that IsReadAvailable and IsTLSReadAvailable are NOT interchangeable on a socket; before, IsTLSReadAvailable was just a wrapper around IsReadAvailable that let you call it with a TLS context instead of a socket(so you could keep track of one less thing). You could still use IsReadAvailable directly, if you wanted to. This is no longer the case; internally, due to the way decryption and buffering work, the DLL may in fact hold on to/buffer/cache more data than is actually returned to LB, for various reasons. This is held back until the next call to DecryptReceive(), where it will check to see if there's still leftover data, and if so, it will return the leftover data instead of doing a network recv() call. IsTLSReadAvailable has been adjusted to check to see if those buffers still have data in them, and will return TRUE if that's the case. Otherwise, you may end up in a situation where the DLL has received information that the program doesn't have yet, but the program keeps trying to read from the network to see if any more data is available(IsReadAvailable). If the client on the other end is waiting for a response from the server, both sides will be locked up. Therefore, once you've created a TLS context, you should only use IsTLSReadAvailable() to check for data. (The DLL did this data buffering previously; I just forgot to take that into account in the IsTLSReadAvailable() function.) For those curious, there are two situations where data will be potentially held back by the DLL: 1) The DecryptMessage() Windows API call did not decrypt everything that was passed in. This is usually due to either hitting a buffer limit, or not having a complete message. The API will give a response of how much of the input was actually processed and decrypted; it is up to the caller of the API to call the function again with the leftover data(plus additional data, if any more was gathered from the client before calling DecryptMessage()). Since the whole point of this DLL is to act as a go-between for LB and the SSPI Windows API, the DLL will handle this transparently, saving any extra still-encrypted bytes and sending them in to DecryptMessage() the next time DecryptReceive() is called. 2) The buffer that the DLL user(LB user) passed in to the DecryptReceive() DLL call was not large enough to hold all of the decrypted data. If this happens, the DLL will copy as much as it can to the LB buffer, and the rest of the unencrypted data will be saved for the next call to DecryptReceive().
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 21, 2019 16:56:23 GMT -5
Haven't had chance to try the latest commit yet but I think I follow (some of) what you're saying. So, let's say we set up a webserver. If we establish a TLS context on, say, port 443, this would only need to listen for encrypted connections. Likewise, if we establish a non-TLS context on, say, port 80, we would not need to listen for encrypted connections. In these examples, losing the ability the change between IsReadAvailable and IsTLSReadAvailable would not be an issue. Am I correct in understanding what you are saying, and that in these examples, it would not be an issue? On that note, if we wanted to do the above, is it safe and possible to establish a non-TLS context on port 80 AND a TLS contest on port 443, both at the same time? And check one, then the other, then the first, then the second, until we see a connection on one or the other and respond appropriately? In terms of the buffer, I recall in an early iteration of this wrapper there was a potential issue we discussed with a buffer size being set, if we wanted to download a file or web page that was larger. Does any of what you talked about above negate this issue, or is it not related to that issue? IIRC it was the buffer element of these functions: Function Receive(sock, byref buf$, bufLen) CallDLL #LBSchannelWrapper, "Receive",_ sock as long,_ buf$ as ptr,_ bufLen as long,_ Receive as long End Function
Function DecryptReceive(hTLS, byref buf$, bufLen) CallDLL #LBSchannelWrapper, "DecryptReceive",_ hTLS as ulong,_ buf$ as ptr,_ bufLen as long,_ DecryptReceive as long End Function but you might be talking about something else entirely so forgive me if I am getting lost! I think you have established that you are NOT an idiot, contrary to your first statement in your last post... otherwise we would not have got to this stage with the project! Keith
|
|
|
Post by Chris Iverson on May 21, 2019 18:36:51 GMT -5
If I'm understanding what you're asking for the first question, that's correct. IsReadAvailable() and IsTLSReadAvailable() is tracked per connected socket/TLS context, so as long as you're not using both on the same socket, you should be fine. Using IsReadAvailable() on a Port 80 server socket at the same time as using IsTLSReadAvailable() on a Port 443 TLS context should work just fine. Just don't use IsReadAvailable() on a socket tied to a TLS context, and vice-versa.
And yes, I do believe there would be no problem with using different sockets to handle both ports at the same time. My DLL should not have a problem with that, although it's admittedly untested at the moment. Not a bad idea of something to test next, actually... shouldn't be hard to convert my existing plaintext/TLS on the same port example into plaintext and TLS on different ports simultaneously.
As for the buffer, the only discussion regarding buffers we had earlier was, I believe, about the feasibility of duplicating the way MeSock handled data returned to LB. While it was very convenient for the LB side(didn't require preallocating, just take the return value and pass it through winstring()), that method would only work for textual data. Binary data, especially data that could potentially have null bytes in it, would fail. (Also, you'd need to call an additional method to free the return buffers once you're done with them. I'm honestly unsure how the MeSock DLL handled this, since it didn't have the option to do so. Without a method to do so, more and more memory would be used by the application, and if left running long enough, it would crash with an OutOfMemory exception.)
As for needing to download files/web pages/data larger than a buffer size, that's pretty much unrelated to the issue I was seeing, and unfortunately, that's something that the application layer(LB, in this case) simply needs to handle. You have to write your applications to be able to handle receiving incomplete messages, and to handle reassembling them if that's the case. It's not just because you might not have a large enough buffer for incoming data; it's because, due to the way network transmission works, your computer may not even have received all of the data from the connected machine at the moment you make the Receive() call. If that's the case, you're going to get an incomplete message back. The OS will simply hand you whatever data is available from the socket. This may not be one complete message.
Actually, it may be MORE than one complete message! If the other side of the network sent multiple "messages" at once, the actual data you receive from the Receive() call at the time you make it may have only part of one message, it may have the complete first message and part of the second, or it may have both the first and second complete messages.
The ONLY guarantee you get from a TCP stream is that the bytes arrive and will be received in the order they were sent. Any further division than just a stream of bytes has to be handled by the application.
(This is actually the cause of the data-held-back scenario #1, in my previous post. It's possible you will either not receive an entire encrypted message, or will receive a bit more than just one encrypted message. If you don't have the whole encrypted message, the DecryptMessage() API will likely be unable to decrypt it. If it can't decrypt anything, you'll get a SEC_E_INCOMPLETE_MESSAGE error. If you have more data than just one encrypted message, the bytes that can be decrypted will be decrypted, and the DecryptMessage API will indicate what portion of the input buffer was unable to be decrypted. It's up to the user of the DecryptMessage() API call(my DLL, in this case) to send those bytes back in later.)
Now, if you were interacting with something at a higher layer than just TCP sockets, like an HTTP protocol library or an SMTP protocol library, something that sits on top of sockets, those libraries would probably handle data reassembly for you, and would be able to pass complete messages back to you(assuming your own buffers were large enough), and not pass back more than one complete message at once. However, those are niceties that are provided by the application-layer libraries, and they are not available if you're interacting with sockets directly.
This is why most established protocols that allow the transfer of arbitrary payloads will have some way of communicating when the payload transfer is complete.
SMTP has it's main payload in the DATA command. In the original SMTP spec, anything following the DATA command was considered to be part of the message, until the client sent a period(.) on a line by itself, followed by an empty line. This was the signal that the DATA payload was at the end, so that the server knew where the end of the data was.
Later revisions to the SMTP spec added a SIZE parameter to the MAIL command, so that the client can specify the exact size of the data that will be transferred.
In HTTP, an empty line is used to indicate the end of the headers of a request or response(after the request/response command/code and any headers), and both the client and server use a Content-Length header if there's a body of content being provided besides just the one-line response code and related headers(such as file content being transferred after a GET request, like an HTML file, or the request parameters for a POST request). You'll notice that I include the Content-Length header in my little server test, as well as making absolutely sure to add an additional crlf$ to the end of the headers.
This lets the other side know when they have received all of the data, and if they should expect more. If you're going to be using a custom protocol over sockets, I recommend building into the protocol some way of knowing the size of the data being sent.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 21, 2019 19:02:21 GMT -5
Thanks for that detailed answer. All of what I am doing is based on common protocols, HTTP, SMTP, FTP etc. so generally this would not be an issue anyway. When using HTTP to GET from an external server, I guess the most obvious thing to do is to check the Content Length header and see if the preset buffer size will be large enough. If not, make the buffer larger and then request the download again. I apologise if I keep complicating this discussion by asking about different scenarios - but I really fancy pushing your wrapper to its limit! Keith
|
|