fwm
Full Member
Posts: 105
|
Post by fwm on May 3, 2019 16:02:46 GMT -5
Don't read too much into it, I ran it for about 60 seconds and probably missed something! I was just excited to try it but I'll try and have a proper look over the weekend.
Keith.
|
|
|
Post by Chris Iverson on May 7, 2019 14:43:57 GMT -5
Well, independent files are being worked on, but right now it's still at the "I'm just trying to figure out how to get this to work" stage.
Also, I was able to confirm that passing in an initial buffer to PerformServerHandshake works, but only if the initial buffer is large enough. Otherwise, it fails out. Need to figure out where it's going wrong to fix it.
(Basically, if I started with a 512-byte read buffer, I was able to to HTTP with Chrome just fine, but HTTPS would fail if I detected a ClientHello and tried to switch over. However, if I made it a 1024-byte read buffer, both HTTP and HTTPS worked fine.)
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 8, 2019 15:32:54 GMT -5
Sounds good - hope you can make some progress with it once you work it all out I still haven't had chance to try out the latest code yet - too busy working! At some point I'll get round to it... Keith.
|
|
|
Post by Chris Iverson on May 9, 2019 11:40:35 GMT -5
Well, I have it working with a specified PFX file(combined cert + key in PKCS#12 format, supported by OpenSSL, also called .p12), because the API supports PFX files directly - I figured out what the previous problem I was having was. (I misunderstood one of the API options, and I was telling the API to discard the private key, and then trying to use the private key to authenticate a session, which obviously won't work.)
However, it's not sending the intermediate certificate with it; it's only sending the end-entity cert. While this is fine for basic purposes, I want to make sure intermediate certs get sent by the server properly.
New function BeginTLSServerWithPFX() has two additional parameters, one for specifying the file path/name, and one for specifying the password on the PFX file, so you can have the cert/key encrypted and still use it with LB. If there's no password, use an empty string("").
I don't support individual PEM/KEY files yet, I need to figure out what needs to be done for those to be supported.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 9, 2019 12:31:23 GMT -5
This is very exciting...! I hope to have time to try this tonight or tomorrow Thanks Chris! Keith
|
|
|
Post by Chris Iverson on May 9, 2019 16:39:54 GMT -5
Well, while researching the intermediate certificates issue, I came across some sample code that pointed out the exact issue I was having with the ClientHello switchover, and exactly how to fix it.
So, I fixed that in master branch, and modified the test-https source code to include the ability to switch over to TLS.
Basically, now, if you run the test-https.bas program that's in the master branch, it will detect if the incoming message is a TLS ClientHello message. If it is, it passes the received buffer over to PerformServerHandshake() before continuing server processing.
If not, it does everything in plaintext. Should demonstrate how simple it is to switch between the two!
EDIT: And I've just now realized that, with the additional logic I've had to put into RunHandshakeLoop(), I could probably remove all initial read code and the first AcceptSecurityContext() call from the PerformServerHandshake() function entirely, vastly simplifying it.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 13, 2019 13:34:18 GMT -5
Hi Chris, Finally had chance to try the test.bas and test-https.bas programs. The test.bas program ran perfectly and did what it should do, so I must have done something wrong with OpenSSL the other day when I tried it out. When running test-https.bas, I couldn't convince Chrome to accept the certificate when it was in the local machine cert store, even if I ran LB as admin etc., but it worked no problem once I added the certificate to the local user store and Chrome was happy. I tried restarting Chrome (not the PC) but maybe system permissions etc. also prevent LB accessing certs stored in the local machine cert store. However I am thinking that "proper" use of this would involve a "real-life" TLS certificate from a recognised certification authority - or at least that's how I am hoping to use it. So for example, I would like to be able to point my program at a Let's Encrypt certificate and then have LB serve up some pages to a remote visitor, so long as they are using the domain that matches the cert. Do you have any example code for using certificate files instead of the certificate stores or are you still working on this bit? At the risk of repeating myself, I will say again that this is an amazing library and I think the work you are doing with this is amazing! I just hope you are enjoying researching and creating it as much as we (or I? Anyone else?) are enjoying testing it out Keith.
|
|
|
Post by Chris Iverson on May 13, 2019 14:11:44 GMT -5
Right now, the work for using files instead of the stores is on the CertFromFile branch. Considering PFX files are, in their most basic form, working, I might just merge what I have now into the master branch. If you switch to the CertFromFile branch, the example test-https-pfx.bas file will load from the PFX file successfully. The only issue is, as mentioned above, I have so far been unable to have SChannel send the intermediate certificates when performing the server handshake if the intermediate certificates aren't loaded into the certificate store, even if said intermediate certs are in the PFX file. I'm trying things out, and as a workaround, I can add the intermediate certs as a temporary workaround(in fact, Internet Explorer will automatically load valid intermediate certs it encounters on the web into your intermediate store, for caching), and remove them if wanted after the session closes. As for the issue with Chrome, maybe a cert didn't get imported properly, or something? It's always worked for me. *shrug* Overall, I am very much enjoying this, it's fun to puzzle out. The times when I'm just trying to dig for information that doesn't seem to exist anywhere, though, is sometimes like pulling teeth.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 17, 2019 7:37:16 GMT -5
Ok, here's what I have tried. I may well be "jumping the gun" and expecting this to work when it isn't ready to, but thought I would give some feedback. This is on Windows 7, just in case it makes a difference (although I doubt it at this point). Using the test-https-pfx.bas program, I wanted to try with a PFX certificate for an actual domain. I created a PFX file without password using one of my existing domain .crt and .key files, using OpenSSL to do so. The .crt already contained an intermediate certificate which I left in. I also set my hosts file for this particular domain to 127.0.0.1 just to avoid changing actual DNS server settings etc. In the .bas file I updated the filename path for the PFX file, and in the BeginTLSServerWithPFX() function I changed "localhost" to the domain name covered by the certificate. Setting the port to 27016, and using Chrome: https://test.mydomain.co.uk:27016/ resulted in LB getting stuck at "Acquiring TLS credentials..." and doing nothing else. http://test.mydomain.co.uk:27016/ resulted in LB getting to "PerformServerHandshake() failed. - -2146893055 - Error: -7FF6FCFF" and then sometimes crashing and sometimes not crashing. Trying with port 443, and using Chrome: https://test.mydomain.co.uk/ resulted in LB getting to "PerformServerHandshake() failed. - -2146893055 - Error: -7FF6FCFF" and then sometimes crashing and sometimes not crashing. I am probably charging way ahead but thought I'd see what happened. Internet Explorer seems to crash out regardless of which port and whether I use http or https. Maybe that helps, maybe it complicates things.... Keith
|
|
|
Post by Chris Iverson on May 17, 2019 12:06:25 GMT -5
Hmm, I'll try to do some similar testing myself. Using a "localhost" self-generated cert has always worked, but I'll try using a LetsEncrypt-issued cert for one of my own sites.
EDIT: I think I may be missing something, because I pulled the live cert from one of my websites, thearcaneanomaly.net, converted it to a PFX, opened it in LB, and used it successfully.
Does it also fail using the script-generated localhost cert?
Also, that return error code matches 0x80010301, which is SEC_E_INVALID_HANDLE. I'm pretty sure that means it failed to set the cert up in some way.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 17, 2019 18:21:36 GMT -5
The problem using the localhost certificate is that Chrome won't recognise it unless I add the CA cert into the certificate store. I am not in front of the machine I can play about with adding certs to the store right now but I will try it over the weekend.
I wanted to test something that Chrome will accept as being from a real CA, so I wonder what you did differently to get it to work for you? Did you include any intermediate certs in creating the PFX file, or just purely the domain cert?
Keith.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 17, 2019 20:54:01 GMT -5
I just retried the real domain certificate, both a PFX made from the pure domain certificate and also one including the intermediate certificate. I also tried creating PFX files both with and without a password. LB just keeps crashing as soon as Chrome gets to "Establishing secure connection". This was on a Windows 10 PC.
I also ran LB as admin to see if this was an issue, but this did not seem to make any difference.
I wonder if the certificate file I am generating is not correct in some way? I am using this command:
openssl pkcs12 -export -inkey mydomain.key -in mydomain.crt -out certificate.pfx
I have also tried the s_client instead of Chrome:
openssl s_client -connect test.mydomain.co.uk:27016
which gets to CONNECTED(00000214) before LB crashes.
Keith.
|
|
|
Post by Chris Iverson on May 17, 2019 23:04:56 GMT -5
It's worked with both including the intermediate cert, and not including it. (I will note, however, that I suspect the reason it worked not including the intermediate is because my system already has the Let's Encrypt intermediate added to it's local store, due to the way IE handles intermediates.)
These are the commands I used to generate both PFX files that worked, from my live cert:
No intermediate:
openssl pkcs12 -export -in cert.pem -inkey privkey.pem -out taa-nosub.pfx -passout pass:
Intermediate:
openssl pkcs12 -export -in cert.pem -inkey privkey.pem -certfile chain.pem -out taa.pfx -passout pass:
(The "passout pass:" part is equivalent to just hitting "enter" when asked for an export password.)
I copied both PFX files to the CA-test folder(though it shouldn't matter where they are, as long as you get the file path correct).
I changed the line:
fileName$ = "CA-test\localhost\localhost.pfx"
to
fileName$ = "CA-test\taa-nosub.pfx" ' or "CA-test\taa.pfx"
I changed the line
ret = BeginTLSServerWithPFX(hTLS, "localhost", fileName$, "")
to
ret = BeginTLSServerWithPFX(hTLS, "thearcaneanomaly.net", fileName$, "")
And it worked(after adjusting the hosts file to point thearcaneanomaly.net to 127.0.0.1, of course.)
...
I did just think of something, though.
When you say "domain name covered by the certificate", do you mean it's the actual subject of the cert? Like, it says "Issued to: <domain name you're using>", and the details tab says "Subject: <domain name you're using>"? Or is the domain name you're using listed as a Subject Alternative Name? Because I don't actually know if the current code will with with subjectAltNames.
EDIT: Pretty sure the subjectAltNames thing is exactly the issue. I modified the custom script to generate a localhost cert with "thearcaneanomaly.net" as a subjectAltName, and got exactly the behavior you described above(hanging at "Acquiring TLS credentials". Looking into this.
EDIT 2: And confirmed that if you modify the BeginTLSServerWithPFX() call to use the actual subject of the cert, the current code will work. (It will also connect properly because, once the cert is loaded, SChannel will handle searching subjectAltNames when actually accepting a ClientHello. So, for my own example, even if I use "localhost" as the name to call BeginTLSServerWithPFX(), it will still work and output the correct cert when I navigate to "thearcaneanomaly.net". Still need to fix this, but it's at least a workaround to be able to test with.
EDIT 3: Wow, multiple bugs. First, the whole "subjectAltName" thing. Second, a bug in the PFX code that caused an exception down the line, due to accidentally freeing the CertStore handle twice. This is what causes the hang, I think, since the call to the function never actually returns.
Then, there's a bug in the wrapper function code in LB, that causes the BeginTLSServerWithPFX() wrapper function to ALWAYS return zero - success. This is what causes the program to keep trying to connect instead of realizing that acquiring the cert failed, and stopping.
Second two bugs are fixed, working on the first one.
In related news, I thank God for Visual Studio's ability to debug DLLs running in any process. It's a snap to connect to the LB process, run the LB sample code, and then get a full debugger session on the DLL code, to step through and see what's going on.
|
|
fwm
Full Member
Posts: 105
|
Post by fwm on May 17, 2019 23:45:05 GMT -5
And I thank God that you know what you're doing...!!! Yes, I didn't think of it until you mentioned it but the certificate is a LE wildcard certificate. Doh! I imagine there is extra technical magic needed to get such a thing working... I will go and hang my head in shame, while testing it again using a simple certificate where the domain name EXACTLY matches the subject of the certificate. But hopefully it has helped you find other bugs to work out - which of course was my intention all along Keith
|
|
|
Post by Chris Iverson on May 18, 2019 0:22:18 GMT -5
Ugh. I hadn't even considered wildcard certs. That's going to be even more trouble. Plus that's going to have to be string manipulation in C/++, which is always !!FUN!!.
Very tempted to just say you have to use the common name/subject name of the cert, and let SChannel handle actually matching the cert to the name in the ClientHello request. At this point, I'm just duplicating effort that will have to be done by SChannel anyway. That's the only thing that the domain name passed into BeginTLSServer*() is used for, anyway; it's not actually used by the server itself. It's just used for certificate searching. After it has the cert to check, SChannel handles parsing the certs and looking through for matching names from the ClientHello request, anyway. (And if you need the server name the client's connecting with, you can use the Host: header provided by the client themselves(in HTTP), or implement a similar header if using your own custom protocol.)
Out of curiosity, go ahead and try to use your wildcard cert, but specify the name as the subject in LB, including the wildcard flag, like:
ret = BeginTLSServerWithPFX(hTLS, "*.mydomain.co.uk", fileName$, "")
EDIT: Or, even better, just auto-use the first non-CA cert in the PFX file.
|
|