|
Post by donnybowers on Sept 22, 2020 1:46:21 GMT -5
I've been trying for hours to use httpget$ to download a page from one of the weather services so that I can parse out the current temperature for a temperature comparison module (outside vs greenhouse temps) I'm writing for my gardening program. I always get the error "the parameter is incorrect". I'm just using the command in the help file and replacing "www.libertybasic.com" with various weather page urls from different weather sites. Once I figure out how to use the command to get the data I need then I'll incorporate it into my program. I'm using LB 4.5.1 in a Windows XP virtual machine.
About the only page I can get httpget$ to download is the Liberty BASIC site. I've found a few others, all http: rather than https:
|
|
|
Post by tenochtitlanuk on Sept 22, 2020 5:39:58 GMT -5
Can you give an example URL where it gives this error? It works for me on LB4.5.1 on htpp and https sites...
print httpget$("http://www.diga.me.uk/index.html") print "_________________________________________________" print httpget$("https://www.bbc.co.uk/weather/2654675")
EDIT It fails in LB5- which is only an ALPHA version...
|
|
|
Post by meerkat on Sept 22, 2020 5:55:42 GMT -5
I tried a couple weather web pages and they worked for me.
Some had so many tags in the web pages that it takes forever. And not much left after striping the tags.
print "===== Underground weather ====" a$ = httpget$("http://www.wunderground.com/weather/us/ca/redding") print a$ a$ = tags$(a$,"|") print "-------- after strip tags -----------------" print a$
print "======= google weather ====================" a$ = httpget$("http://www.google.com/search?q=weather&oq=weather&aqs=chrome..69i57j69i60j69i61j69i65j69i60.11470j0j4&sourceid=chrome&ie=UTF-8") print a$ a$ = tags$(a$,"|") print "-------- after strip tags -----------------" print a$ end ' -------------------------------- ' Replace HTML <tags> with$ ' -------------------------------- FUNCTION tags$(tag$,with$) i = instr(tag$,"<") j = instr(tag$,">",i) WHILE j > i x$ = mid$(tag$,i,j+1-i) tag$ = replstr$(tag$,x$,"|") i = instr(tag$,"<") j = instr(tag$,">",i+1) WEND tags$ = tag$ END FUNCTION
|
|
|
Post by Carl Gundel on Sept 22, 2020 8:25:18 GMT -5
Can you give an example URL where it gives this error? It works for me on LB4.5.1 on htpp and https sites...
print httpget$("http://www.diga.me.uk/index.html") print "_________________________________________________" print httpget$("https://www.bbc.co.uk/weather/2654675")
EDIT It fails in LB5- which is only an ALPHA version...
It fails in LB5 on Windows, Linux, MacOS?
|
|
|
Post by tenochtitlanuk on Sept 22, 2020 9:55:02 GMT -5
/home/john/LBandJB/lb5x64-351/lb5alpha-351 on Linux Unhandled exception: URL not found......
but works under Wine
|
|
|
Post by Carl Gundel on Sept 22, 2020 10:44:29 GMT -5
/home/john/LBandJB/lb5x64-351/lb5alpha-351 on Linux Unhandled exception: URL not found......
but works under Wine
Thanks. Interesting about it working under Wine.
|
|
|
Post by donnybowers on Sept 23, 2020 19:09:41 GMT -5
I'm getting the feeling that perhaps there's something missing in a virtual XP machine on LinuxLiteOS, Virtualbox or a compatibility issue somewhere.
'These worked in virtual XP machine (LB 4.5.1 Pro). Unfortuantely I can't scrape the temperature info I want from them.
'a$ = httpget$("https://www.bbc.co.uk/weather/2654675") 'a$ = httpget$("http://www.diga.me.uk/index.html") 'a$ = httpget$("http://www.google.com/search?q=weather&oq=weather&aqs=chrome..69i57j69i60j69i61j69i65j69i60.11470j0j4&sourceid=chrome&ie=UTF-8")
'These ones give me "parameter is incorrect" error ("invalid parameter" in Wine). These contain the info I want to parse a temperature from.
'a$ = httpget$("https://weather.com/weather/today/l/d704a04e3b4ba603a9a59ced71821c7a94a0c038e305a10f5c50593649d5b07e") 'a$ = httpget$("https://www.wunderground.com/weather/us/mi/shelby/49455") a$ = httpget$("https://www.accuweather.com/en/us/ray-township/48096/weather-forecast/2258411")
print a$
With the Google search I don't get the weather data, but rather what looks like maybe page data being called up with some web code that I don't understand, not the actual weather data. I would be interested in knowing if my weather.com, accuweather.com and wunderground.com urls work with other machines. That might help me narrow it down a little, but I'm pretty much convinced at this point that either I have some compatibility issues or else these websites simply don't want anyone to download them. Maybe I need Java or something installed on the virtual Windows XP machine? But there are some https sites I can download, but many more that I can't. It seems like there were some sites I used to be able to download pages from and now I can't.
|
|
|
Post by Chris Iverson on Sept 24, 2020 11:55:11 GMT -5
For the ones you say don't work, the first two work for me.
The third one fails with a weird error message, but I'm pretty sure that the error message is a result of LB misinterpreting the resulting error code, and not the actual error.
The error I get is:
--------------------------- Warning --------------------------- Runtime Error: The resource loader cache doesn't have loaded MUI entry. ( OS error 16r3B01 )
(see error.log for more information) --------------------------- OK ---------------------------
The reason I think this is misinterpreted is because it takes a long time for the error to come up, making me think the connecting is timing out. And further, attempting to connect to that accuweather page using the Powershell command Invoke-WebRequest(powershell version of "curl", just issues a web request and retrieves the response), I get a timeout error.
PS C:\Users\cjnoo> iwr https://www.accuweather.com/en/us/ray-township/48096/weather-forecast/2258411 Invoke-WebRequest: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond..
(iwr is a shorthand alias for Invoke-WebRequest)
Additionally, Windows 10 v1803 ships with a copy of the curl utility. Using that also produces a timeout.
PS C:\Users\cjnoo> curl https://www.accuweather.com/en/us/ray-township/48096/weather-forecast/2258411 curl: (56) Send failure: Connection was reset
Lastly, attempting to use curl to connect from within Linux(Ubuntu WSL, in this case), results in a 403 Forbidden response from Accuweather.
chris@Sayaka:/mnt/c/Users/cjnoo$ curl https://www.accuweather.com/en/us/ray-township/48096/weather-forecast/2258411 <HTML><HEAD> <TITLE>Access Denied</TITLE> </HEAD><BODY> <H1>Access Denied</H1>
You don't have permission to access "http://www.accuweather.com/en/us/ray-township/48096/weather-forecast/2258411" on this server.<P> Reference #18.856019b8.1600965782.10835067 </BODY> </HTML>
All that said, I think Accuweather is explicitly preventing scraping by denying connecting user-agents that aren't typical web browsers, either by ignoring/dropping the connection or returning a 403 Forbidden.
|
|
|
Post by David Drake on Sept 25, 2020 7:36:36 GMT -5
If you can settle for current temperature at an airport, I've had success using httpget to pull the NWS RSS feed for a given location. I can then parse out the temp, current conditions, etc. To get the RSS URL, go here (https://w1.weather.gov/xml/current_obs/), pick your state and airport. Use the URL as the source. For example, the simple RSS feed for the Birmingham, Alabama airport is: w1.weather.gov/xml/current_obs/KBHM.rssI use this code, and it works fine: a$=httpget$("https://w1.weather.gov/xml/current_obs/KBHM.rss") Good luck!
|
|
|
Post by David Drake on Sept 25, 2020 7:43:43 GMT -5
More specifically, this will print the temperature:
a$=httpget$("https://w1.weather.gov/xml/current_obs/KBHM.rss") character=instr(a$, " F ") temp$=mid$(a$,character-3,5) print "Temp in Birmingham:";temp$
|
|