Page 1 of 2
download data from internet
Posted: Sunday 20th May 2018 4:50am
by westconn
i am just starting with gambas, i had previously started some code in vb6 to download data from a website, in which i had successfully downloaded the data, decided to try to complete in gambas, but so far have not really understood how to login to the website then download the data from a selected page, i believe i should use the curl library but any examples for logging in first would be helpful
Re: download data from internet
Posted: Sunday 20th May 2018 10:44am
by cogier
Hi westconn and welcome to the forum.
There is the easy way to do this in Gambas using tools you probably already have installed on your system. The following example uses
curl and
wget to go and get your external IP address but one or both will need to be installed on your system.
Public Sub Form_Open()
Dim sResult As String
Shell "curl http://ipecho.net/plain" To sResult
Print sResult
''OR
Shell "wget -O- http://ipecho.net/plain" To sResult
Print sResult
End
Then there is the harder way, the following code requires
gb.net.curl. To activate this component you will need to go to the Gambas menu
Project>Properties>Components and activate
gb.net.curl. This code does the same as the above.
''Needs gb.net.curl
Public Sub Form_Open()
Print Get("http://ipecho.net/plain") 'Find this computers external IP address
End
Public Sub Get(sURL As String) As String 'To 'Get' stuff from the web
Dim hClient As HttpClient 'To create a HTTP Client
Dim sResult As String 'To store the result
hClient = New HttpClient As "hClient" 'Create a HTTP Client
With hClient 'With the Client..
.URL = sURL 'Set up the URL
.Async = False 'No Asynchronous transmission?
.TimeOut = 60 'Don't hang around waiting for more than 60 seconds
.get 'Get the data
End With
If Lof(hClient) Then sResult = Read #hClient, Lof(hClient) 'When all the data is downloaded store it in sResult
Return sResult 'Return the result
End
If you go to the Gambas Farm and download
Your_location (I need to get the downloads up!
) this may help you as it gets a CSV file from the net and displays the data.
Re: download data from internet
Posted: Sunday 20th May 2018 11:04am
by westconn
i have no problem using wget or curl from a terminal, so i could shell out to either
my problem is that i need to login, then navigate to the correct url, unless there is some way to shortcut that, if i just try to go directly to the download i am redirected to the login page
can i pass parameters to the url to bypass the login page?
Re: download data from internet
Posted: Sunday 20th May 2018 12:19pm
by westconn
(I need to get the downloads up!
so far i have been unable to find any link to the farm, the search on gambas one does not seem to be working, the gambasfarm.org does not seem to have the samples
do you have a direct link to it?
also thanks for your reply and samples
Re: download data from internet
Posted: Sunday 20th May 2018 2:43pm
by cogier
Re: download data from internet
Posted: Monday 21st May 2018 11:01am
by jornmo
What do you mean by login?
Logging in by a form on a website, or the "pop-up" box that comes with web browsers?
For the first scenario I believe you can use Curl's HTTP FORM component:
http://gambaswiki.org/wiki/comp/gb.net.curl/httpform
(I am not sure if it can handle sessions. If it can't it will not work. It seems there might be a possibility to store the session in a cookie, and then load it again)
For the second you can use Curl's password property:
http://gambaswiki.org/wiki/comp/gb.net.curl/curl
Re: download data from internet
Posted: Wednesday 23rd May 2018 8:39am
by westconn
@cogier, so easy with pictures lol, i did not think of the farm being built into gambas pogram
@jornmo, yes a login form on the webpage, any examples anywhere on using httpform, the manual did not seem to help me much
Re: download data from internet
Posted: Wednesday 23rd May 2018 9:54am
by westconn
client.URL = myurl
client.async = False
data = "login_username=myusername&login_password=mypass&submit=Login"
z[1] = "content - length: " & Str(Len(data))
' client.get ' this works
client.post("application/x-www-form-urlencoded", data, z)
If client.Status < 0 Then
Return ""
Endif
' no data available
If Not Lof(client) Then
Return ""
Endif
' Reads the data from the server and returns it as a String
d = Read #client, Lof(client)
TextArea1.Text = d
i have spent some time to get this far, but no result, nothing is returned to client, if i use client.get instead, the html code of the web page is returned
Re: download data from internet
Posted: Wednesday 23rd May 2018 2:43pm
by jornmo
Since I do not know how the site is built, it is difficult to know what to do. But, when you have got a successfull login, you need somehow to store the session, and then load the URL where the data you want to retrieve is stored with the current session loaded. (I have never attempted such a thing myself)
However, I do not think this is the normal way this kind of operation normally is done. Your webiste should have some API that your program can talk to, and get a JSON/XML/CSV file from. But that again depends on if this is your website?
If it is your webiste, I would probly create a folder that is password protected with HTACCESS/HTPASSWD (if using Apache?) that has a .php file (or what ever language you are using) that spits out a JSON file.
Re: download data from internet
Posted: Wednesday 23rd May 2018 9:30pm
by westconn
if this is your website?
not my web site, i am trying to get the usage data from my isp, so i can display as graph, afaik they have no api or other to get the data
there is an option on the url xml=yes, but that only returns a summary of the data, not the detail and still requires to login first
i have successfully returned and parsed the data using vb, but now am trying to get the same working for linux
while there would be thousands of examples using several methods for vb, i can find almost nothing for gambas, or even other linux commands (curl or wget) i could shell to
though on further searching through the farm (and google) i have found several other working (or not) examples that are helping me with other parts