download data from internet
download data from internet
i am just starting with gambas, i had previously started some code in vb6 to download data from a website, in which i had successfully downloaded the data, decided to try to complete in gambas, but so far have not really understood how to login to the website then download the data from a selected page, i believe i should use the curl library but any examples for logging in first would be helpful
- cogier
- Site Admin
- Posts: 1157
- Joined: Wednesday 21st September 2016 2:22pm
- Location: Guernsey, Channel Islands
Re: download data from internet
Hi westconn and welcome to the forum.
There is the easy way to do this in Gambas using tools you probably already have installed on your system. The following example uses curl and wget to go and get your external IP address but one or both will need to be installed on your system.
There is the easy way to do this in Gambas using tools you probably already have installed on your system. The following example uses curl and wget to go and get your external IP address but one or both will need to be installed on your system.
Public Sub Form_Open()
Dim sResult As String
Shell "curl http://ipecho.net/plain" To sResult
Print sResult
''OR
Shell "wget -O- http://ipecho.net/plain" To sResult
Print sResult
End
Then there is the harder way, the following code requires gb.net.curl. To activate this component you will need to go to the Gambas menu Project>Properties>Components and activate gb.net.curl. This code does the same as the above.
''Needs gb.net.curl
Public Sub Form_Open()
Print Get("http://ipecho.net/plain") 'Find this computers external IP address
End
Public Sub Get(sURL As String) As String 'To 'Get' stuff from the web
Dim hClient As HttpClient 'To create a HTTP Client
Dim sResult As String 'To store the result
hClient = New HttpClient As "hClient" 'Create a HTTP Client
With hClient 'With the Client..
.URL = sURL 'Set up the URL
.Async = False 'No Asynchronous transmission?
.TimeOut = 60 'Don't hang around waiting for more than 60 seconds
.get 'Get the data
End With
If Lof(hClient) Then sResult = Read #hClient, Lof(hClient) 'When all the data is downloaded store it in sResult
Return sResult 'Return the result
End
If you go to the Gambas Farm and download Your_location (I need to get the downloads up! ) this may help you as it gets a CSV file from the net and displays the data.Re: download data from internet
i have no problem using wget or curl from a terminal, so i could shell out to either
my problem is that i need to login, then navigate to the correct url, unless there is some way to shortcut that, if i just try to go directly to the download i am redirected to the login page
can i pass parameters to the url to bypass the login page?
my problem is that i need to login, then navigate to the correct url, unless there is some way to shortcut that, if i just try to go directly to the download i am redirected to the login page
can i pass parameters to the url to bypass the login page?
Re: download data from internet
so far i have been unable to find any link to the farm, the search on gambas one does not seem to be working, the gambasfarm.org does not seem to have the samples(I need to get the downloads up!
do you have a direct link to it?
also thanks for your reply and samples
- cogier
- Site Admin
- Posts: 1157
- Joined: Wednesday 21st September 2016 2:22pm
- Location: Guernsey, Channel Islands
Re: download data from internet
The Gambas Farm on Gambas.One is for viewing only. You need to start Gambas and select the Farm from the menu, note the 'Installed software' option, you may need it later: -
Regarding adding login in details you need to add the details to the URL
If you need more help click here
https://www.google.com/search?client=ub ... 15&bih=936
Then select "All software"
Then from the list select the software you want to download
Note that the software will be available from the main menu under "Installed software"Regarding adding login in details you need to add the details to the URL
Code: Select all
https://xxx.university.edu/portal/server.pt?userName=SomeUser&password=somePassword
https://www.google.com/search?client=ub ... 15&bih=936
Re: download data from internet
What do you mean by login?
Logging in by a form on a website, or the "pop-up" box that comes with web browsers?
For the first scenario I believe you can use Curl's HTTP FORM component:
http://gambaswiki.org/wiki/comp/gb.net.curl/httpform
(I am not sure if it can handle sessions. If it can't it will not work. It seems there might be a possibility to store the session in a cookie, and then load it again)
For the second you can use Curl's password property:
http://gambaswiki.org/wiki/comp/gb.net.curl/curl
Logging in by a form on a website, or the "pop-up" box that comes with web browsers?
For the first scenario I believe you can use Curl's HTTP FORM component:
http://gambaswiki.org/wiki/comp/gb.net.curl/httpform
(I am not sure if it can handle sessions. If it can't it will not work. It seems there might be a possibility to store the session in a cookie, and then load it again)
For the second you can use Curl's password property:
http://gambaswiki.org/wiki/comp/gb.net.curl/curl
Re: download data from internet
@cogier, so easy with pictures lol, i did not think of the farm being built into gambas pogram
@jornmo, yes a login form on the webpage, any examples anywhere on using httpform, the manual did not seem to help me much
@jornmo, yes a login form on the webpage, any examples anywhere on using httpform, the manual did not seem to help me much
Re: download data from internet
client.URL = myurl
client.async = False
data = "login_username=myusername&login_password=mypass&submit=Login"
z[1] = "content - length: " & Str(Len(data))
' client.get ' this works
client.post("application/x-www-form-urlencoded", data, z)
If client.Status < 0 Then
Return ""
Endif
' no data available
If Not Lof(client) Then
Return ""
Endif
' Reads the data from the server and returns it as a String
d = Read #client, Lof(client)
TextArea1.Text = d
i have spent some time to get this far, but no result, nothing is returned to client, if i use client.get instead, the html code of the web page is returnedRe: download data from internet
Since I do not know how the site is built, it is difficult to know what to do. But, when you have got a successfull login, you need somehow to store the session, and then load the URL where the data you want to retrieve is stored with the current session loaded. (I have never attempted such a thing myself)
However, I do not think this is the normal way this kind of operation normally is done. Your webiste should have some API that your program can talk to, and get a JSON/XML/CSV file from. But that again depends on if this is your website?
If it is your webiste, I would probly create a folder that is password protected with HTACCESS/HTPASSWD (if using Apache?) that has a .php file (or what ever language you are using) that spits out a JSON file.
However, I do not think this is the normal way this kind of operation normally is done. Your webiste should have some API that your program can talk to, and get a JSON/XML/CSV file from. But that again depends on if this is your website?
If it is your webiste, I would probly create a folder that is password protected with HTACCESS/HTPASSWD (if using Apache?) that has a .php file (or what ever language you are using) that spits out a JSON file.
Re: download data from internet
not my web site, i am trying to get the usage data from my isp, so i can display as graph, afaik they have no api or other to get the dataif this is your website?
there is an option on the url xml=yes, but that only returns a summary of the data, not the detail and still requires to login first
i have successfully returned and parsed the data using vb, but now am trying to get the same working for linux
while there would be thousands of examples using several methods for vb, i can find almost nothing for gambas, or even other linux commands (curl or wget) i could shell to
though on further searching through the farm (and google) i have found several other working (or not) examples that are helping me with other parts