download data from internet

westconn
Posts: 10
Joined: Saturday 19th May 2018 11:23pm

download data from internet

Post by westconn » Sunday 20th May 2018 4:50am

i am just starting with gambas, i had previously started some code in vb6 to download data from a website, in which i had successfully downloaded the data, decided to try to complete in gambas, but so far have not really understood how to login to the website then download the data from a selected page, i believe i should use the curl library but any examples for logging in first would be helpful

User avatar
cogier
Site Admin
Posts: 165
Joined: Wednesday 21st September 2016 2:22pm
Location: Guernsey, Channel Islands

Re: download data from internet

Post by cogier » Sunday 20th May 2018 10:44am

Hi westconn and welcome to the forum.

There is the easy way to do this in Gambas using tools you probably already have installed on your system. The following example uses curl and wget to go and get your external IP address but one or both will need to be installed on your system.
Public Sub Form_Open()
Dim sResult As String

Shell "curl http://ipecho.net/plain" To sResult
Print sResult

''OR

Shell "wget -O- http://ipecho.net/plain" To sResult
Print sResult

End
Then there is the harder way, the following code requires gb.net.curl. To activate this component you will need to go to the Gambas menu Project>Properties>Components and activate gb.net.curl. This code does the same as the above.
''Needs gb.net.curl
Public Sub Form_Open()

Print Get("http://ipecho.net/plain")                                          'Find this computers external IP address

End
Public Sub Get(sURL As String) As String                                      'To 'Get' stuff from the web
Dim hClient As HttpClient                                                     'To create a HTTP Client
Dim sResult As String                                                         'To store the result

hClient = New HttpClient As "hClient"                                         'Create a HTTP Client
With hClient                                                                  'With the Client..
  .URL = sURL                                                                 'Set up the URL
  .Async = False                                                              'No Asynchronous transmission?
  .TimeOut = 60                                                               'Don't hang around waiting for more than 60 seconds
  .get                                                                        'Get the data
End With

If Lof(hClient) Then sResult = Read #hClient, Lof(hClient)                    'When all the data is downloaded store it in sResult

Return sResult                                                                'Return the result

End
If you go to the Gambas Farm and download Your_location (I need to get the downloads up! :o ) this may help you as it gets a CSV file from the net and displays the data.

westconn
Posts: 10
Joined: Saturday 19th May 2018 11:23pm

Re: download data from internet

Post by westconn » Sunday 20th May 2018 11:04am

i have no problem using wget or curl from a terminal, so i could shell out to either

my problem is that i need to login, then navigate to the correct url, unless there is some way to shortcut that, if i just try to go directly to the download i am redirected to the login page
can i pass parameters to the url to bypass the login page?

westconn
Posts: 10
Joined: Saturday 19th May 2018 11:23pm

Re: download data from internet

Post by westconn » Sunday 20th May 2018 12:19pm

(I need to get the downloads up!
so far i have been unable to find any link to the farm, the search on gambas one does not seem to be working, the gambasfarm.org does not seem to have the samples
do you have a direct link to it?

also thanks for your reply and samples

User avatar
cogier
Site Admin
Posts: 165
Joined: Wednesday 21st September 2016 2:22pm
Location: Guernsey, Channel Islands

Re: download data from internet

Post by cogier » Sunday 20th May 2018 2:43pm

The Gambas Farm on Gambas.One is for viewing only. You need to start Gambas and select the Farm from the menu, note the 'Installed software' option, you may need it later: -
0.png
0.png (265.01 KiB) Viewed 192 times
Then select "All software"
2.png
2.png (243.98 KiB) Viewed 192 times
Then from the list select the software you want to download
3.png
3.png (416.11 KiB) Viewed 192 times
Note that the software will be available from the main menu under "Installed software"

Regarding adding login in details you need to add the details to the URL

Code: Select all

https://xxx.university.edu/portal/server.pt?userName=SomeUser&password=somePassword
If you need more help click here
https://www.google.com/search?client=ub ... 15&bih=936

User avatar
jornmo
Site Admin
Posts: 154
Joined: Wednesday 21st September 2016 1:19pm
Location: Norway

Re: download data from internet

Post by jornmo » Monday 21st May 2018 11:01am

What do you mean by login?

Logging in by a form on a website, or the "pop-up" box that comes with web browsers?

For the first scenario I believe you can use Curl's HTTP FORM component:
http://gambaswiki.org/wiki/comp/gb.net.curl/httpform
(I am not sure if it can handle sessions. If it can't it will not work. It seems there might be a possibility to store the session in a cookie, and then load it again)

For the second you can use Curl's password property:
http://gambaswiki.org/wiki/comp/gb.net.curl/curl

westconn
Posts: 10
Joined: Saturday 19th May 2018 11:23pm

Re: download data from internet

Post by westconn » Wednesday 23rd May 2018 8:39am

@cogier, so easy with pictures lol, i did not think of the farm being built into gambas pogram

@jornmo, yes a login form on the webpage, any examples anywhere on using httpform, the manual did not seem to help me much

westconn
Posts: 10
Joined: Saturday 19th May 2018 11:23pm

Re: download data from internet

Post by westconn » Wednesday 23rd May 2018 9:54am

  client.URL = myurl
    client.async = False
  data = "login_username=myusername&login_password=mypass&submit=Login"
  z[1] = "content - length: " & Str(Len(data))
'    client.get   ' this works
     client.post("application/x-www-form-urlencoded", data, z)

    If client.Status < 0 Then
        Return ""
    Endif

    ' no data available
    If Not Lof(client) Then
        Return ""
    Endif

    ' Reads the data from the server and returns it as a String
    d = Read #client, Lof(client)
    TextArea1.Text = d
i have spent some time to get this far, but no result, nothing is returned to client, if i use client.get instead, the html code of the web page is returned

User avatar
jornmo
Site Admin
Posts: 154
Joined: Wednesday 21st September 2016 1:19pm
Location: Norway

Re: download data from internet

Post by jornmo » Wednesday 23rd May 2018 2:43pm

Since I do not know how the site is built, it is difficult to know what to do. But, when you have got a successfull login, you need somehow to store the session, and then load the URL where the data you want to retrieve is stored with the current session loaded. (I have never attempted such a thing myself)

However, I do not think this is the normal way this kind of operation normally is done. Your webiste should have some API that your program can talk to, and get a JSON/XML/CSV file from. But that again depends on if this is your website?

If it is your webiste, I would probly create a folder that is password protected with HTACCESS/HTPASSWD (if using Apache?) that has a .php file (or what ever language you are using) that spits out a JSON file.

westconn
Posts: 10
Joined: Saturday 19th May 2018 11:23pm

Re: download data from internet

Post by westconn » Wednesday 23rd May 2018 9:30pm

if this is your website?
not my web site, i am trying to get the usage data from my isp, so i can display as graph, afaik they have no api or other to get the data
there is an option on the url xml=yes, but that only returns a summary of the data, not the detail and still requires to login first

i have successfully returned and parsed the data using vb, but now am trying to get the same working for linux
while there would be thousands of examples using several methods for vb, i can find almost nothing for gambas, or even other linux commands (curl or wget) i could shell to
though on further searching through the farm (and google) i have found several other working (or not) examples that are helping me with other parts

Post Reply