Pass an array from one Gambas program to another?

User avatar
Godzilla
Posts: 35
Joined: Wednesday 26th September 2018 11:20am

Pass an array from one Gambas program to another?

Post by Godzilla » Saturday 21st September 2019 8:04am

I've written a Gambas program that does four CPU-intensive tasks in sequence. One task can't begin until the previous one ends, etc.

Everything works well. But from start of the first task, till the end of the last task, takes about 2 minutes. I've considered buying a new computer with a cutting-edge processor that could do the same sequence of tasks in a fraction of the time.

But here's the thing. The computer I'm now using has a 4-core CPU. While these CPU-intensive tasks are being carried out, the CPU usage on one core remains at 100% while the others remain essentially idle. The overall CPU usage, of course, never goes above 25%, because each Gambas program only uses one thread.

It occurred to me that I could avoid having to buy a new computer if I could better utilize this CPU.

If I could create several "slave" programs, and communicate an array, or at least a huge block of text that can be Split() into an array by each slave, then all four CPU cores could be utilized at once. Done in parallel, the same work could in theory be done in 30 seconds.. Once finished, each slave could communicate its resulting arrays, or huge blocks of text, back to the main program in the same manner.

In my VB6 days, this kind of thing could be done fairly easily. I could find a slave program's hWnd with the FindWindow API call, find its child textbox hWnd with a nifty function FindChildObjectByClass. Then use

Code: Select all

Call SendMessageByString(ChildhWnd, WM_SETTEXT, 0, "This is the text to be sent.")
The slave could then Split() that text into an array, perform a task with it, and then send the result back to the main program's textbox in the same manner.

Does anyone know if Gambas can send arrays, or at least hugs blocks of text, between separate running programs?

If not, a workaround that should provide the same functionality would be to use the clipboard. But it would mean the clipboard would always be tied up and unusable, with data always being bounced around back and fourth in it. If I were to mindlessly copy something to the clipboard while these programs are operating, chaos could ensue. So I'd rather avoid using the clipboard if I can. But it may be my best option.

Anyways, thanks for reading. Keep on Gambasing.

User avatar
Quincunxian
Posts: 66
Joined: Sunday 25th June 2017 12:14am
Location: Western Australia

Re: Pass an array from one Gambas program to another?

Post by Quincunxian » Saturday 21st September 2019 11:29pm

Hi Godzilla,
You will also need to have a look at the Linux command taskset.
taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.
https://linux.die.net/man/1/taskset

Not aware of any way to pass information from multiple child applications to a parent application.
Using Shell or Exec functions will require the use of a Wait statement so that negates the outcome you need.

I can think of only 2 ways to do this:
  • Each child application writes the data to a text file and once finished all of the text files from each child is collated by the parent.
  • Write the output data to a database and then collate. There may be more IO cycles with this so overall effect is a little slower but it would give the best design option if you wanted to compare individual runs
I'd just send a general email to the Gambas users email list and if there other solutions then someone will respond.
Cheers - Quin.
I code therefore I am

User avatar
cogier
Site Admin
Posts: 322
Joined: Wednesday 21st September 2016 2:22pm
Location: Guernsey, Channel Islands

Re: Pass an array from one Gambas program to another?

Post by cogier » Sunday 22nd September 2019 11:32am

Hi Godzilla. Your question got a friend and I working on a very interesting Gambas tool called 'Task'. You can find some details here. As usual the documentation could do with a little work and an example would also help. Experimentation produced the attached program that when run on an Intel NUC 4 Core i7 is able to max out all 4 cores, all within the one program, see the 'System Monitor' image below.

Image

This offers the opportunity to do some really powerful stuff.

Have a look at the program, we have deliberately kept it as simple as possible to hopefully make it easy to understand. Reading, and then rereading, the documentation will also help. I don't pretend to understand it all yet but it is a start. You should be able to modify it to process 4 tasks at the same time. If you need more help let us know.

The program was run on Linux Mint 19.2 with the Cinnamon Desktop. If you are running an Ubuntu derived distro then you can install System Monitor with 'sudo apt-get -y install gnome-system-monitor'.
TaskCPUMax-0.0.1.tar.gz
(11.55 KiB) Downloaded 13 times
EDIT: -
Program modified to use 8 cores.
Image

User avatar
Godzilla
Posts: 35
Joined: Wednesday 26th September 2018 11:20am

Re: Pass an array from one Gambas program to another?

Post by Godzilla » Monday 23rd September 2019 5:32am

Quincunxian wrote:
Saturday 21st September 2019 11:29pm
Hi Godzilla,
You will also need to have a look at the Linux command taskset.
taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.
https://linux.die.net/man/1/taskset

Not aware of any way to pass information from multiple child applications to a parent application.
Using Shell or Exec functions will require the use of a Wait statement so that negates the outcome you need.

I can think of only 2 ways to do this:
  • Each child application writes the data to a text file and once finished all of the text files from each child is collated by the parent.
  • Write the output data to a database and then collate. There may be more IO cycles with this so overall effect is a little slower but it would give the best design option if you wanted to compare individual runs
I'd just send a general email to the Gambas users email list and if there other solutions then someone will respond.
Hey Quin, thanks for your reply.

Using taskset with the launch of slave programs, to ensure each program uses a separate CPU core, would be a great idea. Its something I didn't think of.

Using text files or a database would be a viable option. Reading and writing from a ram drive would have several benefits also. It would save wear & tear on the disk, and the speed would be unmatched.

Thank you again for your reply, Quin

User avatar
Godzilla
Posts: 35
Joined: Wednesday 26th September 2018 11:20am

Re: Pass an array from one Gambas program to another?

Post by Godzilla » Monday 23rd September 2019 6:20am

cogier wrote:
Sunday 22nd September 2019 11:32am
Hi Godzilla. Your question got a friend and I working on a very interesting Gambas tool called 'Task'. You can find some details here. As usual the documentation could do with a little work and an example would also help. Experimentation produced the attached program that when run on an Intel NUC 4 Core i7 is able to max out all 4 cores, all within the one program, see the 'System Monitor' image below.

Image

This offers the opportunity to do some really powerful stuff.

Have a look at the program, we have deliberately kept it as simple as possible to hopefully make it easy to understand. Reading, and then rereading, the documentation will also help. I don't pretend to understand it all yet but it is a start. You should be able to modify it to process 4 tasks at the same time. If you need more help let us know.

The program was run on Linux Mint 19.2 with the Cinnamon Desktop. If you are running an Ubuntu derived distro then you can install System Monitor with 'sudo apt-get -y install gnome-system-monitor'.

TaskCPUMax-0.0.1.tar.gz

EDIT: -
Program modified to use 8 cores.
Image
Hello cogier, thank you for your reply.

I'm really amazed by this project created by you and your friend. I appreciate the time and thought you both put into this.

Your project works on both my computers (both having Intel). The single task of counting from 1 to 5000000000000 maxes out all cores at once. I'm very impressed!

Using this method, separate CPU-intensive tasks would still be performed sequentially. But each one would be performed in parallel on 4 cores, instead of a single core. That has to be a significant performance gain.

I wanted to see exactly how much of a performance gain that 10.4 GHz (collectively) gives over 2.6 GHz.

I added another button which simply performs

Code: Select all

For a As Integer = 1 To 5000000000000
Next
I added a timer to this task, as well as adding a timer the method provided in your project.

To my dismay, however, there was no gain. In every test, the times were essentially the same.

This leads me to wonder if the Task method is actually only performing copies of the same task on each core at once? Each task unrelated to the other, with no collating?

I don't know. But this is a very promising thing that warrants further looking into. I hope those who have downloaded this project can report back as to whether they get a performance gain using this method vs the conventional method. This is something that has great potential.

Thank you once again cogier for your reply, and your fascinating project.

User avatar
cogier
Site Admin
Posts: 322
Joined: Wednesday 21st September 2016 2:22pm
Location: Guernsey, Channel Islands

Re: Pass an array from one Gambas program to another?

Post by cogier » Monday 23rd September 2019 4:36pm

I have managed to get the 'Kill' event working which is raised when the background task is complete. Using different values in the loop you can see the different processes finish at different times.

Image
Attachments
TaskCPUMax.tar.gz
(12.96 KiB) Downloaded 11 times

stevedee
Posts: 124
Joined: Monday 20th March 2017 6:06pm

Re: Pass an array from one Gambas program to another?

Post by stevedee » Monday 23rd September 2019 4:53pm

Godzilla wrote:
Monday 23rd September 2019 5:32am

...Using text files or a database would be a viable option. Reading and writing from a ram drive would have several benefits also. It would save wear & tear on the disk, and the speed would be unmatched....
I made my bat logger run faster by splitting the original program into two; one to record to a RAM drive file every 10 seconds, and a second program to analyse each file (the RAM drive could hold maybe 4 or 5 files which were analysed in date/time order).

The second (analysis) program would check for suitable recording files on the RAM drive, and either delete them (if they did not contain a bat call) or process them, write out a new file, and delete the original.

If the second program could not keep up with the first, then the first (recording) program would have to wait until there was enough space on the RAM drive to continue.

I know nothing about your application, but you may be able to write data to text (CSV) files and pass it around in a similar fashion, e.g. the master program could delete the data files when it has taken what it needs from them. Each file would have a suitable ID so the master knows which child program it came from.

User avatar
Godzilla
Posts: 35
Joined: Wednesday 26th September 2018 11:20am

Re: Pass an array from one Gambas program to another?

Post by Godzilla » Tuesday 24th September 2019 7:38am

cogier wrote:
Monday 23rd September 2019 4:36pm
I have managed to get the 'Kill' event working which is raised when the background task is complete. Using different values in the loop you can see the different processes finish at different times.

Image
Hey cogier,

Thank you again for your reply to this thread, and for modifying your awesome code.

It actually doing different tasks in parallel. Amazing! Check out these time benchmarks.

First is a time benchmark result of a sequence of four loops done in the conventional way, using your exact set of numbers:

Code: Select all

00d 00h 00m 41s for task "Using only 1 core"
Next is a time benchmark result using your Task method, included in your latest project upload:

Code: Select all

00d 00h 00m 13s for task "Using all 4 cores"
I would call that a great success! You're really onto something very cool here, and I hope you will continue to develop it and improve on it.

This is absolutely something I can use as a solution to my problem. Its just a question of having each New CPU call separate subroutines, instead of passing numbers. I think the subroutines would need to be relocated into the CPU.class file for them to utilize the Task method? I don't know, it will take some trial and error. But I'm very excited about the possibilities this opens up for those of us whose code requires multiple CPU-intensive tasks.

Once again, thank you cogier for further developing your code and sharing the results of your efforts here.

User avatar
cogier
Site Admin
Posts: 322
Joined: Wednesday 21st September 2016 2:22pm
Location: Guernsey, Channel Islands

Re: Pass an array from one Gambas program to another?

Post by cogier » Tuesday 24th September 2019 12:56pm

Thanks Godzilla. I am also excited by the possibilities this feature opens up.

I was also interested in your original question and came up with these 2 programs named 'One' and 'Two' (original naming don't you think! ;)). They will pass a string between themselves, but could be altered to send arrays.

Image
One.tar.gz
(8.08 KiB) Downloaded 10 times
Attachments
Two.tar.gz
(12.79 KiB) Downloaded 8 times

vuott
Posts: 25
Joined: Wednesday 05th April 2017 6:07pm

Re: Pass an array from one Gambas program to another?

Post by vuott » Tuesday 24th September 2019 6:55pm


Post Reply