Offloading computing tasks to another computer with SSH – getting started using a raspberry pi

SSH into a raspberry pi – why bother?

I am interested in learning how to upload to, run programmes on, and download files from other computers – it is a common situation at work that long analyses are run, clogging up a computer that could be used for other tasks whilst that analysis is running. This could be solved by comandeering a spare computer across the office…

However, the ability to push files across the network, run analysis on them, then receive the results for post processing has much greater power and interest – many analyses could be run simultaneously for a grid-search optimisation, increasing their value in parametric design, or passing much larger analyses to faster computers. In both cases this communication with raspberry pi is a stepping stone to communication with computers in the cloud. Either very many small computers or a single very large computer. Amazon Web Services offers small computers (about the power of an average desktop) for about £0.02-£0.20/hour. The fastest computers, at about £1-£5/hour, appear to be about 10-20 times faster than a typical desktop for applications that use many cores effectively. These prices can be significantly discounted for ‘spot pricing’ and ‘reserved instances’ where immediate use of computing power isn’t guarenteed. This seems like a great way of not only getting bigger jobs done a little faster, but getting much more detailed results from grid-searches and other optimisations – where single programs are run many times with different inputs. This is standard practice in many engineering and scientific environments where most of the work is performed on large servers – with the human interface at low power laptop in a coffee shop.

Image from my final year project on the indentation of tubes – a grid search of different failure mechanisms of tubes subject to an indenting force. All of the processing of these models was performed on a server overnight, and the models generated parametrically.

Result of grid-search, not as dense as I would have liked. This was run on a server using a similar technique, but ran sequentially rather than using many machines at once.

As an aside – this method of sending files was considered for the LightTraceMemory project, but was rejected in favour of shared windows folders which were easier and faster to set up.

Connecting to the Raspberry Pi

Getting started with the Raspberry Pi 3 and the Raspberry Pi Zero-W. I installed Raspbian on both using Noobs, then used system configuration to turn on SSH, connected to the internet, either via an Ethernet cable or using the built in wifi. Finding the IP Address of the Raspberry Pi’s just needed ‘ifconfig’ in the command prompt on the pi – n.b. NOT ‘ipconfig’ – that is for windows.

At this point point ‘ping 192.168.1.XX’ from my PC was working when I pinged the Pi’s address.

To find the IP address of a Pi when I didn’t have a way of finding the IP address from the Pi itself (so when I was running the pi ‘headless’) I used ‘Zenmap’. This program pings every IP address in a range to find the ones that respond, though I’m sure I will find more features of how it works over time. Zenmap is from here: https://nmap.org/zenmap/. As I am working on my home network, every IP address will start with 192.168.1.XX where XX is between 1 and 255 inclusive (I think). To search this range, I searched 192.168.1.1/24 in zenmap. The ‘ping scan’ option worked fine and was fairly quick to run, about 5 seconds – it returns the names of the computers on the network and their IP addresses.

To establish a connection with the Raspberry Pi I used PuTTY – which allows you to log onto another computer on the network. In particular, it allows you to communicate with a computer running linux when you are running windows. I have had some experience with PuTTY during my final year university project, where I needed to run large ABAQUS simulations on the server, but I was blindly following instructions to make it work, rather than trying to understand so I could use it to its full potential for other uses. Typing in the IP address of the pi and opening a window gives you a terminal window onto the pi, where you can execute commands – so this is just like operating the pi in terminal mode.

Nmap window showing computers on the network

 

PuTTY setup window ready to connect to the Raspberry Pi


PuTTY window for Raspberry Pi

Sending and receiving files to and from the Raspberry Pi:

Use PSCP (Putty Secure Copy Program) in windows with the format:

pscp C:\Users\david\OneDrive\Projects\SSHTesting\example.txt pi@192.168.1.106:example2.txt

then using ‘ls’ in the Raspberry Pi’s PuTTY window will list the directory, and show the file has been sent. Without more complicated setup, the password of the raspberry pi will need to be typed each time – I solve this later by setting up keys that allow secure authentication.

To retrieve a file from the Raspberry Pi and copy it onto the local computer, pscp is used again, for example:

pscp pi@192.168.1.106:write_example2.txt C:\Users\david\OneDrive\Projects\SSHTesting\write_example.txt

So the same pscp – source – target structure, when the source or the target isn’t you on your computer, you need to specify who you are – so user@ipaddress.

Another method that is easier, but doesn’t build to the final goal of uploading, running and downloading manipulating files, is a FTP programme like Filezilla, which allows transfer of files using a drag and drop interface.

Running a program on the raspberry pi to manipulate file:

We have now achieved the minimum behaviour wanted – we have a way of sending files to and from the pi, and running commands on the pi. Therefore, we can upload an input file and a program to manipulate it, run that program in the pi, and download the file output. There are several refinements we might want to add to this – running the programme from the same window or batch file. A start is with ‘plink’ [PuTTY Link] which is part of PuTTY.

For example we can upload a text file: example.txt.

Then manipulate it with the following python script which just removes half the letters and saves a file – sys.argv[1] gives the first argument provided after a call, the zeroth is the name of the python file itself:


import sys
readfilepath = sys.argv[1]
writefilepath = sys.argv[2]
readfile = open(readfilepath, 'r')
writefile = open(writefilepath, 'w')
letters = readfile.read()
for i,letter in enumerate(letters):
    if i%2==0:
        #print letter
        writefile.write(letter)
readfile.close
writefile.close

The download the written file back to the PC.

The following commands do that all from one window:


pscp C:\Users\david\OneDrive\Projects\SSHTesting\example.txt pi@192.168.1.106:example2.txt

plink pi@192.168.1.106 python text_halver.py example2.txt write_example2.txt

pscp pi@192.168.1.106:write_example2.txt C:\Users\david\OneDrive\Projects\SSHTesting\write_example.txt

There are still issues before this can become one batch file, the computer still asks for the Raspberry Pi’s password at every command – one way is to pass the password as an argument to plink – but this isn’t recommended for obvious reasons. Therefore, a more permanent connection to the Pi needs to made so that plink can give commands as part of a windows batch file.

Creating a secure connection to the Raspberry Pi:

A way to create a more secure connection between the PC and the Raspberry Pi is with a set of Private-Public keys – something built into PuTTY closely through Puttygen (PuTTY Key Generator) (https://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter8.html#pubkey) and Pagent (stores keys in memory to avoid typing in passwords several times over a session) (https://the.earth.li/~sgtatham/putty/0.58/htmldoc/Chapter9.html#pageant).

Firstly, make the private and public keys in puttygen, then save each in the .ssh folder in windows (User/David/.ssh in my case – I had already made this from a previous experiment with encrypted email), for example as myprivatekey.ppk and mypublickey.pub. Copy the text of the public key into the authorized_keys file in /.ssh folder in the Raspberry File – if you don’t want to do this on the pi itself, then copy in it using vim (itself an education) over the putty command line, or through Filezilla or pscp (I had issues getting into the .ssh folder) as a file transfer. https://dvpizone.wordpress.com/2014/03/02/how-to-connect-to-your-raspberry-pi-using-ssh-key-pairs/ was very useful and I followed its instructions on changing the authorized_keys file and the sshd_config file on the pi. https://askubuntu.com/questions/204400/ssh-public-key-no-supported-authentication-methods-available-server-sent-publ was also helpful – I saved directly from Puttygen and had something that sat in the wrong format for this task. It needs to be in the format “ssh-rsa AAAAB3yc2EA…kqal= rsa-key-20121022” for example – as per the linked page.

Once the public key was on the pi, and private key loaded into pageant, pageant is running, the private key location loaded into putty, the username loaded into PuTTy and password turned off on the pi login was straightforward, just open in PuTTy and straight in securely!

Successful login to Raspberry using keys

This clears the way for plink and other commands to run without requiring a password, which in turn clears the path for them to run in a batch file.

Testing with plink and other commands went well – as expected pageant needs to have been started, but PuTTY doesn’t seem to need to have already been used to make it work.

Finally, I can put those three commands into a batch file (just save a text file as test.bat containing the commands as you would type them), then run the batch file to upload a text document to the raspberry pi, run the program on the file to create a new one, then download the result – which is what was wanted.

More can be done to create more complex batch files, or using the capabilities of windows powershell, to automate the passing and processing of files.


 

Leave a Reply