xenon3 tutorial – Table of Contents
Note we have GoogleAnalytics enabled for this tutorial to help show our funders that the work we do is useful to others.
Getting started¶
On your system, start VirtualBox.
This tutorial uses a virtual machine to help avoid issues that are due to system configuration.
In case you don’t have a copy of the virtual machine, you can
download it from Zenodo here. After the download finishes, click File
in VirtualBox, then
Import appliance
, then select the file you downloaded.
During the import, you’ll see an initialization wizard. Make sure that the virtual machine is configured with two CPUs.
Start the virtual machine and log in as user travis
with password password
.
Once the system has booted, start both a terminal and Firefox by clicking their respective icons. Use Firefox to navigate to the tutorial text at https://xenon-tutorial.readthedocs.io.
In the terminal, confirm that the xenon
command line interface
program can be found on the system:
xenon --help
xenon --version
Xenon CLI v3.0.4, Xenon library v3.0.4, Xenon cloud library v3.0.2
If you run into trouble, here are some pointers on what you can do.
Interacting with filesystems¶
Essentially, xenon
can be used to manipulate files and to interact with schedulers, where either one can be local
or remote. Let’s start simple and see if we can do something with local files. First, check its help:
xenon filesystem --help
The usage line suggests we need to pick one from {file,ftp,s3,sftp,webdav}
.
Again, choose what seems to be the simplest option (file
), and again, check its help.
xenon filesystem file --help
xenon filesystem file
’s usage line seems to suggest that I need to pick one
from {copy,list,mkdir,remove,rename}
. Simplest one is probably list
, so:
xenon filesystem file list --help
So we need a path
as final argument.
In case you hadn’t noticed the pattern, stringing together any number of xenon
subcommands and appending --help
to it will get you help on the particular combination of subcommands you supplied.
The focus of this tutorial is on using Xenon’s command line interface, but be aware that you can use xenon’s functionality from other programming languages through xenon’s gRPC extension.
Where relevant, we have included equivalent code snippets, written in Java and Python, as a separate tab.
Let’s try listing the contents of /home/travis/fixtures/
.
xenon filesystem file list /home/travis/fixtures
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;
public class DirectoryListing {
public static void main(String[] args) throws Exception {
String adaptor = "file";
FileSystem filesystem = FileSystem.create(adaptor);
Path directory = new Path("/home/travis/fixtures");
Boolean recursive = false;
Iterable<PathAttributes> listing = filesystem.list(directory, recursive);
for (PathAttributes elem : listing) {
if (!elem.isHidden()) {
System.out.println(elem.getPath());
}
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | import xenon
from xenon import Path, FileSystem
def run_example():
xenon.init()
filesystem = FileSystem.create(adaptor='file')
path = Path("/home/travis/fixtures")
listing = filesystem.list(path, recursive=False)
for entry in listing:
if not entry.path.is_hidden():
print(entry.path)
filesystem.close()
if __name__ == '__main__':
run_example()
|
The result should be more or less the same as that of ls -1
.
xenon filesystem file list
has a few options that let you specify the details of the list operation, e.g.
--hidden
xenon filesystem file list --hidden /home/travis/fixtures
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;
public class DirectoryListingShowHidden {
public static void main(String[] args) throws Exception {
String adaptor = "file";
FileSystem filesystem = FileSystem.create(adaptor);
Path dir = new Path("/home/travis/fixtures");
boolean recursive = false;
Iterable<PathAttributes> listing = filesystem.list(dir, recursive);
for (PathAttributes elem : listing) {
System.out.println(elem.getPath());
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | import xenon
from xenon import Path, FileSystem
def run_example():
xenon.init()
filesystem = FileSystem.create(adaptor='file')
path = Path("/home/travis/fixtures")
listing = filesystem.list(path, recursive=False)
for entry in listing:
print(entry.path)
filesystem.close()
if __name__ == '__main__':
run_example()
|
and --recursive
xenon filesystem file list --recursive /home/travis/fixtures
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;
public class DirectoryListingRecursive {
public static void main(String[] args) throws Exception {
String adaptor = "file";
FileSystem filesystem = FileSystem.create(adaptor);
Path dir = new Path("/home/travis/fixtures");
boolean recursive = true;
Iterable<PathAttributes> listing = filesystem.list(dir, recursive);
for (PathAttributes elem : listing) {
if (!elem.isHidden()) {
System.out.println(elem.getPath());
}
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | import xenon
from xenon import Path, FileSystem
def run_example():
xenon.init()
filesystem = FileSystem.create(adaptor='file')
path = Path("/home/travis/fixtures")
listing = filesystem.list(path, recursive=True)
for entry in listing:
if not entry.path.is_hidden():
print(entry.path)
filesystem.close()
if __name__ == '__main__':
run_example()
|
Now let’s create a file and try to use xenon
to copy it:
cd /home/travis
echo 'some content' > thefile.txt
Check the relevant help
xenon filesystem file --help
xenon filesystem file copy --help
So, the copy
subcommand takes a source path and a target path:
xenon filesystem file copy /home/travis/thefile.txt /home/travis/thefile.bak
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
public class CopyFileLocalToLocalAbsolutePaths {
public static void main(String[] args) throws Exception {
// use the local file system adaptor to create a file system representation
String adaptor = "file";
FileSystem filesystem = FileSystem.create(adaptor);
// create Paths for the source and destination files, using absolute paths
Path sourceFile = new Path("/home/travis/thefile.txt");
Path destFile = new Path("/home/travis/thefile.bak");
// create the destination file only if the destination path doesn't exist yet
CopyMode mode = CopyMode.CREATE;
boolean recursive = false;
// perform the copy and wait 1000 ms for the successful or otherwise
// completion of the operation
String copyId = filesystem.copy(sourceFile, filesystem, destFile, mode, recursive);
long timeoutMilliSecs = 1000;
CopyStatus copyStatus = filesystem.waitUntilDone(copyId, timeoutMilliSecs);
// print any exceptions
if (copyStatus.getException() != null) {
System.out.println(copyStatus.getException().getMessage());
} else {
System.out.println("File copied.");
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | import xenon
from xenon import Path, FileSystem, CopyRequest, CopyStatus
def run_example():
xenon.init()
filesystem = FileSystem.create(adaptor='file')
source_file = Path("/home/travis/thefile.txt")
dest_file = Path("/home/travis/thefile.bak")
# start the copy operation; no recursion, we're just copying a file
copy_id = filesystem.copy(source_file, filesystem, dest_file, mode=CopyRequest.CREATE, recursive=False)
# wait this many milliseconds for copy operations to complete
wait_duration = 1000 * 60 * 10
# wait for the copy operation to complete (successfully or otherwise)
copy_status = filesystem.wait_until_done(copy_id, wait_duration)
assert copy_status.done
# rethrow the Exception if we got one
assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message
filesystem.close()
if __name__ == '__main__':
run_example()
|
Note that the source path may be standard input, and that the target path may be standard output:
# read from stdin:
cat thefile.txt | xenon filesystem file copy - mystdin.txt
# write to stdout:
xenon filesystem file copy thefile.txt - 1> mystdout.txt
xenon filesystem file
has a few more subcommands, namely mkdir
, rename
and remove
. You can
experiment a bit more with those before moving on to the next section.
Access to remote filesystems¶
Of course the point of xenon
is not to move around files on your local filesystem. There are enough tools to help you with
that. The idea is that you can also use xenon
to move files to and from different types of remote servers, without having to
learn a completely different tool every time.
First, let’s check which types of file servers xenon
currently supports:
xenon filesystem --help
The usage line shows that besides file
we can also choose ftp, s3, sftp
or webdav
. Let’s try ftp
first.
xenon filesystem ftp --help
The usage line tells us that ftp
has an mandatory parameter --location
which we haven’t seen yet. We can use this to specify
which server to connect to. Additionally, there are also optional --username
and --password
options in case we need
to log into the machine.
Let’s see if we can use this to connect to a real machine on the internet. A public FTP server for testing should be available at
test.rebex.net
with the credentials demo
and password
:
# list the files on the ftp server
xenon filesystem ftp --location test.rebex.net --username demo --password password list /
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;
public class FTPDirectoryListingShowHidden {
public static void main(String[] args) throws Exception {
String adaptor = "ftp";
String location = "test.rebex.net";
String username = "demo";
String password = "password";
try (FileSystem filesystem = FileSystem.create(adaptor, location, new PasswordCredential(username, password))) {
Path dir = new Path("/");
boolean recursive = false;
Iterable<PathAttributes> listing = filesystem.list(dir, recursive);
for (PathAttributes elem : listing) {
System.out.println(elem.getPath());
}
}
}
}
|
This should give you a listing of the server at test.rebex.net
.
Besides the commands we have already seen (copy
, list
, etc.), ftp
also supports a few new ones, namely upload
and download
. We can use these to transer files to and from the server. For example, this command will download a file from
our example FTP server:
# download a file from the ftp server
xenon filesystem ftp --location test.rebex.net --username demo --password password download /readme.txt `pwd`/readme.txt
You can even print the remote file on your screen by copying it to stdout:
# print a file from the ftp server on the screen
xenon filesystem ftp --location test.rebex.net --username demo --password password download /readme.txt -
Note that when using copy
on remote servers, xenon
will attempt to copy the file on the server itself. Since we don’t have write
access to this FTP server, the command will fail.
The strength of xenon
is that you can now use the same syntax to access a different type of server. For example, the test.rebex.net
server also offers a secure FTP (sftp
) service for testing. We can access that service with xenon
by simply changing ftp
into sftp
:
# list the files on the sftp server
xenon filesystem sftp --location test.rebex.net --username demo --password password list /
# download a file from the sftp server
xenon filesystem sftp --location test.rebex.net --username demo --password password download /readme.txt `pwd`/readme2.txt
In case you are reluctant to type plaintext passwords on the command line, for example because of logging in
~/.bash_history
, know that you can supply passwords from a file, as follows:
# read password from the password.txt file
xenon filesystem sftp --location test.rebex.net --username demo --password @password.txt list /
in which the file password.txt
should contain the password. Since everything about the user xenon
is public
knowledge anyway, such security precautions are not needed for this tutorial, so we’ll just continue to use the
--password PASSWORD
syntax.
You can also transfer data from and to other types of file servers (such as WebDAV
and S3
) in a similar fashion. We are working to add
support for other types such as GridFTP and iRODS. We will come back to transferring files in the sections below.
Interacting with schedulers¶
Now let’s see if we can use schedulers, starting with SLURM. For this part, we need access to a machine that is running SLURM. To avoid problems related to network connectivity, we won’t try to connect to a physically remote SLURM machine, but instead, we’ll use a dockerized SLURM installation. This way, we can mimic whatever infrastructure we need. The setup will thus be something like this:
A copy of the SLURM Docker image (xenonmiddleware/slurm:17) has been included in the virtual machine. Bring it up with:
docker run --detach --publish 10022:22 --hostname slurm17 xenonmiddleware/slurm:17
Use docker ps
to check the state of the container
docker ps
The last column in the resulting table lists the name of the container. By
default, docker
assigns automatically generated names, like
practical_robinson
, keen_goodall
or infallible_morse
. You can use
this name to stop and remove a container once you’re done with it, as follows:
docker stop practical_robinson
docker rm practical_robinson
Anyway, that’s for later. For now, check that the container’s status is
healthy
, and see if you can ssh
into it on port 10022
as user
xenon
with password javagat
:
ssh -p 10022 xenon@localhost
# if that works, exit again
exit
Be aware that ssh
can sometimes be a little picky. We’ve assembled a list of
tips and tricks for troubleshooting SSH.
Check the help to see how the slurm
subcommand works:
xenon scheduler slurm --help
Let’s first ask what queues the SLURM scheduler has. For this, we need to specify
a location, otherwise xenon
does not know who to ask for the list of queues. According to the help,
LOCATION
is any location format supported by ssh
or local
scheduler.
Our dockerized SLURM machine is reachable as ssh://localhost:10022
.
We’ll also need to provide a --username
and --password
for that location, as follows:
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat queues
# returns:
# Available queues: mypartition, otherpartition
# Default queue: mypartition
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.Credential;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.Scheduler;
public class SlurmQueuesGetter {
public static void main(String[] args) throws XenonException {
String host = "localhost";
String port = "10022";
runExample(host, port);
}
public static void runExample(String host, String port) throws XenonException {
String adaptor = "slurm";
String location = "ssh://" + host + ":" + port;
// make the password credential for user 'xenon'
String username = "xenon";
char[] password = "javagat".toCharArray();
Credential credential = new PasswordCredential(username, password);
// create the SLURM scheduler
try (Scheduler scheduler = Scheduler.create(adaptor, location, credential)) {
// ask SLURM for its queues
System.out.println("Available queues (starred queue is default):");
for (String queueName : scheduler.getQueueNames()) {
if (queueName.equals(scheduler.getDefaultQueueName())) {
System.out.println(queueName + "*");
} else {
System.out.println(queueName);
}
}
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | import xenon
from xenon import Scheduler, PasswordCredential
def run_example():
xenon.init()
credential = PasswordCredential(username='xenon',
password='javagat')
scheduler = Scheduler.create(adaptor='slurm',
location='ssh://localhost:10022',
password_credential=credential)
default_queue = scheduler.get_default_queue_name()
print("Available queues (starred queue is default):")
for queue_name in scheduler.get_queue_names():
if queue_name == default_queue:
print("{}*".format(queue_name))
else:
print(queue_name)
scheduler.close()
if __name__ == '__main__':
run_example()
|
Besides queues
, other slurm
subcommands are exec
, submit
, list
, remove
, and wait
. Let’s try
to have xenon
ask SLURM for its list of jobs in each queue, as follows:
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list
# should work, but we don't have any jobs yet
Now, let’s try to submit a job using slurm submit
. Its usage string suggests that we need to provide (the path
of) an executable
. Note that the executable should be present inside the container when SLURM starts its execution.
For the moment, we’ll use /bin/hostname
as the executable. It simply prints the name of the host on the command line.
For our docker container, it should return the hostname slurm17
of the Docker
container, or whatever hostname you specified for it when you ran the docker run
command earlier:
# check the slurm submit help for correct syntax
xenon scheduler slurm submit --help
# let xenon submit a job with /bin/hostname as executable
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit /bin/hostname
# add --stdout to the submit job to capture its standard out so we know it worked:
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout hostname.stdout.txt /bin/hostname
# check to see if the output was written to file /home/xenon/hostname.stdout.txt
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat download hostname.stdout.txt -
Below are a few more examples of slurm submit
:
# executables that take options prefixed with '-' need special syntax, e.g. 'ls -la'
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout /home/xenon/ls.stdout.txt ls -- -la
# check to see if the output was written to file /home/xenon/ls.stdout.txt
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat download ls.stdout.txt -
# submit an 'env' job with environment variable MYKEY, and capture standard out so we know it worked
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout /home/xenon/env.stdout.txt --env MYKEY=myvalue /usr/bin/env
# check to see if the output from 'env' was written to file /home/xenon/env.stdout.txt
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat download env.stdout.txt -
Combining filesystems and schedulers¶
So far, we’ve used xenon
to manipulate files on the local filesystem, and to run system executables on the remote
machine. In typical usage, however, you would use xenon
to run executables or scripts of your own, which means that
we need to upload such files from the local system to the remote system.
A typical workflow may thus look like this:
upload input file(s)
submit job
download generated output file(s)
Use an editor to create a file sleep.sh
with the following contents (the virtual machine comes with a bunch of editors
like gedit
, leafpad
, and nano
, but you can install a different editor from the repositories if you like):
#!/usr/bin/env bash
echo `date`': went to bed'
sleep $1
echo `date`': woke up'
You can test if your file is correct by:
# last argument is the sleep duration in seconds
bash sleep.sh 5
We need to upload sleep.sh
to the remote machine. We can’t use xenon filesystem file
like we did before,
because we’re copying between file systems, so let’s look at what other options are available:
xenon filesystem --help
# let's try sftp protocol
xenon filesystem sftp --help
# we're interested in 'upload' for now
xenon filesystem sftp upload --help
We’ll also need to tell xenon
what location we want to connect to, and what credentials to use. The SLURM Docker
container we used before is accessible via SFTP using the same location, username and password as before, so let’s use
that:
# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload /home/travis/sleep.sh /home/xenon/sleep.sh
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
public class UploadFileLocalToSftpAbsolutePaths {
public static void main(String[] args) throws XenonException {
String host = "localhost";
String port = "10022";
runExample(host, port);
}
public static void runExample(String host, String port) throws XenonException {
// use the local file system adaptor to create a file system representation
String adaptorLocal = "file";
FileSystem filesystemLocal = FileSystem.create(adaptorLocal);
// define what file to upload
Path fileLocal = new Path("/home/travis/sleep.sh");
// use the sftp file system adaptor to create a file system representation; the remote
// filesystem requires credentials to log in, so we'll have to create those too.
String adaptorRemote = "sftp";
String location = host + ":" + port;
String username = "xenon";
char[] password = "javagat".toCharArray();
PasswordCredential credential = new PasswordCredential(username, password);
FileSystem filesystemRemote = FileSystem.create(adaptorRemote, location, credential);
// define which file to upload to
Path fileRemote = new Path("/home/xenon/sleep.sh");
// create the destination file only if the destination path doesn't exist yet
CopyMode mode = CopyMode.CREATE;
// no need to recurse, we're just downloading a file
boolean recursive = false;
// perform the copy/upload and wait 1000 ms for the successful or otherwise
// completion of the operation
String copyId = filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, mode, recursive);
long timeoutMilliSecs = 1000;
CopyStatus copyStatus = filesystemLocal.waitUntilDone(copyId, timeoutMilliSecs);
// print any exceptions
if (copyStatus.getException() != null) {
System.out.println(copyStatus.getException().getMessage());
} else {
System.out.println("Done");
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | import xenon
from xenon import FileSystem, PasswordCredential, CopyRequest, Path, CopyStatus
def run_example():
xenon.init()
# use the local file system adaptor to create another file system representation
local_fs = FileSystem.create(adaptor='file')
# use the sftp file system adaptor to create a file system
# representation; the remote filesystem requires credentials to log in,
# so we'll have to create those too.
credential = PasswordCredential(username='xenon',
password='javagat')
remote_fs = FileSystem.create(adaptor='sftp',
location='localhost:10022',
password_credential=credential)
# define which file to upload
local_file = Path('/home/travis/sleep.sh')
remote_file = Path('/home/xenon/sleep.sh')
# overwrite the destination file if it exists
mode = CopyRequest.REPLACE
# no need to recurse, we're just uploading a file
recursive = False
# perform the copy/upload and wait 1000 ms for the successful or
# otherwise completion of the operation
copy_id = local_fs.copy(local_file, remote_fs, remote_file,
mode=mode, recursive=recursive)
copy_status = local_fs.wait_until_done(copy_id, timeout=1000)
assert copy_status.done
# rethrow the Exception if we got one
assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message
# remember to close the FileSystem instances
remote_fs.close()
local_fs.close()
print('Done')
if __name__ == '__main__':
run_example()
|
Now that the script is in place, we can submit a bash
job using xenon scheduler slurm submit
like before, taking
the newly uploaded sleep.sh
file as input to bash
, and using a sleep duration of 60 seconds:
# step 2: submit job
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60
# (should return an identifier for the job)
With the job running, let’s see if it shows up in any of the SLURM queues:
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list
# should have the job identifier in it that was printed on the command line
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.Scheduler;
public class SlurmJobListGetter {
public static void main(String[] args) throws XenonException {
String host = "localhost";
String port = "10022";
runExample(host, port);
}
public static void runExample(String host, String port) throws XenonException {
String adaptor = "slurm";
String location = "ssh://" + host + ":" + port;
// make the password credential for user 'xenon'
String username = "xenon";
char[] password = "javagat".toCharArray();
PasswordCredential credential = new PasswordCredential(username, password);
// create the SLURM scheduler
Scheduler scheduler = Scheduler.create(adaptor, location, credential);
// ask SLURM for its queues and list the jobs in each
for (String queueName : scheduler.getQueueNames()) {
System.out.println("List of jobs in queue '" + queueName + "' for SLURM at " + location + "");
String[] jobs = scheduler.getJobs(queueName);
if (jobs.length == 0) {
System.out.println("(No jobs)");
} else {
for (String job : jobs) {
System.out.println(job);
}
}
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | import xenon
from xenon import PasswordCredential, Scheduler
def run_example():
xenon.init()
credential = PasswordCredential(username='xenon',
password='javagat')
location = 'ssh://localhost:10022'
scheduler = Scheduler.create(adaptor='slurm',
location=location,
password_credential=credential)
for queue_name in scheduler.get_queue_names():
print("List of jobs in queue {queue_name} for SLURM at {location}"
.format(queue_name=queue_name, location=location))
jobs = scheduler.get_jobs([queue_name])
if not jobs:
print("(No jobs)")
else:
for job in jobs:
print(" {}".format(job))
scheduler.close()
if __name__ == '__main__':
run_example()
|
When we submitted, we did not specify any queues, so the default queue mypartition
was used:
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list --queue mypartition
# should have the job identifier in it that was printed on the command line
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list --queue otherpartition
# this queue is empty
With step 1 (upload) and step 2 (submit) covered, step 3 (download) remains:
# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download /home/xenon/sleep.stdout.txt /home/travis/sleep.stdout.txt
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
public class DownloadFileSftpToLocalAbsolutePaths {
public static void main(String[] args) throws XenonException {
String host = "localhost";
String port = "10022";
runExample(host, port);
}
public static void runExample(String host, String port) throws XenonException {
// use the sftp file system adaptor to create a file system representation; the remote
// filesystem requires credentials to log in, so we'll have to create those too.
String adaptorRemote = "sftp";
String location = host + ":" + port;
String username = "xenon";
char[] password = "javagat".toCharArray();
PasswordCredential credential = new PasswordCredential(username, password);
FileSystem filesystemRemote = FileSystem.create(adaptorRemote, location, credential);
// define which file to download
Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");
// use the local file system adaptor to create a file system representation
String adaptorLocal = "file";
FileSystem filesystemLocal = FileSystem.create(adaptorLocal);
// define what file to download to
Path fileLocal = new Path("/home/travis/sleep.stdout.txt");
// create the destination file only if the destination path doesn't exist yet
CopyMode mode = CopyMode.CREATE;
// no need to recurse, we're just downloading a file
boolean recursive = false;
// perform the copy/download and wait 1000 ms for the successful or otherwise
// completion of the operation
String copyId = filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, mode, recursive);
long timeoutMilliSecs = 1000;
CopyStatus copyStatus = filesystemRemote.waitUntilDone(copyId, timeoutMilliSecs);
// print any exceptions
if (copyStatus.getException() != null) {
System.out.println(copyStatus.getException().getMessage());
} else {
System.out.println("Done");
}
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | import xenon
from xenon import FileSystem, PasswordCredential, CopyRequest, Path, CopyStatus
def run_example():
xenon.init()
# use the sftp file system adaptor to create a file system
# representation; the remote filesystem requires credentials to log in,
# so we'll have to create those too.
credential = PasswordCredential(username='xenon',
password='javagat')
remote_fs = FileSystem.create(adaptor='sftp',
location='localhost:10022',
password_credential=credential)
# use the local file system adaptor to create another file system representation
local_fs = FileSystem.create(adaptor='file')
# define which file to download
remote_file = Path('/home/xenon/sleep.stdout.txt')
local_file = Path('/home/travis/sleep.stdout.txt')
# create the destination file only if the destination path doesn't exist yet
mode = CopyRequest.CREATE
# no need to recurse, we're just uploading a file
recursive = False
# perform the copy/download and wait 1000 ms for the successful or
# otherwise completion of the operation
copy_id = remote_fs.copy(remote_file, local_fs, local_file,
mode=mode, recursive=recursive)
copy_status = remote_fs.wait_until_done(copy_id, timeout=5000)
assert copy_status.done
# rethrow the Exception if we got one
assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message
# remember to close the FileSystem instances
remote_fs.close()
local_fs.close()
print('Done')
if __name__ == '__main__':
run_example()
|
By this time you may start to consider putting those 3 commands in a script, as follows:
#!/usr/bin/env bash
# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload /home/travis/sleep.sh /home/xenon/sleep.sh
# step 2: submit job
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60
# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download /home/xenon/sleep.stdout.txt /home/travis/sleep.stdout.txt
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.Scheduler;
public class AllTogetherNowWrong {
public static void main(String[] args) throws XenonException {
String host = "localhost";
String port = "10022";
runExample(host, port);
}
public static void runExample(String host, String port) throws XenonException {
/*
* step 1: upload input file(s)
*/
// create the local filesystem representation
String fileAdaptorLocal = "file";
FileSystem filesystemLocal = FileSystem.create(fileAdaptorLocal);
// the remote system requires credentials, create them here:
String username = "xenon";
char[] password = "javagat".toCharArray();
PasswordCredential credential = new PasswordCredential(username, password);
// create the remote filesystem representation and specify the executable's path
String fileAdaptorRemote = "sftp";
String filesystemRemoteLocation = host + ":" + port;
FileSystem filesystemRemote = FileSystem.create(fileAdaptorRemote,
filesystemRemoteLocation, credential);
{
// specify the behavior in case the target path exists already
CopyMode copyMode = CopyMode.CREATE;
// no recursion, we're just copying a file
boolean recursive = false;
// specify the path of the script file on the local and on the remote
Path fileLocal = new Path("/home/travis/sleep.sh");
Path fileRemote = new Path("/home/xenon/sleep.sh");
// start the copy operation
filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, copyMode, recursive);
}
/*
* step 2: submit job and capture its job identifier
*/
// create the SLURM scheduler representation
String schedulerAdaptor = "slurm";
String schedulerLocation = "ssh://" + host + ":" + port;
Scheduler scheduler = Scheduler.create(schedulerAdaptor, schedulerLocation, credential);
// compose the job description:
JobDescription jobDescription = new JobDescription();
jobDescription.setExecutable("bash");
jobDescription.setArguments("sleep.sh", "60");
jobDescription.setStdout("sleep.stdout.txt");
scheduler.submitBatchJob(jobDescription);
/*
* step 3: download generated output file(s)
*/
{
// specify the behavior in case the target path exists already
CopyMode copyMode = CopyMode.CREATE;
// no recursion, we're just copying a file
boolean recursive = false;
// specify the path of the stdout file on the remote and on the local machine
Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");
Path fileLocal = new Path("/home/travis/sleep.stdout.txt");
// start the copy operation
filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, copyMode, recursive);
}
System.out.println("Done.");
}
}
|
However, if you create the script above and run it, you’ll find that:
Xenon complains about some destination paths already existing.
The script finishes suspiciously quickly;
The first error is easily avoided by adding a --replace
optional argument after upload
and download
, but
that does not address the real issue: that of Xenon not waiting for the completion of our sleep job.
Not to worry though, we can use xenon scheduler slurm wait
to wait for jobs to finish. In order to make this work,
we do need to capture the identifier for a specific job, otherwise we don’t know what to wait for.
Adapt the script as follows and run it:
#!/usr/bin/env bash
# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload --replace /home/travis/sleep.sh /home/xenon/sleep.sh
# step 2: submit job and capture its job identifier
JOBID=$(xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60)
# add blocking wait
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
wait $JOBID
# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download --replace /home/xenon/sleep.stdout.txt /home/travis/sleep.stdout.txt
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | package nl.esciencecenter.xenon.tutorial;
import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.JobStatus;
import nl.esciencecenter.xenon.schedulers.Scheduler;
public class AllTogetherNow {
public static void main(String[] args) throws XenonException {
String host = "localhost";
String port = "10022";
runExample(host, port);
}
public static void runExample(String host, String port) throws XenonException {
/*
* step 1: upload input file(s)
*/
// create the local filesystem representation
String fileAdaptorLocal = "file";
FileSystem filesystemLocal = FileSystem.create(fileAdaptorLocal);
// the remote system requires credentials, create them here:
String username = "xenon";
char[] password = "javagat".toCharArray();
PasswordCredential credential = new PasswordCredential(username, password);
// create the remote filesystem representation and specify the executable's path
String fileAdaptorRemote = "sftp";
String filesystemRemoteLocation = host + ":" + port;
FileSystem filesystemRemote = FileSystem.create(fileAdaptorRemote,
filesystemRemoteLocation, credential);
// when waiting for jobs or copy operations to complete, wait indefinitely
final long WAIT_INDEFINITELY = 0;
{
// specify the behavior in case the target path exists already
CopyMode copyMode = CopyMode.REPLACE;
// no recursion, we're just copying a file
boolean recursive = false;
// specify the path of the script file on the local and on the remote
Path fileLocal = new Path("/home/travis/sleep.sh");
Path fileRemote = new Path("/home/xenon/sleep.sh");
// start the copy operation
String copyId = filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, copyMode, recursive);
// wait for the copy operation to complete (successfully or otherwise)
CopyStatus status = filesystemLocal.waitUntilDone(copyId, WAIT_INDEFINITELY);
// rethrow the Exception if we got one
if (status.hasException()) {
throw status.getException();
}
}
/*
* step 2: submit job and capture its job identifier
*/
// create the SLURM scheduler representation
String schedulerAdaptor = "slurm";
String schedulerLocation = "ssh://" + host + ":" + port;
Scheduler scheduler = Scheduler.create(schedulerAdaptor, schedulerLocation, credential);
// compose the job description:
JobDescription jobDescription = new JobDescription();
jobDescription.setExecutable("bash");
jobDescription.setArguments("sleep.sh", "60");
jobDescription.setStdout("sleep.stdout.txt");
String jobId = scheduler.submitBatchJob(jobDescription);
// wait for the job to finish before attempting to copy its output file(s)
JobStatus jobStatus = scheduler.waitUntilDone(jobId, WAIT_INDEFINITELY);
// rethrow the Exception if we got one
if (jobStatus.hasException()) {
throw jobStatus.getException();
}
/*
* step 3: download generated output file(s)
*/
{
// specify the behavior in case the target path exists already
CopyMode copyMode = CopyMode.REPLACE;
// no recursion, we're just copying a file
boolean recursive = false;
// specify the path of the stdout file on the remote and on the local machine
Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");
Path fileLocal = new Path("/home/travis/sleep.stdout.txt");
// start the copy operation
String copyId = filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, copyMode, recursive);
// wait for the copy operation to complete (successfully or otherwise)
CopyStatus status = filesystemRemote.waitUntilDone(copyId, WAIT_INDEFINITELY);
// rethrow the Exception if we got one
if (status.hasException()) {
throw status.getException();
}
}
System.out.println("Done.");
}
}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | import xenon
from xenon import (FileSystem, PasswordCredential, CopyRequest,
Path, CopyStatus, Scheduler, JobDescription, JobStatus)
def upload(local_fs=None, remote_fs=None):
# define which file to upload
local_file = Path('/home/travis/sleep.sh')
remote_file = Path('/home/xenon/sleep.sh')
# overwrite the destination file if the destination path exists
mode = CopyRequest.REPLACE
# no need to recurse, we're just uploading a file
recursive = False
# perform the copy/upload and wait 1000 ms for the successful or
# otherwise completion of the operation
copy_id = local_fs.copy(local_file, remote_fs, remote_file,
mode=mode, recursive=recursive)
copy_status = local_fs.wait_until_done(copy_id, timeout=5000)
assert copy_status.done
assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message
print('Done uploading.')
def submit(credential=None):
description = JobDescription(executable='bash',
arguments=['sleep.sh', '60'],
stdout='sleep.stdout.txt')
scheduler = Scheduler.create(adaptor='slurm',
location='ssh://localhost:10022',
password_credential=credential)
job_id = scheduler.submit_batch_job(description)
print('Done submitting.')
# wait for the job to finish before attempting to copy its output file(s)
job_status = scheduler.wait_until_done(job_id, 10*60*1000)
assert job_status.done
# rethrow the Exception if we got one
assert job_status.error_type == JobStatus.ErrorType.NONE, job_status.error_message
print('Done executing on the remote.')
# make sure to synchronize the remote filesystem
job_id = scheduler.submit_batch_job(JobDescription(executable='sync'))
scheduler.wait_until_done(job_id)
scheduler.close()
def download(remote_fs=None, local_fs=None):
# define which file to download
remote_file = Path('/home/xenon/sleep.stdout.txt')
local_file = Path('/home/travis/sleep.stdout.txt')
# create the destination file; overwrite local file
mode = CopyRequest.REPLACE
# no need to recurse, we're just uploading a file
recursive = False
# perform the copy/download and wait 1000 ms for the successful or
# otherwise completion of the operation
copy_id = remote_fs.copy(remote_file, local_fs, local_file,
mode=mode, recursive=recursive)
copy_status = remote_fs.wait_until_done(copy_id, timeout=5000)
assert copy_status.done
# rethrow the Exception if we got one
assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message
print('Done downloading.')
def run_example():
xenon.init()
# use the local file system adaptor to create a file system representation
local_fs = FileSystem.create(adaptor='file')
# use the sftp file system adaptor to create another file system representation;
# the remote filesystem requires credentials to log in, so we'll have to
# create those too.
credential = PasswordCredential(username='xenon',
password='javagat')
remote_fs = FileSystem.create(adaptor='sftp',
location='localhost:10022',
password_credential=credential)
upload(local_fs=local_fs, remote_fs=remote_fs)
submit(credential=credential)
download(remote_fs=remote_fs, local_fs=local_fs)
# remember to close the FileSystem instances
remote_fs.close()
local_fs.close()
print('Done')
if __name__ == '__main__':
run_example()
|
After about 60 seconds, you should have a local copy of sleep.stdout.txt
, with the correct contents this time.
Congratulations – you have successfully completed the tutorial!
Cleanup¶
Use docker ps
to check the state of the container:
docker ps
The last column in the resulting table lists the name of the container. By
default, docker
assigns automatically generated names, like
practical_robinson
, keen_goodall
or infallible_morse
. You can use
this name to stop and remove a container once you’re done with it, as follows:
docker stop practical_robinson
docker rm practical_robinson
What’s next?¶
If you want, you can continue reading about relevant subjects, or try some of the suggested exercises.
Further reading¶
Xenon’s homepage on GitHub
Xenon’s JavaDoc on github.io
PyXenon: The Python interface to Xenon (github.com, readthedocs.io)
Suggested exercises¶
Repeat selected exercises, but test against a physically remote system instead of a Docker container. Requires credentials for the remote system.
Repeat selected exercises using WebDAV instead of SFTP. We included the Docker container xenonmiddleware/webdav as part of the virtual machine for testing.
Use the
s3
file adaptor to connect to Amazon’s Simple Storage Service. Either use the Docker container xenonmiddleware/s3 (included in this virtual machine) for testing on your own machine, or use an existing Amazon Web Services account for testing against the real thing.
This document was generated from its source files using Sphinx.