xenon-logo



Note we have GoogleAnalytics enabled for this tutorial to help show our funders that the work we do is useful to others.




Getting started

On your system, start VirtualBox.

This tutorial uses a virtual machine to help avoid issues that are due to system configuration. In case you don’t have a copy of the virtual machine, you can download it from Zenodo here. After the download finishes, click File in VirtualBox, then Import appliance, then select the file you downloaded.

During the import, you’ll see an initialization wizard. Make sure that the virtual machine is configured with two CPUs.

Start the virtual machine and log in as user travis with password password.

Once the system has booted, start both a terminal and Firefox by clicking their respective icons. Use Firefox to navigate to the tutorial text at https://xenon-tutorial.readthedocs.io.

In the terminal, confirm that the xenon command line interface program can be found on the system:

xenon --help
xenon --version
Xenon CLI v3.0.4, Xenon library v3.0.4, Xenon cloud library v3.0.2

If you run into trouble, here are some pointers on what you can do.


Interacting with filesystems

Essentially, xenon can be used to manipulate files and to interact with schedulers, where either one can be local or remote. Let’s start simple and see if we can do something with local files. First, check its help:

xenon filesystem --help

The usage line suggests we need to pick one from {file,ftp,s3,sftp,webdav}. Again, choose what seems to be the simplest option (file), and again, check its help.

xenon filesystem file --help

xenon filesystem file’s usage line seems to suggest that I need to pick one from {copy,list,mkdir,remove,rename}. Simplest one is probably list, so:

xenon filesystem file list --help

So we need a path as final argument.

In case you hadn’t noticed the pattern, stringing together any number of xenon subcommands and appending --help to it will get you help on the particular combination of subcommands you supplied.

The focus of this tutorial is on using Xenon’s command line interface, but be aware that you can use xenon’s functionality from other programming languages through xenon’s gRPC extension.

Where relevant, we have included equivalent code snippets, written in Java and Python, as a separate tab.

Let’s try listing the contents of /home/travis/fixtures/.

xenon filesystem file list /home/travis/fixtures
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;


public class DirectoryListing {

    public static void main(String[] args) throws Exception {

        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);
        Path directory = new Path("/home/travis/fixtures");
        Boolean recursive = false;
        Iterable<PathAttributes> listing = filesystem.list(directory, recursive);

        for (PathAttributes elem : listing) {
            if (!elem.isHidden()) {
                System.out.println(elem.getPath());
            }
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import xenon
from xenon import Path, FileSystem


def run_example():

    xenon.init()

    filesystem = FileSystem.create(adaptor='file')
    path = Path("/home/travis/fixtures")

    listing = filesystem.list(path, recursive=False)

    for entry in listing:
        if not entry.path.is_hidden():
            print(entry.path)

    filesystem.close()


if __name__ == '__main__':
    run_example()

The result should be more or less the same as that of ls -1.

xenon filesystem file list has a few options that let you specify the details of the list operation, e.g. --hidden

xenon filesystem file list --hidden /home/travis/fixtures
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;

public class DirectoryListingShowHidden {

    public static void main(String[] args) throws Exception {

        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);

        Path dir = new Path("/home/travis/fixtures");
        boolean recursive = false;

        Iterable<PathAttributes> listing = filesystem.list(dir, recursive);

        for (PathAttributes elem : listing) {
            System.out.println(elem.getPath());
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import xenon
from xenon import Path, FileSystem


def run_example():
    xenon.init()

    filesystem = FileSystem.create(adaptor='file')
    path = Path("/home/travis/fixtures")

    listing = filesystem.list(path, recursive=False)

    for entry in listing:
        print(entry.path)

    filesystem.close()


if __name__ == '__main__':
    run_example()

and --recursive

xenon filesystem file list --recursive /home/travis/fixtures
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;

public class DirectoryListingRecursive {

    public static void main(String[] args) throws Exception {

        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);

        Path dir = new Path("/home/travis/fixtures");
        boolean recursive = true;

        Iterable<PathAttributes> listing = filesystem.list(dir, recursive);

        for (PathAttributes elem : listing) {
            if (!elem.isHidden()) {
                System.out.println(elem.getPath());
            }
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import xenon
from xenon import Path, FileSystem


def run_example():

    xenon.init()

    filesystem = FileSystem.create(adaptor='file')
    path = Path("/home/travis/fixtures")

    listing = filesystem.list(path, recursive=True)

    for entry in listing:
        if not entry.path.is_hidden():
            print(entry.path)

    filesystem.close()


if __name__ == '__main__':
    run_example()

Now let’s create a file and try to use xenon to copy it:

cd /home/travis
echo 'some content' > thefile.txt

Check the relevant help

xenon filesystem file --help
xenon filesystem file copy --help

So, the copy subcommand takes a source path and a target path:

xenon filesystem file copy /home/travis/thefile.txt /home/travis/thefile.bak
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class CopyFileLocalToLocalAbsolutePaths {

    public static void main(String[] args) throws Exception {

        // use the local file system adaptor to create a file system representation
        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);

        // create Paths for the source and destination files, using absolute paths
        Path sourceFile = new Path("/home/travis/thefile.txt");
        Path destFile = new Path("/home/travis/thefile.bak");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;
        boolean recursive = false;

        // perform the copy and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = filesystem.copy(sourceFile, filesystem, destFile, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = filesystem.waitUntilDone(copyId, timeoutMilliSecs);

        // print any exceptions
        if (copyStatus.getException() != null) {
            System.out.println(copyStatus.getException().getMessage());
        } else {
            System.out.println("File copied.");
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import xenon
from xenon import Path, FileSystem, CopyRequest, CopyStatus


def run_example():

    xenon.init()

    filesystem = FileSystem.create(adaptor='file')
    source_file = Path("/home/travis/thefile.txt")
    dest_file = Path("/home/travis/thefile.bak")

    # start the copy operation; no recursion, we're just copying a file
    copy_id = filesystem.copy(source_file, filesystem, dest_file, mode=CopyRequest.CREATE, recursive=False)

    # wait this many milliseconds for copy operations to complete
    wait_duration = 1000 * 60 * 10

    # wait for the copy operation to complete (successfully or otherwise)
    copy_status = filesystem.wait_until_done(copy_id, wait_duration)

    assert copy_status.done

    # rethrow the Exception if we got one
    assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message

    filesystem.close()


if __name__ == '__main__':
    run_example()

Note that the source path may be standard input, and that the target path may be standard output:

# read from stdin:
cat thefile.txt | xenon filesystem file copy - mystdin.txt

# write to stdout:
xenon filesystem file copy thefile.txt - 1> mystdout.txt

xenon filesystem file has a few more subcommands, namely mkdir, rename and remove. You can experiment a bit more with those before moving on to the next section.

Access to remote filesystems

Of course the point of xenon is not to move around files on your local filesystem. There are enough tools to help you with that. The idea is that you can also use xenon to move files to and from different types of remote servers, without having to learn a completely different tool every time.

First, let’s check which types of file servers xenon currently supports:

xenon filesystem --help

The usage line shows that besides file we can also choose ftp, s3, sftp or webdav. Let’s try ftp first.

xenon filesystem ftp --help

The usage line tells us that ftp has an mandatory parameter --location which we haven’t seen yet. We can use this to specify which server to connect to. Additionally, there are also optional --username and --password options in case we need to log into the machine.

Let’s see if we can use this to connect to a real machine on the internet. A public FTP server for testing should be available at test.rebex.net with the credentials demo and password:

# list the files on the ftp server
xenon filesystem ftp --location test.rebex.net --username demo --password password list /
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;

public class FTPDirectoryListingShowHidden {

    public static void main(String[] args) throws Exception {

        String adaptor = "ftp";
        String location = "test.rebex.net";
        String username = "demo";
        String password = "password";

        try (FileSystem filesystem = FileSystem.create(adaptor, location, new PasswordCredential(username, password))) {

           Path dir = new Path("/");
           boolean recursive = false;

           Iterable<PathAttributes> listing = filesystem.list(dir, recursive);

           for (PathAttributes elem : listing) {
               System.out.println(elem.getPath());
           }
        }
    }
}

This should give you a listing of the server at test.rebex.net.

Besides the commands we have already seen (copy, list, etc.), ftp also supports a few new ones, namely upload and download. We can use these to transer files to and from the server. For example, this command will download a file from our example FTP server:

# download a file from the ftp server
xenon filesystem ftp --location test.rebex.net --username demo --password password download /readme.txt `pwd`/readme.txt

You can even print the remote file on your screen by copying it to stdout:

# print a file from the ftp server on the screen
xenon filesystem ftp --location test.rebex.net --username demo --password password download /readme.txt -

Note that when using copy on remote servers, xenon will attempt to copy the file on the server itself. Since we don’t have write access to this FTP server, the command will fail.

The strength of xenon is that you can now use the same syntax to access a different type of server. For example, the test.rebex.net server also offers a secure FTP (sftp) service for testing. We can access that service with xenon by simply changing ftp into sftp:

# list the files on the sftp server
xenon filesystem sftp --location test.rebex.net --username demo --password password list /

# download a file from the sftp server
xenon filesystem sftp --location test.rebex.net --username demo --password password download /readme.txt `pwd`/readme2.txt

In case you are reluctant to type plaintext passwords on the command line, for example because of logging in ~/.bash_history, know that you can supply passwords from a file, as follows:

# read password from the password.txt file
xenon filesystem sftp --location test.rebex.net --username demo --password @password.txt list /

in which the file password.txt should contain the password. Since everything about the user xenon is public knowledge anyway, such security precautions are not needed for this tutorial, so we’ll just continue to use the --password PASSWORD syntax.

You can also transfer data from and to other types of file servers (such as WebDAV and S3) in a similar fashion. We are working to add support for other types such as GridFTP and iRODS. We will come back to transferring files in the sections below.




Interacting with schedulers

Now let’s see if we can use schedulers, starting with SLURM. For this part, we need access to a machine that is running SLURM. To avoid problems related to network connectivity, we won’t try to connect to a physically remote SLURM machine, but instead, we’ll use a dockerized SLURM installation. This way, we can mimic whatever infrastructure we need. The setup will thus be something like this:

babushka


A copy of the SLURM Docker image (xenonmiddleware/slurm:17) has been included in the virtual machine. Bring it up with:

docker run --detach --publish 10022:22 --hostname slurm17 xenonmiddleware/slurm:17

Use docker ps to check the state of the container

docker ps

The last column in the resulting table lists the name of the container. By default, docker assigns automatically generated names, like practical_robinson, keen_goodall or infallible_morse. You can use this name to stop and remove a container once you’re done with it, as follows:

docker stop practical_robinson
docker rm practical_robinson

Anyway, that’s for later. For now, check that the container’s status is healthy, and see if you can ssh into it on port 10022 as user xenon with password javagat:

ssh -p 10022 xenon@localhost

# if that works, exit again
exit

Be aware that ssh can sometimes be a little picky. We’ve assembled a list of tips and tricks for troubleshooting SSH.

Check the help to see how the slurm subcommand works:

xenon scheduler slurm --help

Let’s first ask what queues the SLURM scheduler has. For this, we need to specify a location, otherwise xenon does not know who to ask for the list of queues. According to the help, LOCATION is any location format supported by ssh or local scheduler. Our dockerized SLURM machine is reachable as ssh://localhost:10022. We’ll also need to provide a --username and --password for that location, as follows:

xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat queues
# returns:
# Available queues: mypartition, otherpartition
# Default queue: mypartition
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.Credential;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class SlurmQueuesGetter {


    public static void main(String[] args) throws XenonException {

        String host = "localhost";
        String port = "10022";
        runExample(host, port);
    }

    public static void runExample(String host, String port) throws XenonException {

        String adaptor = "slurm";
        String location = "ssh://" + host + ":" + port;

        // make the password credential for user 'xenon'
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        Credential credential = new PasswordCredential(username, password);

        // create the SLURM scheduler
        try (Scheduler scheduler = Scheduler.create(adaptor, location, credential)) {
            // ask SLURM for its queues
            System.out.println("Available queues (starred queue is default):");
            for (String queueName : scheduler.getQueueNames()) {
                if (queueName.equals(scheduler.getDefaultQueueName())) {
                    System.out.println(queueName + "*");
                } else {
                    System.out.println(queueName);
                }
            }
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import xenon
from xenon import Scheduler, PasswordCredential


def run_example():

    xenon.init()

    credential = PasswordCredential(username='xenon',
                                    password='javagat')

    scheduler = Scheduler.create(adaptor='slurm',
                                 location='ssh://localhost:10022',
                                 password_credential=credential)

    default_queue = scheduler.get_default_queue_name()

    print("Available queues (starred queue is default):")
    for queue_name in scheduler.get_queue_names():
        if queue_name == default_queue:
            print("{}*".format(queue_name))
        else:
            print(queue_name)

    scheduler.close()


if __name__ == '__main__':
    run_example()

Besides queues, other slurm subcommands are exec, submit, list, remove, and wait. Let’s try to have xenon ask SLURM for its list of jobs in each queue, as follows:

xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list
# should work, but we don't have any jobs yet

Now, let’s try to submit a job using slurm submit. Its usage string suggests that we need to provide (the path of) an executable. Note that the executable should be present inside the container when SLURM starts its execution. For the moment, we’ll use /bin/hostname as the executable. It simply prints the name of the host on the command line. For our docker container, it should return the hostname slurm17 of the Docker container, or whatever hostname you specified for it when you ran the docker run command earlier:

# check the slurm submit help for correct syntax
xenon scheduler slurm submit --help

# let xenon submit a job with /bin/hostname as executable
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit /bin/hostname

# add --stdout to the submit job to capture its standard out so we know it worked:
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout hostname.stdout.txt /bin/hostname

# check to see if the output was written to file /home/xenon/hostname.stdout.txt
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat download hostname.stdout.txt -

Below are a few more examples of slurm submit:

# executables that take options prefixed with '-' need special syntax, e.g. 'ls -la'
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout /home/xenon/ls.stdout.txt ls -- -la

# check to see if the output was written to file /home/xenon/ls.stdout.txt
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat download ls.stdout.txt -

# submit an 'env' job with environment variable MYKEY, and capture standard out so we know it worked
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout /home/xenon/env.stdout.txt --env MYKEY=myvalue /usr/bin/env

# check to see if the output from 'env' was written to file /home/xenon/env.stdout.txt
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat download env.stdout.txt -



Combining filesystems and schedulers

So far, we’ve used xenon to manipulate files on the local filesystem, and to run system executables on the remote machine. In typical usage, however, you would use xenon to run executables or scripts of your own, which means that we need to upload such files from the local system to the remote system.

A typical workflow may thus look like this:

  1. upload input file(s)

  2. submit job

  3. download generated output file(s)

Use an editor to create a file sleep.sh with the following contents (the virtual machine comes with a bunch of editors like gedit, leafpad, and nano, but you can install a different editor from the repositories if you like):

#!/usr/bin/env bash
echo `date`': went to bed'
sleep $1
echo `date`': woke up'

You can test if your file is correct by:

# last argument is the sleep duration in seconds
bash sleep.sh 5

We need to upload sleep.sh to the remote machine. We can’t use xenon filesystem file like we did before, because we’re copying between file systems, so let’s look at what other options are available:

xenon filesystem --help

# let's try sftp protocol
xenon filesystem sftp --help

# we're interested in 'upload' for now
xenon filesystem sftp upload --help

We’ll also need to tell xenon what location we want to connect to, and what credentials to use. The SLURM Docker container we used before is accessible via SFTP using the same location, username and password as before, so let’s use that:

# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload /home/travis/sleep.sh /home/xenon/sleep.sh
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class UploadFileLocalToSftpAbsolutePaths {


    public static void main(String[] args) throws XenonException {

        String host = "localhost";
        String port = "10022";
        runExample(host, port);
    }

    public static void runExample(String host, String port) throws XenonException {

        // use the local file system adaptor to create a file system representation
        String adaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(adaptorLocal);

        // define what file to upload
        Path fileLocal = new Path("/home/travis/sleep.sh");

        // use the sftp file system adaptor to create a file system representation; the remote
        // filesystem requires credentials to log in, so we'll have to create those too.
        String adaptorRemote = "sftp";
        String location = host + ":" + port;
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);
        FileSystem filesystemRemote = FileSystem.create(adaptorRemote, location, credential);

        // define which file to upload to
        Path fileRemote = new Path("/home/xenon/sleep.sh");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;

        // no need to recurse, we're just downloading a file
        boolean recursive = false;

        // perform the copy/upload and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = filesystemLocal.waitUntilDone(copyId, timeoutMilliSecs);

        // print any exceptions
        if (copyStatus.getException() != null) {
            System.out.println(copyStatus.getException().getMessage());
        } else {
            System.out.println("Done");
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import xenon
from xenon import FileSystem, PasswordCredential, CopyRequest, Path, CopyStatus


def run_example():

    xenon.init()

    # use the local file system adaptor to create another file system representation
    local_fs = FileSystem.create(adaptor='file')

    # use the sftp file system adaptor to create a file system
    # representation; the remote filesystem requires credentials to log in,
    # so we'll have to create those too.
    credential = PasswordCredential(username='xenon',
                                    password='javagat')

    remote_fs = FileSystem.create(adaptor='sftp',
                                  location='localhost:10022',
                                  password_credential=credential)

    # define which file to upload
    local_file = Path('/home/travis/sleep.sh')
    remote_file = Path('/home/xenon/sleep.sh')

    # overwrite the destination file if it exists
    mode = CopyRequest.REPLACE

    # no need to recurse, we're just uploading a file
    recursive = False

    # perform the copy/upload and wait 1000 ms for the successful or
    # otherwise completion of the operation
    copy_id = local_fs.copy(local_file, remote_fs, remote_file,
                            mode=mode, recursive=recursive)

    copy_status = local_fs.wait_until_done(copy_id, timeout=1000)

    assert copy_status.done

    # rethrow the Exception if we got one
    assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message

    # remember to close the FileSystem instances
    remote_fs.close()
    local_fs.close()

    print('Done')


if __name__ == '__main__':
    run_example()

Now that the script is in place, we can submit a bash job using xenon scheduler slurm submit like before, taking the newly uploaded sleep.sh file as input to bash, and using a sleep duration of 60 seconds:

# step 2: submit job
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60

# (should return an identifier for the job)

With the job running, let’s see if it shows up in any of the SLURM queues:

xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list
# should have the job identifier in it that was printed on the command line
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class SlurmJobListGetter {

    public static void main(String[] args) throws XenonException {

        String host = "localhost";
        String port = "10022";
        runExample(host, port);
    }

    public static void runExample(String host, String port) throws XenonException {


        String adaptor = "slurm";
        String location = "ssh://" + host + ":" + port;

        // make the password credential for user 'xenon'
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the SLURM scheduler
        Scheduler scheduler = Scheduler.create(adaptor, location, credential);

        // ask SLURM for its queues and list the jobs in each
        for (String queueName : scheduler.getQueueNames()) {
            System.out.println("List of jobs in queue '" + queueName + "' for SLURM at " + location + "");
            String[] jobs = scheduler.getJobs(queueName);
            if (jobs.length == 0) {
                System.out.println("(No jobs)");
            } else {
                for (String job : jobs) {
                    System.out.println(job);
                }
            }
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import xenon
from xenon import PasswordCredential, Scheduler


def run_example():

    xenon.init()

    credential = PasswordCredential(username='xenon',
                                    password='javagat')

    location = 'ssh://localhost:10022'

    scheduler = Scheduler.create(adaptor='slurm',
                                 location=location,
                                 password_credential=credential)

    for queue_name in scheduler.get_queue_names():
        print("List of jobs in queue {queue_name} for SLURM at {location}"
              .format(queue_name=queue_name, location=location))
        jobs = scheduler.get_jobs([queue_name])
        if not jobs:
            print("(No jobs)")
        else:
            for job in jobs:
                print("    {}".format(job))

    scheduler.close()


if __name__ == '__main__':
    run_example()

When we submitted, we did not specify any queues, so the default queue mypartition was used:

xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list --queue mypartition
# should have the job identifier in it that was printed on the command line

xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat list --queue otherpartition
# this queue is empty

With step 1 (upload) and step 2 (submit) covered, step 3 (download) remains:

# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download /home/xenon/sleep.stdout.txt /home/travis/sleep.stdout.txt
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class DownloadFileSftpToLocalAbsolutePaths {

    public static void main(String[] args) throws XenonException {

        String host = "localhost";
        String port = "10022";
        runExample(host, port);
    }

    public static void runExample(String host, String port) throws XenonException {


        // use the sftp file system adaptor to create a file system representation; the remote
        // filesystem requires credentials to log in, so we'll have to create those too.
        String adaptorRemote = "sftp";
        String location = host + ":" + port;
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);
        FileSystem filesystemRemote = FileSystem.create(adaptorRemote, location, credential);

        // define which file to download
        Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");

        // use the local file system adaptor to create a file system representation
        String adaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(adaptorLocal);

        // define what file to download to
        Path fileLocal = new Path("/home/travis/sleep.stdout.txt");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;

        // no need to recurse, we're just downloading a file
        boolean recursive = false;

        // perform the copy/download and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = filesystemRemote.waitUntilDone(copyId, timeoutMilliSecs);

        // print any exceptions
        if (copyStatus.getException() != null) {
            System.out.println(copyStatus.getException().getMessage());
        } else {
            System.out.println("Done");
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import xenon
from xenon import FileSystem, PasswordCredential, CopyRequest, Path, CopyStatus


def run_example():

    xenon.init()

    # use the sftp file system adaptor to create a file system
    # representation; the remote filesystem requires credentials to log in,
    # so we'll have to create those too.
    credential = PasswordCredential(username='xenon',
                                    password='javagat')

    remote_fs = FileSystem.create(adaptor='sftp',
                                  location='localhost:10022',
                                  password_credential=credential)

    # use the local file system adaptor to create another file system representation
    local_fs = FileSystem.create(adaptor='file')

    # define which file to download
    remote_file = Path('/home/xenon/sleep.stdout.txt')
    local_file = Path('/home/travis/sleep.stdout.txt')

    # create the destination file only if the destination path doesn't exist yet
    mode = CopyRequest.CREATE

    # no need to recurse, we're just uploading a file
    recursive = False

    # perform the copy/download and wait 1000 ms for the successful or
    # otherwise completion of the operation
    copy_id = remote_fs.copy(remote_file, local_fs, local_file,
                             mode=mode, recursive=recursive)

    copy_status = remote_fs.wait_until_done(copy_id, timeout=5000)

    assert copy_status.done

    # rethrow the Exception if we got one
    assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message

    # remember to close the FileSystem instances
    remote_fs.close()
    local_fs.close()

    print('Done')


if __name__ == '__main__':
    run_example()

By this time you may start to consider putting those 3 commands in a script, as follows:

#!/usr/bin/env bash

# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload /home/travis/sleep.sh /home/xenon/sleep.sh

# step 2: submit job
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60

# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download /home/xenon/sleep.stdout.txt /home/travis/sleep.stdout.txt
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class AllTogetherNowWrong {

    public static void main(String[] args) throws XenonException {

        String host = "localhost";
        String port = "10022";

        runExample(host, port);
    }

    public static void runExample(String host, String port) throws XenonException {

        /*
         * step 1: upload input file(s)
         */

        // create the local filesystem representation
        String fileAdaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(fileAdaptorLocal);

        // the remote system requires credentials, create them here:
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the remote filesystem representation and specify the executable's path
        String fileAdaptorRemote = "sftp";
        String filesystemRemoteLocation = host + ":" + port;
        FileSystem filesystemRemote = FileSystem.create(fileAdaptorRemote,
                filesystemRemoteLocation, credential);

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.CREATE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the script file on the local and on the remote
            Path fileLocal = new Path("/home/travis/sleep.sh");
            Path fileRemote = new Path("/home/xenon/sleep.sh");

            // start the copy operation
            filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, copyMode, recursive);

        }



        /*
         * step 2: submit job and capture its job identifier
         */

        // create the SLURM scheduler representation
        String schedulerAdaptor = "slurm";
        String schedulerLocation = "ssh://" + host + ":" + port;
        Scheduler scheduler = Scheduler.create(schedulerAdaptor, schedulerLocation, credential);

        // compose the job description:
        JobDescription jobDescription = new JobDescription();
        jobDescription.setExecutable("bash");
        jobDescription.setArguments("sleep.sh", "60");
        jobDescription.setStdout("sleep.stdout.txt");

        scheduler.submitBatchJob(jobDescription);


        /*
         *  step 3: download generated output file(s)
         */

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.CREATE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the stdout file on the remote and on the local machine
            Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");
            Path fileLocal = new Path("/home/travis/sleep.stdout.txt");

            // start the copy operation
            filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, copyMode, recursive);

        }

        System.out.println("Done.");

    }
}

However, if you create the script above and run it, you’ll find that:

  1. Xenon complains about some destination paths already existing.

  2. The script finishes suspiciously quickly;

The first error is easily avoided by adding a --replace optional argument after upload and download, but that does not address the real issue: that of Xenon not waiting for the completion of our sleep job.

Not to worry though, we can use xenon scheduler slurm wait to wait for jobs to finish. In order to make this work, we do need to capture the identifier for a specific job, otherwise we don’t know what to wait for.

Adapt the script as follows and run it:

#!/usr/bin/env bash

# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload --replace /home/travis/sleep.sh /home/xenon/sleep.sh

# step 2: submit job and capture its job identifier
JOBID=$(xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60)

# add blocking wait
xenon scheduler slurm --location ssh://localhost:10022 --username xenon --password javagat \
wait $JOBID

# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download --replace /home/xenon/sleep.stdout.txt /home/travis/sleep.stdout.txt
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
package nl.esciencecenter.xenon.tutorial;

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.JobStatus;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class AllTogetherNow {

    public static void main(String[] args) throws XenonException {

        String host = "localhost";
        String port = "10022";

        runExample(host, port);
    }

    public static void runExample(String host, String port) throws XenonException {

        /*
         * step 1: upload input file(s)
         */

        // create the local filesystem representation
        String fileAdaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(fileAdaptorLocal);

        // the remote system requires credentials, create them here:
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the remote filesystem representation and specify the executable's path
        String fileAdaptorRemote = "sftp";
        String filesystemRemoteLocation = host + ":" + port;
        FileSystem filesystemRemote = FileSystem.create(fileAdaptorRemote,
                filesystemRemoteLocation, credential);

        // when waiting for jobs or copy operations to complete, wait indefinitely
        final long WAIT_INDEFINITELY = 0;

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.REPLACE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the script file on the local and on the remote
            Path fileLocal = new Path("/home/travis/sleep.sh");
            Path fileRemote = new Path("/home/xenon/sleep.sh");

            // start the copy operation
            String copyId = filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, copyMode, recursive);

            // wait for the copy operation to complete (successfully or otherwise)
            CopyStatus status = filesystemLocal.waitUntilDone(copyId, WAIT_INDEFINITELY);

            // rethrow the Exception if we got one
            if (status.hasException()) {
                throw status.getException();
            }

        }



        /*
         * step 2: submit job and capture its job identifier
         */

        // create the SLURM scheduler representation
        String schedulerAdaptor = "slurm";
        String schedulerLocation = "ssh://" + host + ":" + port;
        Scheduler scheduler = Scheduler.create(schedulerAdaptor, schedulerLocation, credential);

        // compose the job description:
        JobDescription jobDescription = new JobDescription();
        jobDescription.setExecutable("bash");
        jobDescription.setArguments("sleep.sh", "60");
        jobDescription.setStdout("sleep.stdout.txt");

        String jobId = scheduler.submitBatchJob(jobDescription);

        // wait for the job to finish before attempting to copy its output file(s)
        JobStatus jobStatus = scheduler.waitUntilDone(jobId, WAIT_INDEFINITELY);

        // rethrow the Exception if we got one
        if (jobStatus.hasException()) {
            throw jobStatus.getException();
        }


        /*
         *  step 3: download generated output file(s)
         */

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.REPLACE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the stdout file on the remote and on the local machine
            Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");
            Path fileLocal = new Path("/home/travis/sleep.stdout.txt");

            // start the copy operation
            String copyId = filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, copyMode, recursive);

            // wait for the copy operation to complete (successfully or otherwise)
            CopyStatus status = filesystemRemote.waitUntilDone(copyId, WAIT_INDEFINITELY);

            // rethrow the Exception if we got one
            if (status.hasException()) {
                throw status.getException();
            }

        }

        System.out.println("Done.");

    }
}
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
import xenon
from xenon import (FileSystem, PasswordCredential, CopyRequest,
    Path, CopyStatus, Scheduler, JobDescription, JobStatus)


def upload(local_fs=None, remote_fs=None):

    # define which file to upload
    local_file = Path('/home/travis/sleep.sh')
    remote_file = Path('/home/xenon/sleep.sh')

    # overwrite the destination file if the destination path exists
    mode = CopyRequest.REPLACE

    # no need to recurse, we're just uploading a file
    recursive = False

    # perform the copy/upload and wait 1000 ms for the successful or
    # otherwise completion of the operation
    copy_id = local_fs.copy(local_file, remote_fs, remote_file,
                            mode=mode, recursive=recursive)

    copy_status = local_fs.wait_until_done(copy_id, timeout=5000)

    assert copy_status.done

    assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message

    print('Done uploading.')


def submit(credential=None):

    description = JobDescription(executable='bash',
                                 arguments=['sleep.sh', '60'],
                                 stdout='sleep.stdout.txt')

    scheduler = Scheduler.create(adaptor='slurm',
                                 location='ssh://localhost:10022',
                                 password_credential=credential)

    job_id = scheduler.submit_batch_job(description)

    print('Done submitting.')

    # wait for the job to finish before attempting to copy its output file(s)
    job_status = scheduler.wait_until_done(job_id, 10*60*1000)

    assert job_status.done

    # rethrow the Exception if we got one
    assert job_status.error_type == JobStatus.ErrorType.NONE, job_status.error_message

    print('Done executing on the remote.')

    # make sure to synchronize the remote filesystem
    job_id = scheduler.submit_batch_job(JobDescription(executable='sync'))
    scheduler.wait_until_done(job_id)

    scheduler.close()


def download(remote_fs=None, local_fs=None):

    # define which file to download
    remote_file = Path('/home/xenon/sleep.stdout.txt')
    local_file = Path('/home/travis/sleep.stdout.txt')

    # create the destination file; overwrite local file
    mode = CopyRequest.REPLACE

    # no need to recurse, we're just uploading a file
    recursive = False

    # perform the copy/download and wait 1000 ms for the successful or
    # otherwise completion of the operation
    copy_id = remote_fs.copy(remote_file, local_fs, local_file,
                             mode=mode, recursive=recursive)

    copy_status = remote_fs.wait_until_done(copy_id, timeout=5000)

    assert copy_status.done

    # rethrow the Exception if we got one
    assert copy_status.error_type == CopyStatus.ErrorType.NONE, copy_status.error_message

    print('Done downloading.')


def run_example():

    xenon.init()

    # use the local file system adaptor to create a file system representation
    local_fs = FileSystem.create(adaptor='file')

    # use the sftp file system adaptor to create another file system representation;
    # the remote filesystem requires credentials to log in, so we'll have to
    # create those too.
    credential = PasswordCredential(username='xenon',
                                    password='javagat')

    remote_fs = FileSystem.create(adaptor='sftp',
                                  location='localhost:10022',
                                  password_credential=credential)

    upload(local_fs=local_fs, remote_fs=remote_fs)
    submit(credential=credential)
    download(remote_fs=remote_fs, local_fs=local_fs)

    # remember to close the FileSystem instances
    remote_fs.close()
    local_fs.close()

    print('Done')


if __name__ == '__main__':
    run_example()

After about 60 seconds, you should have a local copy of sleep.stdout.txt, with the correct contents this time.

Congratulations – you have successfully completed the tutorial!

Cleanup

Use docker ps to check the state of the container:

docker ps

The last column in the resulting table lists the name of the container. By default, docker assigns automatically generated names, like practical_robinson, keen_goodall or infallible_morse. You can use this name to stop and remove a container once you’re done with it, as follows:

docker stop practical_robinson
docker rm practical_robinson



What’s next?

If you want, you can continue reading about relevant subjects, or try some of the suggested exercises.

Further reading

Suggested exercises

  • Repeat selected exercises, but test against a physically remote system instead of a Docker container. Requires credentials for the remote system.

  • Repeat selected exercises using WebDAV instead of SFTP. We included the Docker container xenonmiddleware/webdav as part of the virtual machine for testing.

  • Use the s3 file adaptor to connect to Amazon’s Simple Storage Service. Either use the Docker container xenonmiddleware/s3 (included in this virtual machine) for testing on your own machine, or use an existing Amazon Web Services account for testing against the real thing.







This document was generated from its source files using Sphinx.