Using the rclone software for accessing dCache InfiniteSpace (dIS)

The following sections detail how to access dIS using the rclone command line tool. We also explain how to configure and utilize the oidc-agent software for authentication.

Disclaimer

This documentation is under development. Report issues to us or directly contribute here.

Quick overview

To access dIS storage, users must authenticate via the Helmholtz AAI, which uses OIDC technology. Originally designed for web-based access, OIDC requires additional steps to enable its use with command-line applications. With the oidc-agent, users need to log into the Helmholtz AAI just once to obtain credentials. These credentials are then used by the oidc-agent to authenticate command-line tools, such as rclone.

Getting and installing rclone and oidc-agent.

Please refer to their official sources for further information on downloading and installing both rclone and oidc-agent.

• Getting and installign the rclone software
  • rclone supports WebDAV/dCache and the Helmholtz AAI technology (oidc-agent, see below)
  • rclone Software Description
  • Downloads and installation instructions for various platforms
• Getting and installing the oidc-agent software
  • oidc-agent handles the Helmholtz AAI for command line systems like rclone.
  • oidc-agent Software Description
  • Downloads and installation instructions for Linux, Mac and Windows.

Starting the oidc-agent

The oidc-agent allows access to dIS on a per-user, per-computer basis. This means each user must run an instance of the agent on any computer where rclone will be used to access dIS.

When launched without options, oidc-agent displays instructions on how its tools can connect to the agent daemon. It then continues running as a background process. The output from the program consists of shell variable assignments, which must be executed, as displayed, in all Linux shells where oidc-agent or rclone tools will be used. This is an example output from a MAC:

$ oidc-agent
OIDC_SOCK=/var/folders/my/<very long string>/oidc-agent.91826;
export OIDC_SOCK;
OIDCD_PID=9004;
export OIDCD_PID;

The OIDC_SOCK represents a socket endpoint in the filesystem. This endpoint allows the oidc-agent tools to communicate with the agent and is exclusively accessible by the user who initiated the oidc-agent.

The next step is to register the Helmholtz AAI and user credentials with the oidc-agent.

Registering the Helmholtz AAI to the oidc-agent

Although we’ll be using rclone as a command-line application, we must initially configure the oidc-agent through a browser. This is primarily because the OIDC mechanism is designed for web-based access.

When setting up access to the dIS for rclone via the oidc-agent, there are two main scenarios you might face:

• Case I You want to use rclone on your laptop or desktop where you are running your browser.
• Case II You want to use rclone on a computer without a browser but you can use a browser on your desktop or laptop computer.

Generic

To generate an access token for rclone we’ll use the oidc-gen command. oidc-gen is part of the oidc-agent package, you already installed and started. oidc-gen registers an identity to the oidc-agent which subsequently provides access tokens to command line tools without further human intervention.

You can either just start the oidc-gen command and it will guide you through some questions or you can provide the right options directy when executing oidc-gen. Independently of the two use cases, the most important command line options for oidc-gen are listed in the table below. Please refer to the two use cases for details.

key	value
--flow	code
--client-id	desy-public
--client-secret	openid profile offline_access
--iss	https://keycloak.desy.de/auth/realms/production/
--no-webserver	Depends on use case
--no-url-call	Depends on use case
--redirect	http://localhost:4242

Case I

Prerequisite: The computer on which you will use the rclone command is running your browser.

Note: The rclone command must be executed with the same User ID (UID) as the oidc-agent, mentioned in one of the previous sections.

Caution: Using oidc-gen, the temporary login information (e.g. token) will be stored in the paths depicted above (presumably a temporary folder). The tool will ask you for a password or you can provide one via e.g. a variable. In the following examples, we presume that you trust the confidentiality of your file system, so you can leave the password (variable) empty. In all other cases, provide a secure password.

To provide your credentials from the Helmholtz AAI to the oidc-agent please run the command below. Make sure that you assigned the two shell variables OIDC_SOCK and OIDCD_PID as described in section “Starting the oidc-agent“

# leave empty only if you are sure that temporary information is stored
# on a trustworthy file system:
export OIDC_PASSWD=""

oidc-gen --flow=code \
         --client-id="desy-public" \
         --client-secret="" \
         --scope="openid profile offline_access" \
         --iss=https://keycloak.desy.de/auth/realms/production/ \
         --redirect-uri=http://localhost:4242 \
         --pw-env=OIDC_PASSWD \
         HIFIS

You may use a service name of your choice instead of “HIFIS”.

If everything is set up correctly, the oidc-get program will open a browser window prompting you to log into the Helmholtz AAI. Search for your Identity Provider from the list and sign-in using your site credentials. It’s crucial not to stop the oidc-gen process while you’re using the browser. Once you’ve completed the login, the webpage will display a status—indicating either success or providing troubleshooting steps in case of failure. If the process is successful, return to the terminal displaying the oidc-gen command.

Case II

If you intend to use rclone on a computer without a browser, but you have access to a browser on another desktop or laptop, there’s a workaround to facilitate communication between your browser and the oidc-gen command. Here’s a brief overview before we delve into the specifics:

Overview

In a situation like Case I, where the oidc-gen command typically launches your browser for Helmholtz AAI login, the absence of a browser on the target system will instead provide you with a URL. Here’s what you need to do:

• Copy this URL.
• Paste it into your accessible browser.
• Complete the login process.
• After a successful login, your browser will attempt a final redirect to communicate with the oidc-gen command. Since there’s no proper network connection, an error will display. You can disregard this error.
• Copy the URL from the address bar, which contains the encoded access token.
• Provide this URL to the oidc-agent to inform it about the access token.

Now, for a step-by-step guide:

Issue the following command on the target computer, were you intend to run the rclone command and where you started the oidc-agent command under your User ID.

# leave empty only if you are sure that temporary information is stored
# on a trustworthy file system:
export OIDC_PASSWD=""

oidc-gen --flow=code \
         --no-webserver \
         --no-url-call \
         --client-id="desy-public" \
         --client-secret="" \
         --scope="openid profile offline_access" \
         --iss=https://keycloak.desy.de/auth/realms/production/ \
         --redirect-uri=http://localhost:4242 \
         --pw-env=OIDC_PASSWD \
         HIFIS

The extra options --no-webserver -no-url-call instruct the oidc-gen not to start a browser on this machine.

Copy this URL and paste it into the address bar of your browser!

This URL directs you to the Helmholtz login process. Choose your preferred IdP and log in. If successful, the browser will attempt to reach an inaccessible URL. This results in an error message, with the browser unable to connect to a server (likely localhost:4242 or similar).

Provide the content of your browser URL bar to the oidc-gen command!

To proceed, relay the redirect URL from the browser’s address bar to the oidc-agent as described next:

oidc-gen --codeExchange='<url>'

where <url> is the full content browser’s address bar. It will likely start with:

http://localhost:4242/?state=

Linking rclone to the credentials in the oidc-agent.

With the steps outlined above, the authentication process is now complete. Next, we need to configure the rclone tool to retrieve the token from the oidc-agent. To do this, the rclone configuration file must be updated.

You can refer to the rclone documentation for the precise location of its configuration file. However, it’s typically found here:

$HOME/.config/rclone/rclone.conf

Add the following section to the end of rclone.conf or create the file if it doesn’t exist yet.

[HIFIS]
type = webdav
url = https://hifis-storage.desy.de:2880/Helmholtz/<your space>
vendor = other
bearer_token_command = oidc-token HIFIS

• Instead of “HIFIS” please use the keyword you picked for the oidc-gen command.
• <your space> is the path you got when requesting dIS space. It’s most likely the name of your VO.

To verify that everything is setup properly you can run

rclone ls HIFIS:

Again, replace ‘HIFIS’ with the keyword you picked for that space. It should display the content of your storage space.

Please refer to the help pages of rclone or type

rclone --help

for upload, download and name space commands.

Mounting into your file system

If wanted and provided you have sufficient local rights, you can use rclone to mount the folder(s) into your local file system for further access:

rclone mount "HIFIS:" "<path_to_your_mountpoint>" --vfs-cache-mode full