Data Science Lab

rclone is tool for using remote cloud based storage systems. You must have a valid account on one of the remote cloud systems to use rclone See the rclone page for more information.

The following example demonstrates how to use rclone with Google Drive.

The following steps must also be performed using a terminal window on one of the DSL Lab workstations. Once configured, however, the rclone commands can be run within Notebooks like Zeppelin.

Open text window under your account and enter rclone config. The following will guide the Google Drive configuration.

$ rclone config
2022/02/14 13:32:45 NOTICE: Config file "/home/USER/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> mygdrive

Enter n for new and the name of your “remote” drive (in this case mygdrive. Also USER will be your account name.

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Citrix Sharefile
   \ "sharefile"
 9 / Compress a remote
   \ "compress"
10 / Dropbox
   \ "dropbox"
11 / Encrypt/Decrypt a remote
   \ "crypt"
12 / Enterprise File Fabric
   \ "filefabric"
13 / FTP Connection
   \ "ftp"
14 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
15 / Google Drive
   \ "drive"
16 / Google Photos
   \ "google photos"
17 / Hadoop distributed file system
   \ "hdfs"
18 / Hubic
   \ "hubic"
19 / In memory object storage system.
   \ "memory"
20 / Jottacloud
   \ "jottacloud"
21 / Koofr
   \ "koofr"
22 / Local Disk
   \ "local"
23 / Mail.ru Cloud
   \ "mailru"
24 / Mega
   \ "mega"
25 / Microsoft Azure Blob Storage
   \ "azureblob"
26 / Microsoft OneDrive
   \ "onedrive"
27 / OpenDrive
   \ "opendrive"
28 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
29 / Pcloud
   \ "pcloud"
30 / Put.io
   \ "putio"
31 / QingCloud Object Storage
   \ "qingstor"
32 / SSH/SFTP Connection
   \ "sftp"
33 / Sugarsync
   \ "sugarsync"
34 / Tardigrade Decentralized Cloud Storage
   \ "tardigrade"
35 / Transparently chunk/split large files
   \ "chunker"
36 / Union merges the contents of several upstream fs
   \ "union"
37 / Webdav
   \ "webdav"
38 / Yandex Disk
   \ "yandex"
39 / Zoho
   \ "zoho"
40 / http Connection
   \ "http"
41 / premiumize.me
   \ "premiumizeme"
42 / seafile
   \ "seafile"
Storage> 15

Enter 15 to choose Google Drive. Note: this is different than “Google Cloud Storage”

Enter “return” (default) for the next two questions

** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id> 
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret> 
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ "drive.metadata.readonly"
scope> 1

Enter 1 to allow full read/write access to the account.

Enter “return” (default) for the next two questions

ID of the root folder
Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.

Enter a string value. Press Enter for the default ("").
root_folder_id> 
Service Account Credentials JSON file path 
Leave blank normally.
Needed only if you want use SA instead of interactive login.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").
service_account_file> 
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n

Enter n for no advanced config.

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
y/n> N

Enter N for “No auto config”

A long URL will be provided, enter this in a browser on the workstation. It will ask you to authenticate your Google account and provide a verification code.

Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=iXda7MFJhdkbyGNTZoR85w
Log in and authorize rclone for access

Copy the verification code from the web browser and past it into this dialog

Enter verification code> 4C1AX4XfWj8OnqbEqROvTsvPfxCZvdPQbj10TkIwliVrHQ8nAK1O%IjXFfvXzA
Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No (default)
y/n> n

Choose n for single user.

--------------------
[mygdrive]
type = drive
scope = drive
token = {"access_token":"ya29.A0ARrdaM8fYFXIMvBa6ZCfnqmstFtmy-xWPZ2DQ4xLZBKt7odLFtV9BuqcmHMmZylXjqTaddaOWOcgPaghVsowqQfnNxoLBkcwGE4SGFcWvlLCCiIISArJFf1FKjKm4ed2xgQBh9O-V9qvAbQ7-9Hv_nGs3bdO","token_type":"Bearer","refresh_token":"1//0dprGI9j2opAaCgYIARAAGA0SNwF-L9Irj8Y4Z9GckUWTbx8M7TFpYfZsh6RgPrh_wetO7N206IgvBq8EC5w5j8aAAC-O4ESTT3A","expiry":"2022-02-14T14:37:48.369874335-05:00"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Enter y and a list of your current remotes will be provide

Current remotes:

Name                 Type
====                 ====
mygdrive             drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Enter q to quit.

If the configuration was successful rclone should be able to “see” your remote drive.

List top level directories in my Google Drive

rclone lsd mygdrive:
 -1 2017-01-16 12:33:25        -1 Data Science book
 -1 2021-09-07 13:39:13        -1 Logicmoo_Devel
 -1 2022-02-14 13:55:03        -1 data

List files in the data directory

rclone ls mygdrive:data
   461474 bank.csv
 

Long list the files in the data directory

rclone lsl mygdrive:data
   461474 2019-09-05 11:52:36.000000000 bank.csv

Copy the file (bank.csv) from Google Drive data directly to the local directory on the cluster

rclone copy mygdrive:data/bank.csv .

Create a copy (bank2.csv) and copy it to the Google Drive, list files in data directory

cp bank.csv bank2.csv 
rclone copy bank2.csv mygdrive:data
rclone ls mygdrive:data
   461474 bank2.csv
   461474 bank.csv

There are other commands and features of rclone available. The examples shown here are for basic file transfer. See the rclone page for more information.