Data Science Lab

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
=====Configuring and Using rclone ======

''rclone'' is tool for using remote cloud based storage systems. You must have a valid account on one of the remote cloud systems to use ''rclone'' See the [[https://rclone.org/| rclone page]] for more information. 

The following example demonstrates how to use ''rclone'' with **Google Drive**. 

The following steps must also be performed using a terminal window on one of the DSL Lab workstations. Once configured, however, the ''rclone'' commands can be run within Notebooks like Zeppelin.

Open text window under your account and enter ''rclone'' config. The following will guide the Google Drive configuration.

<code>
$ rclone config
2022/02/14 13:32:45 NOTICE: Config file "/home/USER/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> mygdrive
</code>

Enter **''n''** for new and the **''name''** of your "remote" drive (in this case ''mygdrive''. 
Also ''USER'' will be your account name.

<code>
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
   \ "s3"
 5 / Backblaze B2
   \ "b2"
 6 / Box
   \ "box"
 7 / Cache a remote
   \ "cache"
 8 / Citrix Sharefile
   \ "sharefile"
 9 / Compress a remote
   \ "compress"
10 / Dropbox
   \ "dropbox"
11 / Encrypt/Decrypt a remote
   \ "crypt"
12 / Enterprise File Fabric
   \ "filefabric"
13 / FTP Connection
   \ "ftp"
14 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
15 / Google Drive
   \ "drive"
16 / Google Photos
   \ "google photos"
17 / Hadoop distributed file system
   \ "hdfs"
18 / Hubic
   \ "hubic"
19 / In memory object storage system.
   \ "memory"
20 / Jottacloud
   \ "jottacloud"
21 / Koofr
   \ "koofr"
22 / Local Disk
   \ "local"
23 / Mail.ru Cloud
   \ "mailru"
24 / Mega
   \ "mega"
25 / Microsoft Azure Blob Storage
   \ "azureblob"
26 / Microsoft OneDrive
   \ "onedrive"
27 / OpenDrive
   \ "opendrive"
28 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
29 / Pcloud
   \ "pcloud"
30 / Put.io
   \ "putio"
31 / QingCloud Object Storage
   \ "qingstor"
32 / SSH/SFTP Connection
   \ "sftp"
33 / Sugarsync
   \ "sugarsync"
34 / Tardigrade Decentralized Cloud Storage
   \ "tardigrade"
35 / Transparently chunk/split large files
   \ "chunker"
36 / Union merges the contents of several upstream fs
   \ "union"
37 / Webdav
   \ "webdav"
38 / Yandex Disk
   \ "yandex"
39 / Zoho
   \ "zoho"
40 / http Connection
   \ "http"
41 / premiumize.me
   \ "premiumizeme"
42 / seafile
   \ "seafile"
Storage> 15
</code>

Enter ''**15**'' to choose Google Drive. Note: this is different than "Google Cloud Storage"


Enter "return" (default) for the next two questions
<code>
** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id> 
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret> 
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ "drive.metadata.readonly"
scope> 1
</code>
Enter ''**1**'' to allow full read/write access to the account.

Enter "return" (default) for the next two questions
<code>
ID of the root folder
Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.

Enter a string value. Press Enter for the default ("").
root_folder_id> 
Service Account Credentials JSON file path 
Leave blank normally.
Needed only if you want use SA instead of interactive login.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").
service_account_file> 
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
</code>

Enter **''n''** for no advanced config.
<code>

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
y/n> N
</code>

Enter **''N''** for "No auto config"

A long URL will be provided, enter this in a browser on the workstation. It will ask you to authenticate your Google account and provide a verification code. 

<code>
Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=iXda7MFJhdkbyGNTZoR85w
Log in and authorize rclone for access
</code>

Copy the verification code from the web browser and past it into this dialog

<code>
Enter verification code> 4C1AX4XfWj8OnqbEqROvTsvPfxCZvdPQbj10TkIwliVrHQ8nAK1O%IjXFfvXzA
Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No (default)
y/n> n
</code>

Choose **''n''** for single user.

<code>

--------------------
[mygdrive]
type = drive
scope = drive
token = {"access_token":"ya29.A0ARrdaM8fYFXIMvBa6ZCfnqmstFtmy-xWPZ2DQ4xLZBKt7odLFtV9BuqcmHMmZylXjqTaddaOWOcgPaghVsowqQfnNxoLBkcwGE4SGFcWvlLCCiIISArJFf1FKjKm4ed2xgQBh9O-V9qvAbQ7-9Hv_nGs3bdO","token_type":"Bearer","refresh_token":"1//0dprGI9j2opAaCgYIARAAGA0SNwF-L9Irj8Y4Z9GckUWTbx8M7TFpYfZsh6RgPrh_wetO7N206IgvBq8EC5w5j8aAAC-O4ESTT3A","expiry":"2022-02-14T14:37:48.369874335-05:00"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
</code>

Enter **''y''** and a list of your current remotes will be provide
<code>
Current remotes:

Name                 Type
====                 ====
mygdrive             drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
</code>

Enter **''q''** to quit.

If the configuration was successful ''rclone'' should be able to "see" your remote drive.


===== Example Commands=====

** List top level directories in my Google Drive **

  rclone lsd mygdrive:
   -1 2017-01-16 12:33:25        -1 Data Science book
   -1 2021-09-07 13:39:13        -1 Logicmoo_Devel
   -1 2022-02-14 13:55:03        -1 data

** List files in the ''data'' directory**

  rclone ls mygdrive:data
     461474 bank.csv
   
** Long list the files in the ''data'' directory**

  rclone lsl mygdrive:data
     461474 2019-09-05 11:52:36.000000000 bank.csv
 
** Copy the file (''bank.csv'') from Google Drive data directly to the local directory on the cluster**

  rclone copy mygdrive:data/bank.csv .

** Create a copy (''bank2.csv'') and copy it to the Google Drive, list files in ''data'' directory**
  cp bank.csv bank2.csv 
  rclone copy bank2.csv mygdrive:data
  rclone ls mygdrive:data
     461474 bank2.csv
     461474 bank.csv
     
**Delete a file (''bank2.csv'') in the ''data'' directory on the Google Drive (list the files before and after)**
  rclone ls mygdrive:data
     461474 bank2.csv
     461474 bank.csv
  rclone delete  mygdrive:data/bank2.csv 
  rclone ls mygdrive:data
     461474 bank.csv

=====More Information about rclone=====

There are other command options and features of ''rclone'' available. The examples shown here are for basic file transfer. See the [[https://rclone.org/| rclone page]] for more information.