무한한 용량의 Amazon Drive

무한한 용량의 Amazon Drive에 중요한 데이터를 백업하고 싶습니다.
그래서,
정말 제대로 되어 있는지 확인하고 싶다.

Amazon Drive



htps //w w. 아마존. 코 m / c ぉ 흠 d ゔ ぇ


rclone


brew install rclone
% rclone -V
rclone v1.34-DEV
Modified time and MD5SUMs
Amazon Drive doesn’t allow modification times to be changed via the API so these won’t be accurate or used for syncing.

It does store MD5SUMs so for a more accurate sync, you can use the --checksum flag.

AmazonDrive는 파일 업데이트 시간을 저장할 수 없기 때문에 MD5에서 확인할 수 있습니다.

설정



h tp : // rc ぉ. 오 rg / 아마 존 c ぉ 흠 d ゔ ぇ /
따라 설정.

~/.rclone.conf에 설정이 저장됩니다.

~/.rclone.conf
[amz]
type = amazon cloud drive
client_id =
client_secret =
token = {.....}

amazon.co.jp와 amazon.com의 로그인이 충돌하고 거동 의심스러웠지만 몇 번이나 다시 시작하면 할 수 있었다. 수수께끼.

복사


rclone -v copy ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/

이 때는 v1.33이었다.
420GB로 1주일 걸렸다.


확인


rclone check --checksum ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee check_`ts`.txt
2016/11/15 13:24:33 Local file system at /Users/takeru/Pictures/写真 Library.photoslibrary: Building file list
2016/11/15 13:24:33 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Building file list
2016/11/15 13:25:33
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m6.4s

......

2016/11/16 02:59:33
Transferred:      0 Bytes (0 Bytes/s)
Errors:              3933
Checks:            324417
Transferred:            0
Elapsed time:  13h35m6.4s
Checking:
 * Database/Faces/Detected/%tE6BEKPQZubOu%1nMo+tw.apdetected
 * Database/Places/B9zGz6YFSHinJ9eJI+6%bw.applace
 * Database/Versions/2011/05/14/20110514-125249/hopKwBNjR+W77Li7vfxUhQ/Master.apmaster
 * Database/Versions/2013/08/22/20130822-132448/wC6E6H9WR%iXgAqV0dDmEw/Version-0.apversion
 * Database/Versions/2013/11/12/20131112-135919/sFpvdWdgTi6n9zWkjvQYtg/Master.apmaster
 * Database/Versions/2015/05/05/20150505-171157/NDtysoLkSDmMs0SZ6xgdgw/Version-0.apversion
 * Masters/2014/09/17/20140917-135845/P1120427.JPG
 * resources/modelresources/117/72/uHJ%6BPWQWmg2BHFAhu81Q/P1250301.jpg

2016/11/16 03:00:08 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': 3933 differences found
2016/11/16 03:00:08 Failed to check: 3933 differences found

13.5시간이 걸렸다.
% grep "File not in amazon drive" check_20161115_132426.txt | wc -l
    3882
% grep "Sizes differ" check_20161115_132426.txt | wc -l
      40
% grep "MD5 differ" check_20161115_132426.txt | wc -l
       7
% grep "File not in Local file system" check_20161115_132426.txt | wc -l
       3
% grep "Failed to calculate src hash" check_20161115_132426.txt | wc -l
       1

3882+40+7+3+1 = 3933

copy와 check 사이에 흡수를 하고 있기 때문에 차이가 있다.

목록 취득



check가 아니고 수동으로 확인하려면 ls와 md5sum을 사용할 수 있을 것.
% rclone ls -c amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/
     2048 iPhotoAux.db
      341 ProjectDBVersion.plist
       12 Library6.iPhoto
       16 .ipspot_update
       22 Library.iPhoto
    20097 com.apple.iPhoto.plist
       51 iPhoto.ipspot
        1 iPhotoLock.data
     6144 iPhotoMain.db
       22 Library.data
   950272 Projects.db
   149641 Thumbnails/IMG_2258_1024.jpg
    65982 Thumbnails/IMG_2260.jpg
    36507 Thumbnails/IMG_2259.jpg
   340738 Thumbnails/IMG_2256_1024.jpg
    36298 Thumbnails/IMG_2255.jpg
   147391 Thumbnails/IMG_2259_1024.jpg
    68173 Thumbnails/IMG_2262.jpg
    67724 Thumbnails/IMG_2261.jpg
   242106 Thumbnails/IMG_2263_1024.jpg
    50023 Thumbnails/IMG_2263.jpg
    72239 Thumbnails/IMG_2264.jpg
   361674 Thumbnails/IMG_2261_1024.jpg
    73621 Thumbnails/iMovie Data
   399933 Thumbnails/IMG_2264_1024.jpg
   305229 Thumbnails/IMG_2257_1024.jpg
    36485 Thumbnails/IMG_2258.jpg
...
13:43:07 tkrimac2:~% rclone md5sum amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/
10f81bf64edc6c17e8c2c4bcfa1ff2c1  iPhotoAux.db
3d4a1ff0b4efbc59cbe1e05e6b1f7dfd  ProjectDBVersion.plist
9174a025431e8cd040f310f6e9383942  Library6.iPhoto
5bcd17a07bb9b4cd0a739da0501d03b0  .ipspot_update
8c52f4fe8975dbf90921baf1e049a276  Library.data
7215ee9c7d9dc229d2921a40e899ec5f  iPhotoLock.data
8e81d2b8fd3f851cb0e2d6dfe1c7f47d  com.apple.iPhoto.plist
710683e6a2520eb2605757f2c3c3027f  iPhotoMain.db
011b040a3a0de00abd50aee2f81f4ce2  iPhoto.ipspot
008d9f97c161f149c098f767f29f5c4b  Projects.db
8c52f4fe8975dbf90921baf1e049a276  Library.iPhoto
d8d97fc3a0c60e97ac4a66e57a1778f8  Thumbnails/IMG_2257_1024.jpg
a3680df33b27355bd18967860043a5b3  Thumbnails/IMG_2260.jpg
5d59b305703d861e299e06996509442d  Thumbnails/IMG_2258.jpg
ec27b1aa9afe097d397a0538d41d5006  Thumbnails/IMG_2256.jpg
674b50262670adf9b4198d0db8cb3d17  Thumbnails/IMG_2257.jpg
2ea4bae372e10f7677cfebf252d7ce00  Thumbnails/IMG_2255_1024.jpg
f202a9f4343aeac363fa4be8d21a0fc4  Thumbnails/IMG_2262.jpg
fc2e86d04eda8bd2c6b543ed9d266c52  Thumbnails/IMG_2262_1024.jpg
b1e0f5eebe2724d7627eb009a0f6ec67  Thumbnails/IMG_2263_1024.jpg
1acf5f06a9ea072d2de894e312f0be62  Thumbnails/IMG_2263.jpg
34e7f1cc0c191827cd69a2315400e7f8  Thumbnails/IMG_2264.jpg
b8fa406478f6e21d411f9d60e7a901e7  Thumbnails/IMG_2261_1024.jpg
01af20f2141bc7e45d299ee582126170  Thumbnails/iMovie Data
ca38101864190d4be827df219815052d  Thumbnails/IMG_2264_1024.jpg
3e98c476cf43639dbea49121450d8063  Thumbnails/IMG_2258_1024.jpg
611e1d0692dfe5535be10a0990c0608c  Thumbnails/IMG_2259.jpg
fb624e6bef00061223d0b3f8ce0d4850  Thumbnails/IMG_2256_1024.jpg
...

ls로 파일 사이즈, md5sum에서 해시 값의 리스트를 취할 수 있으므로 이것으로 확인할 수 있을 것 같다.

sync



sync --dry-run


% rclone sync --checksum --dry-run ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee sync-dry_`ts`.txt


2016/11/16 10:35:42 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Not making directory as dry run is set
2016/11/16 10:36:42
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:        1m5s

.....

2016/11/16 20:05:42
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:            323414
Transferred:         3932
Elapsed time:     9h30m5s
Checking:
 * Database/Versions/2014/04/22/20140422-110751/Vu2gSEhAQ0epbvs3tBrGAg/Version-0.apversion
 * Database/Versions/2014/10/05/20141005-123438/Zm7l3UzmTYG+WXNYXXhhog/Master.apmaster
 * Database/Versions/2014/10/05/20141005-123438/Zm7l3UzmTYG+WXNYXXhhog/Version-0.apversion
 * Database/Versions/2014/10/05/20141005-123438/Zm7l3UzmTYG+WXNYXXhhog/Version-1.apversion
 * Thumbnails/2014/04/22/20140422-110751/VZUHErpbTXuf5ou4RVYppw/P1090602_1024.jpg
 * Thumbnails/2014/04/22/20140422-110751/VyYA5CICSBSdI8uyYn7KQw/P1100171_1024.jpg
 * Thumbnails/2014/04/22/20140422-110751/W%BECsCHTIa2icv4VrBgfg/P1100271.jpg
 * Thumbnails/2014/04/22/20140422-110751/W%BECsCHTIa2icv4VrBgfg/P1100271_1024.jpg

2016/11/16 20:06:13 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for checks to finish
2016/11/16 20:06:13 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for transfers to finish
2016/11/16 20:06:13 Waiting for deletions to finish
2016/11/16 20:06:13 Thumbnails/2016/11/05/20161105-013504/d60KJ75wRfiHXtIWs1fDfQ/thumb_IMG_5728.jpg: Not deleting as --dry-run
2016/11/16 20:06:13 resources/moments/needsanalysis: Not deleting as --dry-run
2016/11/16 20:06:13 Masters/2016/11/05/20161105-013504/IMG_5728.JPG: Not deleting as --dry-run
2016/11/16 20:06:13 Thumbnails/2016/11/05/20161105-013504/d60KJ75wRfiHXtIWs1fDfQ/thumb_IMG_5728_1024.jpg: Not deleting as --dry-run
2016/11/16 20:06:13
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:            324670
Transferred:         3932
Elapsed time:  9h30m36.4s
% grep "dry-run" sync-dry_20161116_103537.txt | wc -l
    3936
% grep "dry-run" sync-dry_20161116_103537.txt | grep "Not copying" | wc -l
    3932
% grep "dry-run" sync-dry_20161116_103537.txt | grep "Not deleting" | wc -l
       4

대체로 있다.

sync


% rclone sync --checksum ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee sync_`ts`.txt

2016/11/18 12:14:04
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m2.5s

.....

2016/11/19 00:03:40 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for checks to finish
2016/11/19 00:03:40 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Waiting for transfers to finish
2016/11/19 00:03:40 Waiting for deletions to finish
2016/11/19 00:03:42
Transferred:   31.606 GBytes (777.236 kBytes/s)
Errors:                 0
Checks:            324674
Transferred:         3947
Elapsed time:  11h50m39.5s

재확인


% rclone check --checksum ~/Pictures/写真\ Library.photoslibrary/ amz:/Backups/Pictires/20161105_写真\ Library.photoslibrary/ 2>&1 | tee check_`ts`.txt
2016/11/19 08:11:43 Local file system at /Users/takeru/Pictures/写真 Library.photoslibrary: Building file list
2016/11/19 08:11:43 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': Building file list
2016/11/19 08:12:43
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m6.8s

.......

2016/11/19 22:57:43
Transferred:      0 Bytes (0 Bytes/s)
Errors:                22
Checks:            328185
Transferred:            0
Elapsed time:  14h46m6.8s
Checking:
 * Database/Versions/2010/11/10/20101110-003755/4Z559LDtSXyuqbgL8FDWmg/Version-0.apversion
 * Database/Versions/2013/05/09/20130509-014056/IrQpTcJKTgCPwSwfc56+Yw/Version-1.apversion
 * Masters/2010/11/10/20101110-003720/r0300657.jpg
 * Masters/2014/04/22/20140422-110751/P1090489.JPG
 * Masters/2015/12/16/20151216-005122/IMG_3809.JPG
 * Masters/2016/05/04/20160504-092529/P1200785.JPG
 * Thumbnails/2013/12/07/20131207-142941/miwgrFYCRtOusU+R7Nuvlw/IMG_1373_1024.jpg
 * iPod Photo Cache/F43/T5241.ithmb

2016/11/19 22:58:22 amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary': 22 differences found
2016/11/19 22:58:22 Failed to check: 22 differences found
9:32:21 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "File not in amazon drive"
2016/11/19 10:34:42 Masters/2016/11/18/20161118-124000/IMG_5925.PNG: File not in amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary'
2016/11/19 10:34:42 Thumbnails/2016/11/18/20161118-124000/WFpqzcDvTN6hE3b0dvoJVQ/thumb_IMG_5925.jpg: File not in amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary'
2016/11/19 10:34:42 Thumbnails/2016/11/18/20161118-124000/WFpqzcDvTN6hE3b0dvoJVQ/thumb_IMG_5925_1024.jpg: File not in amazon drive root 'Backups/Pictires/20161105_写真 Library.photoslibrary'
9:33:37 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "Sizes differ"
2016/11/19 11:23:41 resources/recovery/RKVersion/0000000015.lij: Sizes differ
2016/11/19 11:28:34 resources/recovery/RKMaster/0000000001.lij: Sizes differ
2016/11/19 13:30:09 Apple TV Photo Cache/Apple TV Photo Database: Sizes differ
2016/11/19 14:14:52 Database/apdb/Library.apdb-wal: Sizes differ
2016/11/19 15:45:11 Database/metaSchema.db-wal: Sizes differ
2016/11/19 18:39:56 resources/recovery/RKImportGroup/0000000000.lij: Sizes differ
2016/11/19 20:16:57 resources/recovery/RKImageProxyState/0000000002.lij: Sizes differ
9:35:08 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "MD5 differ"
2016/11/19 11:30:20 resources/recovery/RKPlaceForVersion/0000000000.lij: MD5 differ
2016/11/19 14:17:08 Database/apdb/Person.db: MD5 differ
2016/11/19 14:27:13 Database/apdb/Person.db-wal: MD5 differ
2016/11/19 15:31:52 Database/apdb/Properties.apdb: MD5 differ
2016/11/19 15:43:03 Database/apdb/ImageProxies.apdb: MD5 differ
2016/11/19 15:44:51 Database/metaSchema.db: MD5 differ
2016/11/19 17:40:35 Database/RKVersion_searchIndexText.skindex: MD5 differ
2016/11/19 18:37:04 Database/apdb/Properties.apdb-wal: MD5 differ
2016/11/19 18:48:50 Database/apdb/Library.apdb: MD5 differ
2016/11/19 20:02:35 Database/apdb/ImageProxies.apdb-wal: MD5 differ
2016/11/19 20:22:31 resources/moments/historicalmarker.plist: MD5 differ
9:35:21 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "File not in Local file system"
9:35:39 tkrimac2:~/tmp/acd-test% cat check_20161119_081136.txt | grep "Failed to calculate src hash"
2016/11/19 22:14:37 Database/metaSchema.db.lock: Failed to calculate src hash: hash: failed to stat: lstat /Users/takeru/Pictures/写真 Library.photoslibrary/Database/metaSchema.db.lock: no such file or directory

OK같다.

좋은 웹페이지 즐겨찾기