Helpful File Copy Script

Voyager DragScript
1

Hello all,

Wanted to share a very simple batch script I wrote to help move your data from your collection machine to another destination.

You can download the script from github below. There are two versions, one for the base users and one for Array 2 node. If there are 3 node and 4 node users I can add more. Just ask.

What this does is very basic and simple. It sets Windows Environment Variables for My Documents\Voyager, and uses Robocopy to move whatever is in the Sequence folder, to a destination. I picked Robocopy because it will not blindly copy files to the destination that already exist (it will skip them) and it is a very robust file copy solution built into Windows already.

The reason I am setting the environment variables is that this is where new users files will default to (and where I continue to have my data stored) so its an easy plug and play for new folks.

You can add this to DragScript very easily during your END block. So when you wake up in the morning you will have all of your data waiting for you on your processing machine, or wherever you sent it to! :slight_smile: It will log its output to whatever directory you place the script into, and call it from. For example, if you put the script on your desktop, the log will be on your desktop. The log is named VOYROBOCOPY.LOG.

Here is a minor sample of the log output. The log is useful to look at to ensure it has completed, since the command prompt window will not actually be visible when it is launched from DragScript.

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows                              
-------------------------------------------------------------------------------

  Started : Sunday, May 24, 2020 3:06:53 PM
   Source : C:\Users\bill\Documents\Voyager\Sequence\
     Dest : \\ASTROBEAST\INBOX\

    Files : *.*
	    
  Options : *.* /S /E /DCOPY:DA /COPY:DAT /NP /R:1000000 /W:30 

------------------------------------------------------------------------------

	                   6	C:\Users\bill\Documents\Voyager\Sequence\
	*EXTRA Dir        -1	\\ASTROBEAST\INBOX\EMPTY\
	*EXTRA Dir        -1	\\ASTROBEAST\INBOX\FLI Library\
	*EXTRA Dir        -1	\\ASTROBEAST\INBOX\Sequence\
	  *EXTRA File 		  61.8 m	60 Secs No Cal NEG25 - FLI16200.fit
	    New File  		 116.6 m	Target_AutoFlat_FLAT_L_BIN1_-5C_001_20200511_192027_106_GA_0_OF_50_W.FIT
	    New File  		 116.6 m	Target_AutoFlat_FLAT_L_BIN1_-5C_002_20200511_192035_410_GA_0_OF_50_W.FIT
	    New File  		 116.6 m	Target_AutoFlat_FLAT_L_BIN1_-5C_003_20200511_192043_668_GA_0_OF_50_W.FIT
	    New File  		 116.6 m	Target_AutoFlat_FLAT_L_BIN1_-5C_004_20200511_192051_929_GA_0_OF_50_W.FIT
	    New File  		 116.6 m	Target_AutoFlat_FLAT_L_BIN1_-5C_005_20200511_192116_532_GA_0_OF_50_W.FIT
	                   0	C:\Users\bill\Documents\Voyager\Sequence\4628-237\
	                  15	C:\Users\bill\Documents\Voyager\Sequence\BIAS\
	                 102	C:\Users\bill\Documents\Voyager\Sequence\Calib\
	                 130	C:\Users\bill\Documents\Voyager\Sequence\M101\
	                  11	C:\Users\bill\Documents\Voyager\Sequence\Polaris\

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         6         0         6         0         0         3
   Files :       264         5       259         0         0         1
   Bytes :  26.635 g  583.41 m  26.065 g         0         0   61.81 m
   Times :   0:02:00   0:01:59                       0:00:00   0:00:00


   Speed :             5112402 Bytes/sec.
   Speed :             292.534 MegaBytes/min.
   Ended : Sunday, May 24, 2020 3:08:53 PM

I hope this is helpful to people and feel free to chime in, suggest some updates, ask questions, bake cookies, etc.

Clear Skies,
Bill

6 Likes
2

I should probably add that Robocopy will maintain the integrity of the destination. So, on the destination machine, if you delete a file that is still on the source, the next time Robocopy runs, it will copy it back over for you.

You can see a list of things Robocopy can do by typing robocopy /? at a command prompt. You can then update the line in the script above that calls robocopy and add additional switches or changes to it as you see fit.

3

Nice, this will be quite helpful.

1 Like
4

Thanks Mike. If you run into any issues let me know. :slight_smile:

5

I got one question about how to use this, and you would call this from the External Script block in Voyager DragScript like this:

RoboCopy

You can place this where you would like in your script. I personally have it set to run after the mount is parked and the camera is warming at the end of the session.

1 Like
6

Small change made to set the Env Variables prior to setting the script variables, as this would cause an error on the first run (but not subsequent runs).

Also added a Github download link and removed the script code from the original post.

7

Bill,

Thanks for the script! This will make life much easier. Is it possible to have Robocopy download files from two different directories from one batch file? I’m running Voyager in a Array so there’s two Voyager directories that images get stored in, Voyager\Sequence and Voyager#2\Sequence. Any way to get this to work or would I need two different batch scripts, one for each folder?

Mark

1 Like
8

I’m working with Array now myself but have not started any sequences yet. What is the default configuration for Array on a single PC for writing FITS to disk? I’ll create another script version for Array. If it just appends #2, #3, #4 this will be easy to do but should be done correctly, rather than just copying the script and hard coding the new location.

9

I have uploaded a new script to Github.

Look for ROBOCOPYFITS_ARRAY2.BAT. Please test this and let me know if it works. Be sure to update the destination location of course, as its currently coded for my machine. :wink:

Do note that now two log files will be created (one for each node) and that the directory copies will intentionally not occur in parallel. The reason for this, is to ensure that the Robocopy process is not clogging up the network with more than one instance of it open at a time.

Also, on the destination the FITS from both nodes will end up going into the same directory, since the project sequence directories are named the same thing (i.e. \Heart Nebula).

Please feel free to provide some feedback and I can adjust as needed.

10

I’ll try this out tonight and let you know how it works.

1 Like
11

Thanks.

I am going to work on setting up some tests with my copy of Array as well.

12

Array save data in node sequence directory. If you using one pc with 2 node you are on the same PC.
If you using 2 pc or more (1 for node) you must copy from network, better to have a sync backup service

1 Like
13

Thanks Leo, I did manage to set up a test of the Array and I can see the directory structure now. I totally agree that a backup service would be most ideal for this, but a quick Robocopy of the data will save me a lot of time, and I will likely be more inclined to process my data faster if it is already sitting on my PixInsight processing machine. :joy: :rofl:

14

The robocopy script worked perfectly for my array setup. I did change some parameters as I wanted the files moved and not just copied so I used /MOV /S /NP as my arguments. Thanks again Bill for writing this.

2 Likes
15

Awesome.

I figured people would want to change the arguments to meet their needs. It was just safer to do a copy and not have any data removed by default.

Glad to hear all is well. I ran some calibration data last night and woke up with all of my data nicely waiting for me on my main machine.

16

I finally set this up this afternoon so first test tonight. It took me a while to set up the permissions and I’m not sure if my approach is correct - what do you recommend for setting up the shared folder permissions on the destination PC?

17

It would need write access.

18

Thanks Bill. There seems to be quite a few options to achieve this level of access - I think I’ve probably gone beyond what was really needed. The file transfer worked flawlessly last night and it was great to wake this morning and find all the files waiting on the processing machine.

Peter

1 Like