Request: DragScript watchdog

As I mentioned in a thread on CN, I image 5-10 objects per evening … as each object comes into the ideal “cone” for imaging (<15deg from zenith) the scope is slewed to it, I take images and then I move to the next object that enters the “cone”. This means two things for me: first, that the time spent imaging an object is relatively short, usually less than 90min, and second that if I spend too long on an individual object then I will miss another object being in the “cone”.

I currently use DragScript to chain together slew/ focus/ guide/ expose operations into blocks, with each block being similar to the sample “Multi Sequence Robot” DragScript with a difference being the sequence start time is set to an absolute time rather than an interval. There’s a problem with that that I face though, in that there are multiple actions outside of the Sequence that will still affect my being able to follow my “hard deadline” of how long to work on one object before I must abandon it and move on to the next object.

My current very crude method is to keep track of how long I have for a particular object, say 7000sec and create a block for that object. At the start of that block an external program is called to which I write the start time (0:10:00) and the time for the block to execute (7000 sec) as well as an array of “milestone” names and expected seconds … “Slew:10, Focus:60, Tracking:60, Expose:6000, Maximum:7000”. At the end of each “milestone” event like a slew I call the external program with the milestone name, and the external program returns “go” or “no-go” depending on how much time is left to complete the rest of the tasks in the maximum time.

So, following this example, at the end of the “slew” action after 30sec of runtime after it would call the program and the program would return “go” as the time remaining (7000 - 30) is more than the total still remaining (60 + 60 + 6000). After the “focus” milestone completed after 200sec of runtime the program would return “go” as time remaining is (7000 - 200) and the task remaining are (60 + 6000) seconds, less than the maximum time remaining.

However, let’s assume that it takes a really long time to focus, 1200 seconds. Now when I make the call to the program it compares the time remaining of (7000 - 30 - 1200) against the task remaining time of (60 + 6000) and finds there isn’t enough time to complete the milestones remaining … and it returns “no-go” to the DragScript. The block then terminates and I wait for the next object sequence to start, ensuring that a delay in a part of one object doesn’t goof up the other objects for that evening

As a request, I would like to have the ability to assign a “watchdog” time to a block, which would prevent the action(s) in that block from occurring if there was no expectation that the action(s) could complete within that timeframe. You already keep statistics on individual actions, so you “know” how long it will take to complete various things as the DragScript executes … if you treat each action as a “milestone” and add all of the incomplete action times together you can do roughly what I’m doing, estimating if there is enough time to complete an action(s) within the time remaining. The watchdog value could be listed in the Block configuration dialog and be in seconds …

As an additional nicety and following from my logging request one could specify which value to use for the time check: low, high or average … that way a user could be aggressive (maybe it completes in time), conservative (never let it go over time) or in the middle …

As always, great stuff and thanks!

I need to read well when i have more time, watchdog is for internal action.
Action can have costumize watchdog time but only internally in Voyager…give control to external DragSCript is dangerous because abort is a complex operations. What you ask is really hard to obtain.

Voyager in the next months will be released in and Advanced version where you put only target filters amount of exposure per filter, min alt and slot time. Nothing else. A list of target with this data and AI in Voyager realize what you ask choosing the best conditions. I think this is the way you can follow.

Is this a product similar to ACP scheduler ?

Thanks

Max

I don’t know ACP scheduler

Hey Leonardo! Agreed, abort is dangerous and I wouldn’t add that … once an action is underway then that’s it, it runs until completion, period.

That is why in my crude program today I keep track of how long the various actions in the block are expected to take and compare the sum of those numbers to how much time I have left … if there are four action milestones left which sum to 400 sec and there is only 300 sec left to complete them then the watchdog signals “no-go” and I don’t run those actions in the first place …

Hi,

I run two systems - ACP Scheduler on one, Voyager on the other. So I have a similar approach - image when an object is +/- an hour or two of the meridian.

I achieve this with Voyager by defining a sequence for each object. I create a DragScript with as many blocks as I have objects to image. In each block, I use a Sequence DragScript object and specify a Start and End time based on when the object is “in the cone” as you put it for that night. In this example, I check the “Finish Running Exposure in case of elapsed time for Sequence Execution” which means " go ahead and finish an exposure which extends past the “end time” you specified. I don’t mind because my exposures are no longer than five minutes. If you don’t check this box, Voyager will not start an exposure that would extend past the end time you specified.

You could also specify the altitude at which you want to exit from the sequence.

Would this achieve your goal without calling an external program? I am not trying to compute how much time focusing, etc. will take. I just let them happen and specify the start and end time based on when the target is within the “cone.”

Cheers,
Rowland

Hey Rowland! Thanks for helping with some ideas on this!

You have some good ideas and look like you’re doing close to what I am, but it’s a little more complex than that … to illustrate I’ve popped in a screen shot of my scheduling program for a few nights ago at my Bay Area “wet site” using a C11 there …

There’s a lot going on here, but the main takeaway is that it’s a full dance card from 21:02 until 04:47 with the telescope flitting about like a hummingbird getting pix … so each of the “groups” from “slew” to “slew” will become a Voyager sequence with all of the trimmings … slew, focus, guide star lock, then the exposure. The time frame for these are pretty exacting to be able to hit all of the objects that I want (so, “wait until altitude” is way too coarse) and there’s not a lot of wiggle room from “group” to “group” (so, “go ahead and finish that extends past end time”) might as well read “ignore the next few groups and see if you can catch up later”.

Like you I use the sequence “start” dialog for immediate start at absolute time; so, the first SNR 30 Orion pix sequence will start at exactly 21:02:06 … the next at exactly 21:22:04, and so on

Notice that I also have about 4min in each “group” for “slew and change filter” … that’s my filler for the actual time it takes for Voyager to execute a slew, do the focus, get a guide star and any other housekeeping before it HAS to start taking pix. Note also that although the first “group” has 15min total for exposures, it’s allocated 16min … that’s for downloading and other housekeeping …

There are two big problems that I face with this … first, there are a number of steps in the middle that ST4 doesn’t let me easily factor in like the actual slew time (which isn’t that bad as most objects are in the “cone” and so not too far away), the focus time (which can vary depending on the type of focus used and if there is a failover or retry of the focus), the time to acquire a guide star and focus steps between exposures (if needed). To be able to properly “fake out” the ST4 delay I need to know what those times are in the first place … best case, worst case, average case … that’s the earlier request for an after action report for a sequence, to let me gather that information

The second problem is that for the first “group” if I can’t complete my slew/ focus/ guide star dance before the 21:06:04 mark then I won’t be able to complete my exposures … something will have to give, either I take fewer exposures or I abort the sequence entirely and wait until the “group” at 21:22:04.

That’s where the milestones and the watchdog program come in … each “action” that takes time in the sequence (slew, focus, guidestar acquire, exposure 1, exposure 2, etc.) gets a name and an expected time to complete and I have a total time to complete … so for the first “group” the watchdog program is started with a string “1198;Slew:30;Focus:60:GuideStar:30;Exp1:330;Exp2:330;Exp3:330” which tells it the DrawScript “block” has to be done in 1198sec and a system timestamp is taken. When the Slew is done the watchdog gets called with the string “Slew” and the program compares the current runtime into the block (say +20sec) against the remaining action times (60 + 30 + 330 + 330 + 330) against the time remaining (1198 - 20) … if there is enough time to complete the action then the program returns “go”, if not then “no-go”. The DrawScript then evaluates that return to determine if it should continue or abort and wait for the next sequence to start at 21:22:04. Note that I’m checking the time BEFORE I begin the action, which means I don’t have to attempt to abort an action (no bueno) and I can decide if I want to “risk it” (this is also where the worst/ best/ average times come in handy, I can decide how aggressive I want to be with my times based on historical data)

In the case of not enough time, then by checking before each action I can ensure that at least SOMETHING is done … so, suppose that focusing took a really long time and and when the last exposure time came I only had 210 sec remaining for the sequence but the Exp3 action remaining needed 330 sec to complete … the watchdog would return “no-go” and I would terminate the sequence and wait for the one at 21:22:04 but at least Exp1 and Exp2 would have been taken.

The watchdog program also lets me get additional information for a “go/no-go” decision beyond a simple time check … for instance, currently Voyager doesn’t have an easy way for me to get SQM data or seeing data, but both of those are available from the watchdog program via ASCOM calls to my SQM meter for those values as well as calls to PhD through the OpenPhD interface to get the SNR values for the guide star, from which I can compute the seeing … that way if the object is getting obscured by a cloud or the skyfog has increased for some reason I can abort the sequence without wasting time. It also lets me synchronize the application of my dew heater to when I’m not going to be in the middle of of an exposure to power it up, monitor the strain gauges on my windbreak to see if it’s going to be too windy to image successfully and a few other things but those are beyond the scope of the watchdog functionality for a block …

Thanks again for your advice, and I might ping you with some questions about DragScript!

Really complex workflow Craig … for sure i can do something to help you.
I can do small step in your direction in Voyager log or server. Just ask.

All the best
Leo

I appreciate that, Leo! I know you’re a busy guy and this isn’t your day job, so any little nugget you put into Voyager is a goldmine for us users! The workflow is complex and when you realize that this is happening for a total of 11 OTA’s each evening, well, it’s good that I do this kind of stuff for my day job too! :slight_smile:

The main reasons for my walls of text is to both possibly trigger an “aha!” from another user facing a similar challenge as I do with ST4 or to hopefully help you understand my somewhat off the wall feature requests (like the temperature reading so I can adjust my motor slew speed when it gets too cold for the grease, or the SQM variable so I can determine if it’s just too bright to image those galaxy dust lanes) … but as always, I really appreciate all that Voyager lets me do and any little bits you add to it …

Hi Craig,

And my friends tell me I’m trying to do too much by going after seven objects in a night! Are you doing survey work or just trying to optimize SNR and frames per night?

I am currently not closing the loop between ST4 and Voyager - i.e., Voyager is getting smarter with each run about timings, but I’m not improving my estimates in ST4, so I’m just letting ST4 give me a plan and then letting the absolute start and end times for the sequence put boundaries around each shot. And by choosing the status of the “Finish Running Exposure…” checkbox, I can decide whether or not to let a sequence run past the end time or not.

Am I understanding correctly here that if you were trying to do this all inside of Voyager, you would need a “don’t even try the next step in this block if it’s going to take me past the absolute end time?” type of decision? I.e., not just whether to take an exposure or not, but whether to start anything - focus, slew, etc - if the end time would extend past a block-level end time?

That doesn’t factor in your SQM meter reading so I guess that would have to be another input to the “give up on this block and skip ahead to the next one” decision.

Cheers,
Rowland

Tell me how can i help you … if is possible i’ll try to do.

Mmm… This is getting pretty too complex. Startools is a planner, calculates moon phases,object’s altitudes, divide a plan into multiple Nights.

The sqm addition can be very powerful (start sequence if sqm… All the related locical functions).

Not easy at all

Are you doing survey work or just trying to optimize SNR and frames per night?

It’s been an interesting journey! Originally I just wanted to get some pix of some pinwheel galaxies and planetary nebulae … so I took my trusty C11 and started to take some pix. I got M33, I was hooked but it took me about two weeks to gather all the needed lights for a really nice (SNR 44 or so) shot (oy!). Going through the various lists I put together a list of a bit over 1400 objects that I could get from my darksite that I wanted to image … but a little bit of math tells me that my grandson would probably be finishing this up! In addition, even though there are some simply perfect nights there are a lot more “meh” nights … so if I waited to image only on “wow” nights it would now be my great-great grandson who would be finishing it up and I’m OK with “ok” or “meh” images! So, what I’m doing now is putting 11 scopes to work rather than just one and gathering lights of different SNR values … “wow” nights and “meh” nights and many in between and stacking them by SNR … so all of my “wow” night lights make a “super wow” pix, all of my “meh” nights makes an “ok” pix, etc. … so I am actually doing both, maximizing SNR per object and going for as many frames as possible per night …

Am I understanding correctly here that if you were trying to do this all inside of Voyager, you would need a “don’t even try the next step in this block if it’s going to take me past the absolute end time?” type of decision? I.e., not just whether to take an exposure or not, but whether to start anything - focus, slew, etc - if the end time would extend past a block-level end time?

Correct; to get as high of an SNR for a particular object as possible I need to start imaging at an exact time and finish by an exact time (the start time is based on airmass, moon phase and location, and a whole lot of other variables that ST4 calculates) … so if I can’t finish a block by that finish time it’s going to throw off my whole schedule … like missing a train in the morning, if you miss that then you’ve goofed up an entire schedule. Aborting an action in the middle is not possible/ safe (abort a slew or an exposure?) so you need to catch it before then … if I need 500sec to do an exposure but there are only 400sec left then I had better not even try it. That’s what my watchdog program is doing, calculating that number and returning a go/no-go response which Voyager can do a branch decision on

That doesn’t factor in your SQM meter reading so I guess that would have to be another input to the “give up on this block and skip ahead to the next one” decision.

Correct as well, and that’s because the skyfog makes a big difference in if I’m really going to be able to get a planned exposure SNR or not. Daily exposures (not multi-night, ST4 doesn’t do those properly and Greg’s not exactly the easiest to point this out to) are then stacked, but if the stacking program finds that 5 of the 10 taken are below a certain threshold it will reject those, so I’m really just stacking 5 and throwing away 5. Those are wasted shots and it’s better to just move on than “take 'em/ toss 'em”. SQM will vary by up to .4 if there are high ice clouds that drift on by, so getting an SQM number before each exposure is worth the effort …

Totally agreed, it’s very complex and complicated … but then the same is true of AP! When I first started out I had a CPC11 on a tripod driving a Nikon D5100 running BackyardNikon on a laptop … now I’ve got C14’s with HyperStar using CCD and CMOS camera’s through an OAG running PhD, Voyager, MaximDL, CCDInspector on a laptop, plus a RaspPi3B embedded computer running a system of dew heaters reading from windscreen strain gauges, temperature and humidity probes and a separate ASCOM watchdog program getting SQM readings, mount absolute position, focuser temperature readings and PhD HFD log info … and that’s just to be a single light! So yes, quite complicated this hobby is! :slight_smile:

I really appreciate that Leo! You’re a very thorough, conscientious fellow who is trying to help people to better succeed at a very demanding hobby and while holding down a demanding day job too so again, I really appreciate the offer! I also want to make sure any suggestions I make are reasonable in terms of effort, will fit into the overall philosophy and direction you’ve set and are going to be of value to other people … and possibly when you look at my wall of text you might have an “aha” idea that would be easy to put in, helpful to others, fit into the design nicely!

Currently my biggest “pain points” (which I am overcoming with custom programming, BTW, so no pressure at all!) are:

  • Communicating between the watchdog program and Voyager … right now it’s an external program call with a command line, then evaluate the response. That’s fine, but that means I have to retain state for the watchdog program each time it starts/ stops. One thing that has occurred to me while I’m working on implementing the MoveAxis helper app is that as an ASCOM safety driver app my helper app can respond to base ASCOM methods like DirectCommand as well as talk to other ASCOM apps (like the Celestron telescope driver) … that means that a DirectCommand sent from Voyager doesn’t have to simply go to the Celestron driver, it could go to my helper app instead. So, one potential “help” would be to allow an ASCOM DirectCommand call to a given ASCOM target and put the return into a variable. That would mean my watchdog app would now be something like an ASCOM safety driver and would get/ respond to DirectCommand calls for the information flow and not only would it be much cleaner than shelling out to an external program each time but I could keep the watchdog running and not have to deal with retaining/ reloading state

  • The ability to have an after-action report of a DragScript sequence, as I’ve already mentioned, to let me get timing information for all of the various events that happen. I’ve already written a wall of text about that … currently I’ve written a program that when a DragScript ends copies off the current log into a temporary one, chops off the bits before the DragScript started and after it ended and parses it into an XML file just like in my wall of text and saves that off in a report directory. I will later run another program against a series of those logs to get the average/ fastest/ slowest timings for each “action” and use that for programming the ST4 delays …

  • The ability to get HFD, or FWHD, or something from the focus operation or even better from the guiding program, the current focuser temperature and the SQM meter readings all as a DragScript variable so that I can make a go/no-go branch decision … if the seeing has now gone to 2" and my object has detail I need 1" seeing to get then again, I’m wasting my time taking pix, same as if the SQM has gone from 21.5 to 21 … better to just not even try. The focuser temperature request I mentioned in an earlier wall of text. Currently getting those numbers is a serious PITA … my watchdog program needs to make a call into the PhD instance to get the current HFD number, it makes an ASCOM call into my Unihedron SQM meter to get the current SQM and then another ASCOM call into the focuser to get the temperature property …

Again, these are all things I am doing today in various ways, so I am making forward progress and am LOVING Voyager just as it is today! I don’t just owe you a bottle of Napa wine (like Rowland), I owe you a full case!

:sweat_smile: i’m working for you now… in next daily build you’ll found new variable of type Decimal in DragScript. Observing conditions controls, SQM controls. Possibility to put observing conditions, SQM, focusposition, HFD of last focus in a decimal variable in DragScript. A possibility to put result of external script in a decimal variable in dragscript.

If you decide to communicate with Voyager application server i can create a method to abort action for watchdog and add new action result of type WATCHDOG (OK, ERROR, TIMEOUT, WATCHDOG)

And all the data in FIT Header …

a Napa wine is welcome :relaxed:

Mmm… This is getting pretty too complex. Startools is a planner, calculates moon phases,object’s altitudes, divide a plan into multiple Nights.

Correct, and that’s where it excels … I used to have a “spreadsheet of pain” which recorded the optimum days for an object (when it would be highest), the exposure count and number I had guessed (6 at 10min, or 8 at 5min, etc.) and each day I would look at the calendar, then check the “spreadsheet of pain” to see what object would be visible. I would then look at the moon phase to see if it was even possible to do any imaging, then use the “spreadsheet of pain” numbers to build my sequences for the evening. It was grueling work and more often than not I would miss imaging a small, faint object at the perfect time because I was busy imaging a brighter object at that time. Lot’s of opportunities lost, lot’s of “ok” images that could have been “wow” shots.

Then ST4 came along, which promised to not only calculate exposure times like my “spreadsheet of pain” but also to take into account moon phase, airmass, humidity, seeing, SQM values, all kinds of stuff … it was like a pilot used to using a map, compass, altimeter and stopwatch to navigate being introduced to a cockpit with weather radar, GPS and satellite navigation! And as an extra bonus, ST4 would arrange my objects by imaging quality, making sure that the faint fuzzy things were imaged at the exact right time for them … that’s again like someone plotting a route with a map and being introduced to Waze! I spent about a month comparing the results from the ST4 numbers, found some questionable results (don’t use ST4 for multi-night calculations) but I’ve been using ST4 for all of my numbers now for about 6 months and have turned a 1hr grind each night into a 10minute routine (and it would be faster, but the ST4 developer just refuses to output the plan results as a text file so I have to use AutoIT in character recognition mode to scrape the results into a text file) …

Not easy at all

My day job is similar to Leonardo’s, I design and build digital and analog embedded hardware and systems for industrial control systems and this AP stuff is easily an order of magnitude more complex than things I deal with during the day … so yes, it’s “not easy at all” … but that’s a big part of the fun, being challenged with a seemingly insurmountable challenge and improvising, adapting and overcoming!

1 Like

Hey Leo, surprised you’re still awake! :smile: Many thanks for all that you’re doing, let me know if you like reds or whites (and what kind) and I’ll get a case put together!

I’d need to learn more about the api for the Voyager application server, because I don’t want to do anything that would potentially block Voyager … that’s why the current DragScript works well, it doesn’t block Voyager (blocks the DragScript), it’s a data “push” to the target (through the command line), then a “pull” (through the return) but it’s a synchronous operation which makes it both safer and more controllable. I’m having some good success using DirectCommand from my MoveAxis helper into the Celestron ASCOM target, maybe just putting in a DirectCommand with a target would be fine … it’s synchronous, push/ pull and just needs an ASCOM target (like turning my watchdog into an ASCOM safety driver) … but again, I’d need to know more about the API to your server …

Again, much much thanks! Voyager has already taken my little “sky-roomba” thing to the next level …

Hi Craig –

I read all your replies, thanks for taking the time to supply all those details.

As you know, I’m using ST4 to decide what to image and when to do it, same as you.
So while the following is a bit simplified, I’m still not quite sure why this wouldn’t be an 80/20 if not 95/5 approach to achieve your goals:

  1. Build the sequence with the absolute start and end times specified based on ST4’s plan – start time = “start slewing to target” and end time = “start slewing to next target.”
  2. Set the DragScript sequence option so Voyager will not start an exposure that would extend past the end time. Unless your exposures are super short, this step should keep many if not most of your sequences from going past the desired end time.
  3. If the sequence runs a bit past the end time because it was doing an autofocus or slew, the next sequence starts a minute or so late. But only that sequence is affected, assuming there is enough time to capture at least one image.

And overarching all of this, even if ST4 suggests shooting a target for a couple of 3 minute frames, I just don’t schedule anything that is less than 30 minutes of exposures. It’s just not worth the overhead in my opinion. I try to schedule targets that will run at least an hour. ST4 also often suggests taking “1 frame of H, 2 of R, 3 of L” and I just say “bah humbug” and take 5 of R that night to avoid the filter change and refocus time for so few frames. When I feed back to ST4 what I actually got, it still knows what I need to reach the SNR goals for each filter.

But then again, I don’t have a farm of scopes, just two :slight_smile: !

I’m 50% trying to be helpful and 50% questioning my own methodology - since we’re trying to do roughly the same thing, but I’m taking a much simpler approach. At the end of the day, we want to get as much data as possible from as good conditions as possible. Perhaps because I’m using CMOS and shorter exposures, tossing a few subs doesn’t bother me horribly.

Cheers!
Rowland

Hey Rowland! First, thanks for digging through my walls of text, most people’s eyes glaze over! :slight_smile:

When I originally started using Voyager I basically did what you’re suggesting, with the difference that I had a sequence with an absolute end rather than an absolute begin … and I’ll also add the disclaimer that I likely didn’t use DragScript/ sequences properly, so there’s that … take the ST4 total time, create a sequence that did the suggested time/ count thing, set my absolute end time, done.

And you’re right, I’m definitely to the right of the 80/20 curve, so there’s that …

That said, there are some objects that only show up for a few weeks a year in an ideal imaging A position … and good nights are few and far between! Usually I have B position targets, which means I need to gather a lot of signal to get something worth stacking … like, 40-60hr of signal … so that lost time really hurts when you realize it’s probably going to be another year before you can get back to gather enough signal to finish the 50 light stack (another justification for putting lot’s of scopes on target, I’d like to finish this stuff now rather than leave it to the great grandkids) …

What I discovered when I did what you suggest (absolute end time, ST4 programmed with estimates of actual time) was that the Voyager sequence time didn’t include block execution time, only sequence execution time … times for independent block actions like focusing and guide star acquisition (even though they were noted in the sequence, like for focus failover) weren’t being covered by the sequence, which means they happened outside of the absolute end (or start) time. Unfortunately those two actions are both necessary and variable in time … so if it took longer to focus than expected due to a failover or retry, I ended up not doing a sequence I had just gotten focused for due to hitting the time limit … maddening …

The other problem was the granularity; sequences are designed for groups of pix (take 10 at 5 min) and if the sequence won’t trigger due to not enough time for 10 at 5 min and there’s enough time for 9 at 5 min it exits and, well, that’s just even more maddening … and I’m not going to make a singleton sequence for each light and chain them one after each other, did that enough with SGP!

Again, part of my problem could be that I don’t know the program well enough (maybe there’s a simple answer that I missed, easily possible!) … but fundamentally I need to do this:

Given a time allotted for an object … slew to that object, focus on that object, get a guidestar, start to track and take as many pix as possible before I need to move on to the next. That sounds simple enough, but the only way I found to do it reliably with the variable slew/ guidestar/ focusing times was as I’ve described. If I only rely on the sequence timings then I can’t factor in the slew/ focus/ guidestar times, if I take “worse case” times for those things and simply delay my sequences by those times I’m going to be having dark time go to waste …

Oh, and check your pixel scales to make sure you’re getting the right exposures with CMOS … I have an ASI 183MM CMOS for my L channel, and since I’m hideously oversampled on that with my C14 I need to have really LONG exposures to get good signal, on the order of 15-20min per light …

Thanks again for your suggestions, hopefully between you, me and Leonard we can get me out of the WatchDog program business! :slight_smile:

This is not true Craig, Voyager report time for each single actions (each blocks are done from actions). Also report complete time for finish sequence (focus,guide,calibration,shot, download time based on bin, dithering and so and so) … realigned at each action inside sequence. See attached image.

post-73074-0-71750600-1547343548

In anycase sequence is not deterministic and the time could change.

All the best
Leo