Note
This technote is a work-in-progress.
1 Abstract¶
This document describes the concept of operations for running the Auxiliary Telescope (AuxTel) during survey data collection. Survey mode utilizes semi-autonomous, scheduler-driven operations and the roles, responsibilities, and expectations for this mode are defined.
2 Guidelines for Operations¶
Survey mode is intended to be the primary mode for Auxiliary Telescope (AuxTel) operations. The central tenant of operations in this mode is that the AuxTel and its data products are intended to support Simonyi Survey Telescope operations and the Legacy Survey of Space and Time, but AuxTel operations should not interfere with Simonyi Telescope Operations.
This results in the following guidelines for operations in survey mode:
The observing staff are not required to actively monitor or report on incoming AuxTel data quality, annotations can be added on a best-effort basis.
Logging will be minimal and limited to short, narrative statements needed to report on changes during the night or provide context for faults and recovery attempts.
Faults will be recovered when observing staff have available time, and the AuxTel may remain in Standby state during the night for extended periods of time. Recorded Fault Loss time may be inflated due to inactivity.
Remote support will be available for emergencies on on-call and fault resolution on a best-effort basis.
The decision to close the AuxTel early is the discretion of the shift manager, no additional approval is required. Hardware risk, system instability (e.g. repeat faults), or early close of Simonyi Survey Telescope are all examples of valid reasons to close AuxTel early.
3 Daytime activities¶
3.1 Daytime Checkouts¶
Daytime checkouts should be at least once per week. At a minimum they should be run following any major software updates (such as a new cycle release), or hardware changes and fault recoveries. When support is available, daytime checkouts will be run daily. They can be performed any time during the day when the system is available.
3.2 Afternoon Calibrations and Venting¶
Afternoon calibrations and venting, including opening the vent gates and enabling the fan, will be performed daily by the daytime observing specialist on a best-effort basis. If daytime engineering work within AuxTel prevents starting calibrations and venting before 16:00, these tasks can be skipped at the discretion of the daytime observing specialist. At a minimum, afternoon calibrations should be performed once per week. When support is available, afternoon calibrations will be run daily.
4 Nighttime Activities¶
4.1 Nighttime Operations¶
The AuxTel will be entirely scheduler-driven at night while in survey mode. A single scheduler configuration will be provided each night to minimize the need for detailed night plans requiring active observing support. Once the daytime activities are complete, the telescope can be prepared for on-sky observations and the scheduler configured to drive at any time before twilight. In the absence of faults, the scheduler should continue to drive AuxTel observations through to morning twilight. It is important to note that while observations are autonomously queued and executed in this mode, the AuxTel is not intended to be operated as a robotic telescope. The AuxTel does not have the capability to close automatically or protect itself from inclement weather so the observing staff will still be required to monitor weather conditions for close conditions.
4.2 Fault resolution¶
Faults which interrupt observations in this mode can be recovered when the observing specialists on the summit have time and are not occupied by responsibilities related to Simonyi Survey Telescope operations. When faults occur which result only in script failures that do not put hardware at risk or CSC components going into fault state, operations can be recovered by simply playing the script queue. The scheduler and standard scripts used during observations are required to be robust against these failures, and should not leave the system in an inoperable state in the case of script failure alone. Faults that result in CSC components going into fault state and which require intervention from the observing specialists to recover will be recovered when possible. CSC faults may result in the system remaining inoperable for extended periods of time if the observing specialists are unavailable to recover. In this case, the CSCs in fault should be transitioned to Standby to prevent warning messages being broadcast
5 Roles and Responsibilities¶
5.1 Daytime Observing Specialist Shift¶
The Daytime Shift members are responsible for deciding if/when it is appropriate to start daytime calibrations and venting. If Simonyi Survey telescope responsibilities prevent the Daytime Shift from completing calibrations and venting, they may be skipped.
5.2 Flex/Nighttime Observing Specialist Shift¶
The Nighttime Shift is responsible for monitoring AuxTel status throughout the night. This does not include monitoring data quality, only operational status. Images can be flagged for poor quality on a best-effort basis when available. If a fault is encountered during the night, it should be recovered when appropriate. At the end of the night, the telescope should be shutdown using a standard set of procedures.
5.3 Remote Support¶
During operation of AuxTel in survey mode, remote support is required to be on-call only in the case of risk to the hardware. Fault resolution will be handled by the observing specialist on shift and remote support will be provided on a best-effort basis. In the event the fault cannot be recovered, it is acceptable for the shift manager to decide to close the AuxTel early for the night.
6 Example 24hr timeline¶
Below is an example 24hr timeline of AuxTel operations in survey mode. All times are given in local (Chile) time.
09:30 : Summit coordination meeting, attended by member of SITCOM. Any issues from prior night are discussed and resources allocated to AuxTel daytime work if needed.
14:00 : Progress on AuxTel Dome daytime work is assessed by Daytime Observing Specialist. Readiness for daytime checkouts, venting, and calibrations is determined. If it is determined we are unable to proceed with daytime checkouts, calibrations and/or venting, they may be skipped entirely.
15:00 : Afternoon checkout, calibrations, and/or venting scripts are queued up by Daytime Specialist. AuxTel dome is prepared for venting.
17:30 : Start of night scripts (prepare_for_onsky, enable scheduler, resume scheduler) are queued up by Daytime Specialist. These can be added directly to the queue at any time following daytime activities, including before heading down for dinner.
18:30 to 06:00 : AuxTel is operated in scheduler-driven mode and monitored by on-shift specialist. Control of AuxTel is handed over to Flex/Nighttime Observing Specialist at shift change boundary.