2.1.

GMOD and GRB Triggering

The GMOD experiment was designed to further the goal of detecting GRBs in the multimessenger era as introduced in Sec. 1. The EIRSAT-1 operational strategy (Fig. 2) needs to account for the transient nature of GRBs, which makes them unpredictable sources to study.

Fig. 2

GMOD operations: the GMOD instrument detects $\gamma$-ray events, streams them to the OBC, which makes data products and triggers on significant events. Operators downlink and report on the trigger data.

GMOD consists of two main components, the detector assembly and the GMOD motherboard (MB). In the detector assembly, a tiled array of 16 silicon photomultipliers (SiPMs) is coupled to a scintillation crystal, which interacts with the gamma-ray photons to produce photons of characteristic energies that can be collected by the optical detectors. The outputs from the SiPMs are readout and digitized by an application-specific integrated circuit (ASIC), produced by Integrated Detector Electronics AS, called SiPM Readout ASIC (SIPHRA).⁴⁵ The GMOD MB supports the electronics required to power and operate the SiPMs and SIPHRA. The firmware on the GMOD MB has been developed in-house⁴⁶ to interface with the OBC of EIRSAT-1. The GMOD MB converts the channel data into time-tagged events (TTEs). There are TTEs generated for the 16 individual channels and for the sum channel (sum of 16 channels). Both sum and 16 channel TTEs are sent in pages to OBC, which stores them until a downlink of the data is requested from the ground station.

The OBC will use the GMOD TTE data to periodically generate lightcurves and spectra and store these in the GMOD lightcurve and GMOD spectrum storage channels for future downlink.⁴⁷ GMOD trigger generation is calculated based on a rolling average of counts in a specified energy range, the average count is compared to a predefined trigger significance level. If the significance level of the signal exceeds the trigger level, TTEs from a specified time period before and after the trigger time are copied and stored in a specific storage channel by the OBC.¹⁵ Due to communication constraints, it is not feasible to downlink all the data generated on board by GMOD. Priority will be given to the protected TTE data associated with triggers for GRB candidates. After a GRB triggers the on-board software (OBSW), the satellite will emit beacons containing basic information about the trigger (e.g., duration, start time) and low-resolution spectra and lightcurves until the next communication pass with the EIRSAT-1 ground station.⁴⁸ The subsequent communication passes will be used to downlink the higher resolution protected TTEs around the time of the trigger for analysis, classification, and reporting to the community. The EIRSAT-1 operations manual needs to prepare operators to optimize communication with the satellite. In a communication pass of $\sim 5$ to 7 min, the operators will need to confirm the health of the S/C and downlink science data from any or all three experiments depending on the priority of the data to downlink, i.e., GMOD GRB data has a higher priority than EMOD RTD readings.

3.1.

Mission Control Software

The main interface with the S/C is currently via Bright Ascension’s Mission Control Software (MCS).⁵¹ At the start of 2022, the project was selected for the Bright Ascension Bright Start Academic Program and received a year’s sponsorship with their flight and ground software. The month-long EQM MT and previous testing were performed using an older version of the MCS with fewer features. Due to the timing of the sponsorship, the updates to the operations manual based on feedback and lessons learned from the test campaign, discussed in more detail in Sec. 5.2, and the updates for interacting with the new MCS could be implemented simultaneously. The MCS is similar to the previous interface with the S/C with an overview of the interface initially presented and then the main improvements outlined.

The operator loads a spacecraft database (SCDB) into the MCS, which is used to interact with the parameters and actions of the onboard software. Parameters can be set, obtained, queried, downlinked from, or uplinked to, and actions can be invoked. These terms are the building blocks of the interface to the S/C and are defined in Table 1.

Table 1

Summary of main components used to operate and command EIRSAT-1 via the MCS.

The MCS has a number of components, with different roles, that can be displayed. Only the ones used by the EIRSAT-1 project to date will be discussed here. The default EIRSAT-1 layout of the S/C interface, before operator customization for a specific procedure, is presented in Fig. 3.

Fig. 3

Default MCS layout used by EIRSAT-1 team, including packet monitor (green box), transfer window (red), and system/event/debug console (blue). An example of an open parameter (purple) and action (yellow) is presented.

The panel on the left is the mission explorer containing a drop-down of the SCDB and connection settings to the S/C. The packet monitor window (surrounded by a green box) in the center displays the telemetry (TM) and telecommands (TCs) exchanged between the MCS and the S/C. The transfer window (red) is located on the right and displays the percentage complete of data being downlinked from/uplinked to the S/C. The transfer window can be used to abort, suspend, and resume any transfer of data between the MCS and the S/C. At the bottom of the layout is the System/Event/Debug console (blue). The System tab displays logs from the MCS itself, e.g., failing to connect to the S/C. The Event tab displays any events from the OBSW, e.g., transition to safe mode. The debug log shows debug information from the OBSW.

In addition to these components, parameters and actions can be opened and positioned on the console via the SCDB and used to interact with the S/C, with an example of each in Fig. 3. Layouts were created for different operational procedures during the MT as it prevented operators from searching through the SCDB for a specific parameter during a communication pass. The first section of the EIRSAT-1 operations manual contains a comprehensive description of the aforementioned components of the MCS, how to load and change the SCDB, and how to set up communication via the MCS.

The EIRSAT-1 operations manual required a revision after transitioning to the MCS, the major updates to the introductory sections were to describe the new features available on the MCS summarized below:

3.2.

Prerequisite Knowledge for Operators

In addition to describing the interface with the spacecraft, the operations manual contains details of all the knowledge that operators are required to have before commanding EIRSAT-1. This Prerequisite Knowledge section of the operations manual is an outcome of the ODTs, discussed in Sec. 5.1. Any background knowledge that applies to most, if not all, of the operational procedures is described. An in-depth description of the contents of this section can be found in Ref. 36, but the following points summarize the main topics:

Fig. 4

Extract from GMOD configuration procedure in the EIRSAT-1 operations manual. The format (objective, introduction, and step-by-step procedure, TM/TC tables) is displayed.

3.3.

Operational Procedures

The operations manual currently has 37 manual operations procedures for nominal and anomalous mission scenarios. The operational procedures can be split into four groups: early mission, nominal, payload, and fault analysis. Examples of each procedure category are presented, and the full list used in the EQM MT can be found in Ref. 36:

A more detailed description of the objectives of these example procedures is presented in Sec. 5.2. In summary, early mission procedures will only be required in the initial phases of the mission, such as making first contact with the S/C after launch and checking the functionality of the subsystems and payloads in the commissioning phase. Nominal procedures will be used on a daily basis by operators, some will be used during every pass. For example, Start A Communication Pass will be performed every time contact is made with the S/C to determine its current state and if any new GRBs have been detected. Nominal operations will also include the downlinking of housekeeping data to assess the health of the S/C and science data from the three experimental payloads. There are procedures specific to the configuration and experiment running of the two hardware payloads, EMOD and GMOD. The development of the WBC procedures will be part of the future work updates in preparation for the flight-ready operations manual.

The most challenging operational procedures to develop were the fault analysis procedures. Seminars were held to discuss the possible points of failure in the mission. These seminars resulted in a list of off-nominal operational procedures, which can be split into two categories—procedures to determine the cause of entering an unexpected mode or OBC image (e.g., Safe Mode Entered, Failsafe Entered) and procedures to determine the root cause(s) of the anomalous behavior and how to recover the mission (e.g., Low Battery Fault Analysis, S/C Reboot Fault Analysis). Ref. 36 contains a detailed example of a procedure from each of the four categories. This paper provides more detail on the GMOD operational procedures (Sec. 4) and describes the operational manual validation methods employed by the team (Sec. 5).

4.1.

Commissioning

Initially, GMOD was to be assigned one operational procedure during the commissioning phase of the mission. However, the ODTs exhibited that breaking the GMOD commissioning into two sections would be more beneficial. Once the core subsystem health checks (battery and ADCS) are completed, a GMOD health check will be performed to assess the basic functionality after launch. The second subset of commissioning procedures involves setting up the S/C for more routine operations. These include a GMOD operations procedure, which will set the experiment running in-orbit for a short duration of time to verify functionality. Both GMOD commissioning phase procedures are described in detail in Fig. 5. The MT verified this approach to commissioning GMOD, and the updates that are required for the flight operations of GMOD are based on advancements made in the firmware.

Fig. 5

Summary and statement of the main aims of the GMOD health check and GMOD operations commissioning procedures in the EIRSAT-1 operations manual.

4.2.

Experiment Configuration and Running

The GMOD configuration and GMOD experiment running operational procedures were developed and adapted from the commissioning procedures. The GMOD configuration procedure guides the operator through the different parameters and actions that can be used to configure the GMOD payload. There are eight sub-procedures that can be carried out individually to assess and update the GMOD experiment parameters. The sub-procedures contain the basic configuration instructions, e.g., to turn the experiment on/off, get and change the operational mode, and set voltage and current values. It also contains sections to customize the lightcurve and spectrum parameters (e.g., bin width, integration time) and set up and enable GRB triggering.

The GMOD experiment running procedure consists of items detailed in points A, B, C, and F in the outline of the GMOD commissioning operations procedure in Fig. 5. GMOD will primarily run with just sum channel data being generated and streamed to the OBC and the OBC triggers on sum channel data. There is a separate operational mode for the generation and stream of 16 channel data from GMOD to the OBC, this will be performed as a technical demonstration but will not be part of the daily operations of the GMOD experiment.

4.3.

GRB Data Downlink

The scientific objective of the GMOD experiment is to detect GRBs and report triggers to the high-energy astrophysics community. This requires a complex operational strategy as these short-lived (approximately seconds long), and isotropic events need to be reacted to in real time. This operational requirement is unlike many other science-based observatories running on scheduled observations of known sources.

The OBSW includes a GRB trigger counter in the satellite beacon whose format will be made publicly accessible. This is to facilitate large communities of ground stations, e.g., SatNOGS⁵⁴ in decoding the beacons and informing the team of new GRB data. The S/C will beacon basic trigger information, compressed lightcurves, and spectra until it communicates with the ground station. Once a trigger has been identified, the following communication passes will prioritize downlinking the protected higher resolution TTE data around the time of the event for ground-based analysis. This analysis will include confirming if the event was a GRB and reporting the observation to the community. As shown in Fig. 6, knowing about a new trigger before a communication pass will increase the efficiency with which the team can respond to GRBs. If the amateur radio community can inform the operators that a new trigger has been detected (grey panel on left of Fig. 6), the next pass plan can include downlinking GRB data. However, if the operators discover the GRB trigger counter has increased during a pass, the downlink of the data will be included in the subsequent pass plan but this delays initiating data acquisition by approximately hours, i.e., the right-hand side of Fig. 6 will need to be followed before proceeding to the downlink of GRB data (left-hand side) during the next communication pass.

Fig. 6

Approach to determining when there is new GRB data and downlinking this data with time-constrained communication passes.

It will take multiple passes to downlink the data from a GRB, so commencing the downlink rapidly is vital. The downlinking of GRBs will make use of the suspend/resume feature of data transfers in the MCS. At loss of signal, due to the S/C passing beyond the TM/TC horizon, the MCS will automatically suspend the transfer of GRB data, this can then be resumed by the operator during the next pass(es) until all the data have been successfully downlinked. At the start of each communication pass, the most recent housekeeping data will be downlinked before resuming any GRB data transfer to monitor the current health of the S/C.

Once that GRB has been successfully downlinked, the storage channel needs to be wiped manually by an operator to free the storage channel for future GRB trigger data. The OBC has 40 storage channels allocated to GRB data. There will be a trigger log on the ground that operators will be responsible for populating when data have been downlinked, from which storage channel and if the channel was wiped successfully. Once data for a GRB have been successfully transferred from the satellite, operators will need to determine the next highest priority task, which could include downlinking the next GRB, some RTD data from the EMOD experiment, or running the WBC experiment.

5.1.

Operations Development Tests

ODTs were used to initially test the procedures individually or in small subgroups (e.g., Upload Payload Image and Rewrite Payload Firmware). Once an initial draft of a procedure was prepared and reviewed, an ODT was organized. Members of the team not involved in the development of that procedure would follow it with the EQM and provide feedback. Any procedures that required large updates were subject to more than one ODT.

The ODTs revealed the importance of initiating operations development once a full system configuration is accessible. Missing scenarios, procedures, and steps were all revealed in the iterative process of testing the individual operational procedures. ODTs were essential in introducing the team to the operations manual and familiarizing them with the prerequisite knowledge and format of the procedures. ODTs also facilitated all team members interacting with the S/C via an interface similar to the MCS. This S/C interface was used during the ODTs, as at that point, the project did not have access to Bright Ascension’s MCS. This predecessor to the MCS had the same major components but not as many user-friendly features, as discussed in Sec. 3.1. The ODTs were the initial stage of operator training of the EIRSAT-1 team.

The ODT approach also had its limitations, the tests were not conducted in a flight-like configuration to complete the first assessment of the procedures efficiently and with the available resources, i.e., the full operational chain had not been developed, and the analysis tools for processing the downlinked data had not been completed. In flight, there will be constricted communication pass times with approximately minutes to complete procedures. Communication pass times were not implemented during ODTs as the main objective was to improve the step-by-step instructions. Without time pressure, the operators could progress through the procedure, pause for feedback and discussions, and there was time to note the required updates for the operations team to implement. For efficiency, the ODTs were performed using serial communication with the S/C. In flight, all communication will be over radio frequency (RF), increasing the time taken to receive responses and data from the satellite. The EQM MT was a more rigorous assessment of the EIRSAT-1 operations manual, and Sec. 6 outlines the major differences between validating the EIRSAT-1 operations manual using the ODTs and MT.

5.2.

Engineering Qualification Model Mission Test

The EQM MT was conducted during August 2021, and a full description of the approach taken to MT can be found in Ref. 12. The MT is a long-duration test campaign simulating flight-like scenarios, including early mission, nominal, and anomalous scenarios. The MT was designed to run through as many scenarios as feasible in 4 weeks and was therefore the logical advanced test of the EIRSAT-1 operations manual. The first week of the MT simulated initial AOS and commissioning of the subsystems and payloads. The remaining 3 weeks tested nominal operational procedures for EIRSAT-1, experiment running procedures for GMOD and EMOD (WBC was not advanced enough to be included in the EQM MT), and off-nominal scenarios, such as low battery events and unexpected S/C reboots.

Throughout the MT, the spacecraft operators could only interact with the S/C during simulated communication passes of $\sim 3$ to 5 min. There were 4 to 5 passes per day, 7 days a week. Test support operators monitored the test ground support equipment (GSE) and manipulated the S/C to inject non-nominal scenarios, e.g., turning off S/C charging and allowing the battery to fall below a certain threshold to trigger entry into safe mode. The MT was utilized for operator training, and the team was exclusively self-taught. Prior to the start of the test campaign, seminars were run for the team by the lead operations engineers to provide an overview of the EIRSAT-1 operations manual and the operator tools, e.g., MCS and Grafana. The manual was provided to the team in advance of the MT to allow time for them to become familiar with the prerequisite knowledge sections and the procedures. Two operators are always required to command EIRSAT-1, the lead operator commands the S/C via MCS while the supporting operator reads the instructions in the manual. During the EQM MT, there was always one experienced operator present for each communication pass but as the MT progressed, the operators in training took on the lead operator role. For the FM MT, the same approach was taken, where less experienced operators began with the supporting role, then as their confidence and competence in commanding the S/C increased, they were given the responsibility of being lead operator.

The majority of the operational procedures were tested during the MT, and the test campaign revealed two new procedures that needed to be developed around downlinking data in atypical scenarios. Of the 35 procedures tested during the MT, 12 required no updates due to sufficient development in the ODTs, 15 required minor updates, and 8 required major updates. Minor updates included typos or small clarifications in a step of the procedure, whereas major updates incorporated a reformatting of the procedure or adding multiple steps/sections. The procedure updates made from the EQM MT were validated during the FM MT. A frozen version of all of the procedures validated during the FM MT will comprise the EIRSAT-1 operations manual used during flight. Any new procedures or improvements required after the FM MT can be validated with the EQM in “mini-MTs,” which are 1-day MTs focusing on a specific scenario or procedure. Table 2 provides examples from each of the operational procedure categories, the number of ODTs that were conducted, and whether updates were required based on feedback from the MT.

Table 2

Examples of each of the EIR_OPS procedure categories (early operations, nominal, payload, and fault analysis) are presented. For each example procedure, the number of ODTs that were included is given in the third column. The final column states the updates required based on EQM MT feedback, with minor updates including typos, adding single steps in, or clarifying a confusing step. Major updates required multiple step changes, addition of new sections, or re-visiting the entire structure of the procedure. The minor* procedures require updates based on advanced FM firmware, rather than issues revealed by the MT.