Data Module:
Data downloading is managed by AllData.py; at this time AllData.py is manually started via the command line. AllData.py begins by executing the GetAllData() function, which retrieves a list of data centers from the datacenter table in the database (shown below) and starts a separate process for each. Some data centers will support multiple simultaneous requests, for those data centers multiple processes will be started. After starting the data center processes, GetAllData() goes into an infinite loop and monitors the progress of the processes.
Each process runs an instance of the GetByDC() function, which also runs in an infinite loop. This function retrieves the next available event for that data center from the database and retrieves all available data for that event from that data center, using the channels previously stored in the database for that event and data center. Instrument responses are removed to displacement for the downloaded time series, and if possible the traces are rotated to the RTZ orientation. The traces are resampled to 20 samples/second, converted to .sac file format, and a number of phase arrival times are computed and stored in the sac user headers. The resulting .sac data files are written to the event directory for that event. Ongoing progress and any errors are written to the Data Log.
The events which have been retrieved for a data center are tracked in the dcevent table, which stores the data center ID, the event ID, and whether processing is complete for that event and data center. A representation of that table is shown below the datacenter table.
The channel information for all files are re-written if necessary so channel designations become BH*, to reflect the resampled data rate. Blank location codes are defaulted to "00". File names are of the form EventName.Network.Station.Location.Channel.Deconvolved.sac; an example is 202209050452.TW.WCKO.00.BHZ.D.sac. If the system was unsuccessful at deconvolving the instrument response, the .D.sac portion of the file name would be .U.sac.
For the above example, all files for this event would be written to /RZ/adept/202209050452/. Any files for which the rotation to RTZ was unsuccessful are stored in a /ZNE subdirectory for that event.
If the process loops and no new event is available, it waits for 12 hours before trying again.
After each event is processed, the GetByDC() function checks whether the shutdown flag has been set in the database. If so, it exits. This action will be recorded in the Data Log. When all data center processes have exited, the GetAllData() function will record that in the Data Log and will exit.
Here is a representation of the datacenter table in the database:
| column name | data type | notes |
| prikey | bigint | auto-generated numeric unique id |
| dcid | text | the text id for the data center, e.g., "IRIS" |
| type | text | the type of data center: "fdsn" uses a common protocol for data acquisition, others require special handling |
| sortorder | integer | the order in which data centers are displayed |
| region | text | country or regional location for the data center |
| enable | boolean | whether or not to retrieve data from this data center |
| url | text | some data centers are not in the ObsPy module or are not correct, for these we need a url to the data center |
| userid | text | some data centers require user authentication |
| password | text | for those data centers requiring user authentication |
| stationlimit | integer | number of channels per data request. Generally 100, some data centers require smaller requests |
| event | text | the event currently being downloaded and processed |
| enum | numeric | where the current event is located in the event list |
| ecount | numeric | number of events in the event list |
| chanrange | text | which channels are currently being requested for this event |
| updatedt | timestamp | when the current event columns were last updated |
| filerate | numeric | the approximate rate of data download in files/minute |
| pid | bigint | the process id of this data center download process. Only needed if a process has to be manually terminated |
| instances | integer | the number of simultaneous download processes to start for this datacenter |
Here is a representation of the dcevent table:
| column name | datatype | notes |
| dckey | bigint | the data center ID, keyed to the prikey column of the datacenter table |
| eventid | text | the event ID, keyed to the eventid column of the event table |
| lastdone | timestamp | the date/time when this data center/event combination was last run |
| complete | boolean | false = inprocess, true = done |