CTD Metadata Development Notes

CalCOFI CTD data are available from 1993 to present. Currently, a revision of the CTD data file formats are underway. Data quality code columns are being added to the CTD+bottle csvs offered as our final processed data. In addition to processing recent cruises, older cruises are being re-processed to update the data file formats. The re-processed CTD measurements should remain virtually unchanged but the column formats are being updated to include data quality codes (QC) for each core measurement. Data quality codes flag questionable or bad data and allow queries to filter out data using these criteria.

Older information about Metadata development is described below:

Starting with CalCOFI 0907M2, the BTLvsCTD program (developed by SIO-CalCOFI) generates several Metadata-friendly csvs. With Seasave Data Processing software v7.19 and later, sensor calibration coefficients are embedded as xml data in the .hdr file.  BTLvsCTD.exe parses the .hdr for all stations into three csvs: hdr.csv, xmlcoeff.csv, & span.csv.

  • The .hdr csv includes the cast information such as date, time, line, sta, lat & lon, sensor configuration, and all Seasave data processing modules and coefficients applied.
  • The span.csv is a tabulation of the min and max values for each sensor, extracted from the .hdr file. This file/table will be useful for automating basic data-quality controls and sensor behavior assessments, allowing data processors to query data ranges for validity.
  • The last file is xmlcoeff.csv, a tabulation of the coefficients for sensor.

Starting with 0907, the first cruise to use dual SBE43 O2 sensors so the data tables will have these columns by default. Earlier cruises, using single O2 sensors will auto-fill the tables, leaving the second O2 columns blank. 
Since sensor installation may vary from cruise to cruise and sensors may occupy different voltage channels, the voltage column headers "V1" "V2" "V3"... are replaced by "TransV" "FluorV" "AltV"... whenever possible.

Complete FinalQC zip archives (YY-YYMMCC_CTDFinalQC.zip) include CTD data .asc files with both column headers - numeric voltages V0-V7 and relabeled voltages. This archives the Seasoft .asc product unchanged and adds BtlVsCTD Metadata-friendly .asc files. CalCOFI tries to keep the sensor channels consistent unless deploying test or ancillary sensors. On cruises such as 1210NH or 1611SR, the secondary oxygen was replaced with a RINKO Oxygen Optode so the voltage columns on those cruises are different. Typical sensor configuration:

  • V0 = Transmissometer
  • V1 = Fluorometer
  • V2 = PAR
  • V3 = Altimeter
  • V4 = Primary Oxygen
  • V5 = Secondary Oxygen
  • V6 = ISUS Nitrate
  • V7 = pH
  • V15 = Surface PAR, when made available by the vessel; SIO's was retired

BTLvsCTD.exe's main module also generates bottle-corrected upcast and downcast csvs, merging all the individual station up and down csvs into two files.  A dbcoeff.csv file is generated tabulating the cruise-ave bottle vs CTD coefficients as well as the station regression coefficients (each station's CTD data are regressed against bottle samples from that station, reducing the station-to-station variability witnessed using the cruise-ave coefficients).  If bottle data are not available then only cruise-ave coefficients are used to correct the CTD data.

BTLvsCTD.exe has a module to automatically relabel the v0 v1 v2... to the sensor voltage in all .asc files. (I plan on using the hdr to automate this but currently the table in BTLvsCTD tabulating voltages vs sensor is used.)

BTLvsCTD.exe has a file renaming module that renames all dYYMM###.asc & .hdr, uYYMM###.asc & .hdr files to web-format YYMM_LLLLSSSS_###d.asc & .hdr, YYMM_LLLLSSSS_###u.asc & u.hdr. Please refer to the CTD Data Processing web page for more information on BtlVsCTD and CTD file formats.