CalCOFI CTD data are available from 1993 to present. Currently, a revision of the CTD data file formats are underway. Data quality code columns are being added to the CTD+bottle csvs offered as our final processed data. In addition to processing recent cruises, older cruises are being re-processed to update the data file formats. The re-processed CTD measurements should remain virtually unchanged but the column formats are being updated to include data quality codes (QC) for each core measurement. Data quality codes flag questionable or bad data and allow queries to filter out data using these criteria.
Older information about Metadata development is described below:
Starting with CalCOFI 0907M2, the BTLvsCTD program (developed by SIO-CalCOFI) generates several Metadata-friendly csvs. With Seasave Data Processing software v7.19 and later, sensor calibration coefficients are embedded as xml data in the .hdr file. BTLvsCTD.exe parses the .hdr for all stations into three csvs: hdr.csv, xmlcoeff.csv, & span.csv.
Starting with 0907, the first cruise to use dual SBE43 O2 sensors so the data tables will have these columns by default. Earlier cruises, using single O2 sensors will auto-fill the tables, leaving the second O2 columns blank.
Since sensor installation may vary from cruise to cruise and sensors may occupy different voltage channels, the voltage column headers "V1" "V2" "V3"... are replaced by "TransV" "FluorV" "AltV"... whenever possible.
Complete FinalQC zip archives (YY-YYMMCC_CTDFinalQC.zip) include CTD data .asc files with both column headers - numeric voltages V0-V7 and relabeled voltages. This archives the Seasoft .asc product unchanged and adds BtlVsCTD Metadata-friendly .asc files. CalCOFI tries to keep the sensor channels consistent unless deploying test or ancillary sensors. On cruises such as 1210NH or 1611SR, the secondary oxygen was replaced with a RINKO Oxygen Optode so the voltage columns on those cruises are different. Typical sensor configuration:
BTLvsCTD.exe's main module also generates bottle-corrected upcast and downcast csvs, merging all the individual station up and down csvs into two files. A dbcoeff.csv file is generated tabulating the cruise-ave bottle vs CTD coefficients as well as the station regression coefficients (each station's CTD data are regressed against bottle samples from that station, reducing the station-to-station variability witnessed using the cruise-ave coefficients). If bottle data are not available then only cruise-ave coefficients are used to correct the CTD data.
BTLvsCTD.exe has a module to automatically relabel the v0 v1 v2... to the sensor voltage in all .asc files. (I plan on using the hdr to automate this but currently the table in BTLvsCTD tabulating voltages vs sensor is used.)
BTLvsCTD.exe has a file renaming module that renames all dYYMM###.asc & .hdr, uYYMM###.asc & .hdr files to web-format YYMM_LLLLSSSS_###d.asc & .hdr, YYMM_LLLLSSSS_###u.asc & u.hdr. Please refer to the CTD Data Processing web page for more information on BtlVsCTD and CTD file formats.
Count (Excel Column) | Field | Description | Format |
1 (A) | Project | Cruise; usually CalCOFI or CalCOFI sampling area | Text, variable |
2 (B) | Study | YYMMSS (year month ship) Cruise designation | Text, 6 char |
3 (C) | Ord_Occ | Order Occupied | Integer |
4 (D) | Event_Num | CTD-at-Depth event number | Integer |
5 (E) | Cast_ID | Study+Order Occupied+up or downcast | Text, YYMM_###D or U |
6 (F) | Date_Time_UTC | Date Time UTC; downcast = start time; upcast=time @ 1st bottle tripped | DD-MMM-YYYY HH:M:SS |
7 (G) | Date_Time_PST | Date Time PST: downcast = start time; upcast=time @ 1st bottle tripped | DD-MMM-YYYY HH:MM:SS |
8 (H) | Lat_Dec | GPS Decimal Latitude: downcast = start lat; upcast=lat @ 1st bottle tripped | R7.5; ##.##### |
9 (I) | Lon_Dec | GPS Decimal Longitude: downcast = start lon; upcast=lon @ 1st bottle tripped | R8.5; -###.##### |
10 (J) | Sta_ID | CalCOFI Line and Station | Text; LLL.L SSS.S |
11 (K) | Line | CalCOFI Line | R4.1; LLL.L |
12 (L) | Sta | CalCOFI Station | R4.1; SSS.S |
13 (M) | Depth | 1 Meter Binavg Depth (meters) | R7.3; #.### |
14 (N) | Pressure | 1 Meter Binavg Pressure (db) | R7.3; #.### |
15 (O) | PrQ | Pressure Sensor Data Quality | I1 |
16 (P) | Temp1 | 1° Temperature (deg C) | R6.4; ##.#### |
17 (Q) | Temp1Q | 1° Temperature Data Quality Code | I1; see key below |
18 (R) | Temp2 | 2° Temperature (deg C) | R6.4; ##.#### |
19 (S) | Temp2Q | 2° Temperature Data Quality Code | I1 |
20 (T) | TempAve | Average Temperature, deg C; (1°+2°)/2 | R6.4;# #.#### |
21 (U) | Salt1 | 1° Salinity (PSU) | R6.4; ##.#### |
22 (V) | Salt1Q | 1° Salinity Data Quality Code | I1 |
23 (W) | Salt1_Corr | 1° Salinity, Bottle-corrected (PSU) | R6.4 |
24 (X) | Salt2 | 2° Salinity (PSU) | R6.4 |
25 (Y) | Salt2Q | 2° Salinity Data Quality Code | I1 |
26 (Z) | Salt2_Corr | 2° Salinity, Bottle-corrected (PSU) | R6.4 |
27 (AA) | SaltAve_Corr | Average Corrected Salinity; (1°+2°)/2 | R6.4 |
28 (AB) | Ox1 | 1° Oxygen; ml/L | R6.5 |
29 (AC) | Ox1Q | 1° Oxygen Data Quality Code | I1 |
30 (AD) | Ox1_CruiseCorr | 1° Oxygen, cruise-corrected; ml/L | R6.5; see key |
31 (AE) | Ox1_StaCorr | 1° Oxygen, sta-corrected; ml/L | R6.5; see key |
32 (AF) | Ox2 | 2° Oxygen; ml/L | R6.5 |
33 (AG) | Ox2Q | 2° Oxygen Data Quality Code | I1 |
34 (AH) | Ox2_CruiseCorr | 2° Oxygen, cruise-corrected; ml/L | R6.5 |
35 (AI) | Ox2_StaCorr | 2° Oxygen, sta-corrected; ml/L | R6.5 |
36 (AJ) | OxAve_StaCorr | Average Oxygen (ml/L); (1°+2°)/2 | R6.5 |
37 (AK) | Ox1uM | 1° Oxygen; uM/Kg | |
38 (AL) | Ox1uM_CruiseCorr | 1° Oxygen;cruise-corrected;uM/Kg | |
39 (AM) | Ox1uM_StaCorr | 1° Oxygen;sta-corrected; uM/Kg | |
40 (AN) | Ox2uM | 2° Oxygen; uM/Kg | |
41 (AO) | Ox2uM_CruiseCorr | 2° Oxygen;cruise-corrected;uM/Kg | |
42 (AP) | Ox2uM_StaCorr | 2° Oxygen;sta-corrected; uM/Kg | |
43 (AQ) | OxAveuM_StaCorr | Average Oxygen (ml/L); (1°+2°)/2 | |
44 (AR) | FluorV | Fluorometer Voltage (V) | R5.4 |
45 (AS) | FluorQ | Fluorometer Data Quality Code | I1 |
46 (AT) | EstChl_CruiseCorr | Estimated Chlorophyll, cruise-corrected | R6.4 |
47 (AU) | EstChl_StaCorr | Estimated Chlorophyll, sta-corrected | R6.4 |
48 (AV) | ISUSV | ISUS Voltage | R5.4 |
49 (AW) | ISUSQ | ISUS Data Quality Code | I1 |
50 (AX) | EstNO3_CruiseCorr | Estimated Nitrate, cruise-corrected | R5.4 |
51 (AY) | EstNO3_StaCorr | Estimated Nitrate, sta-corrected | R5.4 |
52 (AZ) | SigThetaTS1 | Sigma-theta T1S1 | R6.4 |
53 (BA) | SigThetaTS1Q | Sigma-theta T1S1 Quality Code | I1 |
54 (BB) | SigThetaTS2 | Sigma-theta T2S2 | R6.4 |
55 (BC) | SigThetaTS2Q | Sigma-theta T2S2 Quality Code | I1 |
56 (BD) | BAT | Beam Attenuation Coefficient | R5.4 |
57 (BE) | XMiss | % Light Transmission | R6.4 |
58 (BF) | TransQ | Transmissometer Quality Code | I1 |
59 (BG) | pH | pH Sensor | R6.4 |
60 (BH) | pHQ | pH Sensor Quality Code | I1 |
61 (BI) | SPAR | Surface PAR | #.##E+## |
62 (BJ) | SPARQ | Surface PAR Quality Code | I1 |
63 (BK) | PAR | Remote PAR | #.##E+## |
64 (BL) | PARQ | Remote PAR Quality Code | I1 |
65 (BM) | PoT1 | Potential Temperature | R6.4 |
66 (BN) | PoT2 | Potential Temperature | R6.4 |
67 (BO) | DynHt | Dynamic Height | R5.4 |
68 (BP) | SVA | Specific Volume Anomaly | R6.3 |
69 (BQ) | OxSat1 | Oxygen Saturation | R5.4 |
70 (BR) | OxSat2 | Oxygen Saturation | R5.4 |
71 (BS) | BTL_Depth | Bottle Depth (meters) | I4 |
72 (BT) | BTL_Temp | Bottle Temperature (deg C); usually 1°T | R5.3 |
73 (BU) | SaltB | Bottle Salinity (PSU) | R6.4 |
74 (BV) | OxB | Bottle Oxygen (ml/L) | R4.3 |
75 (BW) | OxBuM | Bottle Oxygen (umol/KG) | R6.2 |
76 (BX) | Chl-a | Bottle Chlorophyll | R4.3 |
77 (BY) | Phaeo | Bottle Phaeopigments | R4.3 |
78 (BZ) | NO3 | Bottle Nitrate | R3.1 |
79 (CA) | NO2 | Bottle Nitrite | R3.2 |
80 (CB) | NH4 | Bottle Ammonium | R3.2 |
81 (CC) | PO4 | Bottle Phosphate | R4.2 |
82 (CD) | SIL | Bottle Silicate | R5.2 |
Color Codes for this table (background color indicates data source) | |||
Color | Data Type | Source | |
White | CTD Cast Information | CTD "At Depth" Metadata | |
Green | CTD Data | 1m binavg CTD Data, Seasave processed | |
Yellow | Bottle-corrected CTD Data | 1m CTD Data, Seasave processed, bottle corrected | |
Blue | Bottle Data | Final, processed bottle data | |
Purple | Data Quality |
Data Quality Codes and Comments: |
|
|
The CTD final data database is finally finished.
The three zip files below contain the final bottle-corrected, Seasoft-processed CTD data from CalCOFI cruises from 1993-2019. These have been reprocessed into the current 84-col CTD csv format adopted by CalCOFI using data-quality codes-flags. 2020+ cruise csvs should import into these tables without issue. SIO-CalCOFI uses Microsoft Access as their relational database and that is the format distributed.
Download:
Please note - these files are distributed in decade-based databases because MS Access has a 2gb limit on table size. A complete CTD database should be imported into a relational database like MySQL that does not have this limitation. Exporting these individual databases in sql format from MS Access would facilitate the migration to a non-Access database. MS Access does allow linking separate databases so this method could be used to keep these data in MS Access tables for queries.
Archives are below - these are the individual cruise csvs used to create the above database files. For the complete 1993-2019 csv table visit this webpage
The files on this web page are database-import-ready csvs which combine 1m bin-averaged CTD sensor data with rosette bottle data. The addition of data-quality code columns required older (than 2013) archived cruises be reformatted with data code columns added. Currently, the original CTD+bottle data 62-column data files (YY-YYMMSS_CTDFinal.zip) are being reprocessed to insure consistency of the csv format. Data codes (see the CTD Database Index for key) are added to the QC columns to flag bad or questionable CTD sensor data. These data-quality codes may be used to exclude bad or questionable data during queries. Bad, missing or questionable data values may be changed to -99.0 or NaN, plus the number of columns has increased from 62 to 82 columns.
CTD data archives labeled with "QC" (YY-YYMMSS_FinalQC.zip) or "DB" (YY-YYMMSS_FinalDB.zip) are the latest format. Individual database-ready csvs will be available for each cruise so users can update an existing database on whatever platform they prefer. The addition of these new columns should not change the data values presented in the earlier formats although bad data may be nulled (NaN or -99.0). Additionally, when a sensor primary measurement is data coded "9", any data derived from it will change to "NaN" or "-99.0" - estimated NO3 from the ISUS sensor, for example. Refer to the CTD Database Index and the CTD.csv with QC Index pages for columns and descriptions. As older CTD data are reprocessed to add the QC columns and data codes, this table listing database-ready csvs will be updated.
Additionally, filenames have or will change to match the current CTD data distribution format. CTD.csvs processed with IEH-formatted bottle data (all cruises prior to 2012) will have an "I" denoting "IEH" in the filenames before "D.csv" or "U.csv": e.g. 20-1104SH_093301200_049Id.csv.
Preliminary versions of newer database csvs are available by request (This email address is being protected from spambots. You need JavaScript enabled to view it.) but this page only lists final versions.
Two versions of the CTD final data for each cruise (YY-YY is year, MM is month, SS is ship code):