CalCOFI processes our CTD casts using the guidelines recommended by Seabird for our 911+ sensor array and depths. Please refer to the CTD Data Processing Protocol web page for the Seasoft modules applied. This page describes the in-house software used to merge CTD sensor data with bottle data, derive bottle corrections to apply to sensor data, & CTD data flagging. Also explained is the organization of data and metadata files distributed online as preliminary & final data products.
Files referenced: '.asc' & '.hdr' files are Seasoft-generated CTD data files. All other files referenced are SIO-CalCOFI data files generated at-sea and ashore from data acquistion & analysis software and SIO-CalCOFI-developed in-house software. SIO-CalCOFI's data processing methods, file formats, and metadata descriptions are available on the Data Formats & Methods web pages.
1A) BTLvsCTD1 Step 1: (files required: sta.csvs with all bottle data columns populated; corrected YYMMcasts.csv, Seasoft-processed non-binavg CTD.asc & .hdr files). Preliminary bottle data ('YYMM###.csv' sta.csvs) are merged with 4-sec-ave (prior-to-bottle-closure) CTD .asc data, creating a single YY-YYMMSS_CTDCSV###-###.csv that is imported into Excel.
1B) Bottle vs CTD sensor data are plotted: primary CTD oxygen vs bottle oxygen ml/L & umol/kg; secondary CTD oxygen vs bottle ml/L & umol/Kg; FlV (usually V1) vs Chl-a; ISUSV (usually V6) vs NO3; CTD Salts > 350m are compared to bottle salts.
Three duplicate spreadsheets are pasted as new sheets to allow sorting and deletion of specific columns (or you can cut-paste specific columns to new sheets, if you prefer)
- 'Salts > 350m' - all data are sorted by Pr or Depth, Largest to smallest and all rows less than 350m deleted. Two columns are added: bottle salt - CTD Salt00 & bottle salt - CTD Salt 11. Mean, SD, Min, Max, & n are calculated at the bottom of the column. Any values above 0.02 are excluded. The mean values of the bottle minus primary & secondary salts become the offsets.
- 'Oxygen umol/kg' - for some reason leaving blank fields in the umol/Kg O2 columns can generate "wavy" scatter plots so deleting rows missing bottle oxygen data and missing CTD oxygen sensor data (unusual) is advised. "Fliers" are removed - either bad or questionable bottle or sensor data. Four plots total - Ox1ML vs OxB, Ox2ML vs OxB, Ox1UM vs OxB, Ox2UM vs OxB.
- 'ISUS vs NO3' - remove missing or 0 ISUS voltages rows and missing bottle NO3 rows then plot & add a trendline.
- 'Fluorometer voltage vs chl-a' plot can be generated from the main spreadsheet without issue. Or you can create a 'Chl-a' spreadsheet sorted by depth and delete all depths > 200m.
On all plots - all bottle-data fliers, such as mistrips or misdraws, are removed before trendlines added. PDF plots of all regressions are generated plus 400px & 1024px jpgs for posting online. Preliminary regressions and plots are posted since it can take a year for final data to become available. Once final, regression coefficients and plots are updated with new data.
1C) Seasoft 'Split' and 'Binavg' modules are applied to CTD.asc files, generating 1m binavg up & downcast asc files. It is important to verify any mislabeled CTD .hdr files have been corrected before step 2A to insure the matching sta.csv merges properly with CTD.asc. When there's a mismatch, it's usually caused by the cast number (###; aka 'Proc #' or 'Order Occupied') or CalCOFI Line Sta (eg 93.3 120.0) values being mis-entered when the cast starts. It is usually corrected at-sea but the .hdr correction may be lost if the .hex file has not been corrected. It requires the cast's .hex file be corrected using a binary editor (HxD Hex Editor) to stop the error from re-appearing when processed by Seasoft. If corrected, the original CTD cast .hex & .hdr files should be updated into the cast.zip (YYMM###.zip) & the cruise's YYMMSS_CTDCast.zip distributed online, & documented on the 'CTD Updates' web page.
1D) Inspect and correct YYMMcasts.csv & YYMMevents.csv. YYMMevents.csv will require "erase previous event" editing and Line & Sta verification. After YYMMcasts.csv and YYMMevents.csv have been manually corrected, run the DECODR2 'Clean Event Log' module to compare casts.csv order occupied numbering to the eventlog. (JLW usually checks & corrects the YYMMcasts.csv so coordinate with her to avoid duplicating effort).
2A) BTLvsCTD Step 2: (requires sta.csvs with all bottle-data columns populated, corrected YYMMcasts.csv, corrected YYMMevent.csv log, Seasoft-processed non-binavg CTD .asc & .hdr files).
After regressing 4sec average sensor data vs bottle data (step 1B), CTD.csvs are generated for point-checking using BTLvsCTD module 'Step 2'. Regression coefficients are entered & applied to CTD sensor data, generating individual up & downcast CTD.csvs. The CTD.csvs are moved to a new directory such as 'csvs' and plotted using the appropriate depth Matlab scripts:
- 'plotCTDDB600SD' - plots station-corrected CTD primary sensor data plus bottle data 0-600m; use 'u' & 'd' in the script commandline for upcast & downcast and '1' & '2' in the script commandline for primary or secondary sensor profiles:
- e.g. plotCTDDB600SD('D:\CODES\2016\1607\CTD\csvs\','D:\CODES\2016\1607\CTD\csvs\','1607','u',1)
will plot CalCOFI 1607 primary CTD upcast profiles with bottle data.
syntax: plotCTDDB600SD(input csvs path, output plot path, YYMM for plot titles, 'u' or 'd' = up or down, 0 = no bottle data 1 = add bottle data, '1' or '2' = primary or secondary sensor
- e.g. plotCTDDB600SD('D:\CODES\2016\1607\CTD\csvs\','D:\CODES\2016\1607\CTD\csvs\','1607','u',1)
- 'plotCTDDB600SD2' - plots station-corrected CTD secondary sensor data plus bottle data 0-600m; use a "u" & "d" in the script commandline for upcast & downcast
- 'plotCTDDB800SD' - plots station-corrected CTD primary sensor data plus bottle data 0-800m; use a "u" & "d" in the script commandline for upcast & downcast
- 'plotCTDDB800SD2' - plots station-corrected CTD secondary sensor data plus bottle data 0-800m; use a "u" & "d" in the script commandline for upcast & downcast
- 'plotCTDDB1200SD' - plots station-corrected CTD primary sensor data plus bottle data 0-1200m; use a "u" & "d" in the script commandline for upcast & downcast
- 'plotCTDDB1200SD2' - plots station-corrected CTD secondary sensor data plus bottle data 0-1200m; use a "u" & "d" in the script commandline for upcast & downcast
- 'plotCTDDB100SD' - plots station-corrected CTD primary sensor data plus bottle data 0-100m; use a "u" & "d" in the script commandline for upcast & downcast
- 'plotCTDDB100SD2' - plots station-corrected CTD secondary sensor data plus bottle data 0-100m; use a "u" & "d" in the script commandline for upcast & downcast
Alternate versions: 'plotCTDDB###AD' or 'plotCTDDB###AD2' plots cruise-corrected CTD sensor data - this is necessary if no oxygen, chl-a, &/or NO3 bottle data are available for the cast.
Note: To re-run previously cruise scripts in Matlab, hit up-arrow to recall the proper script then edit the filepath for the new cruise
2B) Step 2A will generate preliminary CTD.csvs for the point-checking process. Bad or questionable bottle data should be corrected then re-calculated, data coded or deleted. Many problems with the CTD .asc .hdr files are often discovered during the preliminary merge. Be sure to correct problems in all files affected - if you regenerate any CTD files by rerunning Seasoft modules, be sure to re-apply corrections, if necessary. A correct YYMMcasts.csv can be used to help clean-up the YYMMevent.csv log. Line & Sta editing of the event log is recommended before running the DECODR2 'Clean Event Log' module. Manual removal of 'erase previous event' tagged event records should also be performed before running the cleaner module. Be sure to do a final check of the event log after running the cleaner. If good, post the final eventlog.csv to Q:\CODES\_finalEventLogs and copy it to the CTD data processing folder.
2C) Preliminary notes are posted on the cruise web page (eg "1708SR Preliminary CTD Cast Notes"). These notes should include any console ops or clipboard cast notes, any CESL sampling notes, and any problems noted during data acquistion, bottle sampling, or processing. Metadata such as CTD sensors deployed, general setup, equipment used (did we use our deck unit or the ship's, for example), logisitics, or troubleshooting should be included.
3A) Once bottle data are final and sta.csvs updated, the Seasoft-processed non-binavg CTD .asc files are merged with final sta.csv bottle data, repeating step 1A & 1B to generate final bottle regression correction coefficients. Final PDF plots of all regressions are generated plus 400px & 1024px jpgs for posting online, updating the CTD Data Processing web page. (I usually make a copy of the preliminary page to save the initial comparison and relabel 'preliminary' text to 'final', adding final processing notes and information.)
3B) Re-run step 2A with the final coefficients; regenerate final Matlab primary & secondary plots, with appropriate depth axes.
3C) Apply data codes to CTD.csvs, as needed, using "8" for questionable sensor data, "9" for bad data. "NaN" bad or missing data, if appropriate.
BTLvsCTD has modules to:
- check the CTD.csvs for correctness: checking for bad values using basic range criteria; comparing station information columns to the casts.csv or .hdr files.
- data coding module - user selected columns can be data coded interactively; depth range option is available
- column-nulling module;
- concat corrected CTD.csvs into updated, merged CTD up & down db.csvs.
4) Zipping data files for online distribution:
This process is still evolving but currently we are following these steps.
\asc-hdr folder/files: there are currently two versions - the original Seasoft data products: YYMM###.asc .hdr (optionally .btl) files and relabeled .asc .hdr files. The original asc-hdr-btl files are in the "original-asc-hdr-btl" folder.
Maintaining these the simpler filenames allow easier re-merging of CTD .asc data with sta.csvs. But since CTD sensors can move to different voltage channels on the 911+ 'fish', we like to generate a version of the asc files that relabel the voltage headers. Instead of the ambiguous "V0 V1 V2...", voltage headers are relabeled "TransV FluorV AltV..." using BTLvsCTD.
Additionally, renaming the .asc .hdr files to match the CTD.csv format is done: in addition to Seasoft files YYMM###D or U.asc & .hdr files, cruise and cast labeled files are archived.
The format is "YY-YYMMSS_LLLLSSSS_###UD.asc & .hdr - "YY-" is decade; YYMMSS is CalCOFI cruise designation YY:Year MM:StartMonth SS:ShipCode; LLLL:CalCOFI Line (0933 = 93.3), SSSS:CalCOFI Sta (1200 = 120.0)
This filename provides better station identifying information without having to refer to the cruise's station-cast index.
\csv-plot folder/files: these are the individual up & downcast CTD.csvs YY-YYMM_LLLLSSSS_###UD.csv (or shipcoded YY-YYMMSS_LLLLSSSS_###UD.csv) plus upcast & downcast primary and secondary plots of sensor data (T, S, O2, Fl estChl, ISUS estNO3) with bottle data overlaid when available.
\db_csvs folder/files: two files only - all CTD+bottle data upcasts as YY-YYMMSS_LLLLSSSS_###U.csv and all downcasts as YY-YYMMSS_LLLLSSSS_###D.csv. These are data-base friendly csv designed to import into the CalCOFI CTD database (in development).
\metadata folder/files: CTD vs bottle data csv (from BTLvsCTD) & Excel file; all bottle vs CTD sensor regression plots; files auto-generated by BTLvsCTD:
- 'YY-YYMMSS_hdr_###-###S.csv': all the CTD .hdr file information parsed in one record per downcast & one per upcast. ###-### denote beginning and end cast numbers (order occupied); "S" denotes sta.csvs were merged (as opposed to "I", IEHs).
- 'YY-YYMMSS_span_###-###S.csv': parsing the sensor low to high ranges from the .hdr files. May be used for data-quality assessment & control.
- 'YY-YYMMSS_xmlcoeff_###-###S.csv': CTD sensor coefficients, tabulated from the .hdr file into a database/spreadsheet friendly csv
- 'YY-YYMMSS_DBcoeff_###-###.csv': tabulation of bottle correction regression coefficients applied to CTD sensors plus BTLvsCTD's individual (sta_corr) dynamic regression correction coefficients.
BTLvsCTD dynamically regresses bottle data vs sensor data from each station separately and tabulates these dynamically-calculated values, along with the cruise-average (Excel) coefficients. - 'YY-YYMMSS_DBcoeff_###-###.txt': readable text version of the bottle correction regression coefficients.
- 'YY-YYMMSS_Final###-###.cfg': BTLvsCTD configuration file containing regression-correction coefficients used to generate CTD.csvs.
- 'YY-YYMMSS_Events.csv': corrected cruise eventlog
- 'YY-YYMMSS_Data_Processing_Notes.txt': all the data processing notes compiled
- White-papers, included in all CTD data archives:
- 'CTD_Data_Files.pdf': pdf white paper that describes the filenaming protocol (included in all CTD archives)
- 'CTD-CSV-Format.pdf': pdf white paper that indexes the CTD.csv column headers and units
- 'FilenamingAndAscHeaderNotes.txt': white paper that explains the voltage header relabing in CTD .asc files & the general file renaming format.
All these files are zipped, organized in these folders and posted online on several web pages using Joomla:
- 'YYMMSS CTD Data Processing Summary' web page - updating download links from 'prelim' to 'final'; updating regression coefficients & regression plots. In Joomla, copy the 'preliminary page' & edit, saving as 'Summary'.
- 'YYMMSS Cruise Summary' - add download links for 'prelim'; update from 'prelim' to 'final'. In Joomla, copy the 'YYMMSS Cruise Prospectus' to 'YYMMSS Cruise Summary', adding two rows at the top of the table: 'Summary:' & 'Data files/links:'; I also post the actual cruise track map vs scheduled station map.
- a new entry is added or 'Prelim' updated to "FinalQC' on 'CTD Archive 1990-present' web page
- a new entry is added or 'Prelim' updated to "FinalQC' on 'Recent CTD Data' web page
- 'Available Data' web page CTD section is updated with latest 'Prelim' & 'Final' CTD data zip
Currently under development - 'CTD Database' web page needs to be populated with zip archives of just the database-friendly merged up & downcast db.csvs
1 - BTLvsCTD is a SIO-CalCOFI in-house developed program used to merge CalCOFI bottle sample data with Seabird CTD sensor data.
2 - DECODR is a SIO-CalCOFI in-house program used to process seawater samples analyzed at-sea and ashore.
Appendix -CalCOFI CTD FinalQC Zip Folder structure: e.g. 20-1604SH_CTDFinalQC.zip
\ (zip root)
20-1604SH_Data_Processing_Notes.txt
CTD Data Files.pdf
CTD-CSV-Format.pdf
FilenamingAndAscHeaderNotes.txt
\asc-hdr-btl (folder)
20-1604SH_06000530_098d.asc
20-1604SH_06000530_098d.hdr
20-1604SH_06000530_098u.asc ...(typically ~300 files - 75 casts x4: upcast .asc .hdr, downcast .asc hdr
\asc-hdr-btl\original-asc-hdr-btl (folder)
1604001.btl
1604002.btl...
d1604001.asc
d1604001.hdr
d1604002.asc
d1604002.hdr ...(typically ~375 files - 75 casts x5: btls, upcast .asc .hdr, downcast .asc hdr
\csv-plot (folder)
1604SH_06000530_098d.csv
1604SH_06000530_098u.csv
1604SH_06000600_099d.csv ...(typically ~150 files - 75 casts x2: upcast .csv, downcast .csv
\csv-plot\plots\primary (folder)
1604_06000530_098d.pdf
1604_06000530_098d.png
1604_06000530_098u.pdf
1604_06000530_098u.png ...(typically ~150 files - 75 casts x2: upcast .csv, downcast .csv
\csv-plot\plots\secondary (folder)
1604_06000530_098d.pdf
1604_06000530_098d.png
1604_06000530_098u.pdf
1604_06000530_098u.png ...(typically ~150 files - 75 casts x2: upcast .csv, downcast .csv
\db_csvs (folder)
20-1604SH_CTDBTL_001-101D.csv
20-1604SH_CTDBTL_001-101U.csv
\metadata (folder)
20-1604SH-FIVvsChla.pdf
20-1604SH-ISUSVSNO3.pdf
20-1604SH-Ox1MLvsOxBML.pdf
20-1604SH-Ox1UMvsOxBUM.pdf
20-1604SH-Ox2MLvsOxBML.pdf
20-1604SH-Ox2UMvsOxBUM.pdf
20-1604SH-saltoffsets.pdf
20-1604SH_CTDCSV_001-101.xlsx
20-1604SH_DBcoeff_001-101.csv
20-1604SH_DBcoeff_001-101.txt
20-1604SH_EventsF.csv
20-1604SH_Final_001-101.cfg
20-1604SH_hdr_001-101S.csv
20-1604SH_span_001-101S.csv
20-1604SH_xmlcoeff_001-101S.csv
\metadata\regressions (folder)
1604SH-FlVvsChla.jpg
1604SH-ISUSvsNO3.jpg
1604SH-Ox1MLvsOxBML.jpg
1604SH-Ox1UMvsOxBUM.jpg
1604SH-Ox2MLvsOxBML.jpg
1604SH-Ox2UMvsOxBUM.jpg