To reprocess CTD data with bottle oxygen in um/Kg, a DECODR utility for inserting additional columns into existing point-checked, annotated sta.csvs & casts.csv was created. The new DECODR module under the CSV tab - [Add um/kg cols to csvs] updates older cruises whose csvs are missing columns for DIC, Ttl_Alk, and um/kg columns (5: OxBumKg=bottle O2, Ox1umKg=CTD Ox1, Ox2umKg=CTD Ox2, OumDC=DataCode based on bottle OxB (ml/L) datacode, OumPI=Point-check code for future use.
Rerunning the O2 module with sta.csv update checked will populate the new columns. Since only observed data is used to correct CTD sensor data, the module uses & updates the non-standard level sta.csvs. Updating the station t.csvs standard levels with um/Kg may be added to [Derive Bottle Data] module later.

Other fixes - lots of reformatting done to sta & casts csvs so visual parsing of columns with a text editor is easier. Changed some column widths to match headers in both sta & casts csvs.

Reporting CTD oxygen(s) in umole/Kg from the Seabird sensor(s) is standard practice. Reporting bottle oxygens in um/Kg in the CalCOFI Data Reports will formally begin in 2013 (1301).
Using draw temperatures saved by CESL (CalCOFI Electronic Sample Log) into the sta.csvs, we can calculate sigma-theta for each O2 sample at draw time. Using this value, pressure=0, & CTD salinity, the ml/L values are converted to umoles/Kg using Seabird's conversion algorithm (Seabird Application Note 64):
[µmole/Kg] = [ml/L] * 44660 / (sigma_theta(P,T,S) + 1000); P = 0

CESL1304 will generate the latest casts & sta.csv formats which allow oxygen values in um/Kg to be archived.
BtlVsCTD1304 will compile the CTD sensor values plus the bottle values, include O2 in um/Kg. This allows bottle oxygen in um/Kg to be regressed against sensor values. The combined CTD+bottle data csv includes: CTD primary & secondary oxygen sensor values in ml/L & um/Kg. Plus bottle corrected CTD oxygen values - cruise-average corrected oxygen in ml/L & um/Kg; station-corrected oxygen in ml/L & um/Kg. Cruise-average corrected = CTD sensor data vs all cruise bottle oxygen regression coefficients are applied (n = ~1300 samples) . Station-corrected = primary & secondary CTD oxygen sensor data are corrected using the corresponding station bottle oxygen samples only (n = ~20 samples). Station-corrected CTD oxygen values are considered the best since station-to-station variability in sensor performance is addressed. Stations that do not have adequate calibration samples are cruise-corrected only and those CTD oxygen values are the best available.

From Seabird Application Note No. 64: SBE 43 Dissolved Oxygen Sensor
"[µmole/Kg] = [ml/L] * 44660 / (sigma_theta(P,T,S) + 1000)

For the µmole/Kg conversion, there is disagreement in the scientific community about the conversion constant 44660:
The value 44660 is exact for oxygen gas.
The value 44615 is the average value for atmospheric gas (N2,O2,Ar,H2O,CO2,...). It is not exact for any individual gas, but has been used historically by oceanographers.
The argument distills to exact versus historic, with oceanographers split; Sea-Bird uses 44660 in all calculations."

The sta.csv respacer module that reformats the column alignment for easy visual parsing of sta.csv will also work on the casts.csv. The module name was changed from [Sta.csv respacer] to [CSV respacer] and the tooltip mouseover description was updated to reflect the change.

Fixes - fixed some column alignment (esthetics) issues in casts.csv & sta.csvs.
Added an update-reinitialize feature when editing the cfg - if a new cruise is entered into the cruise field, it will prompt you to update all related fields and reset the station & vial counts to 1.

DECODR's legacy module to generate an IEH from sta.csvs was mis-formatting prodo dark samples when the value was greater than 1.0 - fixed.

Fixed - weather data entry bottom depth format issue when depth is less than 1000m

Derive Bottle Data will derive data into t.csvs

[CSV to IEH] module will report Sigma-theta and Oxy Pct in the wild columns if checkbox is checked and values are present in the sta.csvs

Fine-tuning of older vs newer file formats ie different casts.csv & sta.csvs column changes.

Additions/Updates:
[ casts.csv>castdb.csv ] will generate an Access-ready cast-table importable csv. The cst_cnt should increment properly - it will use a database you select to set the beginning cst_cnt OR you can key-enter your desired start cst_cnt (for updates). Report Order & Order Occupied are user-selectable.

[ sta.csvs>bottledb.csv ] will generate an Access-ready bottle-table importable csv. The cst_cnt & btl_cnt should increment - it will use a database you select to set the beginning btl_cnt & cst_cnt OR you can key-enter your desired starting btl_cnt and/or cst_cnt (for updates). Please note that each text field is treated independently so you can enter a starting btl_cnt without a cst_cnt or vice-versa BUT the blank cnt will be set by the user-selected database.
Report Order & Order Occupied are user-selectable as well as 'shallow to deep' or 'deep to shallow' depth order.

The CTDatDepth PST & UTC in the casts.csv was incorrect when generated by CESL's csv rebuilding module (regenerates the casts.csv & all the sta.csvs automatically). This module was developed to batch recreate all the csvs so new columns could be added to older cruises. Unfortunately, the ATDepth PST date & time was UTC and AtDepth UTC was UTC+8. CESL was fixed, new casts.csvs generated for 1104, 1108, 1110, 1202, 1203. The AtDepthDate_PST & AtDepthDate_UTC columns were cut & pasted into the working casts.csvs so point-checking data codes & quality codes would remain intact. New IEHs & Data Reports for all published cruises should be updated as well as database dates and times.

Some major revisions to the casts.csv to cast database.csv module: added a report order option since this is the desired station order; order_occ was the default and is also still available via checkbox. Added a Castnum textbox - if this textbox has a Castnum (database Cast table primary ID), the module will not check or ask for the reference database that it typically would append ie add the next set of castnum values chronologically. This feature allows new final data to be imported, replacing old preliminary data records using an update query.
Will work on sta.csvs to bottle database table next after fixing the casts.csv AtDepthPST glitch in CESL.

Worked on designing a station summary html page generating module. Combining information for casts.csv, sta.csv, CTD sensor ranges (span.csv), & event log, the new Sta Rpt module creates a separate summary page for each station. General station information, CTD cast info, seawater sampling, and station activities are tabulated. This will be used in conjunction with the station activities webpage created using NMFS station activities database. Providing more detailed information about station sampling and station activity sequence.

Prodo data going into the IEH and Prodo Report modules needed a column index fix. The addition of oxygen in um/kg moved the prodo columns over two so indexing had to be adjusted.

Fixed some glitches introduced yesterday when modifying the update csv routines, mainly the update casts.csv. It was updating the casts.csv after every station instead of once at the end of the run, so a casts.csv bak file was attempted every sta. Moving the casts.csv updater outside the station loop was the simple fix.
Oxygen module: adjusted the data quality criteria for flagging disagreement with CTD oxygens.
0.00 - 0.04 = no stars
0.05 - 0.14 = *
0.15 - 0.24 = **
0.25+ = ***

Plus the total *'s/bottle are printed at the end of the .out data processing report. Hopefully, the comparisons to CTD oxygens plus flagging disagreements will help troubleshooting at sea and improve data quality.

Stopped using the Name function to rename bak files and implemented a simple read/write bak file routine in WriteNuCastCsv & WriteStaCsv, used to update the casts.csv & sta.csv, when checked.

The Oxygen data processing output has additional columns: added CTD Ox0 ml/l, OxBtl-CTDOx0 ml/l, CTD Ox1 ml/l, OxBtl-CTDOx1 ml/l, OxBtl in um/kg - converted ml/l using Seabird's routine. If the bottle-ctd oxygen is greater than 0.45 it gets one star; if it is greater than 0.9 it gets two stars.
[µmole/Kg] = [ml/L] * 44660 / (sigma_theta(P,T,S) + 1000) (P = 0; T = draw T, S = Btl or CTD S1 or 2)

Chl - fixed a col error, now it puts the tau & F values in the right column. Still needs more work on pre & post-cruise + ave value entries.

Prodo - fixed the column update misalignment problem created by Ox um/kg additional columns; added scin data (counts & eff%) to sta.csv depths; improved the data processing coefficient-saving and use, when ave spec act &/or blank are entered on the form. It will remember your last run values and whether you checked the box to use ave values. The values are saved in ...\YYMM\YYMMprodcoef.txt file. If you want to use values added during the scintillation asc file import then uncheck the box.

The prodo module has had some significant updates: first, there is a new scintillation asc import module that saves the counts into a station-bottle indexed csv. It uses the specific activity and blank values from the casts.csv unless over-ridden by user-entered values in the asc import/export module.
The Prodo data processing module uses this csv (or up to 3 variations, if needed) to generate the prodo data report. The update sta+casts.csvs checkbox will add prodo values to both prodo station csvs & prodo station cast entries in the casts.csv. There is also a specific activity & blank feature in the prodo data processing module that will supersede the csv values if that option is checked (checkbox must be checked). This allows individual stations to be processed with different values, if necessary.
The prodo data processing file still uses the prd### files generated by the PIC program at sea. It also expects start & end incubation times to be key-entered. If these values are missing, the user will be prompted to use LAN & Civil Twilight from the prd### file. These values are entered when running the PIC program.
All ties to 22 files have been removed (hopefully).
Files needed: YYMMscin.asc > YYMMscin.csv; prd###, casts.csv, sta.csvs. Prodo.cfg is needed by DECODR for use by the PIC module at-sea. Light levels, default specific activity & blank values from the prodo.cfg will be embedded into the prd### by PIC. It is important to remember to save any changes to the bottle numbering as the final step in PIC before generating the prd###.

The 89 column station csv format was put in service on CalCOFI 1301SH. CESL & DECODR have been updated to allow 82 (pre-1210), 84 (1210), & 89 (1301+) column station csvs. This will allow the addition of oxygen in um/kg to be included. These extra columns were added near the end so only effect the comments columns.

A new Thermo Scientific Nanopure filtration system arrives 2 Jan 2013. It will sail on CalCOFI cruises, fed by the Milli-Q system, for mixing nutrient standards.

DECODR will not update the nutrient datacodes when processing nutrients with csv updater checked. Nutrient & chlorophyll datacodes will not be updated when CTD data is added to sta.csvs. Ammonia data will be coded '4' on offshore stations at depths below 200m unless already datacoded '9'.

Other updates: terpling bottle data 'broke' when changing the recind '3' to a range - '>2 <6' - so it was recoded back to '3'. This will cause recind 5 depths to not be used to terple. Recind 5 usually denotes a bottle depth without water samples (mistrip or missed), so use CTD data.

Added an 'Additional Comments' cleanup tool - sometimes duplicate oxygen datacodes get appended so they repeat (ie Ox1S;Ox2S;Ox1S;Ox2S). Under the [CSV] tab is a new module that checks the additional comments column for this issue and saves only the last pair. It checks for duplicate pairs (ie Ox1S;Ox2S; or Ox1C;Ox2C; or Ox1U;ox2U) but not differing pairs (ie Ox1S;Ox2C or Ox1C;Ox2U...). Additional permutations could be added if needed but are not in place currently.

 

CESL1301.exe will generate sta.csvs with columns to accommodate oxygens in umoles/kg from bottles, primary & secondary CTD oxygen sensors. These columns will be updated by DECODR.

CTD asc files contain CTD primary & secondary (when installed) oxygen measurements in umoles/kg. These data were not included in the CTD-bottle data merged csvs. With the mandate to include oxygen in umoles/kg as part of our dataset, these columns are now included in the ctd.csvs. Eight additional columns have been added: Ox1uM, Ox1uM_CruiseCorr, Ox1uM_StaCorr, Ox2uM, Ox2uM_CruiseCorr, Ox2uM_StaCorr, OxuMAve_StaCorr, plus the OxBuM column for the bottle value.

Reporting of bottle oxygens in umoles/kg should begin on CalCOFI 1301SH (Jan 2013). With the addition of bottle oxygens in umoles/kg, bottle-corrected CTD oxygens in umoles/kg will populate the ctd.csvs. These data will be archive with final data.

CalCOFI started running oxygens with the ODF auto-titrator in Oct 2003 so it may be possible to generate umoles/kg values from archived runs as time permits (which it rarely does). Further investigation of this possibility is necessary ie whether older cruise bottle oxygens can be reprocessed to umoles/kg easily.

Database module [db connection] was hardwired to G: drive, this has been fixed. Also Btl_Cnt & Cst_Cnt incremental routines were tweaked - they were adding 2 to the initial entry, not 1.

DECODR respacer module was tweaked to handle 82, 84 or 89 column sta.csvs. Other DECODR modules were modified to handle sta.csvs of 82, 84, or 89 columns properly. 82 column sta.csvs are pre-1210 (no DIC or Total Alkalinity); 84 column sta.csvs are 1210-only (include DIC or Total Alkalinity at cols 80 & 81); 89 column sta.csvs will begin 1301, include DIC, Ttl Alkalinity, oxygen in umoles/kg.

With the sta.csv format evolving with the addition of oxygens in umoles/kg, 4 columns were added at column 82 (82-86; to have less impact on the first 81 cols modules). Only the last 3 comment cols are affected, shifting cols 82-84 to 87-89. Bottle oxygen in umoles/kg are not available yet but CTD data files contain primary & secondary oxygens in umoles/kg.

Other programs involved are: CESL (CESL1301.exe), generating sta.csvs of 89 columns; BtlVsCTD (BtlVsCTD1301.exe), generating ctd.csvs with umoles/kg columns for both bottle, primary, & secondary oxygen. Please refer to the individual entries for these other programs.

After discussing some missing data, it became apparent some documentation of how data are added to casts.csv was needed. This guide outlines all 63 columns: col #,  col width, format, and the original source of the data ie what program/process adds or updates data to the file. Some metadata columns are still under development. All columns are generated & some columns populated by CESL after samples are taken at-sea but updated during the different phases of the data process. DECODR keeps process logs and backup versions of the data files for all processes/updates applied to either casts.csv or sta.csvs.

casts.csv contains pre & post calibration columns for tau & acid ratio for the fluorometer but chlorophyll data are calculated post-cruise using averages of the two values (pre & post cruise aves). Averages can be calculated 'on-the-fly' so it is not necessary to archive calculated averages. BUT it is prudent to archive the average tau & acid ratio to corroborate the numbers used in the final calculations. It also improves flexibility if a calculated average has to be adjusted for a particular station or run (rare).

So the casts.csv is now 63 columns - see guide - inserting average tau & acid ratio columns (51 & 52) next to post-cruise values. This shifts the former 51 - 61 over to 53 - 63. SIO-CalCOFI software has been adjusted to handle either casts.csv format. CESL, DECODR, PSal, & FLog will update either 61-col (pre-1301) or 63-col casts.csv, using "is col 53 O2" test as a criteria. DECODR modules effected are: mainly O2 processing, but also Prodo processing, Int 14C, & Int Chl which populate cols to the right.

Quick fix in CSV to IEH module: header 1 'leg' was being padded with a leading space when already padded by leading zero. Quick check for two digits before padding with leading space fixed the glitch. Also verified Prodo light levels were going into casts.csv - they are. When the prodo processing module is run & update csv is checked, the sta.csvs & casts.csv are updated. Note that actual surface prodo tube incubation light level (~53%; from the prodo.cfg) goes into the casts.csv; the 'desired' surface light level (~92%; to get the bottle at 2m) is added to the sta.csv. The light screen on surface prodo tube prevent 'frying' the samples on sunny days.

Tweaked the latest DECODR Database module (Database tab) - users need to run the [db Connection] 1st to get the bottle and cast counts when using casts.csv or sta.csvs to create a database-friendly import file. DECODR was hardwired to a specific database file. Now it prompts users to select the database to use - click [db Connection], select the database (Q:\database is the default path), exit the form, then run the [Sta.csvs > db csv] or [Casts.csv > db] csv modules to generate properly indexed bottle & cast csvs to import.

Fixed: there was a problem when adding CTD data & dynamic heights to non-terpled sta.csvs. If there were duplicate surface bottles shallower than the main surface bottle, they were dropped when calculating dynamic height without terpling. The solution was to introduce a 0m depth into the array that holds sta.csv as the top bottle. The 0m depth is not saved unless the standard level option is selected.

It is also important to remember to recind '6' or '9' extra bottles otherwise DECODR will drop duplicate depths - the last one with recind 3 will be used. This is not a  flaw so no workaround is in place.  Duplicate bottles will be dropped if duplicates are not flagged accordingly (6 or 9).

CESL DIC station sample highlight was tweaked: sta 80.55 should highlight all depths including the requested 15m mooring calibration bottle; sta 80.80 41m bottle added. Improvements of this feature (DIC depth highlighting) should make the program work better. Also CESL help (F1) is working, describing most aspects of the program.

DECODR's data report module was reporting blank ammonia values as 0.0. The routine was fixed to report all blank values as blanks. This particular scenario - all the ammonia data >200m were flagged '9' (ignore) so interpolated standard level ammonia data below 200m were blank.

Surfer horizontal & vertical plot instruction web pages were edited.

DECODR will build a variety of horizontal and vertical plots from csvs automatically but understanding how to generate plots manually is useful. Before exporting to Illustrator, adjustments to the contours such as smoothing, line thickness, and contour interval can improve the plot and reduce hand editing. The Surfer Plots page describes how to customize the templates for both horizontal and vertical plots with new data. DECODR does most the work but knowing how it interacts with Surfer is worth documenting.

Generating vertical plot csvs from sta.csvs was 'fine-tuned' to generate vertical section plots comparable to those generated by a data report. The main solution was to exclude data deeper than 525m - 3500m & basin casts data - from the vertical section plot csvs. Vertical plot csvs were also inverted (shallow to deep) and rounded (data match data report format). Both horizontal and vertical plot csvs generating routines check the data codes for good values.

Zooplankton table from the Master database exported as csv then zipped (by JLW) added to website. Link added to:

Available Data page - http://calcofi.com/data/195-available-data.html

Net Sample page - http://new.data.calcofi.com/index.php/zooplankton

Vertical section plot.csvs were generated in order occupied (001-075). This update changes the plot.csv data to go from nearshore to offshore ie data report order so vertical plot csvs start nearshore. This update also inverts the depths to match the legacy version - shallow to deep vs sta.csvs deep to shallow.

DECODR1210.exe 2 Oct 2012 18:15