Oxygens in umol/Kg: Derive umol/Kg routines improved in all related modules: Process Oxygens, CTD to CSV, Derive Bottle Data, Derive CTD Data. All umol/Kg oxygens are derived from ml/L using Seabirds algorithm:
oxyum! = (oxy * 44660) / (sigma + 1000)
Running the [Derive Bottle Data] & [Derive CTD Data] will work on t.csvs, preserving point-check datacodes. These routines will add umol/Kg values and other derived values in either the bottle or CTD cols. BE AWARE - if you derive/rederive t.csv file values and not sta.csvs file values, the files may be different so use cautiously.
umol/kg datacodes are the same as bottle oxygen ml/L datacodes since it is derived from that value.
Column labels in data products were updated to µmol/Kg & µM where appropriate, particularly in the Data Reports.
Tweaks to DICs: Reordered the DIC database csv so importing into the database does not require rearranging or adding columns. If DIC filename has a space, you will now be prompted to fix it. Most in-house applications do not like filenames with spaces.
Two DICS modules moved to Database tab - create DIC database csv & create Dickson csv. [Create Dickson csv] searches all the selected sta.csvs and creates a single csv with all the DIC bottle sample data to give to Dickson's lab. [Create DIC Database csv] will combine station csvs with Dickson/Cervantes DIC+TA csv for importing into the database.
Fixed: Extra bottle (recind 6) derived data removed or not calculated.
Fixed: bottle csv to database will include 0m depths
Tweak: cast csv to database calculates distance for all stations, including SCCOOS'
Fixed: Dynamic Height calculating erroneously for extra bottles, recind 6; fix = no value
Fixed: Cast database csv is missing secchi value - fixed.
Fixed: Integrated 14C col was empty - fixed
Fixed: Wind_Spd is missing from cast to database csv - fixed
New feature: DIC database csv module. Merges Cervantes DIC csv (export the tabular portion of the Excel file to csv) with sta.csvs bottle depths with matching DIC bottles.
Tweak: Made the Cast to database and Bottle to database modules create csvs in the same format as CIFT so the Access import specs will work for either.
New features, modules, & bug fixes:
Nutrient QC - the output file to includes both uncorrected and corrected concentrations; if the difference between the corrected and uncorrected concentration is less than the minimum detection limit (MDL), no correction is applies; corrections (including 0 "non-corrections") are tabulated at the bottom of each run results. If the import corrected nutrients option is checked, the corrected nutrient concentrations will be imported into the station.csvs. Ammonia will use the 'NH4 Check' concentration value, not the RMNS or '99' NH4 concentration to correct the ammonia. This concentration is also tabulated at the bottom of each run to keep track of the correction, if any, that is applied.
Bug fix: module was expecting nutrient files to be in the nut subdir - now the module will allow nutrients to be in a subdir under the nut subdir and still find the other files (sta.csvs & casts.csv)
To reprocess CTD data with bottle oxygen in um/Kg, a DECODR utility for inserting additional columns into existing point-checked, annotated sta.csvs & casts.csv was created. The new DECODR module under the CSV tab - [Add um/kg cols to csvs] updates older cruises whose csvs are missing columns for DIC, Ttl_Alk, and um/kg columns (5: OxBumKg=bottle O2, Ox1umKg=CTD Ox1, Ox2umKg=CTD Ox2, OumDC=DataCode based on bottle OxB (ml/L) datacode, OumPI=Point-check code for future use.
Rerunning the O2 module with sta.csv update checked will populate the new columns. Since only observed data is used to correct CTD sensor data, the module uses & updates the non-standard level sta.csvs. Updating the station t.csvs standard levels with um/Kg may be added to [Derive Bottle Data] module later.
Other fixes - lots of reformatting done to sta & casts csvs so visual parsing of columns with a text editor is easier. Changed some column widths to match headers in both sta & casts csvs.
Reporting CTD oxygen(s) in umole/Kg from the Seabird sensor(s) is standard practice. Reporting bottle oxygens in um/Kg in the CalCOFI Data Reports will formally begin in 2013 (1301).
Using draw temperatures saved by CESL (CalCOFI Electronic Sample Log) into the sta.csvs, we can calculate sigma-theta for each O2 sample at draw time. Using this value, pressure=0, & CTD salinity, the ml/L values are converted to umoles/Kg using Seabird's conversion algorithm (Seabird Application Note 64):
[µmole/Kg] = [ml/L] * 44660 / (sigma_theta(P,T,S) + 1000); P = 0
CESL1304 will generate the latest casts & sta.csv formats which allow oxygen values in um/Kg to be archived.
BtlVsCTD1304 will compile the CTD sensor values plus the bottle values, include O2 in um/Kg. This allows bottle oxygen in um/Kg to be regressed against sensor values. The combined CTD+bottle data csv includes: CTD primary & secondary oxygen sensor values in ml/L & um/Kg. Plus bottle corrected CTD oxygen values - cruise-average corrected oxygen in ml/L & um/Kg; station-corrected oxygen in ml/L & um/Kg. Cruise-average corrected = CTD sensor data vs all cruise bottle oxygen regression coefficients are applied (n = ~1300 samples) . Station-corrected = primary & secondary CTD oxygen sensor data are corrected using the corresponding station bottle oxygen samples only (n = ~20 samples). Station-corrected CTD oxygen values are considered the best since station-to-station variability in sensor performance is addressed. Stations that do not have adequate calibration samples are cruise-corrected only and those CTD oxygen values are the best available.
For the µmole/Kg conversion, there is disagreement in the scientific community about the conversion constant 44660:
The value 44660 is exact for oxygen gas.
The value 44615 is the average value for atmospheric gas (N2,O2,Ar,H2O,CO2,...). It is not exact for any individual gas, but has been used historically by oceanographers.
The argument distills to exact versus historic, with oceanographers split; Sea-Bird uses 44660 in all calculations."
The sta.csv respacer module that reformats the column alignment for easy visual parsing of sta.csv will also work on the casts.csv. The module name was changed from [Sta.csv respacer] to [CSV respacer] and the tooltip mouseover description was updated to reflect the change.
Additions/Updates:
[ casts.csv>castdb.csv ] will generate an Access-ready cast-table importable csv. The cst_cnt should increment properly - it will use a database you select to set the beginning cst_cnt OR you can key-enter your desired start cst_cnt (for updates). Report Order & Order Occupied are user-selectable.
[ sta.csvs>bottledb.csv ] will generate an Access-ready bottle-table importable csv. The cst_cnt & btl_cnt should increment - it will use a database you select to set the beginning btl_cnt & cst_cnt OR you can key-enter your desired starting btl_cnt and/or cst_cnt (for updates). Please note that each text field is treated independently so you can enter a starting btl_cnt without a cst_cnt or vice-versa BUT the blank cnt will be set by the user-selected database.
Report Order & Order Occupied are user-selectable as well as 'shallow to deep' or 'deep to shallow' depth order.
Worked on designing a station summary html page generating module. Combining information for casts.csv, sta.csv, CTD sensor ranges (span.csv), & event log, the new Sta Rpt module creates a separate summary page for each station. General station information, CTD cast info, seawater sampling, and station activities are tabulated. This will be used in conjunction with the station activities webpage created using NMFS station activities database. Providing more detailed information about station sampling and station activity sequence.
Prodo data going into the IEH and Prodo Report modules needed a column index fix. The addition of oxygen in um/kg moved the prodo columns over two so indexing had to be adjusted.
Fixed some glitches introduced yesterday when modifying the update csv routines, mainly the update casts.csv. It was updating the casts.csv after every station instead of once at the end of the run, so a casts.csv bak file was attempted every sta. Moving the casts.csv updater outside the station loop was the simple fix.
Oxygen module: adjusted the data quality criteria for flagging disagreement with CTD oxygens.
0.00 - 0.04 = no stars
0.05 - 0.14 = *
0.15 - 0.24 = **
0.25+ = ***
Plus the total *'s/bottle are printed at the end of the .out data processing report. Hopefully, the comparisons to CTD oxygens plus flagging disagreements will help troubleshooting at sea and improve data quality.
Stopped using the Name function to rename bak files and implemented a simple read/write bak file routine in WriteNuCastCsv & WriteStaCsv, used to update the casts.csv & sta.csv, when checked.
The Oxygen data processing output has additional columns: added CTD Ox0 ml/l, OxBtl-CTDOx0 ml/l, CTD Ox1 ml/l, OxBtl-CTDOx1 ml/l, OxBtl in um/kg - converted ml/l using Seabird's routine. If the bottle-ctd oxygen is greater than 0.45 it gets one star; if it is greater than 0.9 it gets two stars.
[µmole/Kg] = [ml/L] * 44660 / (sigma_theta(P,T,S) + 1000) (P = 0; T = draw T, S = Btl or CTD S1 or 2)
Chl - fixed a col error, now it puts the tau & F values in the right column. Still needs more work on pre & post-cruise + ave value entries.
Prodo - fixed the column update misalignment problem created by Ox um/kg additional columns; added scin data (counts & eff%) to sta.csv depths; improved the data processing coefficient-saving and use, when ave spec act &/or blank are entered on the form. It will remember your last run values and whether you checked the box to use ave values. The values are saved in ...\YYMM\YYMMprodcoef.txt file. If you want to use values added during the scintillation asc file import then uncheck the box.
The 89 column station csv format was put in service on CalCOFI 1301SH. CESL & DECODR have been updated to allow 82 (pre-1210), 84 (1210), & 89 (1301+) column station csvs. This will allow the addition of oxygen in um/kg to be included. These extra columns were added near the end so only effect the comments columns.
DECODR will not update the nutrient datacodes when processing nutrients with csv updater checked. Nutrient & chlorophyll datacodes will not be updated when CTD data is added to sta.csvs. Ammonia data will be coded '4' on offshore stations at depths below 200m unless already datacoded '9'.
Other updates: terpling bottle data 'broke' when changing the recind '3' to a range - '>2 <6' - so it was recoded back to '3'. This will cause recind 5 depths to not be used to terple. Recind 5 usually denotes a bottle depth without water samples (mistrip or missed), so use CTD data.
Added an 'Additional Comments' cleanup tool - sometimes duplicate oxygen datacodes get appended so they repeat (ie Ox1S;Ox2S;Ox1S;Ox2S). Under the [CSV] tab is a new module that checks the additional comments column for this issue and saves only the last pair. It checks for duplicate pairs (ie Ox1S;Ox2S; or Ox1C;Ox2C; or Ox1U;ox2U) but not differing pairs (ie Ox1S;Ox2C or Ox1C;Ox2U...). Additional permutations could be added if needed but are not in place currently.
Database module [db connection] was hardwired to G: drive, this has been fixed. Also Btl_Cnt & Cst_Cnt incremental routines were tweaked - they were adding 2 to the initial entry, not 1.
DECODR respacer module was tweaked to handle 82, 84 or 89 column sta.csvs. Other DECODR modules were modified to handle sta.csvs of 82, 84, or 89 columns properly. 82 column sta.csvs are pre-1210 (no DIC or Total Alkalinity); 84 column sta.csvs are 1210-only (include DIC or Total Alkalinity at cols 80 & 81); 89 column sta.csvs will begin 1301, include DIC, Ttl Alkalinity, oxygen in umoles/kg.
With the sta.csv format evolving with the addition of oxygens in umoles/kg, 4 columns were added at column 82 (82-86; to have less impact on the first 81 cols modules). Only the last 3 comment cols are affected, shifting cols 82-84 to 87-89. Bottle oxygen in umoles/kg are not available yet but CTD data files contain primary & secondary oxygens in umoles/kg.
Other programs involved are: CESL (CESL1301.exe), generating sta.csvs of 89 columns; BtlVsCTD (BtlVsCTD1301.exe), generating ctd.csvs with umoles/kg columns for both bottle, primary, & secondary oxygen. Please refer to the individual entries for these other programs.
After discussing some missing data, it became apparent some documentation of how data are added to casts.csv was needed. This guide outlines all 63 columns: col #, col width, format, and the original source of the data ie what program/process adds or updates data to the file. Some metadata columns are still under development. All columns are generated & some columns populated by CESL after samples are taken at-sea but updated during the different phases of the data process. DECODR keeps process logs and backup versions of the data files for all processes/updates applied to either casts.csv or sta.csvs.
casts.csv contains pre & post calibration columns for tau & acid ratio for the fluorometer but chlorophyll data are calculated post-cruise using averages of the two values (pre & post cruise aves). Averages can be calculated 'on-the-fly' so it is not necessary to archive calculated averages. BUT it is prudent to archive the average tau & acid ratio to corroborate the numbers used in the final calculations. It also improves flexibility if a calculated average has to be adjusted for a particular station or run (rare).
So the casts.csv is now 63 columns - see guide - inserting average tau & acid ratio columns (51 & 52) next to post-cruise values. This shifts the former 51 - 61 over to 53 - 63. SIO-CalCOFI software has been adjusted to handle either casts.csv format. CESL, DECODR, PSal, & FLog will update either 61-col (pre-1301) or 63-col casts.csv, using "is col 53 O2" test as a criteria. DECODR modules effected are: mainly O2 processing, but also Prodo processing, Int 14C, & Int Chl which populate cols to the right.
Quick fix in CSV to IEH module: header 1 'leg' was being padded with a leading space when already padded by leading zero. Quick check for two digits before padding with leading space fixed the glitch. Also verified Prodo light levels were going into casts.csv - they are. When the prodo processing module is run & update csv is checked, the sta.csvs & casts.csv are updated. Note that actual surface prodo tube incubation light level (~53%; from the prodo.cfg) goes into the casts.csv; the 'desired' surface light level (~92%; to get the bottle at 2m) is added to the sta.csv. The light screen on surface prodo tube prevent 'frying' the samples on sunny days.
Tweaked the latest DECODR Database module (Database tab) - users need to run the [db Connection] 1st to get the bottle and cast counts when using casts.csv or sta.csvs to create a database-friendly import file. DECODR was hardwired to a specific database file. Now it prompts users to select the database to use - click [db Connection], select the database (Q:\database is the default path), exit the form, then run the [Sta.csvs > db csv] or [Casts.csv > db] csv modules to generate properly indexed bottle & cast csvs to import.
Fixed: there was a problem when adding CTD data & dynamic heights to non-terpled sta.csvs. If there were duplicate surface bottles shallower than the main surface bottle, they were dropped when calculating dynamic height without terpling. The solution was to introduce a 0m depth into the array that holds sta.csv as the top bottle. The 0m depth is not saved unless the standard level option is selected.
It is also important to remember to recind '6' or '9' extra bottles otherwise DECODR will drop duplicate depths - the last one with recind 3 will be used. This is not a flaw so no workaround is in place. Duplicate bottles will be dropped if duplicates are not flagged accordingly (6 or 9).
DECODR's data report module was reporting blank ammonia values as 0.0. The routine was fixed to report all blank values as blanks. This particular scenario - all the ammonia data >200m were flagged '9' (ignore) so interpolated standard level ammonia data below 200m were blank.
Surfer horizontal & vertical plot instruction web pages were edited.
DECODR will build a variety of horizontal and vertical plots from csvs automatically but understanding how to generate plots manually is useful. Before exporting to Illustrator, adjustments to the contours such as smoothing, line thickness, and contour interval can improve the plot and reduce hand editing. The Surfer Plots page describes how to customize the templates for both horizontal and vertical plots with new data. DECODR does most the work but knowing how it interacts with Surfer is worth documenting.
Generating vertical plot csvs from sta.csvs was 'fine-tuned' to generate vertical section plots comparable to those generated by a data report. The main solution was to exclude data deeper than 525m - 3500m & basin casts data - from the vertical section plot csvs. Vertical plot csvs were also inverted (shallow to deep) and rounded (data match data report format). Both horizontal and vertical plot csvs generating routines check the data codes for good values.
Vertical section plot.csvs were generated in order occupied (001-075). This update changes the plot.csv data to go from nearshore to offshore ie data report order so vertical plot csvs start nearshore. This update also inverts the depths to match the legacy version - shallow to deep vs sta.csvs deep to shallow.
Integrated the Fisheries' zooplankton and sta activities database table formatting modules from CIFT into DECODR. Under the Database tab - a new module that will ingest the exported zooplankton displacement volume table from the Fisheries database, formatting it for importing into our hydrographic database. A second module ingests the exported station activities table to use online for tabulating station activities. Used in conjunction with the event log.
added two cols to the sta.csv format for DIC & TA, changing the format from 82 to 84 cols. DECODR can handle both versions of the sta.csv, using TO_DO in either col 82 or 84 to distinguish the two formats. Fixed a glitch that added two commas to the end of the 82 col sta.csv - 82 or 84 test failed initially in the respacer reformat routine.
fine-tuned some glitches with interpolated standard level bottle data calculations ie missing data on OSLs
fine-tuned NH4 datacoded "4"
fine-tuned 14C assimilation prodo precision & column alignment for values < 1 in ieh data product
fixed the data report txt to html header - it was cutting off the sta decimal value (49. instead of 49.0)
wrote a new module that combines sta.csvs with DIC bottles for importing into Access and generating DIC Data Reports
fixed the 200/500 dynamic height extrapolation module so values on shallow station would calculate properly. Solution was inverting the format from deep to shallow to shallow to deep (sta.csv vs ieh format difference).
In order to use the datacodes to control which values are used in an interpolation, the 'terple' routine was switched from row-oriented to col-oriented. Instead of starting at surface and going to terminal depth, the routine starts with a particular datatype, say salts, then interpolates standard depth values for that column. Using the data codes as criteria on which salt values to use. This is a significant change from the legacy terple routine which was changed from quadratic interpolation to linear interpolation and now col-oriented vs row-oriented.
Generating CalCOFI hydrographic database cast & bottle csvs required generating an IEH then CIFTing into two csvs. With the migration to casts.csv & sta.csvs, the ability to generate database-friendly cast & bottle csvs directly eliminates the need for the IEH intermediate step. This module is online and working - generating cast & bottle csvs with proper labeling for direct import into the hydrographic database. It is possible to make DECODR update the database directly, saving a few steps, but this seems 'dangerous' since you cannot undo a bad import.
The sta.csvs, generated by CESL during sample drawing on the cruise, were 82-cols wide. DIC & TA values are now added to the sta.csvs in cols 80 & 81 with DIC bottle # in col 79, generating a new 84-col sta.csv format. This will integrate the DIC data with other bottle data and allow inclusion of those values into data products such as the hydrographic database or data reports.
Under the CSV tab, there is a module to generate a merged DIC stations csv with all cast depths and DIC bottle #'s for the point-checking database. The csv can be sent directly to Dickson's DIC analytical lab for data import/key-entry. Or imported into the point-checking database where the DIC Report query will generate forms for Dickson's analysts with the bottle data formatted.
CTD-derived data, such as sigma-theta, oxsat, dyn height, are available using Seabird's data processing suite. But to get comparable-compatible data to replace bad bottle-derived data, it is necessary to use the bottle algorithms on CTD values. DECODR now uses the same routines for both bottle & CTD data when populating the sta.csvs with derived data. This allows us to use CTD-derived data to QC the bottle-derived data & replace bad values with CTD values. This change is only is the sta.csvs and has not been implemented in the BltVsCTD CTD+bottle csvs yet. The CTD 1m binavg derived-data are calculated using the Seabird algorithms so will differ from the bottle-derived data.
