There have been quite a few tweaks of the DECODR program since last entry. Here is a list of the main addition or changes:

UTC-PST issues - when using RV Sally Ride or any shipboard computer to collect CTD data, the .btl bottle closure times are in the computer's time. If the computer is UTC, the bottle closures imported into the sta.csvs will be UTC. Now BtlVsCTD will replace .btl times with CTD PST when CTD.csv data are imported into the sta.csvs. But on cruises, like 1711, where sta.csvs have been point checked, re-importing the CTD.csvs could negate point-checking corrections. DECODR has a new module in the CSV tab that will update the sta.csvs or staT.csvs with CTD PSTs without changing any other column.

Nutrients - multiple RMNS will be saved: previous versions, only the last RMNS run would be saved to the RMNS tabulation QC file. If additional RMNS runs were done between stations or samples in the same run, only the last one was saved. DECODR will now save and tabulate all RMNS runs.

Nutrients: The zeroing of values below the MDL for nitrite NO2 is working fine except that the ISL values still go in as actual calculated from csv data datacoded "0". Datacodes should follow suit and be "4" when there is an adjacent "4" - fixed. (11/06/2017)

Create New Directories - DECODR can now create all the directories required for a new cruise:

DECODR - new O2 Module - Data Processing & Integration of RINKO O2 Data with CalCOFI CTD Data

Seasave & SBE Data Processing suite which we use to process our Seabird 911+ CTD data does not include RINKO data processing capabilities (as of Sept 2017). At-sea, the RINKO III deployed on CalCOFIs 1210NH & 1611SR were attached to two channels. Removing the secondary SBE43 oxygen sensor and altimeter were necessary to free up two channels (since CTD temperature is used for RINKO oxygen calculations, it's possible that only the RINKO current - single channel - may need to be interfaced).
Using a Matlab script from the Australian Ocean Data Network (AODN) and Integrated Marine Observing System (IMOS) & white paper ("Update of CTD oxygen data for the cruises MR07-04 and MR07-06" September 24, 2009 by Hiroshi Uchida (JAMSTEC)). I adapted the formula for calculating RINKO O2 saturation, mg/l, & umol into the units we typically report - ml/l, %sat, umol/Kg:

'depthR! is actually pressure, A!-F! are coefficients from RINKO manufacturer calibration
DissO2! = DissO2! * (1 + E! * depthR! / 100)
DissO2! = A! / (1 + D! * (T! - 25)) + B! / ((RINKOV! - F!) * (1 + D! * (T! - 25)) + C! + F!)

'aging foil correction - G & H are 0% & 100% coefficients derived by SIO-CalCOFI at-sea using saturated seawater & sodium sulfite solutions
DissO2! = G! + H! * DissO2!

'pressure correction
DissO2! = DissO2! * (1 + E! * depthR! / 100)
HoldCTDCSV$(rowcnt%, CalcCol%(5)) = Format(DissO2!, "0.000")  '%Sat

T1! = T! + 273.15
logO2star! = -173.4292 + (249.6339 * 100) / T1! + 143.3483 * log(T1! / 100) - 21.8492 * (T1! / 100) + S! * (-0.033096 + 0.014259 * (T1! / 100) - 0.0017 * (T1! / 100) ^ 2)
O2star! = Exp(logO2star!)

'ml/L
O2uM! = (O2star! * (DissO2! * 1.42857) / 100) * 0.7  '/ 1.33  '10ml/l = 13.3mg/l  1 mg/l = 22.391 ml/31.998 = 0.700 ml/l
HoldCTDCSV$(rowcnt%, CalcCol%(4)) = Format(O2uM!, "0.000")  'ml/L

'uMol/Kg
oxyum! = (O2uM! * 44660) / (HoldCTDCSV$(rowcnt%, CalcCol%(7)) + 1000)
HoldCTDCSV$(rowcnt%, CalcCol%(6)) = Format(oxyum!, "0.000")  'uMol/Kg

DECODR1705.exe addresses a problem seen on CalCOFI 1704SH where the nutrient data file was missing the NH4 column. The original routine expect there to be the expected number (and order) of nutrient columns: N+N NO2 PO4 Sil NH4 NO3(virtual). Runs of sta 32 - 38 were missing the NH4 channel so the 6th column was NO3, shifting the NO3 column into the NH4 column. DECODR1705 reads in the "METH" column headers, indexing the nutrients accordingly. Although the nutrient output file format is printed in a particular column order. The nutrient input file can have the nutrient columns in any order and DECODR1705 should parse them properly. This could use more testing but it appears to work when NH4 is missing or the data are "999".

DECODR has always expected the base cruise data-path to be Q:\CODES\YYYY\YYMM\ where Q: is the data drive, YYYY is the year, and YYMM is the year-month associated with the cruise ie 1701. The base cruise data-path and other settings are set in C:\DECODR\decodr.cfg (this path is hard-wired and has to be named exactly as shown).
This is our data acquistion directory structure used at-sea so all systems can contribute to the proper directories over the network. Since data-processing is done throughout the cruise for data-quality control, DECODR has been programmed to find data in these folders.

But since the data-path of completed cruises may be different once ashore, DECODR will now accept any data-path as long as "YYMM\" is the last part of the string ie "Q:\CODES\2010S\2013\1311\". As always, users should double-check the Current Data Path: displayed in DECODR is correct before running any module. Each modules add to the base data-path so the directory structure under the base data-path is fixed, for example using base data directory Q:\CODES\2017\1704\:

• Salinities module adds "SALT\" ie Q:\CODES\2017\1704\SALT\
• Oxygens module adds "O2\" ie Q:\CODES\2017\1704\O2\
• Chlorophyll module adds "CHL\" ie Q:\CODES\2017\1704\CHL\
• Nutrients module adds "NUT\ ie Q:\CODES\2017\1704\NUT\
• Prodos module adds "PRODO\" ie Q:\CODES\2017\1704\PRODO\

Other module file paths are:

• CTD data under "CTD\" ie Q:\CODES\2017\1704\CTD\
• station and casts csvs under "CSV\" ie Q:\CODES\2017\1704\CSV\
• cruise eventlog under "EVENTS\" ie Q:\CODES\2017\1704\EVENTS\
• bottle sample logs (CESL) under "LOGS\" ie Q:\CODES\2017\1704\LOGS\
• contour maps files under "MAPS\"; ie Q:\CODES\2017\1704\MAPS\
• CTD .mrk & .hdr files under "MRK\"  ie Q:\CODES\2017\1704\MRK\
• Processing logs (modules run on all data files) under "PROCESS\" ie Q:\CODES\2017\1704\PROCESS\
• Legacy station cast description file under "STACST\" ie Q:\CODES\2017\1704\STACST\
• Legacy weather file under "WEATHER\" ie Q:\CODES\2017\1704\WEATHER\

Other support directories under the base data folder but not used directly by DECODR:

• "0020" - obsolete IEH-data-processing subdir, remove on cruises after 2011; used for IEH data-processing on cruises earlier than Jan 2012
• "Matlab" - current data-processing subdir for GTool, our Matlab data-quality control visualization tool
• "NCOG" - support dir for NCOG samples
• "Planning" - cruise planning and information files
• "UWay" - underway data directory

CSVs for contour mapping can be generated from different hydrographic data sources: sta.csvs, data reports, IEHs, CTD .asc or .mrk files. A character code is appended to the map csvs distinguishing the various sources: "S" for sta.csvs, "D" for data report, "I" for IEHs, "" for CTD sources. In previous versions of DECODR, the character codes were "turned off" so only map.csvs without the character code were plotted (ie "1607_010m.csv" vs "1607_010mS.csv").

Sta.csvs are the recommended source for contour map data but Data Report are used for derived data plots such as Dynamic Height.

Also changed - when using a Data Report (TXT > csv in DECODR) as a source, the data files previously generated were shallow to deep. But DECODR1703 now inverts the Line.csvs from deep to shallow - horizontal plot csvs are unchanged since they only include one depth such as 10m data

It should be noted that when auto-contouring using DECODR, if the Surfer plot "shrinks" to an illegible thumbnail. It's a range problem with the master file used as a template. I've updated the templates in my C:\CODES\map_masters\ subdir to use local files such as "current_coast_w_islands1.bln" and gridded data from a previous cruise that has been copied to the map_masters subdir. 

Ship codes for RV Sally Ride - SR - and RV Reuben Lasker - RL - are now properly outputted to data products of DECODR.

ISLs values bottles were not being calculated using DECODR1611 after a conditional was changed from "recind = 3" to "recind < 6". So the original conditional "=3" was reinstated until the logic can be worked out. The "<6" conditional was used on 1511 for extra stations without bottles.

Added a event log UTC to PST utility - CalCOFI 1611SR CTD data acquisition computer and bridge event tablet (and all shipboard computers) were set to UTC. The PST column in CESL and CELog were computer time ie UTC so -8hrs needed to be subtracted from the UTC event times. DECODR's "Clean Event Log" button checks the PST vs UTC and if they are the same, it subtracts -8hrs from the PST date/time.

Under the CSV tab, there is also an interactive utility that lets the user correct StartCast, AtDepth, EndCast dates-times in the casts.csv by adding or subtracting 8hrs.

Tweak: "Cruise Ave" O2s are selected by default in the [ CTD BTL to Sta.csv ] module since they are preferred over "Sta Corr" CTD O2s.

The new version of DECODR (v1610a) deals with duplicate stations when generating the database. It puts additional casts at the end of the csv. CalCOFI 1511OC had two stations that were done twice, for example.

Also addressed was a problem with the dynamic height deriver which relies on the SVA column being present.

CESL1607 will generate a separate PDA csv data file if the PDA column is annotated then saved. A PDA summary csv may also be generated from the slog.csvs (CESL saved sample logs) after the cruise using DECODR1608 (or later) PDA to csv module - see DECODR1608 technical blog entry.

DECODR 0m ISL salt datacode SDC is hard-wired to "0" if the bottle salt column is not empty. Currently, there is no way to make it equal to the SDC of the surface bottle, which it mirrors.

DECODR1608 has a new button on the MISC tab labeled “[Export PDA from SLog]”. Click the button and select all the sample logs files then [OK]. It will generate a “YYMM_PDAX.csv” in the LOGS subdir that includes:

CRUISE,LINE,STA,LAT,LON,CAST,EVENT#,Nisk,Sample,Depth,Temp,Salt,PDA,Comments,

CalCOFI 1607,93.3,26.7,32.95756,-117.30430,001,8,05,05,14.8,16.09,33.506,05,,

New features:

• Main menu, new tab: a prodo incubation start & stop dataentry button has been added as [Add Prodo Times] on the bottom of the main tab. I migrated the prodo editor from CODES to DECODR, allowing the editing of LAN & Civil Twilight and entry of Incubation Start and Stop times. This editor updates both the selected prd### base files and the YYMMcasts.csv.
• Oxygens tab – Process O2s: Added a user-editable reagent volume box in the Process O2s data processing module. Previously set to 0.7, this value, loaded from the oxygen coefficients file (Q:\CODES\YYYY\YYMM\YYMMOxCoeff.txt), third line, can be changed before processing. Check the "Low Reagent Vol.= " box when a lower reagent volume calculation is warranted; if unchecked, 2.0ml reagent volume will be used when calculating oxygen concentration from auto-titrator data.
• CIFT-like CSV export utility: when you select a csv or group of similarly-formatted csvs like sample-log.csvs (YYMM###SLog.csv), sta.csvs (YYMM###.csv) or CTD.csvs (YYMM_LLLLSSSS_###u or d.csv), you can select any group of columns and output the results to a new, combined csv. This may be used to export NCOG or DA data from the sample logs; export specific depth data for a cruise, like 10m nutrients; or export specific columns from the CTD.csvs.

Fixes:

1. Nutrient terpling procedure was not using the new datacodes (separate datacodes for each nutrients) when calculating the interpolated value - fixed.
2. When a (mistrip) recind 5 record inhabits the sta.csv, the CTD import routine will populate the record with CTD data but not bottle data columns.
3. When a (mistrip) recind 5 record inhabits the sta.csv and the terple option is selected, the recind 5 record will be ignored so a recind 7 interpolated data record will be created. If the recind 5 record is at an OBS, an ISL record at the same depth will be created.
4. Problems with bottle & CTD derived data in the recind 5 record being recalculated when terpling have been corrected. Data derived during the non-terple updating of sta.csv will be copied to the ISL from the OSL-recind-5.
   Fixes 2-4 should only affect the way DECODR handles recind 5 records. Recind 5 identifies a CTD-data only record.

Two updates:

Main tab: a prodo incubation start & stop data entry button has been added as [Add Prodo Times] on the bottom of the main screen. I migrated the prodo editor from CODES to DECODR, allowing the editing of LAN & Civil Twilight and entry of Incubation Start and Stop times. This editor updates both the selected prd### base files and the YYMMcasts.csv.

Oxygens tab – Process O2s: Added a user-editable reagent volume box in the Process O2s data processing module. Previously set to 0.7, this value, loaded from the oxygen coefficients file (Q:\CODES\YYYY\YYMM\YYMMOxCoeff.txt), third line, can be changed before processing. Check the box when a lower reagent volume calculation is warranted; if unchecked, 2.0ml reagent volume will be used when calculating oxygen concentration from auto-titrator data.

Newest feature: added a low volume coefficient toggle in the oxygen module. When 0.7ml of reagents are added instead of the traditional 2.0ml, the "Low Volume" box under Options should be checked. This affects the oxygen concentration calculation by increasing the sample volume: denominator V(reag) = 0.7 instead of 2.0.

O2  =     [V(samp) - V(blk!)] * N(thio) * 5.598 - DO(reag)
             -------------------------------------------------------------------
                         [V(bot) - V(reag)] / 1000.0

The reagent volume used is appended/printed on the last line of the output file.

The latest version of DECODR is not a major upgrade but version 1511W was updated to 1604 to reference the upcoming cruise. This version will work on cruises 1504 and onward.

As of today, a new Master database has been updated with the latest QC'd DIC data from Dicksons lab.  The zipped file should be dated 3/11/2016.

With the addition of the uM/Kg oxygen data columns added to the hydrographic and primary productivity data reports and web page data products in Jan 2013 (1301). The data products headers listing "STATION" were truncated as either "ST" or "S". This was a simple string parsing error that was fixed in version 1511W. No data were affected but "STATION" labeling in the headers were fixed on all affected files.

FIXED - DECODR's DIC import module "Create DIC Database" module (under the Database tab):

The problem: the user used to be able to select all stations whether or not there were DIC's sampled or not and get a file with only DIC stations and depths listed.  DECODR was outputting every station depth in the resulting csv for the DIC stations, not just DIC depths.  DECODR1511V only outputs the sta.csv records with the DIC bottle column populated, then inserts the DIC.csv data into the output. Essentially merging the sta.csv bottle and CTD data record with DIC/TA/pH data.

The latest tweaks of DECODR - version 1511U: when a standard observed level has a mistrip and there are no bottle samples, CESL gives the record a recind "5", meaning CTD-only data. In previous version of DECODR, when interpolated values were desired, interpolated bottle data would be added to this CTD-data-only record (recind 5) without changing the recind to "7".
In version 1511U, which can be used for cruises 1504+, records with recind "5" are treated the same as recind "6", as an extra "bottle". Recind 5 & 6 records are stored but not used in the interpolations but re-inserted into the output files. Note that the recind "7" record generated by the terple routine will be populated with CTD data.

Fixed: CTD2CSV Data Codes, right side options, will replace data codes for interpolated depths (recind 7; col 79 of sta.csv). The previous version would not replace existing data codes.

Fixed: The Cruise_ID requested by Ed Weber is consistent now. Established by the first station, the database CRUISE_ID will not change.

Example: 2015-10-28-C-32OC for CalCOFI 1511OC, data collection started on 28 Oct 2015

The addition of data quality code columns with similar names as the data caused the CSL additions, particularly the Temperature, to not update. DECODR was indexing the data code column, which is normally empty when data are good, instead of the data column itself ie "Temp1Q" instead of "Temp1". Conditionals were added to distinguish the datacode columns from the data, testing for the "Q" in the header.

Use DECODR1511Q for the new CTD.csvs with data quality code columns.

IntC14 (integrated C14) fixed

Since CalCOFI 1504, 3 new nutrient datacode columns have been added to the sta.csv. This shifted the primary productivity columns over by 3 columns. The 14C integration routine was using the older column values and under-calculating the intC14 value. This has been fixed in DECODR1504a & 1507a, earlier cruises and data are fine. (CalCOFI 1504 & 1507 data have not been distributed with the under-calculated values since this error was found during point-checking.)

Datacode "4" instead of "8"

The sta.csv to database csv module was changed - nutrient data codes "4", "P4", "S4", O4", "T4" were formerly flagged with datacode "8" (questionable) in the output file. The logic being datacode "4" was added to values at or below the mean detection limit (MDL) of the nutrient analyzer. But unless the value is reported, the default value is = 0 which does not need a questionable code. The "4" will be retained in the database csv for reference. A "4" datacode means the measurement was set equal to 0.

DECODR1501m works with CalCOFI 1501 data and will generate the correct data products. DECODR1504a & 1507a will also datacode this way.

"D" footnoting of ISL Salinity

When CTD data are used in place of bottle data, a "D" footnote is used in IEH & Data Reports. There was a bug in earlier versions that would "D" annotate the ISL salinity value when the interpolated salinity was reported. This has been fixed in version 1501 and later.

After trying a "cluster" of nutrient datacodes in column 32 in the 2014 sta.csv format. Individual nutrient datacodes have been inserted into the current 1504 sta.csvs. Phosphate, silicate, nitrate, & nitrite now have separate columns for datacodes, adding three columns to the 2014 format. The column total is now 92 - please refer to the sta.csv format web pages: 2014, 89 columns; 2015, 92 columns.

Evolution of sta.csv nutrient datacodes: 1st - one code for PO4, SIL, NO3, & NO2; 2nd: datacode cluster "P0;S0;O0;T0;"; 3rd: P4DC, SIDC, N2DC, N3DC. Note: NH3 has always had its own datacode.

DECODR and other programs developed by SIO-CalCOFI usually have a date-related version attached such as DECODR1509 (DECODR 2015 Sept). This may cause some confusion on which evolving version of DECODR to use to process different cruises. So starting with 2015 cruises (1501, 1504, 1507, 1511), DECODR (and otherSIO-CalCOFI programs) will be named after the cruise it's designed to process. If no changes happen between cruises (rare), the latest version will be renamed regardless. There will be a DECODR1504, DECODR1507, & DECODR1511.

So far only DECODR1501n has be labeled this way. Note - a single alpha character will denote revisions, "n" was used previously to specify "revised nutrient datacode format" so it was kept - next revision for CalCOFI 1501 data processing will be "DECODR1501o".

Added a new "Step 3" button to add Quality Codes to CTD.csv output (see CTD.csv data format web page) -

• PrQ = pressure quality code;
• T1Q & T2Q = primary & secondary temperature quality codes;
• S1Q & S2Q = primary & secondary salinity quality codes;
• Ox1Q & Ox2Q = primary & secondary oxygen sensor quality codes (these carry over to the umol/Kg oxygen values since they are derived from ml/L);
• FlQ = fluorometer quality code;
• ISUSQ = ISUS nitrate sensor quality code;
• SigTheta1Q & SigTheta2 = T1S1 & T2S2 calculated sigma-theta densities
• TransQ = Transmissometer quality code
• pHQ = pH sensor quality code
• SPARQ = Surface PAR quality code
• PARQ = Remote PAR quality code

Codes - the data quality codes will follow the CalCOFI legacy codes

• no code = good data
• 8 = questionable data; bad calibration or bio-fouling
• 9 = bad or no data; sensor or pump failure

These quality codes have been added to flag bad or questionable data collected by our Seabird 911+ CTD sensors. Sensor failure, bio-fouling, or bad calibrations can generate poor or bad data but may still be included in the data file.

User-interface improvement request from JLW & DMW - make the cruise being processed more prominent. So the user interface was changed so the current cruise and datapath are front-n-center. Additional features/tweaks include either manually changing the cruise in the large text box or by selecting the cruises at the bottom of the screen. Either will update the current cruise to the one entered or selected and change the datapath.

Two features - one improved and one added:

Improved - the CSV Respacer module will work properly with the latest 2015 92-column sta.csv format. The additional nutrient datacodes were being excluded from the reformatting routine. This is strictly a visual improvement that allows use of a text editor to view the files easily; columns and values lineup vertically so visual parsing is easier.

Added - ported from the BtlVsCTD program, the column "purge values" routine has been added to DECODR. Using the header to generate the column selection, the user can select a specific column or set of columns to bulk edit or clear. This is designed to purge data that may have imported improperly into the wrong column. As the data processing scheme evolves and new columns are added, the multitude of routines that populate the columns need to be updated. If one isn't, it can import numbers into the wrong column, overwriting the original column occupant. This is rare fortunately, but did happen when derived oxygen in umol/Kg values were imported into the DIC bottle number column. DIC bottle numbers had to be reentered; oxygen in umol/Kg were reprocessed after the correction with the derived data populating the correct column. DIC bottle numbers are from the original CESL sample log.

Note: the BtlVsCTD column clearing routine was updated to work with the latest version of the CTD.csvs as well. It parses the header to generate the column selections.