IEH: "Is Everybody Happy"

CalCOFI's IEH format was developed in 1984 to accomodate additional data measurements added to the CalCOFI time-series. The previous format, sd2, was adequate for 35 years of hydrographic measurements. But in 1984, chlorophyll, primary productivity, & additional nutrients were added. IEH was developed to contain the additional data within the confines of 128 columns, the data width of a computer punch card. Data were stripped of decimal points to maximize the data accuracy in the available space, resulting in column-specific formats. The CalCOFI hydrographic data processing was performed on a mini computer running VMS until the early 1990s. In 1993, the VMS fortran code was ported by Dave Newton to run on an IBM-PC running DOS. In the late 90's, James Wilkinson ported the DOS Fortran code into Visual Basic for Windows. Coincidently, this change coincided with the transition from hanging Niskin bottles with reversing thermometer on the hydro wire to CTD-rosette to collect seawater samples and vertical profile electronic sensor data. As most of the fortran code was converted to Visual Basic to run natively under Windows 9x-7. Careful, meticulous comparisons were done at each transition - VMS to PC, Fortran to Basic to insure there were no changes in measurement calculations. Cruise data were processed using the old & new software and the results scrutinized to insure the values were identical.

Many user of CalCOFI data still use and expect the IEH-formatted data, using tools developed to import its unique format. But with the additional measurements & resolution provided by the Seabird CTD, other formats are available. Bottle-corrected, 1m bin-averaged CTD data improves the resolution of many physical parameters measured during the quarterly cruises. A relational database, which contains all +60 years of hydrographic data, allows data queries that can extract subsets from the time-series.

Data Processing

Data processing pre-2012 - processing bottle samples into IEH & Hydrographic Data Reports requires the analyses of individual data types: salinity, oxygen, chlorophyll, nutrients, and primary productivity. Each analysis generates data measurements housed in flat ascii files. Each data type is processed individually then merged into combined station files (00/20/22 files). Once merged, Andyplot vertical profiles (bottle data vs depth) are generated and the data point-checked. Odd or erroneous data, bottle mistrips for example, are flagged or removed from the data file. Once final, an IEH file is generated for each station. To generate standard depths, IEH data are interpolated using the observed values. This final, interpolated IEH, which contains both observed and interpolated data, is used to generate the CalCOFI Hydrographic Data Report. The IEH data were also converted to csv then imported into the CalCOFI Hydrographic Database. CalCOFI hydrographic data are archived in IEH formats: 'arch.ieh', observed data only, and 't.ieh', observed & interpolated data. Prior to 2000, the IEH and CalCOFI Hydrographic Data Reports were the core data products published by the CalCOFI Technical Group at SIO.

Data processing +2012 - processing bottle samples still requires the analyses of individual data types: salinity, oxygen, chlorophyll, nutrients, and primary productivity. Each analysis continues to generate flat ascii data measurement files. Each data type is processed individually with output files scrutinized for gross errors. After preliminary corrections are applied, all data are merged into station files (sta.csvs for depth data; casts.csv for station data). The main change from old to new at this stage, is the reduction in the number of data files. The sta.csvs include all bottle data plus matching-depth CTD data at bottle depths, combining "00", "20", "22" ascii files. For 75 stations, that reduces 375 files to 76, 75 sta.csvs & 1 casts.csv. The casts.csv combines stacst (Station Cast Description), weather, hdr, and metadata files into one. Casts.csv contains all cast & sta information, weather, & coefficients used to calculate & correct data.

The standard depth bottle data are interpolated using the observed values primarily for nutrient data unavailable from the CTD. CTD data are processed using Seasoft and bottle-corrected CTD sensor data imported into the sta.csvs at bottle & standard depths. These sta.csvs are converted to mat files using GPlot (Matlab) to plot & point-check bottle & CTD data. Odd or erroneous data such as mistrips, for example, are flagged as bad or questionable in the data file by the point-checker using GTool (Matlab). The point-checker, using GTool, annotates the sta.csvs data codes to indicate the best data for each data type - all bottle measurements are preserved. Using the data code index, the best data, from either bottle or CTD, are used to generate all subsequent data products - IEHs for legacy users, CalCOFI Hydrographic Data Reports, CalCOFI Hydrographic Database. The IEH shifts from being the primary data product to an ancillary, legacy data product. Sta.csvs & casts.csv will be archived but the primary archive will be the CalCOFI Hydrographic Database.

CalCOFI's Seabird 911+ CTD includes dual Temperature, Conductivity, and Oxygen sensor arrays (dual O2s since Jul 2009) that are separately plumbed & pumped. Data from these sensors are labeled Temp1, Salt1, Ox1, & Temp2, Salt2, Ox2 respectively (refer to the index below for units). The "FinalQC" version of the CTD csvs also include oxygens in umol/kg (Ox1uM, Ox2uM) and data-quality code columns. Data codes (flags) are used to identify bad (datacode "9") or questionable (datacode "8") data.

CalCOFI collects seawater samples using 24-10L metal-free bottles mounted on an epoxy-coated rosette. The seawater samples are analyzed & processed at-sea then undergo a series of additional post-cruise calibration, data-processing & data-quality steps on shore. Initial merging & comparison of CTD & bottle data generates the "prelim" CTD csvs used for point-checking. Once bottle data have been gone through CalCOFI's data quality control protocols, the final bottle data are merged with 1m-binavg CTD sensor data from the same cast and depths. CTD salinities deeper than 350m are subtracted from matching bottle (depth) salinities. These differences are averaged into offsets for both primary and secondary CTD salinities and applied to all CTD salinities. Bottle-corrected CTD salinities are reported as Salt1_Corr (primary), Salt2_Corr (secondary), and SaltAve_Corr (ave of both).

Oxygen bottle samples are plotted versus the corresponding CTD primary (Ox1) and secondary (Ox2) oxygen sensor data. Regression coefficients to correct CTD oxygens are derived for both sensors in ml/L and µM/kg - 4 plots total. The bottle-corrected CTD sensor data are reported in the Ox1_CruiseCorr (ml/L) and Ox1µM_CruiseCorr (µM/Kg) and Ox2_CruiseCorr and Ox2_µM_CruiseCorr data columns.

One-meter binavg CTD fluorometer voltages are plotted versus the corresponding depth chlorophyll bottle samples to derive regression coefficients. These are applied to the 1m binavg fluorometer voltages from 0-200 meters, estimating chlorophyll-a  & reported as EstChl_CruiseCorr.

A plot of 1m binavg ISUS voltages versus nitrate (NO3) using all the nitrate bottle samples generates regression coefficients. These are applied to the 1m binavg ISUS voltage to derive estimated nitrate data, reported as EstNO3_CruiseCorr
Note: when bottle data are plotted versus CTD sensor data, bottle data "fliers" are excluded so they do not influence the bottle data sensor correction coefficients.

In addition to cruise-corrected CTD data, there are station-corrected data. These data are bottle-corrected CTD data that use the bottle data from only the matching station. These data are generated using CalCOFI-developed software that dynamically calculates the correction coefficients and applies them automatically to CTD sensor data. These data are usually the best data since this method adjusts for station-to-station variablilty in the sensor behavior such as sensor drift. These data are reported as "StaCorr": Ox1_StaCorr, Ox2_StaCorr, EstChl_StaCorr, EstNO2_StaCorr. The coefficients applied dynamically for each station may be different and are tabulated in each cruises CTD metadata file "DBcoeff.csv".

Please note that the FinalQC format with additional data quality code columns was adopted in 2015. CTD data files with the 82-column format are labeled "FinalQC". Cruise files labeled "Final" are being reprocessed to add the data-quality columns, data codes, & O2s in uM/Kg. When updated, they will be relabled "FinalQC".  Note that most data values, particularly T & S, should remain unchanged - only the format is changing and data-flags added.
QCodes: blank or "0" code = good data; "8" = questionable data; "9" = bad data

SBE formats

There are two types of SBE files that CalCOFI provides to users. There are six types of cast data created each cast by SeaBird's Seasave program during each CalCOFI station. ASC-HDR files are created by SBE Data Processing software module ASCII Out. This routine outputs the header portion (.hdr) and the data portion (.asc) of a processed converted data file (.cnv).

Read more ...

DECODR will tabulate all processing steps and coefficients applied to data in two different csv files.

YYMMPLog.csv - compiles all data processing steps, in batch, for each data type: salts, oxygens, chlorophylls, nutrients, primary productivites, & CTD data imports.

YYMM###_PLog.csv - compiles all data processing steps for each station, for each data type: salts, oxygens, chlorophylls, nutrients, primary productivites, & CTD data imports.

Please note that the two logs are virtually identical except for columns 6 & 7: in the Cruise Processing Log, these columns indicate the beginning and end stations numbers of the processing run; in the Sta Processing Log, these columns indicate the Order Occupied and the CalCOFI Line.Sta number. All other columns are identical and populated at the same time.

SIO-CalCOFI uses datacodes in the sta.csv bottle data files to define data processing actions & output.

Read more ...

A station csv (sta.csv) is generated for each hydrographic station occupied during a cruise. Bottle data records for each depth are built & populated during sample analysis and data processing.

Read more ...

The CalCOFI IEH archive includes two versions: observed data only (YYMMarch.ieh) and observed data plus interpolated standard levels (YYMMt.ieh). Interpolated data are added to allow standard depth comparisons & plotting when observed standard depths are not available.

The IEH format was developed in 1982 as the final bottle data archive format for CalCOFI data. The IEH format is 128 columns wide, based on the 128-col computer punch card used at the time. Decimals are removed to maximize data space resulting in a column-specfic format - see key below. In IEH files generated before Aug 1995, columns 64-65 contain format alignment characters (Z*) throughout all records; after Aug 1995, columns 64-65 report niskin bottle numbers on the bottle depth records. Interpolated depth do not have a bottle number so columns 64-65 are blank.

In 2006, the Station Id Line & Sta are reported to tenths (i.e., 81.8 46.9 instead of 82 47 or 86.7 100.0 instead of 87 100). This change allows the inclusion & resolution of coastal SCCOOS station data from nearby standard station data. SCCOOS stations can be within the 2 nautical mile radius of the standard station ie 93.3 26.4 SCCOOS & 93.3 26.7 Standard stations.

IEH stands for "Is Everybody Happy?" coined by programmer David Newton.

IEH Format 21Aug95
Specification:
** First Master Record **
Col. Notes
--------------------------------------------------------------------------
1-6 Latitude DDMMtH t=1/10 min. H = hemisphere (N or S)
7-13 Longitude DDDMMtH H = hemisphere (E or W)
14-19 Date YYMMDD
20-23 Cast time HHMM GMT This should be the time most representative of the data in the station.
24-28 Bottom sounding XXXXX meters
29-30 Wave direction XX 2 digit code. WMO 0885 (00-36,49)
31-32 " height XX feet
33-34 " period XX seconds
35-36 Wind direction XX 2 digit code. WMO 0877 (00-36,49)
37-38 " speed XX knots
39-43 Barometer XXXXX millibars to tenths.
44-47 Dry bulb air temp. XXXX deg. celsius to tenths.
48-51 Wet " " " XXXX " " " "
52 Weather X 1 digit code. WMO 4501.
53 Cloud type. X 1 digit code. WMO 0500.
54 Cloud amount X 1 digit code. WMO 2700. (oktas)
55 Visibility X 1 digit code. WMO 4300.
56-61 Ship code --------------------------------
56-57 XX country code. These 6 characters should indicate
58-59 XX institution code. ship ownership. Who owns the boat?
60-61 XX ship code. ---------------------------------
62-63 Data type XX 2 character code. "PR" = productivity.
   "HY" = hydrographic, "10" = ten-meter.
   "CT" = compressed ctd (low res.)
   "MX" = mixed ctd and bottle data.
64 'Z'
65 '*' Asterisk used for format alignment.
66-69 Processing number XXXX for data processing use only.
70-72 Cruise id XXX three characters for SD2 conversion.
73-74 Leg number XX
75-84 Station id XXXXXXXXXX alphanumeric 10 characters.
85-88 Data origin Whose data is it?
85-86 XX country code.
87-88 XX institution code.
89-103 Cruise name 15 character alphanumeric. Will be printed to data
reports.
104-111 name for data in wild column 1. 8 chars.. printed to data report.
112-119 " " " " " " 2. " " " " "
120-127 " " " " " " 3. " " " " "
128 '1' record indicator.

** Second Master Record **
-----------------------------------------------------------------------------
1 Blank used for format alignment
2-3 Secchi disk depth XX meters
4-5 Water color XX (0-21) Forel-Ule scale
6-13 Incubation times
6-9 HHMM start time (local)
10-13 HHMM end time (local)
14-17 Local apparent noon HHMM (local)
18-21 Civil twilight HHMM (local)
22-24 Time zone +/-HH hours preceeded by '+' or '-'
25-29 Integrated C14 assimilation XXXXX milligrams/m^2 /experiment to 10ths.
30-33 Integrated Chlor-A XXXX milligrams/m^2 to 10ths.
34-37 Integrated Phaeopigment XXXX milligrams/m^2 to 10ths.
38-63 Ship name 26 character alphanumeric. Printed to data report.
64 'Z' alignment character.
65 '*' alignment character.
66-82 Blanks (17)
83-85 Interpolation flags. Used by an interpolation program to signal that you want data in the wild columns to be interpolated.  Defaults to no interpolation.
83 Wild col 1. 'I' to interpolate. Blank for no interpolation.
84 " " 2. " " " " " "
85 " " 3. " " " " " "
86-103 Runtime formats used by programs to fill the wild columns with data. The formats must be of the form "(F7.x)" and be 6 chars long. x can range from 0 to 6. If the wild columns are used these formats must be specified.
86-91 format for wild column 1. Blank if none.
92-97 " " " " 2. " " "
98-103 " " " " 3. " " "
104-111 units for parameter in wild column 1. 8 character alphanumeric.  Will be printed to data report.
112-119 units for wild column 2.
120-127 units for wild column 3.
128 '2' record indicator.

** Detail Record **
---------------------------------------------------------------------------
1-5 Depth XXXXX whole meters
6 Footnote indicator 1 chararcter. Used as a general footnote indicator. Printed to data report.
7-11 Temperature XXXXX deg. celsius. Last column is 1000ths.  Trailing blanks permitted.
12 " precision X 1,2,or3 or blank if no temperature.
13 " quality X quality code.
14-18 Salinity XXXXX Last column is 1000ths. Trailing blanks OK.
19 " precision X 1,2,or3 or blank if no salinity.
20 " quality X quality code.
21-26 Pressure XXXXXX decibars to 10ths.
27 " quality X quality code.
28-31 Oxygen XXXX milliliters/liter to 100ths. range .00->99.99
32 " quality X quality code.
33-36 Phosphate XXXX ugram-atoms/liter to 100ths. range .00->99.99
37 " quality X quality code.
38-41 Silicate XXXX ugram-atoms/liter to 10ths. range .0->999.9
42 " quality X quality code.
43-46 NO2 (Nitrite) XXXX ugram-atoms/liter to 100ths. range .00->99.99
47 " quality X quality code.
48-50 NO3 (Nitrate) XXX ugram-atoms/liter to 10ths. range .0->99.9
51 " quality X quality code.
52-55 NH3 XXXX ugram-atoms/liter to 100ths. range .00->99.99
56 " quality X quality code.
57-60 Chlorophyll-A XXXX milligrams/meter^3 to 100ths. range .00->99.99
61 " quality X quality code.
62-63 Cast number XX
64-65 Bottle id number XX (formerly 'Z*' for alignment prior to Aug 1995)
66-69 Phaeopigment XXXX milligrams/meter^3 to 100ths. range .00->99.99
70 " quality X quality code.
71-75 C14 assimilation #1 XXXXX milligrams/meter^3/experiment.  last column is 100ths. range .00->999.99 trailing blank permitted.
76 " precision X 1,2,or blank if not recorded.
77 " quality X quality code.
78-82 C14 assim. #2 XXXXX milligrams/meter^3/experiment.  last column is 100ths. range .00->999.99 trailing blank permitted.
83 " precision X 1,2,or blank if not recorded.
84 " quality X quality code.
85-87 Dark C14 assim. XXX milligrams/meter^3/experiment. Last column is 100ths. range .00->9.99. trailing blank permitted.
88 " precision X 1,2,or blank if not recorded.
89 " quality X quality code.
90-94 Mean C14 assim. XXXXX milligrams/meter^3/experiment. Last column is 100ths. range .00->999.99 trailing blank permitted.
95 " precision X 1,2,or blank if not recorded.
96 " quality X quality code.
97-100 Elapsed incubation time HHMM
101-103 Light percent (incubation). XXX to 10ths.  punch decimal to record 100ths below 1 percent.
104-110 Wild column 1. XXXXXXX Any data value that can fit. Punch explicit decimal. No trailing blanks allowed.
111 " quality X quality code.
112-118 Wild column 2. XXXXXXX Any data value that can fit. Punch explicit decimal. No trailing blanks allowed.
119 " quality X quality code.
120-126 Wild column 3. XXXXXXX Any data value that can fit. Punch explicit decimal. No trailing blanks allowed.
127 " quality X quality code.
128 Record indicator. 3,4,5,6,or7
3==observed data. 4==educated office guess (ghost)
5==data from std or ctd device (low resolution)
6==multiple depth card.
7==interpolated to a standard depth.

** Footnote Record **
----------------------------------------------------------------------------
1-63 Text
64 'Z'
65 '*'
66-127 Text
128 '8' record indicator

A footnote record may appear only after the detail records for a station and will be printed to the data report. Columns 64 and 65 will be ignored on printout.

** Text Record **
----------------------------------------------------------------------------
1-63 Text
64 'Z'
65 '*'
66-127 Text
128 '9' record indicator

A text record can appear anywhere in the file except directly after the first master record.. The information will not be printed to data reports, but will remain with the data.
-----------------------------------------------------------------------------
** Codes and usage **

1. Quality code. Used on the detail records.
   blank == data ok
   '6' == data ok,but taken from a ctd device.  (used only when data is, on the whole, considered to be bottle data)
   '8' == originator thinks value is suspect.
   '9' == missing data
2. Examples of Calcofi station numbers. b=blank
b93bbb40bb 110bbb40bb
110bbb40.2 b93.1b40bb
b93.2b40.1 110.1100.3
3. Missing or unrecorded data MUST be indicated with a '9' in the quality code.
4. Trailing blanks are only allowed when specifically mentioned and when they are mentioned, their use is required if the precision
is less than the maximum.
5. The ship ownership country and institution can be different from the data ownership country and institution.
6. The format can be folded into a 64 character variety if that makes data entry more convenient, but the specification and any application programs will operate on the 128 character variety listed here.
7. Any interpretation problems should be referred to Jennifer Rodgers-Wolgast at x42888.
8. The first master record should be immediately followed by the second master record.
9. Times in HHMM format should be written with no embedded blanks.
10. Ghostcards (rectyp=4) are used for interpolation but not printed to data reports. Multiple depth cards (rectyp=6) are *not* used for interpolation but *are* printed to data report.

Revisions:
5/12/87 dmn. Probably added some notes.
5/ 9/91 dmn. Added word incubation to light percent explanation.
8/21/95 dmn. added bottle id number to data records. added rcord type 6 (multiple depth card).  added usage note 10.
8/95 jrw. changed col 64-65 from *Z format verification to bottle number