Evaluating Sampling Data

Health assessors will encounter an extremely broad range of sampling data sets over their careers, and the characteristics of these data sets will vary from one site to the next. Some data sets, particularly older ones, may present measured concentrations in hard copy format with limited supporting information. Other data sets, especially those conducted more recently for EPA and other environmental agencies, are typically available in electronic databases and are extensively documented in multi-volume reports. Regardless of these differences, health assessors need to evaluate all sampling data sets before using them in the PHA process.

There is no single, prescriptive approach for how health assessors should evaluate sampling data. The level and extent of data review will vary from site to site and from data set to data set. However, these basic steps will apply to most situations:

1. Become familiar with the site to determine where and how people were exposed to environmental contamination.
2. Identify and understand each sampling program’s underlying objectives.
3. Evaluate the representativeness of the sampling data relevant to the completed and potential exposure pathways and exposure units.
4. Assess the quality of the sampling data.

Become Familiar with the Site

Health assessors should form a conceptual model for the sites they evaluate. This process begins with identifying the source of contamination, understanding how that contamination moves through the environment, and determining how community members might come into contact with the contamination. This step will help health assessors identify a site’s completed and potential exposure pathways and exposure units. This step is important because sampling data for the PHA process should characterize contamination in these pathways and within a site’s exposure units.

Consider these factors when forming a conceptual model and understanding site conditions:

• How did the area originally become contaminated (for instance, because of a spill, a routine release, or a naturally occurring source)?
• What contaminants are present?
• How does the contamination move through the environment?
• What environmental media are affected? Can contamination move from one medium to another?
• Were sampling data collected in the exposure unit?
• What is the land use in the exposure unit (e.g., residential, industrial, commercial)?
• How might community members come into contact with the environmental contamination?
• Has anyone (e.g., EPA, state agencies) investigated the site?

Having answers to these questions – and identifying potential and completed exposure pathways and exposure units – will help health assessors evaluate data, as health assessors do not need to spend time evaluating data for incomplete exposure pathways or from outside site-specific exposure units.

Identify and Understand Each Sampling Program’s Underlying Objectives

Virtually all sampling data are collected with specific objectives in mind. Health assessors need to understand those objectives before using sampling data. Documentation of sampling objectives will vary considerably from study to study.

EPA published the Guidance on Systematic Planning Using the Data Quality Objectives Process, [PDF – 698 KB] which helps those who conduct sampling to articulate their objectives. The guidance encourages individuals to follow seven steps, such as stating the problem that sampling is to address, identifying the goals of the sampling, delineating the boundaries of the sampling, and specifying performance criteria or acceptability criteria. Thus, for those sampling programs that followed this EPA guidance, sampling protocols and sampling reports should clearly state the program’s data quality objectives (DQOs) — which health assessors should review to understand why sampling occurred and what sampling intended to accomplish. This step is important for health assessors to follow whenever reviewing sampling data.

That said, not every sampling report will include DQOs. Sampling reports issued prior to EPA’s guidance may not refer to DQOs. Similarly, more recent sampling not conducted under an EPA program are not required to follow this guidance and also may not refer to DQOs. Examples include sampling conducted for academic research and sampling conducted by community groups. Even if sampling reports do not specifically refer to DQOs, these reports should include “purpose statements” or similar language explaining why sampling was conducted.

Whether or not sampling reports include DQOs, health assessors need to understand why sampling occurred and what the sampling intended to accomplish. This is an easy task for sampling with documented DQOs. In all other cases, health assessors should be able to discern objectives from the sampling reports and plans or through conversations with the sampling programs’ principal investigators. Knowing the sampling program objectives should also give you greater understanding for how, what, where, and under what conditions samples were collected, and how site-specific factors could affect the sampling collection and results.

Sampling data used to support the PHA process can come from sampling programs that vary widely in scope and purpose. In many cases, you will use whatever data are available, even those collected in programs not designed for public health evaluations. Your job is to consider the objectives and document in your reports why you believe a given sampling program’s data are suitable for characterizing exposures at your site’s exposure units.

Evaluating the Sampling Data Representativeness

Sampling will characterize contaminant concentrations at specific locations and times, and generally not for every location and time in an exposure unit. You will therefore need to determine the representativeness of sampling data.

Often environmental data are not collected to define exposures but instead for other purposes such as to define the extent of site contamination for regulatory purposes, including remediation. The health assessor must determine whether the data collected for differing objectives are sufficient to evaluate a site’s exposure pathways and exposure units. Data representativeness refers to the degree that data are sufficient to identify the concentration and location of contaminants at a site. It also refers to how well the data adequately characterize the exposure pathways of concern during the time frame of interest (i.e., when exposures occurred).

A sampling program’s DQOs will help a health assessor determine whether the data can be used in the PHA process and are representative of exposures in an exposure unit. Assessing the representativeness of data is subjective, drawing from your professional judgment and technical understanding of a site’s conceptual model, exposure pathways, and fate and transport of environmental contaminants. Your documents should explain why you think a given sampling data set is or isn’t representative of the exposure pathways and exposure units. In some cases, you may end up using a portion of the sampling data when characterizing exposures (e.g., you might exclude onsite data with a secure perimeter when examining community members’ exposures to contaminated surface soils).

Here are some examples of issues health assessors may need to consider when evaluating data representativeness:

• You are working on a site where several feet of clean fill were placed over waste, and residents have expressed concern about contacting the surface soil (see box on the right). You are reviewing subsurface sampling data (greater than 3 feet in depth) shown to be highly accurate and precise. However, given that samples were collected at a depth greater than 3 feet below ground surface, the data are not representative of an exposure pathway because subsurface soil cannot be used to evaluate residential concerns about surface soil exposure.
• You are evaluating groundwater monitoring data at a site with elevated metals concentrations in drinking water. A community group has questioned the relevance of these data because samples were filtered prior to analysis, per the sampling protocol. The group argues that the filtering step removed metals that are contaminants of concern for private wells. Before making decisions about data representativeness, the health assessor should consider the types of filtering used on well water prior to consumption and how that filtering step compares to the filtering of samples.
• You are evaluating fish tissue contamination at a site, and the only data you have available are whole fish samples (and not fillet samples). These data still might be representative of some exposures – you need to first research residents’ fish consumption patterns, as some individuals do consume most portions of fish that they catch, and be sure to mention this type of data limitation in your document.

If you have questions when deciding whether your site’s sampling data are representative of exposures, get a second opinion! Ask the Associate Director for Science (ADS) group, your Technical Project Office (TPO), or an experienced health assessor whether they agree with your assessment of data representativeness.

Media-Specific Concerns

Health assessors should consider numerous media-specific concerns when evaluating the representativeness of sampling data. Keep in mind that environmental contamination will vary both with location (spatially) and time (temporally).

Samples collected in some media might be representative of contamination over very small areas, while samples from other media might be representative of contamination over broad ranges. As an example, when evaluating air releases of contaminants from a ground-level source, you would expect to see the highest concentrations of the contaminant nearest to the source, with concentrations decreasing considerably with downwind distance. Some other air pollutants, however, are known to have minimal spatial variations over broad ranges: ozone, for instance, forms in the air as a product of photochemical reactions, and its concentrations do not vary considerably over entire cities. In both cases, this understanding would inform your assessment of whether sampling at a given site is representative of the exposures that might occur.

Additionally, temporal variations are an important consideration when reviewing sampling data. For some media, environmental contamination levels are known to vary seasonally — or even daily — and these variations are important to consider when assessing representativeness. If samples were collected in only one season, one day of the week, or a certain time of the day, you should ask yourself if samples collected at other times would be expected to have a different result.

Similarly, health assessors should always be mindful of exposure potential when evaluating data. Many sites will have soil samples collected at depth to determine the extent of contamination; in some cases, these samples are from several feet below the surface, where exposures are not expected. These data will not factor into the PHA process, unless there is evidence of potential contact with the subsurface materials (e.g., residents digging pools or ponds on their properties). On the other hand, surface soil samples collected at a depth of 0–3 inches in an exposure unit are directly relevant to exposures because residents may contact those samples during various routine activities (e.g., playing, gardening) (see this text box).

In all cases, you should question how adequately sampling locations and times represent exposure conditions at points of known or suspected exposure.

Examining the Sampling Data Quality

Health assessors must evaluate data quality whenever using sampling data. This is critical because public health conclusions need to be based on data of a known and high quality. Uncertainties and limitations should be acknowledged when this is not the case.

Data collected from different sources and for different purposes can be used in your evaluation if you deem the data are appropriate and of sufficient quality for examining site-related exposures. You will see a broad range of practices of documenting data quality in sampling reports. Some reports will have data quality narratives that provide an overall data quality assessment, followed by original laboratory data packages (which may be hundreds of pages). On the other hand, some reports may have very limited information on data quality — or perhaps be silent on this topic altogether. Health assessors should review as much information as necessary to confirm that the underlying data are of a known and high quality for supporting the PHA process. During this data review, you may also need to contact sampling program principal investigators for more information, as it is incumbent upon the data provider to demonstrate that data are of a known and high quality.

In some cases, you may not be able to locate any information on data quality, even after contacting principal investigators. Your documents should identify such sampling data, note that the data are of unknown quality, and explain why those data do not factor into your site’s public health conclusions. This step is important because community members may be aware of the data sources that you exclude. Your documents should be transparent in terms of which data sources factor into your conclusions and why.

When evaluating sampling data quality, you will need to consider several components, including the data source, sampling documentation, sampling techniques, and sampling data validation. Also, follow the various tips for gauging sampling data quality.

Source of Sampling Data and Data Use

Additional Sampling Documentation to Review

Another important resource to review is sampling plans, to the extent they are available. In most cases, those who conduct sampling will develop planning documents that describe key elements of sampling programs before they are implemented. These planning documents may be called Sampling and Analysis Plans, Sampling Protocols, or Quality Assurance Project Plans (QAPPs). When reviewing sampling data, health assessors should access all relevant planning documents – but recognize that not every sampling program will have a written plan.

The planning documents should provide important perspective on the environmental media and/or biological materials to be sampled, contaminants to be measured, sampling and analytical methods, sample handling and shipping procedures, proposed locations or individuals to sample, sampling frequency and duration, and health and safety considerations for field personnel. These planning documents should justify the various decisions made when implementing a sampling program.

These documents should also describe the procedures to ensure sampling data are collected in a way to meet DQOs. For sampling programs conducted for EPA and some other environmental agencies, this information will typically be found in a QAPP. Health assessors should therefore review QAPPs for all sampling programs, but they should also recognize that not every program will have one. When available, QAPPs outline specific quality acceptance criteria that relate to measures such as precision, accuracy, sensitivity, completeness, comparability, and representativeness of data.

The key point is that “reviewing sampling data” should not focus on measured concentrations only. For a complete picture, health assessors should review the original planning documents, the sampling reports, and all aspects of the data themselves (e.g., non-detect results).

Sampling Techniques

Sampling techniques generally fall into four categories, and these techniques have bearing on the quality of the measurements that are made. The following table identifies the four techniques and presents general guidelines about strengths, limitations, and data quality. The following table should not be your only basis for evaluating data quality. For every sampling data set, regardless of the sampling technique, health assessors should review various metrics of data quality, which are typically documented in a data quality narrative that accompanied the sampling results.

Common Sampling Techniques

Sampling Data Validation

Sampling results can provide useful information for the PHA process, but health assessors need to be on the lookout for factors that can bias sampling data. For instance, a laboratory might have used a cleaning solvent containing a volatile organic compound that was being tested for, and this practice could contaminate all measurement results. Another example is using a direct reading device in the field that has known interferences (e.g., dusty areas might lead to positive biases in certain colorimetric detection devices).

Given these and other potential problems, you should never assume that all sampling data are accurate. The original data provider must prove their data are valid for the intended purpose. Ask yourself how confident you are that the reported concentrations truly indicate the levels of contamination in the sampled media, or simply, if the sampling data have been verified and validated.

The laboratories that analyze samples typically validate their data. The laboratories will perform various QA/QC procedures to prove that their data are of a known and high quality — and that they meet the sampling program’s DQOs. Validation is an analyte- and sample-specific process, typically performed by an analytical chemist, that determines the quality of a data set. The chemist validates the data using various standard procedures, such as analyzing matrix spikes, blanks, standard reference materials, and duplicate samples. The validation results are typically documented in a data quality narrative or similar section of the sampling report. The section on Tips for Gauging Sampling Data Quality presents some basic data quality assessments that health assessors should consider when evaluating any sampling data set.

In some cases, a laboratory will have third-party validation of data. This could involve splitting samples, having an outside laboratory run concurrent analyses, and comparing results. This third-party validation, when available, can provide higher confidence in data quality. However, health assessors should not expect to see third-party validation for all sampling data; most sampling efforts are conducted without it. In short: third-party validation can provide greater confidence in data quality, but this validation is not a necessary condition for demonstrating data quality.

Tips for Gauging Sampling Data Quality

Health assessors can follow these general tips to help gauge the quality of sampling data. These are intended to cover some basic data quality review considerations. A good reference for assessing data quality is Steps One and Two in EPA’s Data Quality Assessment: A Reviewer’s Guide (EPA QA/G-9R) [PDF – 1,248 KB]. In some cases, health assessors may need to consult with ATSDR subject matter experts (SMEs) or contact the laboratory that generated the sampling data for a more complete understanding of data quality.

The following are important data quality assessment steps:

1. Check the sampling and analytical methods. The methods that site investigators use to collect samples and analyze them for contaminant concentrations have a significant bearing on data quality and use. The selected method usually determines what contaminants can be measured and in what concentration ranges. Identify the methods used in every site investigation and ensure that they are appropriate for that site’s contaminants of concern. For instance, a Toxicity Characteristic Leaching Procedure (TCLP) test tells you if a material (e.g., sludge, contaminated soils) needs to be handled as hazardous waste, but the test doesn’t indicate the contaminant concentrations in the material that you need to know for use in exposure assessment. If you do not know how to evaluate a method, ask an SME for help.

   EPA’s ExpoBox identifies many of the agency’s commonly used environmental sampling and analytical methods. When consulting this toolbox, be sure the sampling collection methods used can measure the contaminant concentrations of concern at a site.

   Also, determine whether the detection limits for the methods used are low enough to enable health assessment. Detection limits generally need to be lower than ATSDR’s comparison values. However, for a few contaminants, comparison values are lower than levels that can be measured with widely used analytical methods.

2. Consult with site investigators, regulators, and technical experts. Generally speaking, the people responsible for collecting and reviewing the data are the best sources for assessing sampling data validity. These individuals often can give insights on strengths and limitations of sampling projects beyond what might be documented in the sampling reports. Contacting site investigators and regulators is particularly important when little or no data validation documentation is available. EPA’s regional laboratories might also have insights.

3. Check data quality documentation to understand the sampling and analytical protocols. In general, data quality documentation addresses field practices, laboratory practices, QA/QC procedures, and data quality indicators. The availability of documentation varies from one sampling project to another. For example, health assessors may encounter data sets from the 1970s that lack data quality narratives and insight into data quality, while on the other hand, samples collected for Superfund remedial investigations are often supported by QAPPs and data validation reports.

Health assessors should review data quality narratives and other documentation, because this is where you can learn about missing samples, abnormal results, instances where field teams deviated from standard operating procedures, and instances where field or laboratory staff applied non-standard methodologies. At a minimum, health assessors should be familiar with these terms, and consider looking into these data quality indicators for the site-specific sampling data.

• • Completeness refers to the fraction of attempted sampling events that have valid results. It is often expressed as a percentage (e.g., 92% of the samples collected were valid). The completeness of a sampling program is a rough measure of how successful it was. A sampling program with low completeness might result from field or laboratory personnel having routine problems resulting in a significant number of samples being invalidated. Such scenarios might cast doubt on the overall validity of a sampling program.
  • Accuracy indicates the extent to which measurements represent their corresponding “true” or “actual” value. Sampling programs can characterize accuracy in many ways. In some cases, the principal investigators will collect and analyze certified audit samples (i.e., samples with known levels of contamination) and compare them against the measured levels of contamination reported by the analytical laboratory. In other cases, they collect and analyze “blank” samples. When a contaminant is detected in both the sample and the blank, the presence of the contaminant in the sample may be a result of equipment contamination. Published sampling methods typically set guidelines for how accurate measurements should be.
  • Precision characterizes the repeatability of measurements. Two measurements of the same material that are in excellent agreement are highly precise. Precision is usually quantified by collecting duplicate samples in the field or by analyzing the sample twice in the laboratory (replicate). Ideally, concentrations of contaminants in duplicate and replicate samples will be in close agreement and within the bounds specified by the published sampling and analytical method. Precise sampling data will have relatively low differences in concentrations between duplicate and replicate samples.

    Precision is usually reported as a relative percent difference (RPD), and most sampling and analytical methods specify acceptable ranges of RPDs. Sampling program DQOs, if available, should also specify acceptable ranges of precision. In cases where precision does not meet QA/QC criteria, health assessors need to examine the data more closely to determine if they may still be usable for the PHA process. For example, if RPDs in duplicate samples are slightly higher than the method acceptability limit but both measurements are at concentrations orders of magnitude less than ATSDR’s comparison values for that contaminant, obtaining more precise data may not be necessary — and the available data may be used in the PHA process. You may want to consult an analytical chemist, however, when making such judgments.

4. Examine information on data qualifiers included in the laboratory data package. Regardless of the approaches used, laboratories and other data validation reviewers assign qualifiers to certain measurements. These qualifiers add important details to the measured concentrations and must be considered whenever working with sampling data. To understand the meaning of data qualifiers, health assessors should first review documentation in sampling reports. In cases where reports do not describe qualifiers, consult other references (such as EPA’s Guidance on Environmental Data Verification and Data Validation [PDF – 364 KB]). Consult with others (e.g., an analytical chemist, site investigators, the laboratory) if you have questions.

   The following table summarizes some of the most used qualifiers and their corresponding definitions. However, over time, conventions for assigning and defining qualifiers have changed. Always check your original data source to see how the reporting laboratory defined these terms.

Common Laboratory Data Qualifiers

Conclusions About Data Quality

Assessing data quality answers two general questions:

1. What sampling data are suitable for making public health decisions?
2. What data are not suitable for this purpose?

Answering these questions requires both an objective evaluation of data quality parameters and use of professional judgment. In some cases, you might decide to use data that other agencies have excluded from their analyses; in other cases, you might reject data that others have used. It is your responsibility to decide what data are appropriate for the PHA process and to justify these decisions.

Generally, you can be confident in the quality of sampling data that went through the following steps:

• Measured by standard laboratory techniques or rigorously tested field methods (e.g., EPA reference methods for particulate matter).
• Collected and analyzed following methods approved by EPA and other agencies.
• Accompanied by thorough QA/QC documentation indicating the data have been verified and validated and that DQOs were met.

However, remember that just because sampling data are of a known and high quality does not mean they are perfectly suited for the purpose of estimating human exposure. Data used in the PHA process should provide insights relevant to an exposure unit and for a completed or potential exposure pathway.

When sampling data do not meet all the quality criteria, health assessors should do the following:

• Discuss the data limitations with SMEs and the ADS group.
• Explain in your documents the nature of the limitations (e.g., incomplete data sets, no QA/QC information, original laboratory data packages not available). Also convey your findings in the context of the data limitations. For example, indicate whether using such data may lead to exposure overestimates or underestimates.
• Consider whether the available data that do not meet quality criteria can be used to reach public health conclusions. Acknowledge, to the extent possible, how conclusions about potential public health hazards may change if additional information was gathered.