Evaluation of the Effectiveness of Pollution Prevention Planning in NJ

III. Study Methodology
Before developing its questionnaire and interview structure, HRA accompanied NJDEP on a site visit and examined a few plans submitted to NJDEP. The site visit demonstrated that HRA would have to examine the plans of all the facilities it selected to survey. Since only plan summaries are submitted to NJDEP and plans are kept on site at the facility, HRA could not determine if the facilities' plans contained the elements it would be asking facilities to comment on unless the plans themselves were reviewed. Also, placing planning activities in light of previous activities at facilities was virtually impossible without site visits.

III. A. Facility Selection
NJDEP provided HRA with a database of 552 facilities that either had submitted a plan summary to NJDEP by July 1, 1994, or were considered to be covered by the planning requirements, based on their 1992 DEQ-114 data. Each facility was assigned a random number from zero to nine, and NJDEP itself visited a random selection of facilities with numbers seven through nine. HRA used the remaining 407 facilities as a basis for selection.

HRA had two considerations in selecting facilities. First, to gauge the effectiveness of planning, it was important to examine as much of the generation of nonproduct output (NPO) as possible, while maintaining representation of the range of facility sizes. Accordingly, facilities selected for HRA's study had to have generated a substantial fraction of New Jersey's total NPO in 1992, the year of the latest DEQ-114 data available at the time HRA selected facilities. Second, to avoid artificially inflating the impact of a particular industry, the distribution of industries had to represent the distribution of industries among facilities available for visits and those submitting plans.

The 1992 DEQ-114 data for the 407 facilities available to HRA showed a high concentration of NPO generation at a relatively small number of facilities: 39 facilities generated at least 2.0 million pounds of NPO in 1992. The remaining 368 facilities each generated less than 2.0 million pounds of NPO in 1992, and only six of them generated between 1.0 and 2.0 million pounds. HRA selected the top 25 facilities for 1992 NPO generation and selected another 25 at random from facilities generating less than 2.0 million pounds of NPO in 1992. Some of the facilities had either closed or dropped below reporting thresholds for 1993 and were not required to submit plan summaries; others were participating in NJDEP's facility-wide permitting program and were receiving NJDEP's assistance in preparing their plans. Therefore, 15 additional facilities were selected at random as alternates from the group generating less than 2.0 million pounds. HRA scheduled a total of 48 site visits: 21 (44 percent) among those generating more than 2.0 million pounds of NPO and 27 (46 percent) chosen at random among the facilities generating less than 2.0 million pounds of NPO. This selection of facilities covered 66% of the 1992 NPO generated by facilities available to HRA (see Table 1).

HRA visited facilities in each of the five industrial classifications required to submit plans by July 1, 1994 (SIC codes 26, 28, 30, 33, and 34). The selection of large facilities and random selection of other facilities resulted in a distribution of facility SIC codes that reasonably mirrors the industrial distribution of facilities available for visiting and thus validated the sample as representative. In addition, HRA visited one facility reporting under SIC code 20 and one under 35 (Food and Electrical, respectively) that submitted plans to NJDEP before their deadline of July 1, 1996 (see Table 2).

III. B. Protocol for Facility Visits
HRA staff met with facility representatives, usually with the person in charge of preparing the plan. Other facility employees who participated in the planning process were usually present for all or part of the visit. In facilities that had plans prepared by outside consultants, the consultant was usually present as well. In a few cases, the person in charge of preparing the plan was no longer employed at the facility and was unavailable for discussion. Visits typically followed a set protocol:

1. HRA began each visit by explaining the purpose of the visit and by assuring each facility of the confidential nature of the visit: no information about specific facilities would be shared with NJDEP or others.
   
   
2. The discussions began with the HRA questionnaire containing a series of questions designed to place creation of the facility's plan in a historical context of planning and general environmental activities at each facility. These were followed by questions about creation of the plan and a series of questions about the planning process, following an outline derived from Industrial Pollution Prevention Planning: Meeting Requirements under the New Jersey Pollution Prevention Act (see Appendix A).
   
   
3. HRA then examined the facility's plan and environmental reporting data (DEQ-114, TRI Form R, and New Jersey Hazardous Waste Generator forms) and completed two questionnaires developed by NJDEP: an Administrative Review and a Plan Quality Review. The Administrative Review steps through the provisions of the planning rule and measures completeness of the plan. The Plan Quality Review expands on the Administrative Review to indicate individual features of facility's plan.
   
   
4. After completing the review of the plan, HRA met with the facility representatives again to discuss the results of the Administrative and Plan Quality Reviews. The facility was offered a photocopy of each review. Most visits ended with a tour of the facility.

III. C. General Analysis
HRA used data from the Administrative Review, the Plan Quality Review, the HRA questionnaire, plan summary forms, TRI Form R, and DEQ-114 reports for each facility to assemble a list of factors such as plan attributes, reactions of facilities to the planning process, types of previous environmental and planning activities for the facilities, and projected reductions in use and NPO. These factors were analyzed against four measures of effectiveness: plan completeness, plan quality, percent reduction goals for use, and percent reduction goals for NPO. (Box 1, in the Introduction, lists factors and measures; Section III discusses plan completeness and plan quality). The goal of these analyses was to determine if any of the factors was associated with more complete plans, better plans, or higher percent reduction goals. Given the complexity and individuality of facilities' plans, HRA decided it was appropriate to use percent reduction goals both as measures and as factors against measures of plan completeness and plan quality. They were not used, however, as both factors and measures in any one analysis.

HRA tested factors against measures for statistical significance using a standard chi-squared 95% confidence interval test. Not all facilities could be used in each analysis, and tables list the facility counts and reasons for excluding one or more of the facilities. For example, when using percent reduction goals as measures, one facility had to be excluded because it represented 40 percent of the NPO for the State of New Jersey in 1993, and whatever reduction goals it set virtually dictated the results for the particular group it happened to be in. HRA also excluded from analyses involving reduction goals those chemical forms with inconsistent goals, i.e., when a facility's stated goal of pounds of NPO to be reduced over five years for a particular chemical was inconsistent with the base year pounds on the DEQ-114 and had not been clarified during the site visit. If all of a facility's goals were inconsistent, the facility was excluded from the analysis.

III.C.1. Measurement of Plan Completeness
HRA selected four categories as a measure of plan completeness:

1. Process-level inventory
   
   
2. Part I and Part II costs
   
   
3. Pollution prevention reduction options beyond those to be implemented
   
   
4. Five-year use and NPO reduction goals, both facility-wide and per unit product

Facilities were assigned to one of four groups based on data from the Administrative and Plan Quality Reviews. These four groups essentially rank plans from most to least complete (see Box 2), and a facility in either of the top two groups would be considered to have a more complete plan than a facility in either of the bottom two groups.

III.C.2. Measurement of Plan Quality
HRA examined the Plan Quality Review forms to determine measurements of plan quality. Like the Administrative Review, the Plan Quality Review lists attributes of plans, but these attributes address issues such as thoroughness, good faith effort, and understanding of facility activities, the planning process, and pollution prevention (see Appendix C). HRA devised six groups of questions from the Plan Quality Review: Sources, Processes, Part I Costs, Targeting, Options, and Part II Costs (see Boxes 3 and 4). HRA then examined the Plan Quality Reviews to see if facilities had the appropriate responses to all questions in each group. Facilities received a "1" for each group completed with all the appropriate responses. An inappropriate answer to any question in the group resulted in zero credit for that group. Thus, facilities could earn plan quality totals from zero to six with a "better" plan having a higher number.

III.C.3. Plan Completeness versus Plan Quality
At first glance, the plan completeness and plan quality criteria seem similar enough to justify having one measure instead of two. However, the plan quality criteria have more specific judgment attached to them, not just "was it there", but "how well was it done?" That is, inputs and outputs should balance, targeting choices should not exclude important processes, etc. Although the plans with the highest possible quality score are also in the top completeness group, the two measures do not track exactly with one another. It is possible to have a reasonably complete plan that is not very good, for example. Although the two measures do overlap somewhat, HRA developed them separately to include as much data collected during site visits as possible.

III.C.4. Measurement of Reduction Goals
Facilities must report facility-wide, five-year reduction goals for use and NPO for each reportable hazardous substance. HRA compared these goals - pound and percent reduction - to the facilities' 1993 DEQ-114 data for consistency. For example, a facility projecting a 30 percent reduction in NPO of dichloromethane in its pollution prevention plan should have listed on its plan summary an amount of reduction in pounds equal to 30 percent of the dichloromethane NPO it reported on its DEQ-114 form for that chemical. Discrepancies arise for two reasons: Either the facility could have had different base-year NPO totals in its plan than it reported on the DEQ-114, or there was an error in calculating pounds or percent reduction. HRA and NJDEP were usually able to resolve these discrepancies in their site visits or by telephone. When discrepancies could not be resolved, HRA dropped from its analysis those individual chemical goals that were inconsistent. In some cases, this meant dropping an entire facility from a particular analysis.

In addition, one facility reported 1993 NPO amounts and reduction goals high enough to overwhelm any analysis using reduction goals as a measure. This facility was not included in such analyses. Tables of analyses using reduction goals as factors list the number of facilities included and reasons for any exclusions, and Box 5 lists the number of facilities included in all analyses.

III.C.5. Reduction Projections on 1991 and 1992 DEQ-114 Forms
New Jersey facilities have been required to make two- and five-year projections for decreases in NPO due to source reduction on their DEQ-114 forms each year since 1988. HRA examined five-year projections made on facilities' 1991 and 1992 DEQ-114 forms and compared them to projections reported on their plan summaries, both for the 115 facilities visited by HRA and NJDEP and the 405 facilities that submitted plan summaries to NJDEP by July 1, 1994.

Facilities made chemical-specific reduction projections on their DEQ-114 forms, and HRA first examined the projections for consistency. Some projected five-year reductions were greater than the base-year NPO total for that chemical; these were discarded. HRA also discovered that the DEQ-114 forms and instructions gave conflicting directions for calculating projections: The forms indicated that projections were to be reported on a per-year basis, while the instructions asked for the total amount to be reduced in two or five years. It was impossible to determine if the five-year projections were the total amount to be reduced or the total reduction divided by five. To have a basis for comparing these projections, made before pollution prevention planning, with reduction goals set in the plan summaries, HRA devised a lower- and upper-bound strategy to calculate the earlier projections.

HRA took the five-year reduction projection as stated on the 1991 and 1992 DEQ-114 forms to be the lower bound (as long as it was lower than the base-year NPO amount; otherwise, the upper and lower bound were zero). If the five-year projection could be multiplied by five and still remain lower than the base-year NPO amount, that multiple became the upper bound. If the multiplication put the figure higher than the base-year NPO, then the non-multiplied figure served as the upper bound (in other words, the lower and upper bound were the same).

Consistency problems occurred when facilities submitting plan summaries did not submit 1991 or 1992 DEQ-114 forms and when a valid NPO amount could not be calculated for 1991 or 1992. Consistency was also a problem for the plan summaries. HRA checked the chemical-specific NPO reduction goals in plan summaries for consistency against the facilities' 1993 DEQ-114 forms. First, the reduction goal in pounds was divided by the percent reduction goal; the result should have equaled the 1993 NPO for that chemical as reported on the DEQ-114. Inconsistent data for facilities that were visited by HRA and NJDEP had already been corrected, if possible. Of the 115 facilities visited, HRA included 86 to 97 in each analysis, depending on whether 1991 or 1992 projections were used. Inconsistent data for any of the 296 facilities that were not visited could not be checked with the facilities, so HRA applied a +/- 10% consistency rule: As long as the data matched within 10% the goal was retained. In analyzing all plan summaries, HRA included 262 to 307 facilities in each analysis, depending on whether 1991 or 1992 projections were used.

III.C.6. Analysis of Previous Environmental Activities and Facility PerceptionsHRA developed a questionnaire to establish the pollution prevention history of each facility, as well as to determine how facility representatives perceived the planning process. Structured interviews were undertaken. Interviewers used the same list of questions for each facility, with freedom to vary the order in which they were asked. Interviewers had studied reporting forms and pollution prevention plans, when available, and were thus sensitive to inconsistencies and omissions of data that required further questioning to clarify. Open-ended questions gave respondents a great deal of liberty to elaborate. After the interviews, HRA developed a coding format for analyzing responses.

When possible, HRA interviewed the facility employee or consultant who had actually prepared the pollution prevention plan. In a few cases this was impossible because that person was no longer employed at the facility or employed as a consultant. It is important to note that responses to interview questions are self-reported and cannot be verified. Nonetheless, survey responses provide valuable information on facilities' perceptions of their activities which serves as an important counterpoint to information obtained via reporting.

III.D. State and Federal Data Used in the Evaluation

DEQ-114:  New Jersey's DEQ-114 database contains chemical- and facility-specific materials accounting, NPO generation, releases and transfers, and source reduction data. Facilities in New Jersey have reported annually since 1987. DEQ-114 is the only database in the United States that contains data on amounts of NPO not generated due to source reduction. In addition, facilities are required to estimate amounts of NPO that will not be generated due to source reduction in two years and five years following the reporting year.

HRA used DEQ-114 data in three specific ways for this report:

1. Verifying that facility-wide NPO totals in facilities' plans were consistent with data reported on the 1993 DEQ-114.
   
   
2. Verifying that facility-wide NPO and use reduction goals listed in the plan summary were consistent; that is, that the reduction goal, in pounds and percent, was consistent with the base year NPO amounts reported on the 1993 DEQ-114.
   
   
3. The projections of NPO to be reduced by source reduction in five years reported in previous years on the 1991 and 1992 DEQ-114 forms were used to compare with facility-wide reduction goals reported in plan summaries. This comparison attempts to show the influence of the planning process on setting goals.

Although the DEQ-114 data are unique and valuable, they also have limitations:

1.) Materials accounting data are not always consistent. The amount of NPO calculated from the materials accounting inputs and outputs should in most cases equal the sum of the releases and transfers on the DEQ-114 forms. In other words, for each chemical:

In practice, this does not always occur, either due to roundoff or errors in facility reporting. In some cases, NPO calculated by materials accounting is less than zero, so NPO for these analyses was calculated using releases and transfers.

2) Source reduction is probably underreported both in the amounts reduced and projections for amounts to be reduced over five years. HRA examined source reduction activities undertaken by facilities as reported on the DEQ-114, U.S. TRI Form R, and New Jersey Pollution Prevention Plan Summary forms. HRA also asked facilities if they had undertaken source reduction activities. While 60 percent of facilities reported prior source reduction on state and federal forms, 83 percent told HRA that they had undertaken source reduction prior to planning. Facilities described these activities and HRA verified that they were indeed source reduction. Other studies have confirmed that facilities do not always consider appropriate reduction projects to be source reduction, especially when the result was unintentional. Thus, when facilities made five-year projections for amounts of NPO to be reduced by source reduction, it is possible that they were not considering all of the effects of projects under development. Nonetheless, these data are a source of projections for NPO reduction by pollution prevention not found in any other U.S. database.

3) Some projections for amounts of source reduction may actually be use reduction projections. HRA considered facilities' source reduction projections to be NPO reductions, as described above. The instructions for DEQ-114 forms give facilities the opportunity to make either NPO or use reduction projections, although facilities do not have to specify whether the reduction projections are for NPO or use. After consultation with NJDEP, HRA decided to treat all reduction projections as NPO reductions for three reasons: (a) Total use calculations are not required on the DEQ-114 forms; (b) although facilities were required to calculate use in their pollution prevention plans, many of them did not, suggesting that they do not typically calculate use and so would not make use reduction projections; (c) facilities' use reduction goals in their plan summaries were often calculated incorrectly (see Plan Summary data, below).

U.S. Toxics Release Inventory (TRI) Form R: TRI is a facility- and chemical-specific system for reporting on chemicals at industrial facilities in the United States. Since 1987, facilities have reported amounts of releases and off-site transfers to U.S. EPA. On-site waste management activities and amounts were added with the 1991 reporting year, along with source reduction activities undertaken during the year (although not amounts of waste reduced due to source reduction). Facilities also estimate amounts of on-site waste management, off-site transfers, and releases plus transfers to disposal for two years following the reporting year.

HRA used TRI data in two ways for this evaluation:

1.  Comparing amounts of NPO listed in facility pollution prevention plans with amounts of NPO listed in the 1993 DEQ-114 forms and 1993 TRI Form Rs. NPO is called "production-related waste" in TRI and is calculated by summing the data in Sections 8.1-8.7 (on-site waste management plus off-site transfers of waste plus on-site releases and transfers to disposal). This total should equal the NPO amounts on the DEQ-114 forms, which in turn should be the amount of base-year NPO listed in the pollution prevention plan. Forty-eight percent of facilities visited by HRA and NJDEP had DEQ-114 or TRI data that needed to be revised due to this comparison or the planning process.
   
   
2. Examining reporting of source reduction activities in 1991, 1992, or 1993. These data were used to determine if facilities had undertaken source reduction prior to planning.

As with DEQ-114 data, source reduction is probably underreported in TRI. Another issue concerning TRI data was whether or not projections for production-related waste could be used to compare goals made before New Jersey planning was required. Since facilities project amounts of waste generation, it might seem logical to compare these projections made in 1991 and 1992 with goals in the plan summary. However, changes in production-related waste can occur for many reasons other than source reduction, such as changes in production or changing calculation methods, so HRA chose to use the source reduction projections from facilities' DEQ-114 forms instead.

New Jersey Pollution Prevention Plan Summaries: As part of the planning requirement, facilities submitted summaries of their plans to DEP on standard forms (see Section IV for some of the data elements required in plan summaries). These summaries contain chemical-specific, facility-wide reduction goals in use and NPO (both pounds and percent), process descriptions, and percent reduction goals for processes targeted for reduction. Facilities may also describe pollution prevention activities since 1987 that led to reductions in use or NPO.

HRA used the plan summary data in three ways for this report:

1. Facility-wide goals for use and NPO were used as measures to evaluate planning factors as described in Box 1.
   
   
2. Voluntary information on source reduction activities prior to planning was used to determine if facilities had done source reduction before planning was required.
   
   
3. Facility-wide goals for use and NPO were compared to source reduction projections made on facility DEQ-114 forms to determine if facilities made greater percent reduction goals through planning than they had before.

Although plan summary data are a valuable resource, not only for evaluation purposes, but for individuals interested in learning about pollution prevention activities at local facilities, the data have two important limitations:

1. Goals are sometimes inconsistent with base-year data. Data consistency was an issue with plan summaries as with DEQ-114 data. Facility-wide reduction goals in pounds were not always equal to those calculated using 1993 DEQ-114 data and the percent reduction goal. HRA and NJDEP attempted to reconcile these discrepancies by contacting facilities; where they could not be resolved, the reduction goal for that specific chemical was not used.
   
   
2. Facilities did not always calculate use reduction goals according to New Jersey's guidelines. Since NPO is a component of use, numerical reduction goals for NPO should not be greater than those for use. According to definitions in the New Jersey guidance document, facilities should assume constant production levels when calculating NPO and use reduction goals, so NPO reduction amounts should not be greater than use reduction amounts. HRA found that some facilities were not assuming constant production, but rather that the amount of production would increase because of the decrease in NPO, so some numerical use reduction goals were smaller than NPO reduction goals. HRA did not revise use reduction goals unless they were inconsistent with base year use amounts calculated from 1993 DEQ-114 data.

III.E. Statistical Method
A statistical test was performed to determine whether factors of the planning process were associated with different levels of plan completeness, plan quality or percent reduction goals. That is, HRA tested whether some factors (for example, prior source reduction activity) are present, on average, at plants with more complete/better plans than at those without complete plans. The possibility of correlation or association between each program feature and the percent reduction goals or plan characteristics was tested using a contingency analysis and the Chi-squared test for independence.

The analysis identified whether the difference in reduction goal, for example, between the number of facilities having a different level of the program feature (such as those reporting prior source reduction and those not reporting prior source reduction) was statistically significant at the 95 percent confidence level. In other words, it determined whether, 95 times out of 100, the same variation would not have occurred in a hypothetical set of plants with just normal variation and no association between the factor and the goals or plan characteristic.

The Chi-squared test was applied to each pair of program feature and plan characteristic or reduction goal. If the difference between each pair and the hypothetical set of plants was small (as measured against the Chi-squared statistic at the 95 percent confidence level), then the pair is considered statistically independent. If the difference is large, the two features are not independent and may be associated at the plants in the study.

It is important to note that the Chi-squared test only shows whether, on average, a program feature tended to occur with the level of plan completeness or reduction goal; it does not show cause and effect. For example, even though plants with greater amounts of NPO were associated with higher percentage use reduction goals, how much larger percentage use reduction goals would be expected to be with larger amounts of NPO are not shown by this test.