
|
|
SEPA HOME |
Report on SEPA Project Checklist Testing (Draft)SummaryThis report provides the results of testing new draft versions of the SEPA environmental checklist for project actions. The Department of Ecology (Ecology) has worked with an advisory committee to investigate changes to the existing SEPA environmental checklist (WAC 197-11-960). The advisory committee previously developed goals or criteria to guide changes to the checklist. Testing took place with the assistance of two volunteer SEPA lead agencies, Thurston County and City of Yakima, and a number of volunteer applicants. The testing simulated a variety of real-world projects. Three versions of the checklist were tested. The results of the testing indicate there is not unanimous agreement about which changes represent an improvement. However, there are elements of all versions that were viewed as an improvement over the existing checklist. At this point in time, Ecology must make a decision about how to proceed with rule making efforts. Ecology must also determine the relationship between any changes to the project checklist and possible changes to the SEPA Rules for nonproject actions. Given the results of testing, Ecology has the following questions for the advisory committee:
Table of Contents
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Brief Overview of Version 4a
This version is intended to walk an applicant, and later the lead agency, through the environmental analysis thought process. Information provided in Part B is intended to create a basis for everyone to understand the proposal. Part C asks screening questions that were designed to allow simple projects to answer fewer questions. For example a simple project that only answered yes to two questions, would only provide additional information on changes, impacts, and mitigation relative to the two "yes" answers. Additionally, information in part B, would allow not only the applicant to answer the questions in Part C, but also the lead agency and any reviewing agencies should have sufficient information to check the "applicant’s" answers to the screening questions in Part C. This checklist focuses on linking the past environmental analysis and any existing regulations to the project under review as part of the effort to avoid duplication of previous environmental analysis and requirements of existing codes, rules, and ordinances. The checklist asks the applicant about relevant studies and plans and about existing requirements and as a result the applicant is asked to become familiar with applicable laws and relevant reports, etc. |
Brief Overview of Version 4b
The purpose of this version is very similar to version 4a. It is intended to walk an applicant, and later the lead agency, through the environmental analysis thought process. Information provided in Part B describes the proposal. Part C asks screening questions that were designed to allow simple projects to answer fewer questions. A simple project that only answered "yes" to two screening questions, would only provide additional information on changes, impacts, and mitigation relative to the two "yes" answers. For areas of the environment that will or might be changed, Part D asks the applicant to "think" about the changes that their proposal will cause and to think about possible mitigation for those changes. This checklist focuses on linking the past environmental analysis and any existing regulations to the project under review as part of the effort to avoid duplication of previous environmental analysis and requirements of existing codes, rules, and ordinances. The checklist asks the applicant about relevant studies and plans and about existing requirements and as a result the applicant is asked to become familiar with applicable laws and relevant reports, etc. |
Brief Overview of Version 5
When developing this version of the checklist, the goals were to keep the language as simple as possible, provide as much white space as possible, and other formatting methods to make the form as unintimidating as possible and to allow the agency room to make comments throughout the form. The use of yes and no boxes was intended to force the applicant to provide some answer to each question, rather than leave it blank or write "NA". There is a rather large amount of repeated questions in this version, first between Part 1 and Part 2 as Part 2 is intended to extract details that may have been omitted in the Project description. Much information is again requested in Part 3 for the site maps. Finally the information is summarized in Part 4 during the identification of potential impacts. The benefit is that agencies have a greater chance of extracting the information they need from the applicant. The drawback is the tedious burdening of the applicant. Consideration has been made of making Part 4 a worksheet for the lead agency, rather than a portion filled out by the applicant. The advantage of having the applicant complete this portion is not only to reduce the burden on the lead agency, but to force the applicant to consider the potential impacts of their proposal. Even if the applicant proposes no mitigation themselves, it is thought Part 4 better prepares them to expect mitigation conditions from the lead agency. |
With the help of a testing consultant, a testing plan was developed see Attachment C, Usability Evaluation Plan - word format). It was recognized that the test needed to account for a number of variables:
The testing was intended to include representation for each of the variables. It was also designed to test real applicants with their own real projects. Efforts were made to identify applicants and agencies willing to test the new checklist. After extensive efforts were generally ineffective, the test plan was revised and, as a result, may not be as comprehensive as originally envisioned.
The final testing was designed to include:
The final testing schedule and locations included:
Two real projects were selected. One project was a Sports Park on 73.17 acres with a wetland on site. It was described as a multiple-use sports and recreation facility, including eleven lighted softball fields, two soccer fields, a picnic area in an existing oak grove, a mini-golf course, four concession stands, a bowling alley, restaurant, retail and maintenance building and 1000 paved parking spaces. The adjacent areas included a mushroom farm, fire station, residential, vacant land, and the Lacey water tower. Potential issues associated with this project included wetland on site and traffic.
The second project was a residential development on 20 acres. It was described as 75 single-family residential lots, 8 townhouse lots and a reserved track for a 42-unit multi-family development. Some of the area would be reserved for landscaping, buffer and recreation. Adjacent areas mostly consisted of open space, one single-family subdivision, and a private park. Potential issues associated with this project included traffic and stormwater.
Two real projects were also selected. The "simple" project was a demolition of a 19,600 square foot building known as "The Armory" in downtown Yakima. There was a long-term plan to build a justice building, although there were no specific plans for this phase of the project. There was some asbestos in the building. A potential issue associated with this project includes waste handling and disposal.
The second project was a phased 54-unit condominium with 108 paved parking spaces proposed on 5.61 acres. The site includes a "degraded" creek running along the south border. Adjacent areas include single-family, light industrial, and a racquet club. Potential issues associated with this project include potential impacts to the creek (water quality/quantity and habitat), stormwater, erosion during construction, views and glare, service extensions and road access .
The process was basically the same during both tests:
Applicants
Lead Agency reviewers
Table A below summarizes the testing at both locations:
Table A - Details from Draft Project Checklist Testing
|
Thurston County Dec 2000 |
City of Yakima March 2001 |
|
| Room size and organization | All testers in 1 large room with 1 large table; 2 observers available to take notes and answer questions | All testers in 1 room with 1 smaller; 3 observers available to take notes and answer questions; |
| Local information and resources | - 1 county person available for applicant questions - help desk and computer access to a county website |
- 1 county person available for applicant questions - help desk and computer access to a city website |
| Number of participants | 4 on the day of the test, 2 completed on their own |
4 on the day of the test |
| Type of applicants | 3 consultants; 3 Ecology employees (w/ water quality, waste, and GMA experience, respectively) | 2 consultants; 2 Ecology employees (w/ SEPA and water quality experience, respectively) |
| Types of projects | 2 - somewhat complex | 1 complex; 1 simple |
| Versions being tested | Version 4a | Version 4b and 5 |
| Questionnaires | Applicant questionnaire Lead agency reviewer questionnaire |
Same as Thurston County questionnaire except with a few extra, specific questions |
| Lead agency reviewing staff | - 2 with approximately 10 years experience with SEPA - 1 with less than 3 years experience with SEPA |
1 with 10 years experience in SEPA |
| Overall process | Room smaller – more conversation between applicants and/or observers regarding checklist and projects |
Site changes (Cover type changes) in Versions 4a, 4b, and 5
This section regarding site changes generally used the same format and requested the same information in all three versions of the form. It asked applicants to provide before and after acreage or square footage for different cover types (impervious surface, forest, meadow, water surface area, etc.). For all versions, applicants generally filled this section in a very haphazard or incomplete manner. The applicants were probably at a disadvantage due to limitations of the test. This type of information request is present in other states’ environmental review forms (New York and Minnesota). City of Yakima agency reviewer suggested this level of detail was unnecessary and to change it. The change would keep the surface cover types but not ask for specific quantities.
Impacts (with minor moderate, and major) in Version 5
Applicants (2) answered this section (Part 4.2) and answered the optional section regarding minor, moderate, and major. Although they did identify some minor impacts, they did not describe any mitigation for these issues or any others in the next section (Part 5).
Is there a way to get more thought and information from applicants on mitigation? If there were a link between the two sections, they might be reminded to think about and may provide some relevant information in the mitigation section. Another option would be to put sections 4.1 and 4.2 in guidance and put the mitigation back with the individual sections in Part 2.
Special Reports in Versions 4a, 4b, and 5
Version 4a had a Special Reports section in the front, similar to versions 4b and 5. In version 4a, though, applicants were reminded to think about their reports and to provide relevant information from them as they later answered questions about specific issues. In the later sections, the applicants provided more information about Special Reports. One applicant (an experienced consultant) even referred to and provided relevant information from the comprehensive plan. It is possible this may have been due more to the circumstances of the test environment (some applicants felt rushed at the end) and test project than due to the design of the form. (The testing numbers were too small to draw any certain conclusions about this).
In version 4b, the reports were listed and referenced later in the document. Applicants did not have a copy of the reports, so it is unknown whether they would have retrieved information from them to insert in Part D, Changes, Impacts and Mitigation.
In version 5, the special reports section functioned at a level similar to the existing checklist. The reports were listed and not referenced later. Applicants did not have a copy of the reports, so it is unknown whether they would have retrieved information from them to insert in the changes and mitigation sections of the checklist.
Version 4a used by Thurston County
Based on the way the form was filled out, it appeared there was some confusion about how the screening questions worked and when to move to the next section or when to finish answering the questions for the current section. Generally applicants used the screening questions correctly.
Applicants provided more information about possible mitigation. Applicants linked the proposal to relevant reports/plans and reiterated the relevant information (it is possible this may have been a result of the testing conditions rather than the result of the form).
Version 4b used by City of Yakima (shortened version of 4a)
This version did not retrieve any more information from applicants than version 5. Answers in part C (screening questions) and D (existing, changes, and mitigation) appeared to be less useful than the similar answers in version 5. Applicants did not offer much description of mitigation, although there was more in this version than in version 5.
Version 5 used by City of Yakima
The sophisticated applicant misunderstood Part 4.1 Existing Conditions and filled out this section as if it referred to project changes. Both applicants left Part 5, Proposed Mitigation, blank. One applicant described some mitigation in Part 4. This applicant was in a hurry at the end and might have spent more time on Part 5 if he had not felt rushed.
The form elicited information about existing conditions regarding existing water quality in the creek, current uses of the site, including public use. Because mitigation was at the end and optional, it is possible applicants would forget to say anything.
Versions 4b and 5 used by Yakima
One key aspect of the more complex proposal was that water from impervious surfaces would go through an oil/water separator before infiltration on site. Neither version appeared to elicit this information from the applicant although they were fully aware of it (they verbally discussed the proposed use of oil/water separators before infiltration) After review of the forms, the city representative noted that it would be helpful to have more questions about stormwater handling because it is such a big issue.
The applicant and lead agency volunteers answered questionnaires after completing their work on the test checklists. The questionnaires asked the volunteers to provide feedback on attributes of the form with a response using a rating scale (usually from 1 to 5, with 5 being best). The questionnaires also allowed the opportunity to provide narrative responses (included with discussion section of this report). Tables 1b through 6 present the compiled responses to the questionnaires. Responses of volunteers are pooled. However, results are separated to distinguish between test versions and between applicant and lead agency.
It should be noted that the sample size for the questionnaire responses is not large enough to strictly rely on the numerical results. However, when combined with feedback from the narrative response and the discussion, the results provide some insight into the effectiveness of the forms. In some cases, the individual response on the questionnaires differs from what was heard during the discussion portion of the testing.
Table 1 and Table 2 show responses to a variety of questions addressing the overall usability of the form. Table 1 shows only small differences between the three test versions. Version 4a seemed to be viewed favorably by the applicants, with the only negative response being the difficulty of getting information to complete the form. This is in contrast to the largely negative response from the lead agency on the same version. Version 5 received slightly higher applicant ratings than Version 4b. The lead agency reviewer for Versions 4b and 5 combined his response for both versions. Table 2 shows markedly differing responses by the two agencies on several of initial format related questions, possibly indicating improvements made prior to the City of Yakima test. At the bottom of Table 2, responses from the City of Yakima indicate how well the form integrated information requirements of other forms.
Table 1: Applicant Responses on Overall Usability of Form
(Average response of all respondents. Responses scored from 1 to 5, with 5 being best)
| Question | Version 4a (Thurston) | Version 4b (Yakima) | Version 5 (Yakima) |
| Number of applicants |
6 |
2 |
2 |
| Order of questions |
3.8 |
3 |
3 |
| Page format |
3.5 |
2.5 |
3.5 |
| Understanding the questions |
3.3 |
3 |
3.5 |
| Duplicate questions |
N/A |
1.5 |
2.5 |
| Getting information to answer the questions |
2 |
2.5 |
2.5 |
|
Form
allowed me to provide necessary information to the lead
agency (yes/no, number of responses) |
Yes = 4 No = 2 |
Yes = 2 |
Yes = 1 No = 1 |
|
Used
instructions (yes/no, number of responses) |
Yes = 3 No = 3 |
Yes = 2 |
Yes = 2 |
| Instructions helpful and understandable |
3.7 |
2.8 |
3.4 |
Table 2: Lead Agency Responses on Overall Usability of Form
(Average response of all respondents. Responses scored from 1 to 5, with 5 being best)
| Question | Version 4a (Thurston) |
Versions 4b and 5 (Yakima) |
| Number of agency staff reviewers |
3 |
1 |
| Order of questions |
2.7 |
4 |
| Page format |
1.7 |
4 |
| Understanding the questions |
2 |
4 |
| Reviewing the answers to questions |
3.3 |
3 |
| Verifying the answers |
3 |
3 |
| Questions asked for needed info |
3 |
4 |
| Completed checklist provided necessary information |
2.5 |
4 |
| No unnecessary information |
2.5 |
3 |
| Need for separate agency guidance |
NA |
Yes |
|
Used instructions (yes/no, number of responses) |
No = 2 |
Yes |
| Instructions helpful and understandable |
NA |
4 |
| Overall ease/difficulty of form review |
2 |
3 |
| Integrates Notice of Application (NOA) information requirements |
NA |
2 |
| Integrates agency's Master Application information requirements |
NA |
4 |
| Integrates Joint Aquatics Resources Permit Application (JARPA) information requirements |
NA |
2 |
Table 3 and Table 4 compare the existing SEPA checklist (WAC 197-11-960) to the versions used in the tests. The Thurston County test showed very different responses by the applicants versus the lead agency; the applicants showed a strong preference for the test version, whereas the lead agency strongly preferred the existing version. The City of Yakima responses were less emphatic, with no clear preference shown by the applicants. There does appear to be some overall agreement that existing checklist is easier and that the test versions are more complete.
Table 3: Applicant Responses Comparing Test Version to Existing Checklist
| Checklist Attributes |
Version 4a (Thurston) |
Version 4b (Yakima) |
Version 5 (Yakima) |
|||
| existing | test | existing | test | existing | test | |
| Easier |
2 |
2 |
2 |
1 |
||
| More logical |
4 |
1 |
1 |
|||
| More helpful |
4 |
1 |
1 |
1 |
||
| More understandable |
4 |
1 |
1 |
|||
| More complete |
4 |
1 |
1 |
1 |
||
| Overall, which do you prefer |
4 |
1 |
1 |
1 |
||
Table 4: Lead Agency Responses Comparing Test Versions to Existing Checklist
| Checklist Attributes |
Version 4a (Thurston) |
Versions 4b and 5 (Yakima) |
||
| existing | test | existing | test | |
| Easier |
2 |
1 |
||
| More logical |
1 |
1 |
||
| More helpful |
1 |
1 |
||
| More understandable |
2 |
1 |
||
| More complete |
1 |
1 |
||
| Overall, which do you prefer |
2 |
1 |
||
Table 5 and Table 6 shows responses to some specific question regarding formatting issues. These questions were added for the City of Yakima test. Table 5 shows that the applicants were generally agreeable on formatting issues, even when the questions were intended to elicit a preference (e.g., two respondents indicated that they liked both lines and white space to write in). The only strongly negative response was on multiple column formatting. The response on questions regarding guidance seems to indicate that the applicants universally like to see guidance in the form itself, although some also like more detail proved in a separate guidance document. Table 6 shows lead agency response to formatting questions that were asked only during the Yakima test.
Table 5: Applicant Responses on Specific Formatting Issues
(Shows number of responses for each question within category)
| Specific format issue |
Version 4b (Yakima) |
Version 5 (Yakima) |
||||
| Like | Don’t Like | Don’t Care | Like | Don’t Like | Don’t Care | |
| Lines to write in |
2 |
1 |
1 |
|||
| White space to write in |
1 |
1 |
2 |
|||
| Boxes organizing questions on page |
2 |
1 |
1 |
|||
| Guidance in the form itself |
2 |
2 |
||||
| Guidance in separate document |
1 |
1 |
1 |
1 |
||
| Check boxes |
2 |
2 |
||||
| Single column of questions |
2 |
2 |
||||
| More than one column |
2 |
2 |
||||
Table 6: Lead Agency Responses on Specific Formatting Issues
(Shows number of responses for each question within category)
| Specific format issue |
Versions 4b and 5 (Yakima) |
||
| Like | Don’t Like | Don’t Care | |
| Organize form separating sections with fact finding questions from sections with analysis questions |
1 |
||
| Organize form by resources issue |
1 |
||
| Screening questions |
1 |
||
| Asking open ended questions requiring narrative response |
1 |
||
| Asking mostly yes/no questions |
1 |
||
Version 4a - Used by Thurston County
Applicant feedback (4 people)
Lead agency feedback (3 staff people)
Versions 4b and 5 - Used in Yakima
Applicant feedback (4 people)
Lead agency feedback (1 person)
This section provides analysis of how well the project checklist versions performed - based on both test results and Ecology's own review of the checklist versions. A key issue for the revision process, and a range of possible Ecology actions related to the project review form is described. Specific questions Ecology requests input from the SEPA Checklist Advisory Committee is also detailed.
Table 7 contains a comparison of the checklist versions and the existing checklist against the evaluation criteria and goals that were previously identified. While the test versions were able to improve on the existing checklist in a number of areas, no version was effective in meeting all the criteria.
Table 7: Checklist Versions Compared to Evaluation Criteria
(How well did checklist version meet criteria? Rated from 1 to 5, with 5 being best)
| Evaluation Criteria | Version 4a | Version 4b | Version 5 |
Existing Checklist |
| Integrates SEPA and GMA and uses same terminology |
5 |
5 |
5 |
1 |
| Starts with decisions already made, rely on previous analysis |
4 |
3 |
1 |
1 |
| Contains good project description |
4 |
4 |
4 |
2 |
| Contains NOA elements |
2 |
2 |
2 |
1 |
| Condensed format for simpler or consistent projects |
4 |
5 |
2 |
2 |
| Form and associated guidance is logical and helpful |
2 |
3 |
4 |
3 |
| Applicants fill out all the information |
3 |
2 |
2 |
1 |
| Applicants can provide the requested information |
2 |
2 |
2 |
3 |
| The form ask for the right and/or sufficient information |
4 |
3 |
4 |
1 |
| The questions get answered correctly |
3 |
3 |
3 |
2 |
A key issue remains unresolved. Should we proceed with a checklist version that does not support "Starts with decisions already made, relies on previous analysis"?
As stated in the Background Section, one of the key objectives of the checklist revision effort was to create a checklist that could be used as a tool to assist agencies to be able to "add" to previous environmental analysis and to fill the "gaps" in environmental regulations. It appears that designing a checklist to accomplish this objective is perceived as making it longer and more difficult. Version 5 and the existing checklist do not accomplish this objective. Changing version 5 to meet this objective would essentially negate many of the strengths of this version.
There are many barriers to accomplishing this objective. The checklist is only one tool that, even if perfectly designed, could not alone result in full success in meeting this objective. Practically speaking, the objective is a difficult one. With the current tracking system for existing environmental data and the challenges of accessing relevant information in plans, rules, and ordinances, it is difficult for any one person (applicant or agency staff) to identify: 1) analysis that is already done and 2) all local, state, and federal regulations that fully or partly mitigate impacts.
The first checklist version, known as version 2 (See Attachment O, Version 2 Draft Environmental Checklist) was probably formatted in a manner that would best accomplish this checklist objective. It was not well received by several committee members. It was felt that average applicants could not fill it out. Guidance was integrated into the checklist, resulting in a lot of text and discussion for each question. Terminology was fairly technical. As a result of various factors, this version did not get tested and was modified and integrated with version 3. Because this version did not get tested, it is unknown how well it would have accomplished the objective.
Ecology would like to move forward to resolution of the project checklist issue, regardless of whether we drop the effort or move forward with broader public review of a proposed form. If we are going to make rule changes to the project checklist, it would best occur simultaneous with nonproject revisions.
At a minimum, Ecology would like to proceed with creating some guidance materials for checklist users and also making some minor changes to the existing checklist. From the testing efforts, we received positive feedback on guidance; ideally the guidance would be included both in the form itself and in a more detailed companion guidance document. Additionally, we would like to further investigate adding a requirement for inclusion of a site map with the checklist.
If the SEPA Checklist Advisory can reach consensus on a version of the checklist to pursue, further improvement can be made to that version and Ecology can proceed with a broader public review. If there is no consensus among advisory committee members, Ecology will likely focus efforts on guidance and minor changes to the existing checklist (e.g., include GMA terminology).
At a future advisory committee meeting, Ecology would like to hear responses from the committee members to the following questions. For members not attending, we would like to receive response by the end of April, 2001.
For general questions or comments concerning SEPA, please e-mail the SEPA Unit or call (360) 407-6922.
Copyright © Washington State Department of Ecology. See http://www.ecy.wa.gov/copyright.html.
|