In our August issue of Newscast, we asked readers for their questions. We heard from Capt. Monty Hawkins, a charter boat operator, blogger, and recreational fishing advocate, who sent us two related queries. For the purposes of space, we've condensed them below. To read the full text with his accompanying commentary, along with his other posts, you can visit Capt. Monty's blog at
http://blog.morningstarfishing.com/ Question:
How can you say MRIP is "better" than MRFSS when there are still so many estimates that appear to be obvious outliers? Specific examples include:
- Wave 2, 2010, New Jersey Shore Mode tautog catch. The estimate was 483,198 pounds. That number is greater than the TOTAL for-hire Wave 2, 2010, catch PLUS the total commercial landings for the WHOLE YEAR.
- Wave 3, Massachusetts Private Boat Mode black sea bass catch. The estimate was 246,973 sea bass in Wave 3 alone. That number is greater than the catch of the entire East Coast for-hire fleet through Wave 3.
Where's the head-count? Where are the statistical stops to prevent wild flyers in the data? Isn't there some way to clearly flag the "bad" numbers and only report the ones that make sense?
MRIP Responds
Dear Capt. Monty,
Thank you for your questions. These are important issues with complex explanations that straddle the line between the science of producing estimates of recreational fishing activity, and the most appropriate use of those estimates to fairly and sustainably manage recreational fisheries.
On the science side, MRIP has implemented a number of significant, peer-reviewed improvements to our previous recreational fishing data collection program. The basis for these improvements is a 2006 review of MRFSS by the National Research Council (NRC), a leading group of independent scientists. One of the chief concerns raised by the NRC was that our catch estimation methods introduced the potential for bias in our results. In statistics, bias can occur when you make assumptions about your data that you haven't tested, such as assuming that catch rates are the same during different parts of the day.
The NRC recommended a number of specific changes to MRFSS to remove the potential for bias from our estimates, which the MRIP team - made up of NOAA representatives, state partners, outside consultants, fishermen and other stakeholders - has systematically worked to implement over the past several years. Complete details of all our projects can be found at our website, www.countmyfish.noaa.gov.
With these improvements in place, we can say with confidence that we have enhanced the quality of our estimates. In fact, the cases you cite are good examples for demonstrating exactly what we mean by that. To begin with, it is important to recognize that when we talk about an "estimate," we're actually talking about two numbers:
- The first is the "Point Estimate," which is the number you refer to in your question.
- The second is the "Precision." In polling, this is often referred to as the "margin of error." In our estimates, we use a measure called "percent standard error" (or PSE). Precision tells us how confident we can be in the point estimate.
For an estimate to have any real-world meaning, BOTH of these numbers have to be taken into account. That's because if there is a high PSE, then we are less certain that the point estimate reflects the true value, a fact that has to be accounted for when using the data. However, less precision is not the same as less accuracy. Because we have removed the potential for bias from the way we estimate catch, MRIP's new numbers - the point estimates combined with the PSEs - are still a more accurate estimation of recreational fishing activity.
In the tautog example you ask about, the PSE was a very imprecise 86.4. One of the reasons the PSE is so high for this species in this mode is because we don't encounter many people catching them. Because of the way that sampling and estimation work, there is a good chance that the point estimate for any individual species and type of fishing (mode) during a single two-month sampling period (wave) may seem unrealistically high or low. Although it is typically the high "outliers" that tend to get the most attention, they must also be taken in context with the low outliers; considering both is an important part of evaluating the bigger picture.
As an example, the table below shows Wave 2/New Jersey/Shore Fishing/Tautog Catch Estimates from 2000 through 2012. In nine of those years, the estimate was zero tautog caught (PSEs cannot be calculated for zero catch). In years where there has been reported catch, the PSE is very high.
Table 1
Screen grab from data query publicly available at www.countymyfish.noaa.gov.
| FINAL | 2000 | MARCH/APRIL | TAUTOG | 4,287 | 100.0 | 0 |
| FINAL | 2005 | MARCH/APRIL | TAUTOG | 0 | . | 0 |
| FINAL | 2009 | MARCH/APRIL | TAUTOG | 13,966 | 96.0 | 0 |
| FINAL | 2010 | MARCH/APRIL | TAUTOG | 483,198 | 86.4 | 0 |
| PRELIMINARY | 2012 | MARCH/APRIL | TAUTOG | 11,388 | 101.7 | 0 |
Both statistically and anecdotally, it is equally unlikely that zero fish were caught during any given year as it is that there was a 35-fold increase in catch in 2010 over 2009. Therefore, what these numbers indicate more than anything is that our samplers encounter very few individuals catching tautog from the shore in New Jersey during Wave 2.
To improve precision we would need to substantially increase the size of our intercept sample, which would mean talking to significantly greater numbers of anglers. That, in turn, would significantly increase the cost of the surveys. As we discuss below, this is certainly an option, but it must be weighed carefully against all the other competing needs for those resources.
With regard to black sea bass, the PSE for Wave 3 in 2012 was 30.9. This is far more precise, but there's still a fairly wide margin in terms of the potential number of fish caught. It's also worth pointing out these are preliminary estimates. Before they're finalized, all of our estimates go through an extensive quality control process, which includes a point-by-point data review with the specific purpose of looking for collection errors.
This process is part of what we do to "flag" outlier numbers. In addition to our own review, preliminary estimates are open to public scrutiny so that individuals, such as yourself, can point out numbers that should get closer scrutiny. We have also added new features to our query outputs that highlight especially high PSEs, which can be output as either graphs or tables.
As you note in the rest of your post about black sea bass, even if these particular point estimates hold, as we begin to look at data over a longer and longer time series, or across broader geographic areas, the PSE declines and the point estimate becomes more precise. (Readers can see the numbers for themselves and run their own queries at www.countmyfish.noaa.gov.)
This leads to the issue of how best to use the data that our surveys produce, a challenge highlighted in the recent decision to close the black sea bass fishery. (More information about the closure decision is available from NOAA Fisheries' Northeast Regional Office (www.nero.noaa.gov)). As managers face new mandates to ensure that overfishing is not occurring, we may find a greater need for more precise estimates delivered more frequently for some species during some parts of the year. Each of these needs has costs associated with it. Ultimately the question of where the money will come from and how to spend it is part of the dialogue that takes place among fisheries managers, scientists, fishermen, coastal community representatives, and other stakeholders. But the tools are being put into place to get the information when it's needed.
In addition, the work to make our surveys even better - and to anticipate the emerging needs and opportunities of the future - is continuing. Numerous MRIP-funded studies are underway looking at everything from how to improve survey response rates, to rethinking how we count for-hire catch, to looking at ways to enable anglers to submit their own data. As each study is completed, the findings are incorporated into the overall program, making the process of improvement incremental and ongoing.
In closing, we'd like to highlight three main points:
- The estimates we produce under MRIP represent a clear and quantifiable improvement and we have confidence in their accuracy, but point estimates always need to be considered in the context of the margin of error.
- We recognize that management sometimes has to occur at a finer scale - either in terms of geographic area or time period - than our estimates are ideally suited for. As we complete the implementation of our fundamental design improvements, we will work with managers, scientists, fishermen, and other stakeholders to evaluate and prioritize investments in programs to meet data user needs for finer precision, timeliness, and geographic resolution.
- MRIP is an ongoing process of making improvements and addressing shifting needs. We know that the best way to improve the system is through an open and interactive process. We appreciate the attention of fishermen who care enough about the future of recreational fishing to remain informed and engaged.
