Nederland uses LiDAR Data for decision-making

Article provided by Lex Ivey, TerraCognito GIS Services Principal. Lex can be reached at 303-258-3515 or [email protected].

The 2014 Denver Regional Aerial Photography Project (DRAPP) is picking up steam. RFPs were released in late August. We received a total of 16 bids: six for imagery, four for DAT, four for LiDAR and two for WMS.

During the first week of October, DRCOG distributed the bids to evaluation teams made up of potential DRAPP 2014 partners. This process of having our partners evaluate, score, and interview our potential vendors is essential to the success of DRAPP because it helps ensure the chosen vendor is acceptable to our varied partner group.

Scoresheets are due back to the DRCOG Project Manager Oct. 24. That means there is still time to participate in the evaluation process! Please get in touch ASAP if you would like to review bids from any of the categories (Imagery, LiDAR, Quality Control, WMS/Resale).

Based on the closeness of the scores, DRCOG will determine how many of the candidates to interview. We usually bring in the top two from each category.  Interviews will be held sometime between Oct. 28 and Nov. 4. Approval of the vendors by the DRCOG Board will be sought Nov. 20.

Our next meeting will be held the morning of Nov. 21. All potential partners should attend. At that point, we will have chosen a vendor, and project costs will be known.  This meeting will be the primary time for you to represent your organization's needs on a few remaining specifications before the final contracts are written with the vendor in January 2014.

Also, please be alerted that Letters of Intent (LOIs) will be going out earlier than normal for this project. The Letter of Intent (LOI) process is the means by which DRCOG solidifies the contract with partners and clarifies the amount due. These will be sent to potential partners in late November/early December and need to be signed and returned as soon as possible.

If you have questions or concerns, please contact Ashley Summers at [email protected] or 303-480-6746. 

Colorado Data Summit brings data community together

The Colorado Secretary of State's Office recently created a Business Intelligence Center (BIC) to "aggregate public data and make it available to the widest audience in the most useful format." To this end, the BIC hosted a Colorado Data Summit in late August to encourage a more cohesive open data community, identify issues and opportunities, and determine next steps for this statewide initiative. Attendance included representatives from cities, counties, regional and state government, non-profits, universities, the health care industry and private technology firms.

 
For those of us in the GIS community, the issues surrounding inaccessible data are well known. Silo-ed or restricted data leads to duplication of effort, as each organization has to mine, "clean," and interpret similar datasets, instead of sharing one amongst themselves.  Due to current data restrictions, each organization in the state that needs this information must request it, pay for it, and prepare it for analysis. This process not only wastes time and money, but creates inconsistencies between datasets and results that are supposed to be comparable.

 
Another identified barrier is lack of a governance structure for open data to guide standardization of data models and associated data products (e.g. metadata). A governance structure could also create a policy framework for data publishing that honors ownership and maintains data integrity.

The group went on to discuss the need for a central repository for data that could serve as a "one-stop-shop" for the State of Colorado. Currently, there are many data portals working toward this goal. The Colorado Information Marketplace, OpenColorado, and the Piton Foundation's Colorado Data Engine all serve out statewide data. DRCOG serves regional-level data through the Regional Data Catalog, and many local governments are beginning to put data on their own websites.

While there were many issues and barriers to be discussed, there are equally as many opportunities for success. For example, better illustrating the benefits of open data versus the perceived risks could help build momentum for change. Celebrating the organizations already ushering in change by sharing their own data, and engaging the data user community to provide organizations with feedback on data usefulness are other strategies that were discussed.

Next steps for this group include meeting again in a few months to continue this discussion and to make an action plan.  

In the world of GIS, we understand the importance of specifying the correct projection and datum of our data. To neglect this could cause misalignments or distortions that lead to error in our analyses.  Even though we know this in theory, how many of us dig deeper into the accuracy issue and ensure that we are specifying the correct Datum Realization, Epoch and Geoid for all our datasets and capturing that in the metadata? Probably not many.  That may need to change in the near future because with increasing capabilities of our data collection technologies (e.g. LiDAR) and more demanding project requirements (e.g. to support survey-level work), failing to understand and address these components could lead us down a rabbit hole of errors.

 
For those of us who haven't been in school for while or those who picked up GIS on the job, here's a little refresher on terms.

An Ellipsoid is a mathematical surface. Its dimensions are determined as a best fit to the world's geoid model and allows for easier calculation of positions.

The Origin in the past (e.g. for NAD27) was a point at Meades Ranch, KS based on the Clarke ellipsoid of 1866. The origin for more recent datums (e.g. NAD83 and WGS84) is the center of the earth's mass based on the Geodetic Reference System of 1980 (GRS 80).

Image from http://www.spatial-ed.com

A Horizontal Datum refers to the reference specifications of a system of coordinates. The datum will specify the ellipsoid and the origin.     

Datum Realization (also referred to as a Datum Adjustment) occurs when the National  Geodetic  Survey (NGS) re-computes the positions of  known points (i.e. survey marks) based on new data observations to account for technological improvements and  movements across the earth's surface.

An Epoch is the date that the coordinates were held in the realization. An epoch is simply an extra level of granularity used to provide a time stamp of the positional information. This has become more critical in places that are very tectonically active (Note that Colorado is fairly tectonically stable, so we don't have to worry much about epochs). 

A Geoid represents the gravity potential that approximates mean sea level.

To summarize those terms, let's dissect this coordinate system description:

NAD 83(2011) epoch 2010.00.

Now the big question: Why do we care about these things?

 
Answer: Because it can affect the accuracy of our work. 

NGS has made four realizations to NAD83.

1. The first realization was released in 1986 before GPS (so it was 2-dimensional). The relative national network accuracy was at the 1m level.
2. The second realization was done on a state-by-state basis due to the advent of GPS, and exceeded the accuracies published in the 1986 realization. This realization occurred in Colorado in 1992. It is referred to as the High Accuracy Reference Network (HARN) realization and it increased accuracy nationally tenfold; to the 10cm level. Relative accuracy was actually at the 2 cm level here in CO from the HARN.
3. The third realization was released in 2007 - called NAD83 (NSRS2007) - and was based on approximately  70,000 monuments. The data from GPS observations on the monuments were tied to 700 stations in the Continuously Operating Reference Stations (CORS) network - called NAD83(CORS96). Due to the continuous GPS data collected by CORS, the nationwide accuracy level increased to 1cm, and again,  Colorado remained at a relative accuracy at the 2 cm level. (Keep in mind that this realization, as well as the first two, has been officially superseded).
4. The fourth realization was released in 2011 - called NAD83(2011) epoch 2010.00. Observed data at approximately 80,000 NGS monuments were tied to  1195 CORS (IGS08)  epoch 2005.00 and produced better absolute accuracy, both horizontally and vertically. If you use any of the NGS products or tools (e.g. Opus, NGS CORS data) it is best to use this realization to ensure consistency.

For much of the analyses that we do in government GIS, we are looking at data trends at a less-detailed level. Plotting incidents of crime, estimating canopy cover in urban landscapes, and determining a walkshed to a train station don't rely on sub-centimeter accuracy.

However, when you are building bridges, putting in water lines, or working with planimetric data (e.g. Edge of Payment or Building Outlines) the difference between 1cm, 10cm, and 1m could be substantial. It all comes down to what you are trying to do with your GIS data.

The most important concepts to take away from this article are: 

1. Consider updating your datasets to the most recent datum realization to stay in sync with NGS.  If you can't do that, then...
2. Document, document, document the full projection and datum for all your datasets and data sources so that you and all users of your data know the level of error when the data is applied.

How to apply this knowledge in ArcMap:
Most of us use a State Plane projection in US Feet - either North or Central, so that leaves us with the following eight options when mapping our data. Here's how we think they correspond to the realizations listed above.*
  

Read about the differences between CORS96 and NSRS2007 here.
Download NAD 1983 NRS2011 projection files here. 

*Pam, Ashley, and ESRI Technical Support collaborated on crosswalking these projections to their corresponding realizations. With that said, the authors have not yet found official documentation that confirms these assertions. 

Next Steps
We will be featuring more articles and presentations on this topic through the Data Consortium over the next several months. In the interim, if you have specific questions or want more information on this complex topic, please contact Pam Fromhertz at NOAA. 

In the case of a public entity that is charging for electronic data, the options suggested by our staff attorneys are:
Option 1.  Work with Data Consortium members to persuade the charging entity that the data is being used for a "public purpose" and request the fee be waived per CRS 24-72-205 (4).

 
Option 2.  Inquire as to other entities receiving this information from the public entity at no charge and convince the entity that the Data Consortium members are "similarly situated" as describe in CRS 24-72-205 (4).

Option 3.  Ask the public entity the basis for the fees they charge so we can determine if the fees are consistent with CORA.

Option 4.  If legally defensible, challenge the fee schedule in court.

The Data Consortium's mission is to provide members with the most current authoritative information to serve our customers.  When some jurisdictions do not participate, we all lose.  We are left scrambling to come up with data to fulfill our roles.  This is why the effort to acquire data for public purposes is so important. 

Moving forward, I suggest we consider Option 1 when faced with government entities that charge for their data.  A collective effort at making a formal request to waive fees for those of us in the public sector could be an effective course of action.  Hopefully these entities would come to realize the greater benefit of sharing data and amend their current policies. 

Options 2 and 3 would be the next steps to help us understand the scope of how data is being sold and to whom.  As the CORA states, a portion of the cost of creating the data can be recovered.  Do charging entities have a review schedule for their fees like many private- sector companies?   Also, what constitutes a case where data is freely given to government entities and is it consistent with CORA?  While jurisdictions like Denver and Broomfield have given up charging for digital data, what  revenue is being  generated by those that do?  These are just some of the questions that would help us make a better case for data sharing. 

The last option is not viable at this time.  Court proceedings are expensive and time-consuming.  Still, if it comes to that, there must be enough participants willing to fund an effort to make the case worthwhile.   If you think of all the time wasted in duplicate effort, the economics become more favorable. 

This is an ongoing effort that will discussed and followed up on at future Data Consortium meetings.  If you have any comments or would like to discuss this effort, please contact Dave Murray at 303-658-2140. 

Additional reading:
21st Century Sunshine: Modernizing CORA

What is a reasonable CORA fee?
Changing technology landscape requires changing open data policies