Description of Data Exchange bug

Contrary to what we previously assumed about the new Data Exchange process using Biotics, that is, that it would work similarly to the way it worked in BCD, we have discovered that some data that should have been combined in your systems was not properly combined. This may have resulted in duplicate element records in your system with the additional inaccuracy of having your Element Subnational (EST) and Element Occurrence data attached to the wrong duplicate.

We have a way to identify these errors, and are working on a solution that will present these errors to you in a manner that makes them easy and simple to rectify. We're in the process of assessing the scope of this issue and our initial results show us that:

1. This duplication has occurred on a small subset of records in certain NHP/CDC's Biotics Tracker.
2. Only botanical records are affected, not zoological records. More specific communication about this issue and its ramifications will be forthcoming.

During the data exchange process, some records come up because a taxonomic change was performed by either the NHP/CDC or NatureServe, and records involved in taxonomic updates often need to be moved to other element records which more accurately represent the NHP/CDC's view of a taxon/element. The resulting record replacements involve changes to the Universal Keys (UIDs), and during the data exchange upload process, updates to Universal Keys enable the NHP/CDC's data to be moved from one element record to another. We have realized that these Universal Key updates have not been taking place correctly and have resulted in record duplication.

Here is an example of what this will look like in your Biotics Tracker:

Two ESTs with SNAMEs that are the same, the Std/Nonstd table relationship (ELEMENT_STD_NONSTD_REL table) implies overlapping circumscriptions:

Taxonomic comments:

Following Kartesz 1999, the Standard Atriplex patula excludes the rare Californian species Atriplex joaquiniana which had been treated by Kartesz in his 1994 checklist as A. patula ssp. spicata. The Nonstandard Atriplex patula is in the broad sense, including ssp. spicata.

Technical bug description written by technical person.

A Member Program's Biotics Tracker should only have one of these elements depending on what taxonomic treatment the program uses, but the data exchange process has been adding the additional element record, resulting in two subnational records with the same names. Please use the following link to access the SQL to identify cases where two ESTs have the same SNAME. SQL to find most common cases.

There are other cases of record duplication where the names are different, and these are more complicated to detect, however, we are working on ways to accurately detect these.

An additional problem related to how record deletions are executed was also discovered. NatureServe communicates with the NHP/CDC during the data exchange, requesting that they delete specific elements in their systems. This is a routine process that, in the BCD, was handled entirely by the software. It turns out those deletions must be handled manually before the data exchange data are loaded into the Biotics Exchanger application holding area.

Previously, it was believed that doing the deletions during or even after the entire data exchange upload would suffice. This is how it should work, but it turns out that in certain cases, not all, waiting to do a deletion actually prevents the Universal Key updates from taking place. In other words, the record that should have been deleted gets in the way of the UID change process and prevents it from occurring so that a record replacement isn't executed. The net effect is the record that should have been deleted is retained and the new incoming record is retained, when a record replacement should have occurred.

The technical team at NatureServe is exploring ways to automate the deletion process in a manner analogous to the way it was handled in the BCD (so that automatically removed elements could be re-imported into the system at the data manager's discretion). Until this problem is resolved by a technical solution, we must be diligent about performing the requested deletions PRIOR to the post-data-exchange upload.

The new Data Exchange Business Process group is evaluating ways to make the data exchange easier and simpler to do, and will be exploring ways to make processing the information less manual and time-consuming. More information about how to detect and resolve these issues is forth coming and will be posted to this site. Resolution of these issues is our highest priority.