Q. How can I be sure that all my BCD data will get into Biotics?
A. One of the most important things you can do is to run the QC programs accessible at http://www.natureserve.org/prodServices/biotics/biotics-learn-more.jsp#impl prior to conversion. The automated batch QC will find data that will not get either (1) out of BCD into XML, or (2) from XML into the Biotics database. In addition, there are several checks not included in the batch QC that should be done—see the Data QC home page at http://www.natureserve.org/prodServices/biotics/biotics-prep-checklist.jsp for a list of manual processes (such as the check to make sure all records have valid keys) as well as other fields that should be examined.
Q. Why won’t the data just go into Biotics as it is?
A. The main reasons are invalid values or characters in controlled-value fields and mismatched data types. The Oracle database enforces controlled-value fields much more rigorously than in BCD. In addition, some fields that will accept only specific types of data, such as numbers or complete dates. For example, the formatted date fields in Biotics require a year, month, and day. If you have a partial date, the field will be null in Biotics. For example, EORANKDATE needs to be qc’ed to make sure it contains a complete date,* whereas LASTOBSDATE and FIRSTOBSDATE do not require a complete date.
*For more information on EORANKDATE, see below under question about other critical QC’s.
Q. The Duplicate Gname section of the QC: If the tally that is run returns no duplicate gname's, then I assume that everything is fine?
Q. Under the Element_Global_Rank conversions, it states that it is recommended that we bypass this QC entirely. Why is that?
A. The global data was QC’ed at NatureServe prior to conversion from Central BCD so that it would convert cleanly. You will receive the cleaned-up data at your next data exchange. Therefore, QC of the global data (such as EGR, ES, xCAG) in your system is not mandatory. You will still convert it, but the small number of fields with invalid data will fail conversion and be null.
Q. If my program chooses to QC global data, can we update the fields that fail QC?
A. If the data are immediately important to your program (for example, the USESA value for a listed element), you can call your point of contact and request the corrected values from the central database. If, on the other hand, the data are not immediately important, you should wait until your next data exchange to get the corrected values.
A. In the custom.cfg document in the BCD-XML conversion process, the default is to take the values from the ET if they exist. Follow the instructions in the custom.cfg document if you want to change the default values.
Q. The QC document shows that rank factor values are different in Biotics Tracker than they were in BCD. Do I have to change them prior to conversion? (Examples of the rank factor data fields are: SABUND, SESTEOS, SFRAGIL, SPROTEOS, SRANGE, STREND.)
A. No. The QC of rank factors in EGR, ENR, and ESR will point out invalid values (such as any value with a question mark, like A?). You need to correct only those values. The conversion will take your present values and automatically change them to new values in Tracker. The new values represent different break points in quantitative rank factors that were adopted in order to bring the rank factors used by NatureServe in closer synchronization with similar data used by external organizations such as IUCN. (New values are defined in the Help files.)
Q. The instructions for the Element_Natl_Rank seem to be identical to the instructions for the Element_Subnatl_rank. I only ran the complete instructions for the Natl_Rank. Are the instructions for the Subnatl_rank necessary?
A. Yes. These instructions involve configuring the .ini file in two different places—once for the National rank section and once for the Subnational rank section—so you'll want to make sure you read through the documentation and make those decisions in both places. You should be more concerned about your subnational ranks, as these are specific to your jurisdiction.
Q. We use 11 digit numbers for our watershed codes and this generates an error in the QC process. We want to keep our codes, what steps do we need to take?
A. You don’t need to do anything. Your codes will be added to the watershed lookup table during the installation.
Q. In checking the EORANKs for conversion to the new standard I found several EOs with a rank of O. In all cases the EO that had this rank was a very old record that hasn't been seen recently (despite being searched for) and it is questionable whether it was correctly identified in the first place. How should I rank these EOs?
A. All existing O ranks should translate to F ranks.
The F rank (failed to find) is used when something that was once known to occur at a particular location has been searched for more recently and not found for some reason [wrong season, wrong time of day, etc], but may still be there [i.e., no evidence of extirpation and another search may turn it up]). O (= obscure) meant searched for and not found (whether due to a dry year, recent mowing, not sure if searching in exact location, etc.), but habitat still exists in the area so additional searching is warranted. It is felt that continuing to have two ranks with essentially the same definition is not justified, and that locational obscurity is more properly indicated through use of the "Locational Uncertainty" field. Therefore, O is no longer considered a valid EO rank. In the case described above, the identification uncertainty is not reflected in the EO Rank itself, but may be mentioned in the EO Rank Comment field.
There is still ongoing methodological discussion of the validity of some EO ranks. Certain currently valid ranks may be eliminated in the future, and others may be added. At present, the full range of valid possible EO ranks is as follows:
A - Excellent estimated viability/ecological integrity
A? - Possibly excellent estimated viability/ecological integrity
AB - Excellent or good estimated viability/ecological integrity
AC - Excellent, good, or fair estimated viability/ecological integrity
B - Good estimated viability/ecological integrity
B? - Possibly good estimated viability/ecological integrity
BC - Good or fair estimated viability/ecological integrity
BD - Good, fair, or poor estimated viability/ecological integrity
C - Fair estimated viability/ecological integrity
C? - Possibly fair estimated viability/ecological integrity
CD - Fair or poor estimated viability/ecological integrity
D - Poor estimated viability/ecological integrity
D? - Possibly poor estimated viability/ecological integrity
E - Verified extant (viability/ecological integrity not assessed)
F - Failed to find
F? – Possibly failed to find
H – Historical
H? – Possibly historical
X – Extirpated
X? – Possibly extirpated
U – Unrankable
NR – Not ranked
Ranks other than the ones listed above will be caught by the pre-conversion EORANK QC and, if they are not changed, will be nulled out upon import of the data to Biotics.
Q. I am confused on the purpose of the "ConfigurePrincipalEO" portion of the instructions. I am not sure what the purpose of this step is. Is this more applicable to programs that don't already have Biotics 3.1 ?
A. Yes. You don't need to do this if you are already in Biotics 3.1. Some programs have been storing Principal/Sub-EO information in BCD optional/non-standard fields, and this is a way to get that information into Biotics 4. If you haven't done this and you are already in Biotics 3.1 you don't need to do anything in this section.
Q. I don’t understand the InvalidConceptReference report or what to do about it.
A. For conversion to Biotics 4, every element must have a concept reference, which is a reference that describes the circumscription of the element. If such a reference is unavailable or unknown, a temporary “placeholder” reference can be used. In BCD, the concept reference is the NAMEREF field. If NAMEREF is empty, a placeholder concept reference must be assigned. Otherwise, the Element Tracking record cannot be created in Biotics Tracker, and all related records (like EORs) will also fail.
Only elements with an ELCODE not beginning in A, I, P, or N will show up in the InvalidConceptReference report. Placeholder references for species elements with a blank NAMEREF field will be assigned automatically during conversion. In subsequent data exchanges these will ultimately be replaced with the concept references from the Central Databases.
A program does need to make decisions about what the concept reference should be for all non-species elements, e.g., communities or other elements. At a minimum, you need to create appropriate Source Abstract records. Then you must either edit your custom.cfg file or populate the NAMEREF field appropriately.
Here’s an example:
Community records. In looking at their community ET records, a program decides they need 2 concept references:
a) Terrestrial communities (elcode starts “CT”) - B90RES01NYUS - Reschke, C. 1990. Ecological communities of New York state. New York Natural Heritage Program, Latham, NY. 96pp.
b) Palustrine communities (elcode starts “CP”) - UNDNHP01NYUS - New York Natural Heritage Program. Unpublished. Concept reference for community elements for which no reference which describes the circumscription has been recorded; to be used as a placeholder until such a citation is identified.
Other records. In looking at these, all are animal assemblages. A placeholder reference similar to the one created for the palustrine communities is sufficient for all of these:
c) Animal Assemblages (elcode starts “O”) - U03NOV01NYUS - Novak, P. 2003. Concept reference for animal assemblages used in the New York Natural Heritage Program.
As noted earlier, you have two choices. You can edit your custom.cfg file (see directions on the General Configuration and QC guidelines [ConfigureTheSystem] and ET_NO_CONCEPT [InvalidConceptReference]). For this example, you would
The other choice is to use the Global Search and Replace feature of BCD to update the NAMEREF field.
Q: Are there any critical data QC checks that I should do prior to conversion, besides the BCD2HDMS QC routines?
A: Yes. You should look at the BESTSOURCE and SOURCECODE fields in the EOR file, as well as EORANKDATE.
BESTSOURCE and SOURCECODE
First, carefully document how they have been used. Based on this information, you and your point of contact can determine the best way to convert these data, as well as whether you will need to do some data clean-up before or after conversion to Biotics.
Conversion of these fields is complicated because of (1) inconsistency among HP/CDCs in what was put into the BESTSOURCE field, and (2) the fact that it was possible to list a SOURCECODE in EOR that did not refer to a real SA record. In the EO References field in Biotics, the Reference Code must refer to an actual Reference record. Therefore, the default data conversion process creates a Reference record if it finds a sourcecode that does not match an existing SA record from your BCD. The problem is that duplicate reference records may be created by this process.
The default conversion procedure works as follows:
1. The program checks the contents of the BESTSOURCE to determine if it contains 12 characters with no spaces, or more than 12 characters.
2. If BESTSOURCE contains exactly 12 characters, the program tries to match them to the sourcecode of an existing SA record. If it does not find a match, it creates a new record with those 12 characters as the sourcecode/reference code and fills in a placeholder citation.
3. If the BESTSOURCE field contains more than 12 characters, the program tries to match them to the citation of an existing SA record. If it does not find an exact match, it creates a new record with a sourcecode/reference code beginning with the letter U and the BESTSOURCE data in the citation field.
For HPs/CDCs following standard field use guidelines:
If you use the BESTSOURCE and SOURCECODE fields as defined in BCD Help files, BESTSOURCE will always contain a name or citation, and the first row in SOURCECODE will contain the code for the information in BESTSOURCE. Sometimes, however, if the BESTSOURCE is a field form, no SA record was created to go with that SOURCECODE. Even if an SA record exists, the citation in that record and the text in the BESTSOURCE field may differ in some way, such as punctuation or spelling. In either case, the default conversion will result in two references records for the same “best source”: one created from the BESTSOURCE field which would be marked as the Primary Reference (see “Converted Data” section, below), and another created from or already existing for the sourcecode on the first line of the SOURCECODE field. This reference will appear in the References grid in the Tracker EO record, but would not be marked as the primary reference (only one can be marked primary).
Pre-conversion solution (strongly recommended):
1. Make sure that every sourcecode entered on the first line of the EOR SOURCECODE field corresponds to an SA record. Create any missing SA records and make sure that all relevant information in the BESTSOURCE field is captured in them.
2. Create a BCD symbolic the reads the first SOURCECODE and make sure this symbolic is exported with your EOR data.
3. Ask your data conversion contact to use this symbolic instead of the BESTSOURCE field during your data conversion. Data in BESTSOURCE will be ignored, and the reference denoted by the first sourcecode will be marked “primary” for the EO.
1. Identify possible duplicate references (e.g., those starting with U or with F, if SAs for field forms did not exist in your BCD) and find the EOs to which they are linked.
2. Remove unnecessary links to duplicate references from the EO References grid in the EO record, and then delete the duplicate reference records from the database. This is a manual operation.
For HPs/CDCs using BESTSOURCE and/or SOURCECODE fields in a nonstandard way:
1. Make sure your use of these fields is consistent! In other words, if you put a sourcecode in BESTSOURCE, make sure you always have only a sourcecode there.
2. If you don’t want possible duplicate reference records created during conversion, make sure that all the sourcecodes refer to actual SA records.
3. Consult your data conversion contact to discuss options. For example, you may need to create a symbolic different from the one described above, or you may elect to let the default conversion happen and do any needed data clean-up later.
In the current version of Biotics Tracker, this field is a date datatype, meaning that it contain month, day, and year. The conversion program will convert any partial dates it find in this field in to a complete date by doing the following: If it finds a year only, it will fill in January 1 as the month and day (e.g., 1999 becomes 1999-01-01); if it finds a month and year, it will fill in the 1st as the day (e.g., June 2001 becomes 2001-06-01). If the EORANKDATE field contains information that the conversion program cannot interpret, like a date range or words (e.g., 1998-99, 2000-Spring), the field in Biotics will be blank.
Prior to conversion, you should check the data in this field to so if you have dates that will not convert correctly. If you can change them to complete dates, you should do so, but if you do not have sufficient information to change them, the best solution is to ask your installer to create an extensible field and migrate EORANKDATE for now. There are plans to change the Biotics field to accept imprecise dates in the near future. Once that is done, the data in the extensible field can be transferred to the “real” EO Rank Date field by SQL update, and the extensible field deleted.
Q: What if my BESTSOURCE and/or other sources are specimens? I haven’t created SA records for all the specimen listed in my EOR.
A: If your EOR SOURCECODE field contains sourcecodes for specimens which don’t link to an actual SA, you should move the sourcecodes to the SPECIMENS field so that SA records for each specimen don’t get created during conversion (unless you want that to happen). If your BESTSOURCE is a specimen, you should create an SA record for it prior to conversion. See previous question for suggestions about how to avoid creation of duplicate reference records during conversion.
Q: In the custom.cfg document, it seems like I’m doing exactly the same configuration twice for elements with ELCODEs starting with C, G, H, or O. Why is this necessary?
A: There are two separate data attributes that must be set for these elements: Name Category (a replacement for Major Group) and Classification Level (e.g., species, subspecies, variety for plants and animals). These attributes don’t have to be configured for taxa because the data exists in BCD. However, in order to use a common data model for taxa, communities, and “other” types of elements, these fields need to be filled in for all elements—they are required fields in the model. Classification Level doesn’t apply to elements that aren’t in a classification hierarchy, so it was decided that the value options would be the same as those for Name Category—basically, it’s a placeholder value. Therefore, the values for Name Category and Classification Level will be identical for C, G, H, and O elements, but you have to specify them for both fields.
Q: How do I categorize communities developed by NatureServe Central Ecology (ELCODE begins with CEGL)? I have some of them in my database.
A: All terrestrial communities in local Biotics databases will start out with Name Category and Classification Level equal to “Terrestrial Community – Other Classification,” including CEGL records. If all the communities in your database are terrestrial and have an ELCODE that begins with C, all you have to do is remove the asterisk from in front of the lines that begin “C=11” under Name Category and “C=52” under Classification Level. If you have different types of community records (e.g., for subterranean communities, complexes, freshwater communities, etc.), uncomment the lines that correspond to those types of records. You may have to add more of the ELCODE to the lines in custom.cfg to distinguish between the types (see examples in that document), but always select the “other classification” choice.
Even though the CEGL records are from the Central Ecology International Vegetation Classification system, it is safer to convert them as “other classification” because taxonomic and nomenclatural changes may have occurred since the data were put in the local database. There has been no formal data exchange of ecological records to keep them up-to-date. After the Ecology Pilot BCD data conversion is complete (scheduled for August 2003), the latest versions of the relevant Central Ecology records will be sent to the Network for loading in local Biotics databases. Reconciliation with local records will be done later.
Following your data conversion, an SQL update will be done by the installer to change any ELCODE beginning in CEGL to CExx, where xx = the 2-letter abbreviation of your Heritage Program/CDC. The reason for the change is that ELCODE must be unique for each element, and the Central Ecology records could not be added to a local database if their ELCODEs already were in there.
Q. Can my community scientific names be exactly the same as my species scientific names?
A. No, if they are the communities will be loaded in as species and it will take a lot of work to clean up. Check to make sure they are not.
Q: I’m not sure exactly what software is installed on my BCD – how can I check?
A: To determine exactly what software a program has installed, execute the following at TCL:
LIST BCDSYS.CONTROL JUSTLEN 50
The results will look something like the report below. The critical pieces are bolded.
PAGE 1 11:38:23 24 FEB 2003
BIOTICS_CORE_NS_MODULE_001C -- NOTE: the “001C” may be different
BCD2HDMS_2003-02-12 – NOTE: the date refers to the software version.
13 Records Processed
Q. Will BIOTICS accept duplicate quadname/quadcode in the instances of multiple tenten values per quad?
A. No, each quadname/quadcode must be unique to the EO. Duplicates will be rejected – therefore, if you wish to retain multiple tenten values per quad, you will need to add them to a single record (quadname/quadcode). The tenten value in BIOTICS is limited to 20 characters, so anything beyond that will be truncated.
Q. Once we freeze the database to do our final conversion, I am assuming that no data entry can occur in Biotics Mapper. Is this true? For the sake of ease of the install I am guessing that the info in Biotics and in BCD need to be identical.
A. Correct. Once the database is frozen no more data entry should occur in BCD.
Q. Which clients machines require their own workspace on the server? Is it only for the clients that actually use mapper, or do those using tracker need a workspace of their own?
A. Only the users who will be using Mapper require a workspace on the server (the workspace is used to store “clones” of the master shapefiles for each user). If it's an issue and each user will ONLY be using Mapper from ONE machine (per person), the workspace can be local on the client. If the user needs the ability to float around from client to client, then the workspace needs to be on the server.
Q. Does Biotics have the ability to access spatial data stored in ArcSDE?
Note: We are not Crystal Reports experts. We encourage you to talk with Crystal Decisions and to take advantange of the Listservs to gather experience from the Network. Please let us know how you address these issues or if you learn more relevant information, as these issues will be common across the Network.
Q. Once Biotics 4 is installed, we are contemplating whether we should get the full-blown version of Crystal Reports 8.5 or upgrade from ArcView 3.2 to 3.3, which comes bundled with Crystal Reports version 8.5. Is there is much difference between the bundled version and the full-blown version of Crystal Reports 8.5? Obviously, upgrading ArcView would be the cheaper route but we don't want our hands tied with less functionality.
A. We do not have a definitive answer. However, we did some testing with a version bundled with ArcView and our Developer version. Each version could connect to the database and use a report created by the other version.
The latest version of Crystal Reports is 9. It has some significant changes. One is the ability to create and modify SQL directly in the tool and the other upgrades a text limitation for formulas from 256 characters to 4000 (or something like that). It also comes bundled with a light version of a web interface that allows users to execute the reports using a browser (no need to have Crystal Reports installed on their computers). We have not yet tested this version here at NatureServe, but plan to soon. Crystal Reports 9 comes bundled with ArcGIS 8.3. ArcGIS 8.3 is not used with Biotics, but many programs own ArcGIS licenses to run other GIS functions.
Q. We have approx 35 biologists that will need access to at least some of the information in Biotics 4, and all of these 35 staff are at remote locations, do not have reliable web access and we won’t be able to get oracle licenses for each of them. The plan that we came up with was use ArcView as the primary reporting tool. To us this makes sense as the biologists think of the information geographically anyway. We also figured that since Crystal Reports came bundled with ArcView we could use that to allow the biologists to generate the reports that they find themselves using on a regular basis. We figured that it was up to us to create a bunch of standard reports that could be accessed easily (like from a drop down menu) and generated in the most push button way possible. Is what I am proposing even possible?
A. Create a view in Oracle (Biotics data model) containing the attributes you want to distribute. Then, in Biotics Mapper, attach the view to the EO shape file and then use the ArcView convert to shapefile function to produce a new shapefile for distribution. How often you refresh and redistribute this is something you'll need to consider. Use Crystal Reports to create your standard set of reports using the .dbf portion of the shapefile as your data source. Include these reports as part of your distribution package. Integrate the execution of the reports with ArcView. (We have not done this here at NatureServe and cannot offer any advice. Perhaps someone in the Network can help. Also try the Crystal Decisions and ESRI support sites.
Q. Are reports (or report templates) generated in Crystal Reports 9 able to be read and used in Crystal Reports 8.5?
A. We have not started using Crystal Reports 9 yet so do not know the limitations. However, previous versions of Crystal reports had backwards compatibility. You would be limited by the inability to save new functionality in an older version. An example would be that in Crystal Reports 9 you can write formulas against text fields exceeding 255 characters but in Crystal Reports 8.5 you could not.
Q. If we purchase the full version of Crystal Reports 9, would we need the Developer's version, or one of the other versions?
A. This is best answered by comparing the versions on the Crystal Decisions website and by talking with one of their sales reps. I do not think you need the developers edition unless you are planning on seamlessly integrating report functionality through custom software development.
Q. What happened to the BESTSOURCE field in the EOR?
A. If you had data in BESTSOURCE, it is now included in the References grid on the EO Documentation tab. It’s the reference with a checkmark in the checkbox called “Primary,” i.e., it’s the primary reference for that EO.
In order to consolidate BESTSOURCE into the reference list, it was necessary to create a real Reference record for that data, with a unique Reference Code (formerly, SOURCECODE), if it did not already exist. The conversion program creates a Reference record, inserts the record in the reference grid, and marks it ‘Primary.’
See the entry earlier in this document for a full discussion of BESTSOURCE conversion.