Q. How can I be sure that all my BCD data will get into
Biotics?
A. One of the most important things you can do is to run the QC programs accessible at http://www.natureserve.org/prodServices/biotics/biotics-learn-more.jsp#impl prior to conversion. The automated batch QC will find data that will not get either (1) out of BCD into XML, or (2) from XML into the Biotics database. In addition, there are several checks not included in the batch QC that should be done—see the Data QC home page at http://www.natureserve.org/prodServices/biotics/biotics-prep-checklist.jsp for a list of manual processes (such as the check to make sure all records have valid keys) as well as other fields that should be examined.
Q.
Why won’t the data just go into Biotics as it is?
A. The main reasons are invalid values or characters in controlled-value fields and mismatched data types. The Oracle database enforces controlled-value fields much more rigorously than in BCD. In addition, some fields that will accept only specific types of data, such as numbers or complete dates. For example, the formatted date fields in Biotics require a year, month, and day. If you have a partial date, the field will be null in Biotics. For example, EORANKDATE needs to be qc’ed to make sure it contains a complete date,* whereas LASTOBSDATE and FIRSTOBSDATE do not require a complete date.
*For more information on EORANKDATE, see below under question about other critical QC’s.
Q. The Duplicate Gname section of the QC: If the tally that is run returns no duplicate gname's, then I assume that everything is fine?
A. Correct.
Q. Under
the Element_Global_Rank conversions, it states that it is recommended that we
bypass this QC entirely. Why is that?
A. The
global data was QC’ed at NatureServe prior to conversion from Central BCD so
that it would convert cleanly. You will receive the cleaned-up data at your
next data exchange. Therefore, QC of the global data (such as EGR, ES, xCAG) in
your system is not mandatory. You will still convert it, but the small number
of fields with invalid data will fail conversion and be null.
Q.
If my program chooses to QC global data, can we update the fields that
fail QC?
A.
If the data are immediately important to your program (for example, the
USESA value for a listed element), you can call your point of contact and
request the corrected values from the central database. If, on the other hand, the data are not
immediately important, you should wait until your next data exchange to get the
corrected values.
A.
In
the custom.cfg document in the BCD-XML conversion process, the default is to
take the values from the ET if they exist. Follow the instructions in the
custom.cfg document if you want to change the default values.
Q. The QC document shows that rank factor values are different in Biotics Tracker than they were in BCD. Do I have to change them prior to conversion? (Examples of the rank factor data fields are: SABUND, SESTEOS, SFRAGIL, SPROTEOS, SRANGE, STREND.)
A. No.
The QC of rank factors in EGR, ENR, and ESR will point out invalid
values (such as any value with a question mark, like A?). You need to correct only those values. The conversion will take your present values
and automatically change them to new values in Tracker. The new values represent different break
points in quantitative rank factors that were adopted in order to bring the
rank factors used by NatureServe in closer synchronization with similar data
used by external organizations such as IUCN. (New values are defined in the
Help files.)
Q. The instructions for the Element_Natl_Rank seem to be identical to the instructions for the Element_Subnatl_rank. I only ran the complete instructions for the Natl_Rank. Are the instructions for the Subnatl_rank necessary?
A.
Yes. These instructions involve
configuring the .ini file in two different places—once for the National rank
section and once for the Subnational rank section—so you'll want to make sure
you read through the documentation and make those decisions in both
places. You should be more concerned
about your subnational ranks, as these are specific to your jurisdiction.
Q. We use 11 digit numbers for our watershed codes and this generates an error in the QC process. We want to keep our codes, what steps do we need to take?
A. You
don’t need to do anything. Your codes will be added to the watershed lookup
table during the installation.
Q. In checking the EORANKs for conversion to the new standard I found several EOs with a rank of O. In all cases the EO that had this rank was a very old record that hasn't been seen recently (despite being searched for) and it is questionable whether it was correctly identified in the first place. How should I rank these EOs?
A. All existing O ranks should translate to F ranks.
The F rank
(failed to find) is used when something that was once known to occur at a
particular location has been searched for more recently and not found for some
reason [wrong season, wrong time of day, etc], but may still be there [i.e., no
evidence of extirpation and another search may turn it up]). O (= obscure)
meant searched for and not found (whether due to a dry year, recent mowing, not
sure if searching in exact location, etc.), but habitat still exists in the
area so additional searching is warranted. It is felt that continuing to have
two ranks with essentially the same definition is not justified, and that locational obscurity is more
properly indicated through use of the "Locational Uncertainty" field.
Therefore, O is no longer considered a valid EO rank. In the case described above, the
identification uncertainty is not reflected in the EO Rank itself, but may be
mentioned in the EO Rank Comment field.
There is still ongoing methodological
discussion of the validity of some EO ranks. Certain currently valid ranks may
be eliminated in the future, and others may be added. At present, the full
range of valid possible EO ranks is as follows:
A - Excellent estimated
viability/ecological integrity
A? - Possibly excellent estimated
viability/ecological integrity
AB - Excellent or good estimated
viability/ecological integrity
AC - Excellent, good, or fair
estimated viability/ecological integrity
B - Good estimated
viability/ecological integrity
B? - Possibly good estimated viability/ecological integrity
BC - Good or fair estimated
viability/ecological integrity
BD - Good, fair, or poor estimated
viability/ecological integrity
C - Fair estimated
viability/ecological integrity
C? - Possibly fair estimated viability/ecological integrity
CD - Fair or poor estimated
viability/ecological integrity
D - Poor estimated
viability/ecological integrity
D? - Possibly poor estimated viability/ecological integrity
E - Verified extant
(viability/ecological integrity not assessed)
F - Failed to find
F? – Possibly failed to find
H – Historical
H? – Possibly historical
X – Extirpated
X? – Possibly extirpated
U – Unrankable
NR – Not ranked
Ranks other than the ones listed
above will be caught by the pre-conversion EORANK QC and, if they are not
changed, will be nulled out upon import of the data to Biotics.
Q.
I am confused on the purpose of the "ConfigurePrincipalEO"
portion of the instructions. I am not
sure what the purpose of this step is.
Is this more applicable to programs that don't already have Biotics 3.1
?
A. Yes. You don't need to do this if you are already in Biotics 3.1. Some programs have been storing Principal/Sub-EO information in BCD optional/non-standard fields, and this is a way to get that information into Biotics 4. If you haven't done this and you are already in Biotics 3.1 you don't need to do anything in this section.
Q.
I don’t understand the InvalidConceptReference report or what to do
about it.
A. For conversion to Biotics 4, every element must have a concept reference, which is a reference that describes the circumscription of the element. If such a reference is unavailable or unknown, a temporary “placeholder” reference can be used. In BCD, the concept reference is the NAMEREF field. If NAMEREF is empty, a placeholder concept reference must be assigned. Otherwise, the Element Tracking record cannot be created in Biotics Tracker, and all related records (like EORs) will also fail.
Only elements with an ELCODE not beginning in A, I, P, or N will show up in the InvalidConceptReference report. Placeholder references for species elements with a blank NAMEREF field will be assigned automatically during conversion. In subsequent data exchanges these will ultimately be replaced with the concept references from the Central Databases.
A program does need to
make decisions about what the concept reference should be for all non-species
elements, e.g., communities or other elements.
At a minimum, you need to create appropriate Source Abstract
records. Then you must either edit
your custom.cfg file or populate the NAMEREF field appropriately.
Here’s an example:
Community records. In looking at their community ET records, a program decides they need 2 concept references:
a)
Terrestrial communities (elcode
starts “CT”) - B90RES01NYUS - Reschke, C. 1990.
Ecological communities of New York state. New York Natural Heritage Program,
Latham, NY. 96pp.
b)
Palustrine communities (elcode
starts “CP”) - UNDNHP01NYUS
- New York Natural Heritage Program. Unpublished. Concept reference for
community elements for which no reference which describes the circumscription
has been recorded; to be used as a placeholder until such a citation is
identified.
Other records. In looking at these, all are animal assemblages. A placeholder reference similar to the one
created for the palustrine communities is sufficient for all of these:
c)
Animal
Assemblages (elcode starts “O”) - U03NOV01NYUS - Novak, P. 2003. Concept reference for animal assemblages
used in the New York Natural Heritage Program.
As noted earlier, you
have two choices. You can edit your
custom.cfg file (see directions on the General Configuration and QC guidelines
[ConfigureTheSystem] and ET_NO_CONCEPT [InvalidConceptReference]). For this example, you would
change
C=HDMSCONVHOLD
G=HDMSCONVHOLD
H=HDMSCONVHOLD
O=HDMSCONVHOLD
to
CT=B90RES01NYUS
CP=UNDNHP01NYUS
O=U03NOV01NYUS
The other choice is to
use the Global Search and Replace feature of BCD to update the NAMEREF field.
Q: Are there any critical data QC checks that I should do prior to conversion, besides the BCD2HDMS QC routines?
A: Yes. You should look at the BESTSOURCE and SOURCECODE fields in the EOR file, as well as EORANKDATE.
BESTSOURCE and SOURCECODE
First, carefully document how they have been used. Based on this information, you and your point of contact can determine the best way to convert these data, as well as whether you will need to do some data clean-up before or after conversion to Biotics.
Conversion of these
fields is complicated because of (1) inconsistency among HP/CDCs in what was
put into the BESTSOURCE field, and (2) the fact that it was possible to list a
SOURCECODE in EOR that did not refer to a real SA record. In the EO References field in Biotics, the
Reference Code must refer to an actual Reference record. Therefore, the default
data conversion process creates a Reference record if it finds a sourcecode
that does not match an existing SA record from your BCD. The problem is that duplicate reference
records may be created by this process.
The default conversion
procedure works as follows:
1.
The
program checks the contents of the BESTSOURCE to determine if it contains 12
characters with no spaces, or more than 12 characters.
2.
If
BESTSOURCE contains exactly 12 characters, the program tries to match them to
the sourcecode of an existing SA record.
If it does not find a match, it creates a new record with those 12
characters as the sourcecode/reference code and fills in a placeholder
citation.
3.
If the
BESTSOURCE field contains more than 12 characters, the program tries to match
them to the citation of an existing SA record.
If it does not find an exact match, it creates a new record with a
sourcecode/reference code beginning with the letter U and the BESTSOURCE data
in the citation field.
For HPs/CDCs
following standard field use guidelines:
If you use the
BESTSOURCE and SOURCECODE fields as defined in BCD Help files, BESTSOURCE will
always contain a name or citation, and the first row in SOURCECODE will contain
the code for the information in BESTSOURCE.
Sometimes, however, if the BESTSOURCE is a field form, no SA record was
created to go with that SOURCECODE. Even if an SA record exists, the citation
in that record and the text in the BESTSOURCE field may differ in some way,
such as punctuation or spelling. In
either case, the default conversion will result in two references records for
the same “best source”: one created from the BESTSOURCE field which would be
marked as the Primary Reference (see “Converted Data” section, below), and
another created from or already existing for the sourcecode on the first line
of the SOURCECODE field. This reference will appear in the References grid in
the Tracker EO record, but would not be marked as the primary reference (only
one can be marked primary).
Pre-conversion solution
(strongly recommended):
1.
Make
sure that every sourcecode entered on the first line of the EOR SOURCECODE
field corresponds to an SA record.
Create any missing SA records and make sure that all relevant
information in the BESTSOURCE field is captured in them.
2.
Create
a BCD symbolic the reads the first SOURCECODE and make sure this symbolic is
exported with your EOR data.
3.
Ask
your data conversion contact to use this symbolic instead of the
BESTSOURCE field during your data conversion.
Data in BESTSOURCE will be ignored, and the reference denoted by the
first sourcecode will be marked “primary” for the EO.
Post-conversion
solution:
1.
Identify
possible duplicate references (e.g., those starting with U or with F, if SAs
for field forms did not exist in your BCD) and find the EOs to which they are
linked.
2.
Remove
unnecessary links to duplicate references from the EO References grid in the EO
record, and then delete the duplicate reference records from the database. This
is a manual operation.
For HPs/CDCs using
BESTSOURCE and/or SOURCECODE fields in a nonstandard way:
1.
Make
sure your use of these fields is consistent!
In other words, if you put a sourcecode in BESTSOURCE, make sure you
always have only a sourcecode there.
2.
If you
don’t want possible duplicate reference records created during conversion, make
sure that all the sourcecodes refer to actual SA records.
3.
Consult
your data conversion contact to discuss options. For example, you may need to
create a symbolic different from the one described above, or you may elect to
let the default conversion happen and do any needed data clean-up later.
In the current version
of Biotics Tracker, this field is a date datatype, meaning that it contain
month, day, and year. The conversion program will convert any partial dates it
find in this field in to a complete date by doing the following: If it finds a
year only, it will fill in January 1 as the month and day (e.g., 1999 becomes
1999-01-01); if it finds a month and year, it will fill in the 1st as the day
(e.g., June 2001 becomes 2001-06-01).
If the EORANKDATE field contains information that the conversion program
cannot interpret, like a date range or words (e.g., 1998-99, 2000-Spring), the
field in Biotics will be blank.
Prior to conversion, you
should check the data in this field to so if you have dates that will not
convert correctly. If you can change them to complete dates, you should do so,
but if you do not have sufficient information to change them, the best solution
is to ask your installer to create an extensible field and migrate EORANKDATE
for now. There are plans to change the
Biotics field to accept imprecise dates in the near future. Once that is done, the data in the
extensible field can be transferred to the “real” EO Rank Date field by SQL
update, and the extensible field deleted.
Q: What if my BESTSOURCE and/or other sources are specimens? I haven’t created SA records for all the specimen listed in my EOR.
A: If your EOR SOURCECODE field contains
sourcecodes for specimens which don’t link to an actual SA, you should move the
sourcecodes to the SPECIMENS field so that SA records for each specimen don’t
get created during conversion (unless you want that to happen). If your BESTSOURCE is a specimen, you should
create an SA record for it prior to conversion. See previous question for
suggestions about how to avoid creation of duplicate reference records during
conversion.
Q: In the custom.cfg document, it seems like I’m doing exactly the same configuration twice for elements with ELCODEs starting with C, G, H, or O. Why is this necessary?
A: There are two separate data attributes that
must be set for these elements: Name Category (a replacement for Major Group)
and Classification Level (e.g., species, subspecies, variety for plants and
animals). These attributes don’t have
to be configured for taxa because the data exists in BCD. However, in order to use a common data model
for taxa, communities, and “other” types of elements, these fields need to be
filled in for all elements—they are required fields in the model.
Classification Level doesn’t apply to elements that aren’t in a classification
hierarchy, so it was decided that the value options would be the same as those
for Name Category—basically, it’s a placeholder value. Therefore, the values
for Name Category and Classification Level will be identical for C, G, H, and O
elements, but you have to specify them for both fields.
Q: How do I categorize communities developed by NatureServe Central Ecology (ELCODE begins with CEGL)? I have some of them in my database.
A: All terrestrial
communities in local Biotics databases will start out with Name Category and
Classification Level equal to “Terrestrial Community – Other Classification,” including
CEGL records. If all the communities in your database are terrestrial and have
an ELCODE that begins with C, all you have to do is remove the asterisk from in
front of the lines that begin “C=11” under Name Category and “C=52” under
Classification Level. If you have different types of community records (e.g.,
for subterranean communities, complexes, freshwater communities, etc.),
uncomment the lines that correspond to those types of records. You may have to
add more of the ELCODE to the lines in custom.cfg to distinguish between the
types (see examples in that document), but always select the “other
classification” choice.
Even though the CEGL
records are from the Central Ecology International Vegetation Classification
system, it is safer to convert them as “other classification” because taxonomic
and nomenclatural changes may have occurred since the data were put in the
local database. There has been no formal data exchange of ecological records to
keep them up-to-date. After the Ecology Pilot BCD data conversion is complete
(scheduled for August 2003), the latest
versions of the relevant Central Ecology records will be sent to the Network
for loading in local Biotics databases. Reconciliation with local records will
be done later.
Following your data
conversion, an SQL update will be done by the installer to change any ELCODE
beginning in CEGL to CExx, where xx = the 2-letter abbreviation of your
Heritage Program/CDC. The reason for the change is that ELCODE must be unique
for each element, and the Central Ecology records could not be added to a local
database if their ELCODEs already were in there.
Q. Can my community
scientific names be exactly the same as my species scientific names?
A. No, if they are the
communities will be loaded in as species and it will take a lot of work to
clean up. Check to make sure they are not.
Q: I’m not sure exactly what software is
installed on my BCD – how can I check?
A: To determine exactly
what software a program has installed, execute the following at TCL:
LIST
BCDSYS.CONTROL JUSTLEN 50
The results will look something like the report below. The critical pieces are bolded.
PAGE 1 11:38:23
24 FEB 2003
Key...............................................
BCD1992
BCD1992.SUPP1
BCD1992.SUPP2
BCD1992.SUPP2.FIX
BCD1996
BCD1996.BCD.UPLOAD.PATCH.1
ADJUNCT_ADDER_2000
EOMO_2000
ADJUNCT_ADDER_2001
BIOTICS_3.1
ADJUNCT_ADDER_2003
BIOTICS_CORE_NS_MODULE_001C -- NOTE: the “001C” may be different
BCD2HDMS_2003-02-12
– NOTE: the
date refers to the software version.
13 Records Processed
Q. Will BIOTICS accept duplicate
quadname/quadcode in the instances of multiple tenten values per quad?
A. No, each quadname/quadcode must be unique to
the EO. Duplicates will be rejected –
therefore, if you wish to retain multiple tenten values per quad, you will need
to add them to a single record (quadname/quadcode). The tenten value in BIOTICS is limited to 20 characters, so
anything beyond that will be truncated.
Q. Once we freeze the database to do our final conversion, I am assuming that no data entry can occur in Biotics Mapper. Is this true? For the sake of ease of the install I am guessing that the info in Biotics and in BCD need to be identical.
A. Correct. Once the database is frozen no more data entry
should occur in BCD.
Q. Which clients machines require their own workspace on the server? Is it only for the clients that actually use mapper, or do those using tracker need a workspace of their own?
A.
Only the users who will be using Mapper require a workspace on the
server (the workspace is used to store “clones” of the master shapefiles for
each user). If it's an issue and each
user will ONLY be using Mapper from ONE machine (per person), the workspace can
be local on the client. If the user
needs the ability to float around from client to client, then the workspace
needs to be on the server.
Q.
Does Biotics have the ability to access spatial data stored in
ArcSDE?
A. No.
Note: We are not Crystal Reports experts. We encourage you to talk with Crystal Decisions and to take advantange of the Listservs to gather experience from the Network. Please let us know how you address these issues or if you learn more relevant information, as these issues will be common across the Network.
Q.
Once Biotics 4 is installed, we are contemplating whether we should get
the full-blown version of Crystal Reports 8.5 or upgrade from ArcView 3.2 to
3.3, which comes bundled with Crystal Reports version 8.5. Is there is much
difference between the bundled version and the full-blown version of Crystal
Reports 8.5? Obviously, upgrading
ArcView would be the cheaper route but we don't want our hands tied with less
functionality.
A.
We do not have a definitive answer.
However, we did some testing with a version bundled with ArcView and our
Developer version. Each version could
connect to the database and use a report created by the other version.
The latest version of Crystal Reports is 9. It has some significant changes. One is the ability to create and modify SQL directly in the tool and the other upgrades a text limitation for formulas from 256 characters to 4000 (or something like that). It also comes bundled with a light version of a web interface that allows users to execute the reports using a browser (no need to have Crystal Reports installed on their computers). We have not yet tested this version here at NatureServe, but plan to soon. Crystal Reports 9 comes bundled with ArcGIS 8.3. ArcGIS 8.3 is not used with Biotics, but many programs own ArcGIS licenses to run other GIS functions.
Q. We have approx 35 biologists that will need access to at least some of the information in Biotics 4, and all of these 35 staff are at remote locations, do not have reliable web access and we won’t be able to get oracle licenses for each of them. The plan that we came up with was use ArcView as the primary reporting tool. To us this makes sense as the biologists think of the information geographically anyway. We also figured that since Crystal Reports came bundled with ArcView we could use that to allow the biologists to generate the reports that they find themselves using on a regular basis. We figured that it was up to us to create a bunch of standard reports that could be accessed easily (like from a drop down menu) and generated in the most push button way possible. Is what I am proposing even possible?
A.
Create a view in Oracle (Biotics data model) containing the attributes
you want to distribute. Then, in
Biotics Mapper, attach the view to the EO shape file and then use the ArcView
convert to shapefile function to produce a new shapefile for distribution. How often you refresh and redistribute this
is something you'll need to consider.
Use Crystal Reports to create your standard set of reports using the
.dbf portion of the shapefile as your data source. Include these reports as part of your distribution package. Integrate the execution of the reports with
ArcView. (We have not done this here at
NatureServe and cannot offer any advice.
Perhaps someone in the Network can help. Also try the Crystal Decisions and ESRI support sites.
Q.
Are reports (or report templates) generated in Crystal Reports 9 able to
be read and used in Crystal Reports 8.5?
A.
We have not started using Crystal Reports 9 yet so do not know the
limitations. However, previous versions
of Crystal reports had backwards compatibility. You would be limited by the inability to save new functionality
in an older version. An example would
be that in Crystal Reports 9 you can write formulas against text fields
exceeding 255 characters but in Crystal Reports 8.5 you could not.
Q.
If we purchase the full version of Crystal Reports 9, would we need the
Developer's version, or one of the other versions?
A.
This is best answered by comparing the versions on the Crystal Decisions
website and by talking with one of their sales reps. I do not think you need
the developers edition unless you are planning on seamlessly integrating report
functionality through custom software development.
Q. What happened to the BESTSOURCE field in the EOR?
A. If you had data in BESTSOURCE, it is now included in the References grid on the EO Documentation tab. It’s the reference with a checkmark in the checkbox called “Primary,” i.e., it’s the primary reference for that EO.
In order to consolidate BESTSOURCE into the reference list, it was necessary to create a real Reference record for that data, with a unique Reference Code (formerly, SOURCECODE), if it did not already exist. The conversion program creates a Reference record, inserts the record in the reference grid, and marks it ‘Primary.’
See the entry earlier in this document for a full discussion of BESTSOURCE conversion.