Mapping and GIS Solutions Community

 View Only
Expand all | Collapse all

DDF (Data Dictionary Utility) tool to generate GDB (Geodatabase) schema, Domains

  • 1.  DDF (Data Dictionary Utility) tool to generate GDB (Geodatabase) schema, Domains

    Posted 12-03-2023 11:50
      |   view attached

    Summary:  Create a stand-a-lone or web-based app to convert DDF files or ASCII  to a file Geodatabase.   This one act would produce, for the world, a method for so many of us with long-standing relationships with Pathfinder Office, Positions and now Terra Office who simply need to convert thousands of highly complex to simple data dictionaries to file geodatabases.

    Over a long history of Pathfinder Office / TerraSync, spanning decades.... there are tens of thousands of Data Dictionary files (DDF's), which contain all the elements for conversion to ESRI fGDB schema but for reasons below are being blocked for reasons of deprecation.   Trimble invented the term Data Dictionary (DDF) and you all should be applauded for the simplicity and POWER of the DDF and the invention of the mighty Data Dictionary utility.

    Trimble invented the gold standard for easy GIS feature collection (1990's), and may I suggest it was the premier reason PFO/TS flourished for so long, regardless of the Windows Mobile OS on devices. You gave us the power to create features, menus, default values, "float" attributes up or down, incredibly powerful photo integration, load large text file menu attribute items with a click of a button and copy across feature types (point to a line or a poly) that has NEVER been duplicated in the ESRI framework.  I have challenged the best ESRI power users (using both Pro or FieldMap designer) many times to beat me in creation of a point/line/polygon.  I'm outside collecting data in a fraction of the time, while the best template ever created is tweaked for a real-world scenario... Oh, you need another domain attribute value?   Sure, hold on....  Sure, were not getting fancy smancy symbology, or relationships, but I'm suggesting, the majority of users don't need a basemap to collect data (Field Maps requirement), pop ups, tasks, over burdensome cloud-based Terra Office project creation processes  TO GET THE JOB DONE!

    Today:

    1. Pathfinder Office due to its lack of updates is no longer being allowed for installation on many high security systems (Federal Government), and the Data Dictionary App, cannot be loaded separately as a standalone.  So when PFO is removed, so is the Data Dictionary utility
    2. ArcMap is, for many in the federal government being being pulled in early 2024 from all computers.  Clearly, ESRI has long made it clear this day has arrived.   Trimble Positions extension for ArcMap and its reader is therefore not available.
    3. TerraOffice for Arcmap (see#2 above) and TerraOffice extension for Pro requires login to TerraOffice, a non FEDRAMPED cloud which many agencies/bureuas cannot access due to security reasons.
    4. SAFE Software, FME use to have a DDF reader, but has long dropped the reader from its library.

    We realize many exports from PFO workflows generated shapefiles, and we can load shapefiles into fGDB's, but the power of default values, menu lists etc are lost when loading data.

    We know we can export an ASCII export from the Data Dictionary Utility, which contains the "schema", which could be an excellent workflow if ESRI could read that schema, but there is not one.

    Could Trimble hear our pain out on the street to create a Data Dictionary Reader (app) that could provide a Reader/Writer capacity to generate features in a fGDB.  Your relationship with ESRI is obviously very strong. Would this be something you could convince ESRI to create a "Create Feature and Domain" tool in Geoprocessing?  Provide sample Python script to read either a DDF natively or even the ASCII output?  

    Attached is a sample of an ASCII out export from the DDF utility that has been used since 2015!  We are losing our access to the Data Dictionary Editor.   Please.

    Thank you Trimble for all you have done in creating the gold standard for hi precision collection over the years and the awesome DDF!



    ------------------------------
    Joel Cusick
    ------------------------------

    Attachment(s)



  • 2.  RE: DDF (Data Dictionary Utility) tool to generate GDB (Geodatabase) schema, Domains

    Posted 12-04-2023 08:19

    Joel,

    I hear you and I agree that a standalone utility would be nice. That being said, I can do this all day with Terra Office for ArcMap. The issue is all of the limitations that you have at DOI. I will continue to run ArcMap for years to come even though it is deprecated by ESRI. I realize that you don't have that luxury. I can also use the TerraSync2TerraFlex utility to create a TerraFlex Map Workspace from a DDF. Again, I realize that you cannot use the cloud but the vast majority of our customers are using it. Perhaps Trimble will come up with a perpetual license for Terra Office eventually that will allow a non-named user sign in to create a GDB from a DDF?  ;) @Matthew Morris? Lastly, could you hire a programmer that can build this for you? It would not be that difficult, as you would take a DDF, export it from the data dictionary editor to ASCII, and then convert from ASCII to GDB. I am just trying to come up with alternatives given your DOI restrictions. I do feel your pain, as I teach a day a week at the Southwest Indian Polytechnic Institute and they fall under the BIE/BIA which falls under the DOI. 



    ------------------------------
    Zach Edwards
    ------------------------------



  • 3.  RE: DDF (Data Dictionary Utility) tool to generate GDB (Geodatabase) schema, Domains

    Posted 12-04-2023 08:37
    Zach,
    Thanks for reply, and thanks for all your assistance.   Yea, feds are left out of the Terra Office (and field sharing on the surveying side).

    I also agree with the ASCII - GDB would be the slickest approach, but discussions with Trimble this weekend points to writing to GDB being the problem.   I know using PFO (v 5.85 or higher) still works in exporting to the version of the GDB API (writer) from a SSF file, and builds features just fine, but the DDF will need to be built well to satisfy constraints in a fGDB.  But most of our users have well built DDF's.   The problem with ASCII out and exporting from PFO, is PFO is going to be pulled due to deprecation, so we, and users everywhere are using PFO less and less.

    If we could get ESRI to build a tool to take either a SSF into GDB (building both schema and data loading) or a DDF (schema only) that would be a great alternative.   FME use to have a Reader/Writer for DDF - GDB schema construction,but pulled that long ago (probably, like Trimble, the access to that API wasn't being updated).

    Like to hear from the community if anyone has found a way besides PFO?




    Joel Cusick
    GIS Specialist
    Region 11 - Alaska Regional Office
    240 W. Fifth Ave. Anchorage, AK 99501-2327
    cell:907 280-9724  wk:907 644-3549 
    Alaska Region GIS Resources:
    Helpdesk: akro_gis_helpdesk@nps.gov
    Intranet Site: https://inpakrovmgis.nps.doi.net/apps/akrgis/ (please note updated link as of 03/22/2023)






  • 4.  RE: DDF (Data Dictionary Utility) tool to generate GDB (Geodatabase) schema, Domains

    Posted 12-05-2023 08:08

    WHile you have access to PFO i recommend that you start doing the DDF to Geodatabase conversion while you can.  All it takes it to run TerraSYnc, collect data (parking lot or digitize) for each feature in the DDF, transfer the file, then export to esri GDB.  From there you can run your esri software and you can pull and save the GDB schema from your export.  Repeat the process as needed.



    ------------------------------
    eric bock
    ------------------------------



  • 5.  RE: DDF (Data Dictionary Utility) tool to generate GDB (Geodatabase) schema, Domains

    Posted 12-12-2023 07:58

    Hi Eric,

    As always, a guy who gets it done.  Great point, were doing that right now, but the clock is ticking down to end of this year for FEDS on many fronts (Windows Mobile, ArcMap).  Your workflow suggestion is perfect.   Only a few moments outside and export to fGDB in PFO will work.   But were talking decades of DDF's out in the landscape.   A tool run from Pro would have a long lasting use that I can predict will be needed for years.  Great scoop my friend.   Your one of a kind.   NEI rocks.



    ------------------------------
    Joel Cusick
    ------------------------------



  • 6.  RE: DDF (Data Dictionary Utility) tool to generate GDB (Geodatabase) schema, Domains

    Posted 12-28-2023 13:50

    To add to Joel's request, we also need the equivalent of the PFO export utility.  The impending loss of PFO and the lack of ssf support in TBC 2023, mean that we will no longer be able to process legacy ssf files or export them to GIS format.  At the center where I work, we have 1000s of ssfs that may not have been correctly exported over the last 20+ years.  I re-correct and export them as I need them, but we don't have the funding or personnel to do all of them now on short notice.  This is federal project data paid for by American taxpayers and we are required to maintain it and make sure that it is available and usable for future research.  I would think that the simplest solution would be to leave ssf support in TBC.



    ------------------------------
    Jill Halchin
    ------------------------------