Trimble Business Center

I am working on creating an original ground surface.  I have a single .vce file that was created by importing the .spj files from the data collectors used to record the surface.  There are very many duplicate points.  Some are duplicate points only in the name because they were recorded on different units so that even though they have the same name they are not the same point.  Other points truly are duplicate because an spj was imported on day one, then the work was continued in the same work order and the same spj was imported again on day two, etc.  What I want to do is delete all but one of each of the points that are identical.  The Merge Points command wants me to go through each one individually which is not practical with thousands of them.  The project cleanup command gets rid of the duplicate lines nicely but doesn't touch the points.  Is there a trick or a command I'm missing here?

If you import a Work Order on Day 1 (using Attach Site and import WO Results or drag and drop the SPJ, and then import it again on Day 2 (with more data in it), the first Work Order import is replaced by the updated one Automatically so there should be no duplicate data created by that process (I just tried this to confirm that that has not changed recently). So if your project has duplicate Points in it they must have come from different Work Orders unless you renamed a Work Order and imported it again. 

Either way, each file that is imported should create a Selection Set and you should be able to show only that selection Set through the View Filter Manager so if you do have two versions of the same file, find the one you imported first and then delete it removing all of those duplicate point and line objects. 

That then leaves duplicate points by name, and you can rename those points - again if you use Selection Sets to isolate a specific Work Order data and then e.g. Prefix the name with W0XX- or something or add 1000 to all the Point Numbers or similar and that should get you unique Point Names - maybe that keep the reference to original point number but with a prefix of the work order number or similar.

The only other Duplicate Points will be where two crews in different WOs recorded the same locations and you want to merge those points into a single point, if the points are in the same XYZ location (within a tolerance that you specify). I would not think that there would be that many of those, however there of course could be, and we don't have a "remove duplicate point function" - I can take a look for duplicate locations in your file, if I knew where to look for those that would be helpful - I can search through for overlapping ID labels to see what I can find.

I cannot do any real process tests with the VCL because the VCL drops the association of the Points to the original SPJ files but you should be able to try the above.

If you want me to take another look at this, please send me the original project (Archived) so I get all of the original WOs and associated Site etc. I think the above should work however.

Then the Merge Duplicate Points is left to merge points that are identical or close to identical if you need to do that on any duplicates from 2 or more work orders.

Alan

Alan,  Thank you for the reply.

I did not realize the .spj files had that functionality of replacing previously imported data from the same spj.  It looks like I need to revise my workflow for bringing in field data.  Currently my process is to open a new blank project, import the spj into that, do any editing, move data to the desired layers, write it out to a vcl, and import that vcl into my main working file for that particular project.  Of course knowing what I know now, that cuts me off from the spj import functionality.  Looks like I'll have to suffer though this job, and know better on the next one.

Cheers.

I do see that you have a lot of duplicate Lines and Points in this file, if you have the original project where the SPJs were imported then you should be able to do the above to clean this out as above. However if you got this as a VCL file (as shared here), the opportunity has gone because of the detachment of the points from the source Work Order files.

While we could I am sure create a "Delete Duplicate Points" function, that would have to go on the wish list and get prioritized accordingly. Let me know if the above helps though Wayne.

Alan

Alan, a Delete Duplicate Points function would be very useful; either as an additional option in the project cleanup or on its own.  In this example, I can go back through my records and get the original spjs and follow the procedure you outlined above, but that is not always possible, primarily when receiving data from an outside company.

I second the Delete Duplicate Points function.

I have also just checked the "Merge Points" Command and it is supposed to process all the points that you select in one pass, but does not appear to work on SCS900 recorded points - I will chat to development on that later today when in the office.

Alan

Alan, I also tried Merge Points in the past but it didn't work for me. I thought it was because I was doing it incorrectly or had my setting wrong. I import all my points from PNEZD .txt or .csv files.

I am checking but I think it is likely tied to the "Quality pf the points"

being Survey or Control Quality or something like that - will let you know

when I find out

Alan

Merge points should do this (while it keeps both points, it brings them together as one point and that one point can be included in the surface etc.). It just appears that SCS900 points after import for some reason will not be processed by that command. I will look into that.

Did you every get a reason on why this doesn't work for spj data. Just tried to use this on a project where I have thousands of points and probably hundreds of duplicate points. We had multiple foreman on site sharing the same spj work orders back and forth and this led to some overlap. 

Is there a work around method that you would suggest if this doesn't work?

Thanks!

One other solution would be to export all points to Excel and let Excel find the duplicates.  There's a number of easy ways to use Excel to find the duplicates, then simply export to CSV and back into your vce file.  This would not retain any links to the correct spj or work orders if that's important.  But would do what you need.  

Corbitt, that is very true and that would definitely work for points that are not connected to a linestring.  But since the linestrings that are imported from SCS900 are defined by the points themselves then the linestrings will cease to exist when the points are removed before re-importing back from excel.

If there was a way to convert the linestrings to being defined by coordinates and elevation, rather than the points that would make your method work?

Offset line with 0 offset will create a linestring that is made up off coordinates not connected to Point IDs - it has to because the points IDs at an offset would have to be different so it uses Coordinates not Point IDs as there are unlikely to be points out there at the offset to connect to.

While it would mean having to do one line at a time, it is at least possible to do.

Or just select all the lines export and reimport into the project file