TBC Macros and Extensions

 View Only
Expand all | Collapse all

Supporting proxy settings and checking date from the internet

  • 1.  Supporting proxy settings and checking date from the internet

    Posted 02-08-2024 02:59
    Edited by Fernando Calvo 02-08-2024 08:33

    Hi guys,

    does any of you have experience with supporting proxy server settings in your TBC macros?

    I´ve used several TML built-in functions in some of my macros to check internet connection and get the current datum from the internet (instead of from the local system) such us:

    public static System.Threading.Tasks.Task<bool> IsInternetConnectionAvailableAsync()
        Member von Trimble.Vce.TbcLicense.HardenedManager

    public static System.DateTime GetCurrentDate()
        Member von Trimble.Vce.TbcLicense.HardenedManager

    However from TBC version to version, these was moved to other places etc, and even there were other functions such us IsInternetConnectionAvailable() that no longer exist, but the new one does not work and gives back always True although there is no internet connection or there is but due to the proxy settings, no real "connection" to the "ouside world" is possible. As this built-in function finish to work properly a couple of TBC versions ago, I changed the internet connection check by using sockets.

    Furthermore since 2023.10 some customers who have been using proxy server settings and my macros worked with the own internet connection check via sockets i.e., it does not work any more:

    ## see if we can resolve the host name -- tells us if there is
    ## a DNS listening
    host = socket.gethostbyname("http://google.com")
    ## connect to the host -- tells us if the host is actually reachable

    s = socket.create_connection((host, 80), 2)
    s.close()

    So basically, there are a lot of Python functions to check the internet connection or read the current date from the internet, however not working for IronPython, so my question would be if some of you knows how to check the internet connection and current date from the internet (obviously if there is an internet connection) from IronPython or if there is any other (working ;-) ) TML built-in function for that. The GetCurrentDate() one works indeed, however not the other one for the internet connection.

    For example, I would like to use the urllib module, however it does not work from the Macro itself although the module and its function is recognized in the code:

    , but giving "ImportError: No module named 'urllib'" error when launching the macro inside TBC.

    Any help or recommendation would be highly appreciated.

    Thanks.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 2.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-06-2024 02:10

    Hi again,

    26 days waiting (as usual as with many other threads and direct questions ) for an answer by the TBC development team, at least to say that it is not possible or whatever, but nothing happens.

    Is it really the goal of this forum to create it and leave the developers alone and if no answer from other TML developer (unfortunately max. 5-10 worldwide), then you can be waiting for years for an answer?

    Is it so difficult to invest 30 Minutes of your time (TBC team) per week or month to check the questions and issue in this forum? There are just 3-5 threads per month, so it´s not a major time investment from your side.

    Thanks.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 3.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-11-2024 12:49
    Edited by Paul Garno 03-11-2024 13:15

    Hello Fernando,

    Sorry for the delay in our response to your question. Gary Lantaff is no longer with Trimble so the monitoring of this community wasn't a high priority. We have put a plan in place to actively monitor and reply to questions in a more timely manor. Your question is currently being researched. We hope to have an answer soon.



    ------------------------------
    Paul Garno
    ------------------------------



  • 4.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-12-2024 00:14

    Hi Paul,

    thanks for coming back to me and for the infos ! Really sad that Gary finally retired, but obviusly fully deserved and we all are really thankful for his fantastic support over the last years.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 5.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-11-2024 15:23

    Hi Fernando,

    Sorry for the late reply - I meant to reply to your post a couple weeks ago...

    Since the public TBC command extension mechanism uses IronPython, you have 3 different sources for coding solutions:
    1) Python (like you are trying above)
    2) .NET Framework 4.8 (this is the .NET version that TBC v5.50+ uses)
    3) TBC API libraries (like the HardenedManager you mentioned above)

    So if you're having trouble with 1) or 3), you can always Google for answers from .NET Framework 4.8 - there should be a large pool of posts (or debates!) since that is a longstanding platform.

    I did a 10-minute browse and found numerous answers - several examples were "pinging" something well-known like Google.com.  Like many code searches, it seemed there was not a quick answer.  It seemed like most people were asking this question (how to check for an internet connection) because they wanted to avoid making a web call and waiting for a timeout.

    In my case, I am starting out with using HttpClient (from System.Net.Http) and using the PostAsync() and GetAsync() methods, and using Task.Wait(5000) for a 5-second timeout.  I appreciate that I'll need fancier code to display a cancelable progress bar in the UI, etc., but this is OK just to get things working (for now).  Your web code may be well beyond this basic usage...

    Also, sort of a side comment, but thanks for sharing (over the years) some of your exploration into using the HardenedManager!

    I had a customer request to be able to associate command license checks with the hardware and software key IDs used by TBC and was able to use it successfully.  I also updated my code to use HardenedManager.GetCurrentDate(), so my license check uses the same date that TBC uses (whatever strategy it uses to determine the date).

    Hope some of that helps,



    ------------------------------
    Quan Mueller
    Revenant Solutions - TBC Extension Developer
    ------------------------------



  • 6.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-12-2024 00:18

    Hi Quan,

    thanks so much for your answer and whole recommendations, and please, you don´t need to apologize, it´s not your job as you are like myself a 3rd party developer, so special thanks for dedicating time to help me.

    I tried indeed dozens of ways that I found on the internet but I didn´t find a way to get it working, especially when working with proxies, although I guess TBC should also support proxies in any way as hundreds or thousands of customers, especially public administration, only work via VPN and/or proxy, however I don´t see any setting inside TBC to set the IP and port as I see in other software packages. Some other solutions ask if they should use the already available proxy settings from Windows.

    As Paul Garno also replied and they are researching it, let´s wait what they say and hopefully there is any function from the sdk that we can easily use similar to the GetCurrentDate but also supporting proxies etc.

    Thanks.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 7.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-12-2024 11:07

    Hi Fernando,

    It's tricky being an extension developer - you have to figure out whether the issue you're wrestling with is a Python issue, a .NET issue, or a TBC API issue.

    Issues with the TBC API (like "HardenedManager.IsInternetConnectionAvailableAsync() doesn't work with proxies") are clearly TBC dev team issues.

    However, "how do I make Python web calls work with proxies" sure sounds like a Python education issue.

    But... as we see from Virginia's response, it looks like TBC has a Python installation error... hopefully they can get that resolved for 2024.00 - and you can get your code working again.



    ------------------------------
    Quan Mueller
    Revenant Solutions - TBC Extension Developer
    ------------------------------



  • 8.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-13-2024 00:49

    Hi Quan,

    yes, indeed you used the perfect wording (wrestling), and obviously fully agree that many things have to be researched and tried dozens of ways and workarounds, and this is what I do for days or weeks before I post my questions to the TBC dev team.

    Python is not equals IronPython and many modules with fantastic tools that work for Python, do not for IronPython, so the whole TBC dev environment is extremely limited for us and adding this to the absolute lack of documentation and usual 0 support by the TBC dev team, it makes it really difficult and frustrating to develop TBC macros that help much more Trimble and Trimble resellers to sell more instruments and software, while on our side, it´s not as profitable as for them, and even though, they usually leave us alone "wrestling" with all these.

    I have been developing plugins for many different software packages such us AutoCAD, Civil 3D, Map, ArcGIS, ArcPad, SketchUp, Spatial Analyzer, PowerCAD, BricsCAD, ARES Commander, Trimble Access, and some more,  for over 25 years now, and to be honest developing plugins for TBC is far the most frustrating one of all, especially due to the lack of enough samples, documentation and support, and obviously there are just a handful 3rd party developers worldwide, so no huge forums information where you can learn from like for example for Autodesk or Esri development and I fully acknowledge it.

    Thanks again for your understanding and support ! we are all on the same boat ;-)

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 9.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-12-2024 07:55

    Hi Fernando,

    To address the urllib import error that you are seeing, please navigate to the library of IronPython files included in your TBC installation (this should be something like C:\Program Files\Trimble\Trimble Business Center\IronPython 3.4\Lib), and rename the urlib folder to urllib.

    This issue should be resolved for future releases.

    Thank you for your patience!



    ------------------------------
    VirginiaHutchinson
    ------------------------------



  • 10.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-13-2024 00:40

    Hi Virgina,

    thanks so much for your reply and information ! Really appreciate it !

    It´s working now after changing the folder name. I think I found something similar during my dozens of tests of different workarounds or ways but I think it was not the folder but the name itself as in former python versions it was called urlib and in newers urllib and also some references to urllib2 vs urllib3 or similar.

    I´ll try much deeper but at the first sight, I have now access to the urllib.requests and can even read the proxy settings etc, so trying to make my internet checks bypassing the proxies, etc. BTW is TBC simply checking the current Windows proxy settings and using them to make the internet license checks etc? or is there any other function inside the SDK to ge an easier access to it such us GetCurrentDate ? However GetCurrentDate needs a "proper" internet connection without checking anything, so if no "real" internet connection, then you get a date of 1999 or so and the builg-in "IsInternetConnection...." does not work as it always gives back True although no internet available.

    Thanks again for your valuable support !

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 11.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-13-2024 04:26
    Edited by Fernando Calvo 03-13-2024 04:27

    Hi again, 

    just a quick correction when using GetCurrentDate, so if no internet connection or not able to go through the proxies, we get back 31.12.9999 as you can see in the following screenshot and the first row showing the date via urllib:

    Thanks.

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 12.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-14-2024 08:24

    Hi Fernando, 

    Sorry for all your troubles.

    For the HardenedManager, I am confused why you believe that IsInternetConnectAvailableAsync always returns true. I can confirm we ping multiple locations to determine if we have a connection. If any succeed, we return true, otherwise false. If you are offline, those calls will fail and return false. 

    I have taken a screenshot below demonstrating IsInternetConnectionAvailableAsync (see wifi connection in toolbar to confirm offline) and see the call working as expected. 

    GetCurrentDate is similar in we ping multiple servers to get the correct location. If this fails (ie offline), we return the DateTime.MaxValue (12/31/9999) as you have demonstrated in your screenshot above. I believe you misread the year value as 1999, not 9999. 

    I hope this helps in understanding what is happening in these calls. You are more than welcome to try to use the HardenedManager or roll your own solution with the python libraries. I do not see any glaring issues at this time with the HardenedManager and its responses. 

    Good Luck!



    ------------------------------
    Bryce Haire
    ------------------------------



  • 13.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-14-2024 11:27

    Hi Bryce,

    thanks a lot for your reply and infos !

    Regarding IsInternetConnectionAvailableAsync(), I have tried now even some "stupid" code (especially the else as it makes no sense but just to see what comes as I got False although I had now an internet connection), that I would never use, but just to see what happens:

    ###########################

    bIsInternet = License.IsInternetConnectionAvailableAsync()

    if bIsInternet == True:

         strInternet = "True"

    elif bIsInternet == False:

         strInternet = "False"

    else:

         strInternet = "Nothing"

    dateCurrent = License.GetCurrentDate()

    VceMessageBox.ShowInformation(TrimbleOffice.TheOffice.MainWindow.Form, "Internet: " + strInternet + "\n" + str(dateCurrent) )

    ###########################

    And I get the following result first having a current internet connection:

    And without a current internet connection:

    So as my original code was assuming I´m getting back either True or False, my else was for getting False and therefore getting always False, however we are getting neither True nor False...Sorry if I´m missing or overseeing anything but  I don´t understand it.

    Furthermore, in my own case without proxies, if I use the GetCurrentDate(), I automatically get within 1 second the maximum date with year=9999 which is fine, however when using Proxies, it takes around 2 minutes while I can´t do anything inside TBC and apparently no timeout or too long set, and after that, I get the corresponding message.

    I´m now supporting proxies by setting IP and Port, and additionally User and Password if also needed, and it works fine using the urllib, so I´ll continue from now using my own code, however it would be highly appreciated if you can check the IsInternetAvailable.. and GetCurrentDate as mentioned above.

    Thanks again for your support.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 14.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-14-2024 11:43

    IsInternetConnectionAvailableAsync is a Task<bool>, not a boolean. You need to await the task for the response. The response of "nothing" makes sense because it is neither true or false, it is a Task<bool>. 

    GetCurrentDate is a synchronous call (not a Task), so it will block the calling thread in this case until it completes. The default timeout for this request is 100 seconds, which makes sense why it returns ~2 minutes later. The call tries for that time period and eventually gives up and returns the DateTime.MaxValue. 

    I would recommend that whatever solution you come up with, you make the HTTP request calls on a background thread and do not block the UI for a better user experience. 

    Hope this helps!



    ------------------------------
    Bryce Haire
    ------------------------------



  • 15.  RE: Supporting proxy settings and checking date from the internet

    Posted 03-14-2024 12:26

    Hi Bryce,

    thanks again for your response, especially because I´m not used to get any response at all, so I highly appreciate it ! ;-)

    I´m pretty sure I used the function in the past in former TBC versions and I got a boolean back and probably this is the reason why it didn´t work in my macros any longer from a special TBC version, or I used another IsInternetConnection...without asynchronus in the past or similar, not sure, now, but anyway now I know that it´s a task and I can check it properly (for sure it can be checked in the object catalogue, however as I say, I´m pretty sure it was a boolean in the past, but through my thousands of tests and changes to get my macros working from version to version due to the changes, I cannot guarantee now without checking all code and all macros skd´s, if it was a boolean or not).

    As I now have access to urllib, I´ll do my internet connection and date checks using it, but thanks for your recommendation for using a background thread, I´ll try it ;-)

    So thanks again for your fantastic support !

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 16.  RE: Supporting proxy settings and checking date from the internet

    Posted 3 days ago

    Hi @Virginia Hutchinson and @Bryce Haire,

    so far it´s working fine all that I did with urllib, however struggling for older versions such us TBC v5.81 and older.

    As you mentioned, for v5.90 or 2023, the urllib is placed on:

    C:\Program Files\Trimble\TBC v2023.10\IronPython 3.4\Lib\urllib

    and the correction of the folder name was done for TBC v2024, which is fine, thanks for that ;-)

    For TBC v5.80 the corresponding folder is:

    C:\Program Files\Trimble\TBC v5.81\IronPython2.7\Lib

    so I see on that folder urllib.py and urllib2.py but the macro fails when trying to load them. Is there anything we need to change to get them loaded on launch?

    At least backwards up to v5.41 I can see that folder and both urllib´s .py

    Thanks.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 17.  RE: Supporting proxy settings and checking date from the internet

    Posted 3 days ago

    Hi Fernando,

    This seems to work for v5.81.

    import json
    from urllib2 import Request, urlopen        #python2

    ....

            url = 'https://jsonfeed.org/feed.json'
            request = Request(url)
            response_json_string = urlopen(request).read().decode('utf8')
            response_json_object = json.loads(response_json_string)
            self.urlResponse.Text = response_json_object['title']



    ------------------------------
    Bryce Haire
    ------------------------------



  • 18.  RE: Supporting proxy settings and checking date from the internet

    Posted 3 days ago

    Hi Bryce,

    thanks so much for your quick response and infos!

    I used so far urllib for my macros and not urllib2, but I´ll have a look at it in case urllib cannot be used for TBC v5.81 or older. Can you confirm that it wouldn´t work with urllib? or if there is any other way to work with urllib instead of urllib2 in order to avoid changing the code again and checking if urllib2 also works on 5.90 and 2023?

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 19.  RE: Supporting proxy settings and checking date from the internet

    Posted 2 days ago

    Hi Fernando,
    I found a couple StackOverflow posts on having a Python 2 and Python 3 compatible version of urllib/urllib2. 

    https://stackoverflow.com/a/60121758

    https://stackoverflow.com/a/17510727

    Unfortunately, urllib appears to be a package that had some changes between the Python versions that you need to account for. 



    ------------------------------
    Bryce Haire
    ------------------------------



  • 20.  RE: Supporting proxy settings and checking date from the internet

    Posted 2 days ago

    Hi Bryce,

    thanks for your reply. I knew a bit about the differences indeed, but my question was more if there is also any known issue with false naming (such us urlib instead of urllib, or false folder or the need to create a subfolder named "urllib" or similar), as my main issue is that TBC cannot load the "urllib" module, so similar to the issue that I had with v5.90 or higher due to the incorrect naming of the urllib folder during TBC´s installation.

    thanks.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 21.  RE: Supporting proxy settings and checking date from the internet

    Posted 2 days ago

    Hey Fernando,

    Upcoming TBC 2024.00 has addressed the naming of the folder to the correct naming (urllib). There should be no need to either rename a folder or use the incorrect module name (urlib). 



    ------------------------------
    Bryce Haire
    ------------------------------



  • 22.  RE: Supporting proxy settings and checking date from the internet

    Posted 2 days ago

    Hi Bryce,

    thanks, I noticed it already while testing the Beta version, which is great, thanks for fixing it.

    The question is if there is anything similar for 5.81 and older as TBC cannot import urllib although the urllib.py exists on the corresponding folder.

    Thanks.

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------



  • 23.  RE: Supporting proxy settings and checking date from the internet

    Posted 2 days ago

    Hey Fernando,

    Personally, I've never used the urllib module before. 

    https://stackoverflow.com/a/41431794

    Looking at this stackoverflow, there is no urllib.request in Python 2. If you want to use the same library, it looks like you should be using urllib2. 

    The code I posted should allow you use the same base code of the class, you need add a try/except and use a different import statement if it fails. 

    try:
        # For Python 3.0 and later
        from urllib.request import urlopen
    except ImportError:
        # Fall back to Python 2's urllib2
        from urllib2 import urlopen
    
    html = urlopen("http://www.google.com/")
    print(html.read())

    Sorry if this hasn't been helpful, but v5.81- uses Python 2 and therefore you need to handle the Python 2 vs 3 changes for your macros. 



    ------------------------------
    Bryce Haire
    ------------------------------



  • 24.  RE: Supporting proxy settings and checking date from the internet

    Posted 2 days ago

    Hi Bryce,

    thanks so much for your fantastic and quick help.

    Yes, I'm handling many things in my macros different if iron python 2 or 3, but I was happy that I finally got the proxy settings working via urllib.request and built all functions behind based on that, but didn't check on 5.81 and older...

    So I'll try to use then urllib2 instead tomtry to keep the same code for all TBC versions (from 5.40 in my case) and also iron python 2 and 3.

    Thanks again for such valuable and quick support that we have been waiting for years for and we all really appreciate.

    Regards,

    Fernando



    ------------------------------
    Fernando Calvo
    calvo@calvo-geospatial.com
    ------------------------------