WARNING - 2024-04-19 10:56:08,909 - __init__ - Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING - 2024-04-19 10:56:08,911 - __init__ - Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING - 2024-04-19 10:56:08,912 - __init__ - Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING - 2024-04-19 10:56:08,959 - __init__ - Limited tf.summary API due to missing TensorBoard installation.
WARNING - 2024-04-19 10:56:08,970 - __init__ - Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING - 2024-04-19 10:56:09,031 - __init__ - Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING - 2024-04-19 10:56:09,067 - __init__ - Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING - 2024-04-19 10:56:10,674 - ag_logging - AutoGraph could not transform <bound method RadiusNorm.call of <ml3d.tf.models.FKAConv.conv_fkaconv.RadiusNorm object at 0x0000027485A09E70>> and will run it as-is.
Cause: Unable to locate the source code of <bound method RadiusNorm.call of <ml3d.tf.models.FKAConv.conv_fkaconv.RadiusNorm object at 0x0000027485A09E70>>. Note that functions defined in certain environments, like the interactive Python shell, do not expose their source code. If that is the case, you should define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.experimental.do_not_convert. Original error: could not get source code
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING - 2024-04-19 10:56:10,680 - ag_logging - AutoGraph could not transform <bound method AddAlphaBeta.call of <ml3d.tf.models.FKAConv.conv_fkaconv.AddAlphaBeta object at 0x0000027485A0A500>> and will run it as-is.
Cause: Unable to locate the source code of <bound method AddAlphaBeta.call of <ml3d.tf.models.FKAConv.conv_fkaconv.AddAlphaBeta object at 0x0000027485A0A500>>. Note that functions defined in certain environments, like the interactive Python shell, do not expose their source code. If that is the case, you should define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.experimental.do_not_convert. Original error: could not get source code
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
INFO - 2024-04-19 10:56:13,362 - semantic_segmentation - <ml3d.tf.models.fkaconv.FKAConv object at 0x00000274FF4C3D30>
INFO - 2024-04-19 10:56:13,362 - trimblereduced - Found 4 pointclouds for training
regular arguments
addon_model: null
batch_size: null
cache_dir: null
cfg_dataset: null
cfg_file: C:\ProgramData\Trimble\PythonDL\training-env\ml3d/configs/fkaconv_customtraining_updated.yml
cfg_model: null
cfg_pipeline: null
ckpt_path: null
dataset: null
dataset_path: null
device: gpu
framework: tf
fused_model_path: null
gpus: 0
grid_size: null
label_map: null
label_smoothing: null
main_log_dir: null
max_epochs: null
mode: null
model: null
model_save_dir: null
pipeline: SemanticSegmentation
seed: 0
split: train
steps_per_epoch_train: null
test_files: null
train_files: null
extra arguments
{}
Number of weights: 5168360
1 Physical GPUs, 1 Logical GPUs
Compute dataset summary:   0%|          | 0/4 [00:00<?, ?it/s]
Compute dataset summary: 100%|##########| 4/4 [00:00<00:00, 49.61it/s]
INFO - 2024-04-19 10:56:13,643 - trimblereduced - Found 0 pointclouds for validation
INFO - 2024-04-19 10:56:13,643 - trimblereduced - Found 2 pointclouds for test
Preparing test data:   0%|          | 0/2 [00:00<?, ?it/s]
  0%|          | 0/54761 [00:00<?, ?it/s]INFO - 2024-04-19 10:56:13,718 - semantic_segmentation - Writing tf board summary in C:\Users\michal.rendla\Documents\Trimble Business Center\TensorBoard logs\auta_2024-04-19_10-56-10.
 35%|###4      | 19058/54761 [00:00<00:00, 85173.20it/s]
 92%|#########2| 50593/54761 [00:00<00:00, 173707.44it/s]
100%|##########| 54761/54761 [00:00<00:00, 151511.28it/s]
Preparing test data:  50%|#####     | 1/2 [00:00<00:00,  2.49it/s]
Loading FKAConv weights...
  0%|          | 0/259943 [00:00<?, ?it/s]
 14%|#4        | 36582/259943 [00:00<00:00, 333744.50it/s]
 31%|###1      | 81232/259943 [00:00<00:00, 369147.23it/s]
 51%|#####     | 132168/259943 [00:00<00:00, 390834.78it/s]
 71%|#######   | 183419/259943 [00:00<00:00, 394036.18it/s]
 87%|########7 | 226785/259943 [00:00<00:00, 374399.54it/s]
100%|##########| 259943/259943 [00:01<00:00, 165114.10it/s]
Preparing test data: 100%|##########| 2/2 [00:02<00:00,  1.16s/it]
Preparing test data: 100%|##########| 2/2 [00:02<00:00,  1.04s/it]
INFO - 2024-04-19 10:56:28,468 - semantic_segmentation - Restored from models/2023-04-03_FKAConv_444e3c5e9e_ep_82_val_miou_55.1
INFO - 2024-04-19 10:56:28,468 - semantic_segmentation - === EPOCH 0/100 ===
Mismatched layers:
Expected Conv_out with [(1, 80, 2), (2,)] weights, got Conv_out with [(1, 80, 13), (13,)]
training:   0%|          | 0/180 [00:00<?, ?it/s]2024-04-19 10:56:30.860997: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2, command: '"ptxas.exe" "--version"'
2024-04-19 10:56:30.862123: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2, command: '"ptxas.exe" "--version"'
2024-04-19 10:56:30.862148: W tensorflow/stream_executor/gpu/asm_compiler.cc:80] Couldn't get ptxas version string: INTERNAL: Couldn't invoke ptxas.exe --version
2024-04-19 10:56:30.869382: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2, command: '"ptxas.exe" "C:\Users\MICHAL~1.REN\AppData\Local\Temp\/tempfile-D1-GEODET1-26a8-2964-6166f438e9351" "-o" "C:\Users\MICHAL~1.REN\AppData\Local\Temp\/tempfile-D1-GEODET1-26a8-2964-6166f438eac02" "-arch=sm_89" "--warn-on-spills"'
2024-04-19 10:56:30.869602: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] INTERNAL: Failed to launch ptxas
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
training:   0%|          | 0/180 [00:03<?, ?it/s]
Traceback (most recent call last):
  File "<frozen __main__>", line 3, in <module>
  File "<frozen run_pipeline>", line 306, in <module>
  File "<frozen run_pipeline>", line 95, in main_spawn
  File "<frozen run_pipeline>", line 296, in main
  File "C:\ProgramData\Trimble\PythonDL\training-env\ml3d\tf\pipelines\semantic_segmentation.py", line 407, in run_train
    log_to_tensorboard(writer, global_step, self.metric_train.iou(), self.losses, names, "train", log={"lr":lr}, simplify=self.tensorboard_simplified_log)
  File "C:\ProgramData\Trimble\PythonDL\training-env\ml3d\tf\pipelines\semantic_segmentation.py", line 48, in log_to_tensorboard
    tf.summary.scalar(key, value, step=step)
AttributeError: module 'tensorflow._api.v2.summary' has no attribute 'scalar'