Discussion: Strange lidar problem

I have a strange lidar problem with a new dataset that I can't make sense of and I'm wondering if anyone here can point me in the right direction.

The problem is that I can't get either OCAD nor Karttapullautin to generate contours. I've tried multiple different tiles and partial tiles and always get the same result...

OCAD runs just fine and it says all the points are there including several million ground points but the end result is 0 contour objects, a solid grey hill shade and white slope image. The veg height image is solid yellow. The intensity and classification raster images are what one would normally expect.

KP gives the following messages:

Preparing input file
.......... done.
Knoll detection part 1
.. done.
Knoll detection part 2
..Illegal modulus zero at script/pullautin.pl line 6163.

Contour generation part 1
.................... done.
Contour generation part 2
.Illegal division by zero at script/pullautin.pl line 4982.

Contour generation part 3
.................... done.
Contour generation part 4
.................... done.
Vegetation generation
....

In the hopes that someone is willing to take a look at the data I've uploaded two tiles to my google drive here: https://drive.google.com/drive/folders/1IiSB3d2nOo...

The entire dataset (100+ sq km - 134 tiles) is downloadable here: https://opendata.nfis.org/mapserver/PRF.html

Many thanks!

all ground points has elevation value 0.0. So it is entirely flat and even.

I think data is pre-processed for some kind of vegetation height analysis, elevation (z) for all points is relative to the ground, not sea level. So you can't use these files to calculate contours.

open the laz file with lasview and tilt it in 3D, it is entirely flat.

Thanks Jagge.

That would obviously do it. They have a dtm available but it's been processed to 1 point /m2 so I was hoping I wouldn't have to use it directly.

Jeff, I hope you get it worked out. I'm told that is an area with great potential as an orienteering venue.
I tried a couple of things with OCAD 2020 Dem Import Wizard . First: those tile areas are not side by side, are they?
The first time I tried the DEM wizard I got contour lines but they were just a jumble - I had chosen a 1 m contour interval and chsoen to clasify vegetation height.
The second time choosing just to get hill shading and contours with a 2.5m interval I got nothing but grey hill shading, same as you. The good thing is that it took virtually no time to process.
The third time besides ground contours I had asked for a vegetation base map. I got a jumble of contours that seem to represent the height of the vegetation, typically 7, 8 and 9 metres. Also almost a solid mass of spot elevations again probably for individual trees and small clusters of trees.
Seems to make sense that thy have processed the LiDAR relative to vegetation. It is after all a forestry research station. But can the data not be reprocessed for the true ground elevation?

It seems like a case where you want to try to get in touch with the owners of the data and ask if they have the original data, before it was processed into whatever it is now.

For some areas in MB I could get DTM's from the web, but to get the all points laz data I had to actually contact someone. They were quite happy to provide it.

Thanks all, I've sent an email to a co tact listed on the site and will see what kind of a response I get.

Gord, those two tiles were chosen at random and are not side by side.

Using the OCAD DEM wizard settings below I was able to generate what looks like a good vegetation height image. The vegetation base map image was not so good. Click on the image to see the whole thing.

vegetation height

vegetation base map - seems to me it works in open areas but not under tree cover. Only first returns - no returns under the canopy to work out vegetation density

This could even be a first return only, -replace_z, i.e. vegetation height file, with no intermediate returns. In either case you really need to get hold of the original scan.

Is this near Ottawa?

Northwest of Ottawa some 160 km / 100 miles between towns of Petawawa and Chalk River.

The Petawawa Research Forest is over 100 years old and a world famous experimental forest for ecology, wildfire behaviour, and forestry.

Worst case - if you can't get the original data you could still make a pretty good base map with veg height from OCAD and contours from the DTM.

contours from the DTM using QGIS 0.5 and 2.5m

Thanks all,

To be clear, I know I can use the DTM to generate contours. I thought I would take advantage of the huge amount of data to experiment with some new (to me) techniques for extracting maximum information from the lidar. Not having the full dataset in the laz files doesn't allow me to do that so unless I hear back from someone saying they can get me the full dataset I'm putting this project aside for now.

Hopefully some day in the future this area will be mapped for orienteering but, as far as I know, it's not on anyone's immediate to do list.

I've had good luck with a kinda roundabout way to get the nice KP contour tweaking from a DEM (the ground DEM = DTM?), that is, I've used KP with the DEM as an input. I'd suggest it for lidar-derived DEMs, but not low-res DEMs, although you can certainly try it.

I use this command on the DEM (here a .tif) to create a uniform grid LAZ format file of ground points:

demzip -i *.tif -o test.laz -set_classification 2

Then I run that LAZ file through KP.

I'd suggest trying both a normal contour generating process as well as KP. But on the two times I've done this, the KP contours seemed like a better starting point for the mapper. But test it yourself.

I use my normal CRT file, which is a text file with the lines below. In Windows Notepad, I change the "save as TXT" to "All files", or you might get a .CRT.TXT file extension. I save this as "KP_out2_formlines_dot_knolls_dxfs.crt", and the filename reminds me to select the "out2.dxf", "formlines.dxf" (if you asked for formlines), and the "dot_knolls.dxf". Here are the lines for that CRT file:

115.000 udepression
206.000 uglydotknoll
112.000 dotknoll
313.000 uglyudepression
103.000 formline
101.000 contour
705.000 depression_intermed
0.0 contour_intermed
102.000 contour_index
707.000 depression_index
704.000 depression

{Edit: If you tiled the KP input and merged the tiled output, the out2.dxf file becomes "contours.dxf" or "merged_contours.dxf".}

Verify your symbol set has numbers that match mine, or update the symbol numbers to match your map symbols.

You can consider not importing the dot_knolls.dxf, which contains depressions, dot knolls, and ugly depressions and ugly dot knolls (which means the contour probably needs to be cut). I import the ugly ones with black boulders and blue U-depressions, and it's often not very good. Especially for a 1m or 2m grid or whatever grid dimension your DEM has, it's probably okay to not import them.

Oh---After you import, ideally in to a clean, new map, hide all the symbols and delete everything that wasn't converted. In settings in OCAD, you can change the color of unconverted symbols to something like 100% magenta, so you can see them. The old default was gray, and it was very easy to miss them. I click "entire map", zoom out one click, and select almost the whole screen when I delete the unconverted symbols.

If you set a smaller basemap interval, like 1.25m, those all import as a single symbol from basemap.dxf. I often use a different color than brown for these, and open them as a template (background map) so the symbols don't mix with the final map.

I'm not sure if this import process works identically in OOMapper. I use OCAD for importing the KP vector contours. I think it can be done in OOMapper.

using this thread for an issue not related to original post

i’m having with lidar though it might not be strange and rather im just a newbie with ocad and lidar.

the area i am processing data for has lots of lakes and when i run the DEM wizard i get lots of contour “artifacts” on the shoreline. i can fix that with lots of manual editing but are there any fancier approaches that can “mask” that issue.

i couldn’t find a discussion of this elsewhere so it might be that im doing something wrong

Are you referring to massing of contour lines?
Good question. I get it all the time, as well, and so far the best I can do is manual deletion. Often if the lake is small enough I can select all the lines and delete them together. My select is set to items comptetely in the rectangle. See OCAD Preferences in Options. But in this case it would also work with partially in and just select a portion of the contour lines, nothing else.

I'd also be interested in hearing better, faster methods.

I haven't tried it, but "Generating graceful contour lines from high resolution DEMs" sounds like it get at what Hammer is describing:

https://hkartor.se/anteckningar/generating_gracefu...

This is a peninsula on a large lake.

Post processing in DEM Wizard.


After manual edits

In Global Mapper, when creating a DTM there is a setting to preserve or interpolate gaps in the data. The below image shows the difference in a DTM created using a tight or a loose setting

https://www.dropbox.com/scl/fi/1prc26wnerqomsx5g7s...

https://www.dropbox.com/scl/fi/4fyesqslszfci8uzr4b...

I expect there would also be a setting in QGIS. Perhaps check to see if there is something similar when creating a DTM in OCAD. I use older version of OCAD.

Mike what you're seeing there is standard. It happens ecause the lidar data for large bodies of water is either removed completely from the point cloud dataset or is classified as water and as such is not included in generating the dtm (from which the contours are generated). You likely get the same thing around the outside of your map. I don't k ow enough about the contour generst8ng algorithms themselves to know exactly why those artifacts are created the way they are.

In OCAD you have the option of setting what point classifications and return numbers are included in the dtm. You can try setting that to all points but last return only or all ground and water points and last return only.

Alternatively can you just cover up the offending data with water in OCAD? Or can you draw the waters edge and use that to crop the map thus cutting out the data that looks funny?

KP uses ground and water for dtm, makes some adaptive smoothing to both dem and contours and then for raster output it just renders water over contours. Here is some patches without vector water so you can see how it looks (some Atlantic waves mapped as knolls):

https://mapant.no/?lat=70.40164256142877&lng=2...

Thanks everyone.

More on the Petawawa Research Forest. I believe sberg contacted them over a year ago about mapping a portion of the forest and using it for O, but did not receive a positive reply. Something for the future.

I've been asked by The Norwegian Mapping Authority to look into algorithms to (semi-)automatically detect lake boundaries: In mountain areas in Norway, with zero vegetation, we're using Dense Photo Matching (i.e. automatic photo-grammetry) to generate a 3D location for every pixel, with a 50x50cm pixel size.
These projects are delivered as RGB LiDAR (laz) files, just like the regular laser scanning projects so they can be processed in the same pipeline.
The problem with lakes is that when you take slanted photos of a lake, from two separate flight line passes, every single photo match from the lake surface will be essentially random and result in huge altitude errors.
In order to fix this, the best method they found included a lot of manual labor looking at the aerial photos to determine the actual lake/pond boundaries and then force all pixel heights within this area to the same value.
Given that we have an approximate location for every pond (from old vector topo maps), my idea was to start with those pixels and look around for the nearest contour line (25 cm equidistance) which almost or completely (95-100%)
defined a closed loop: This should be just above the water surface, right?

I also looked at trying to locate the speckle pattern given by the individual pixel heights, since they contain a lot more noise than any normal (real) surface.

A trick I have used sometimes: if you have one big lake and no part of the map is lower then the lake (like that peninsula example), then you can simply pick one elevation figure slightly (30 cm or so) above the lake and set all dtm elevation (or lidar point z values) to that if the original value is lower. Then you should get no contours on lakes, because the lake would have no elevation varaiation at all, countours would just represent situation whenthe water level is that 30cm higher than what it was when it was scanned.

The problem with DPM-generated point clouds covering water is that you get bogus elevations many meters above and below the actual water surface. :-(