DTM generation with Points2Grid using Docker and PDAL

Owned by tswetnam
Last updated: Jun 10, 2016
4 min read

Bare-Earth & Canopy Height Models

To generate bare-earth layers that are (1) appropriate for canopy height model generation, and (2) utilize the highest resolution possible for hydrological flow routing, you can use either local-gridding or TIN (triangular irregular network) techniques.

The Woolpert data were delivered classified with bare earth returns (ASPRS LAS v 1.4 class 2), the 2003 WGEW lidar are not classified (nor are they in LAS). However, the bare-earth model generated by Woolpert still has errors (e.g. shrubs that are classed as ground).

Using the Woolpert ground classification. I created a new set of .las and .laz files using LAStools las2las which uses only the ground returns.

las2las -i "%1.las" -remove_all_vlrs -keep_class 2 -odir "F:\Woolpert\las_ground" -o "%~n1.las"

where %1 is the identifier for the file location and name. I also removed the VLRs (variable length records) for the LAS 1.4

for /F "eol=; tokens=1* delims=,. " %%i in (F:\Woolpert\LAS\filelist.txt) do call wgew_las_ground %%i

I merged these files into a single .las and .laz layer.

C:\LAStools\bin>lasmerge -lof F:\Woolpert\las_ground\filelist.txt -verbose -o F:\Woolpert\las_ground\wgew_ground.las
merging headers took 5.116 sec. there are 557582614 points in total.
merging files took 181.11 sec.

Et cetera

FUSION has a TINSurfaceCreate feature which you can use.

An option to explore in Docker and Ubuntu: http://applied-geosolutions.github.io/lidar2dems/doc/installation.html 

PDAL uses points2grid

http://www.pdal.io/workshop/exercises/analysis/dtm/dtm.html  

Using DOCKER and PDAL

http://www.pdal.io/tutorial/docker.html

Using PDAL to attribute the point cloud

I am interested in attributing the point clouds with various extra valued fields including height (m) above ground level (agl), and NAIP NIR,R,G,B. 

http://www.pdal.io/tutorial/calculating-normalized-heights.html

PDAL has a set of features which use binary point format (bpf)

I also want to attribute the files using the 2015 NAIP imagery - but this is not possible with the libLAS solution I was using previously, so I need to find a new way to map the 4 bands into the las/laz v. 1.4 format files. 

Generating a grid surface using Points2Grid (in Docker with PDAL)

Start an instance of Docker Quickstart Terminal

Check Docker status

docker-machine env default
docker pull pdal/pdal

Get data from CyVerse 

 curl -o 'wgew_ground.laz' 'https://de.iplantcollaborative.org/dl/d/A9B20D8B-AE27-4D18-8AF1-E6409E0C612C/wgew_ground.laz'

Following the instructions from the recent PDAL Workshop

I installed Docker and OSGEO4W64 (QGIS)

Note: Because I'm going to be working with huge datasets (>100GB) I need to delete any old Docker Containers and increase the size of the Docker image/containers

To create a new container in Docker:

$ docker-machine create -d virtualbox --virtualbox-memory 65536 --virtualbox-cpu-count 16 --virtualbox-disk-size "80000" default

To remove old images:

tswetnam@DarkTower MINGW64 ~
$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   -        virtualbox   Running   tcp://192.168.99.103:2376           v1.11.1
testA     -        virtualbox   Running   tcp://192.168.99.101:2376           v1.11.1
testB     -        virtualbox   Running   tcp://192.168.99.102:2376           v1.11.1

tswetnam@DarkTower MINGW64 ~
$ docker-machine rm testA
About to remove testA
Are you sure? (y/n): y
Successfully removed testA

tswetnam@DarkTower MINGW64 ~
$ docker-machine rm testB
About to remove testB
Are you sure? (y/n): y
Successfully removed testB

tswetnam@DarkTower MINGW64 ~
$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   -        virtualbox   Running   tcp://192.168.99.103:2376           v1.11.1

To remove old containers:

docker ps -a | awk 'NR > 1 {print $1}' | xargs docker rm

Pull the most up-to-date version of PDAL (Note: the OSGEO4W64 installation of PDAL is only 1.1):

docker pull pdal/pdal:1.2

I started an instance of Docker and installed the necessary dependencies for PDAL

I then modified the p2g.json file which contains PDAL's 'pipeline' instructions for PDAL points2grid. 

{
    "pipeline": [
        "/data/wgew_ground.las",
        {
            "filename":"/data/wgew_ground_p5m",
            "output_format":"tif",
            "output_type":"all",
            "radius":"1.0",
            "grid_dist_x":"0.50",
            "grid_dist_y":"0.50",
            "type": "writers.p2g"
        }
    ]
}

In the Shell (MINGW64)

docker -run -v /c/Users/tswetnam/PDAL:/data -t pdal/pdal pdal pipeline /data/p2g.json

After fighting and failing to figure out Docker - one of the issues is that the data must be in the User/username/ directory on the disk where Docker is installed. I moved back into using basic file list call scripts

Statistical Outlier removal example for a single file:

pdal translate F:\Woolpert\las_colored\12RWA890090_colored.las F:\Woolpert\las_filtered\12RWA890090_colored_filt.las statisticaloutlier -v 5 --debug --filters.statisticaloutlier.extract=true


The pipeline is more efficient than writing the entire command out at one time. If you want to run the command without a pipeline you would type:

pdal translate F:\Woolpert\las_ground\12RWA890090.las F:\Woolpert\dem\12RWA890090_dem_1m.tif p2g --writers.p2g.grid_dist_y=1 --writers.p2g.grid_dist_x=1 --writers.p2g.radius=1 --writers.p2g.output_format=tif -writers.p2g.output_type=mean
pdal translate -i F:\Woolpert\laz_ground\wgew_ground.laz --writer=writers.p2g --writers.p2g.filename=F:\Woolpert\wgew_ground_1m -writers.p2g.grid_dist_y=1 --writers.p2g.grid_dist_x=1 --writers.p2g.radius=1 --writers.p2g.output_format=tif --writers.p2g.output_type=all

I had uploaded the merged Walnut Gulch bare ground .las and .laz files to Cyverse using Cyberduck.

I used a curl command to pull the data into my Docker image

curl -o 'wgew_ground.laz' 'https://de.iplantcollaborative.org/dl/d/A9B20D8B-AE27-4D18-8AF1-E6409E0C612C/wgew_ground.laz'

In Docker I executed this command - where the data are stored on the C-drive along with the p2g.json 

docker run -v /c/Users/tswetnam/PDAL:/data -t pdal/pdal pdal pipeline /data/p2g.json

 

Getting started with GNU Parallel

https://www.gnu.org/software/bash/manual/html_node/GNU-Parallel.html 

 wget ftp://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2