Monday, May 22, 2017

Merging multiple comma separated values CSV files

If there are many comma separated value CSV files with headers and you want to merge them into a single CSV file, it can be a pain having to do it by hand. Fortunately, there are ways to automate the task. One method which I like is to use the tail command from Unix. For Windows, a tail utility can be downloaded from;There are others which a Google search can reveal.

Below are screenshots of a few CSV files to merge.

  1. Using a text editor, create a script or batch file. In the editor, type in the tail command to create a new merged file with a header.

    tail -n +1 -q green.csv > merge.csv

    Note: -n +1 means to start from the first line.
    A single > means to output to a new file

  2. Type in the commands to append lines from subsequent files without headers to the output file.

    tail -n +2 -q orange.csv >> merge.csv
    tail -n +2 -q transfer.csv >> merge.csv
    Note: -n +2 means to start from the 2nd line
    >> means to append to the output file

  3. Run the script or batch file in a Command Prompt.

    The CSV files are merged.

Sunday, May 14, 2017

Resolving Python Module 'numpy' has no 'xxxx' member error message in Visual Studio Code

While writing some Python code in Visual Studio Code that calls some Numpy classes, Pylint error messages appear complaining about non-existing Numpy members, e.g. the log2 method, and red wavy lines underline the offending code, as shown in the screenshot below.

One way to clear this up is to get Pylint to white-list Numpy using Visual Studio Code's Settings.

  1. In Visual Studio Code, select File | Preferences | Settings.

    The settings.json file is displayed in the editor.

  2. Click the Workspace Settings tab.

    The settings appear in the editor.

  3. In the editor, type in the following and save the file.

    { "python.linting.pylintArgs" : [ "--extension-pkg-whitelist=numpy" ] }

  4. Close and reopen the folder in Visual Studio Code.

    The missing member messages and the red wavy underlines no longer appear.

Monday, May 1, 2017

Create labels from multi-columns in QGIS

It is quite simple to create labels from one or more database columns in QGIS; just that you have to type in the correct syntax as shown below.

  1. Run QGIS. Display some layers, e.g. stations.

  2. In the Layer Panel, mouse right click on the layer to be labelled, e.g. stations.

    A pop up menu appears

  3. Choose Properties.

    The Layer Properties dialog box appears.

  4. Choose Labels. In the combo box, select Show labels for this layer.

  5.  In the Label with field, click the Expression icon.

    The Expression dialog box appears.

  6. In the Expression tab, type in the syntax to form the multi-column label e.g.
    name || ' ' || code.

    Note: name and code are two columns from the layer database table, and || is the string concatenation operator and ' ' is a space character.
  7. Click OK. And Click OK to apply the changes.

    The labels appear.
  8. If multi-line labels are desired, then the newline character '\n' needs to be concatenated with the database fields in the Expression dialog box, as shown below.
    name || '\n' || code

    And the resultant map look like this.

Thursday, March 23, 2017

LiDAR data coordinate readout in Cloud Compare

It is useful to obtain a readout of LiDAR data points in Cloud Compare, perhaps to determine the height value, or the intensity value of a LiDAR point. This post illustrates how to read out coordinate values in Cloud Compare.
  1. In Cloud Compare, load in a LAS file, e.g. Serpent Mound Model.las.

    The Global Shift/scale dialog box appears.

    Note: Real LiDAR data have large coordinate values requiring double data types, but most 3-D software like Cloud Compare works with coordinates in float data types internally. So in order to preserve the LiDAR coordinates in Cloud Compare, it may be necessary to apply some coordinate offset and/or scaling so that the LiDAR data points can fit into float data types. So when you load a LAS file in Cloud Compare, it will calculate and recommend an offset/scaling values before importing, as shown in the screen shot below.
  2. Click Yes.

    The LAS file is loaded; its data points are globally shifted and scaled.
  3. In the toolbar, click the Point picking icon.

    A point picking toolbar appears on the top right corner.
  4. By default, the first command (Select one point and displays its info) on the left is active. If not, click on it.
  5. Click on one LAS point.

    A balloon appears showing the X, Y, Z and a scalar value.

    Note: the xl, yl, zl on the left are local float data type coordinates. The xg, yg, zg on the right are the actual LiDAR data point vertices. The Point Source ID is the current scalar value.
If you want to get the read out to show another scalar field, you have to set the desired field before entering the Point Pick command, as described below.
  1. In the Tree pane, select the point cloud node, e.g. Serpent Mound Model.

    The Properties appear below the Tree pane.
  2. In the Scalar Fields | Active field, choose the desired field, e.g. Intensity.

    The point colors change to intensity values.
  3. Now, use the Point Pick command to readout a point.

    The readout balloon now shows the selected scalar field - Intensity.

Monday, March 13, 2017

Batch sample or convert videos to animated GIF files WebApp

I wrote a WebApp VidzGif for batch sampling or converting video files into animated GIF images that works with just a Web browser without uploading the videos to a server for processing.

To use this WebApp, do the following:

  1. Open up the WebApp at with a Web browser.

  2. Optional. Click Settings. Set the desired parameters, e.g. sample interval, duration, etc.
  3. Click the Browse button and select one or more video files. Or drag and drop one or more video files into the dashed box.

    The videos are sampled and rendered into GIFs.
  4. To save out the GIF, right click on the image and choose Save image as.

Monday, March 6, 2017

Interactive cross-section in Cloud Compare

After loading LiDAR point clouds in Cloud Compare, it helps sometimes to cut a cross section or profile to get a feel for the land. There is a couple of Segment commands but the one I use for this purpose is the Cross Section Tool with the graphic interactors.

The following steps show how to use the Cross Section Tool.
  1. Load a point cloud file, e.g. Serpent Mound Model LAS Data.las, into Cloud Compare.
  2. Select the cloud in the DB Tree pane.

  3. Select Tools | Segmentation | Cross Section.

    The graphic interactors (the large 3D arrows pointing to the X, Y, Z axes) and Cross Section box appear
  4. Drag a cone of an interactor to thin the cross section box.

  5. Drag the torus of an interactor to rotate the cross section box.

  6. Adjust the interactors and camera view until the desired orientation is achieved.

    Note: if you wish to segment out the cross section into another cloud, click the Tick icon on the Cross Section tool box. Otherwise, click the X icon to exit the command when done.

Monday, February 27, 2017

How to enable Cloud Compare's Fine Registration (ICP) command

For the longest time, I tried to align two point clouds with CloudCompare's Fine Align (ICP) command. But unfortunately, the command would not enable for me no matter what I did. I selected one cloud, two cloud (or so I thought). Neither reading the documentation nor watching the YouTube videos helped. The screenshot below showed the deactivated Fine Align command in the tool bar.

It turned out that proper selection is key. I was using the SHIFT key to select one cloud folder node and another cloud folder node, which selected folders, and clouds among other things.

  1. The proper way was to press down the CTRL key while clicking on only cloud nodes in the tree view pane, as shown in the screenshot below. Once two cloud nodes have been selected, the File Align icon in the toolbar will become enabled.

    The File Align icon is enabled.
  2. Click the Fine Align icon.

    The Clouds registration dialog box appears.

Monday, February 20, 2017

Clear Javascript, CSS, and HTML web cache in Chrome

When developing Javascript code for web pages, the Chrome browser caches the Javascript code including cascading style sheets and HTML files. Any changes made to the code may not be reflected when the page is refreshed. I found this frustrating. That is, until I discovered that Chrome has a way to clear its cache before reloading the web page.

To enable this, do the following:

  1. Select Options (the 3 vertical dots icon on the top right) | More Tools | Developer Tools.

    The Developer Tools pane appears.
  2. Now, do a mouse-right click on the Reload Page icon, as shown in the screenshot below.

    A popup menu appears.

  3. Choose Empty Cache and Hard Reload.

    The cache is cleared and any any changes made to the code is reflected in the reloaded page.

Monday, February 13, 2017

Show Myanmar Datum 2000 coordinates with this Mapplet

Myanmar uses a Universe Transverse Mercator projection at Zone 46 (with a Myanmar Datum 2000). For convenience, this mapplet has been written to display and locate Myanmar easting, northing coordinates on Google Maps. To show the coordinates of a location on the map, just click a point on the map, as shown in the screenshot below. A marker Info window will pop up displaying the Myanmar Datum 2000 easting and northing coordinates, as well as the Geographic latitude, and longitude values.

Try out this mapplet at this location

Related Posts with Thumbnails