Monday, May 22, 2017

Merging multiple comma separated values CSV files

If there are many comma separated value CSV files with headers and you want to merge them into a single CSV file, it can be a pain having to do it by hand. Fortunately, there are ways to automate the task. One method which I like is to use the tail command from Unix. For Windows, a tail utility can be downloaded from http://tailforwin32.sourceforge.net/;There are others which a Google search can reveal.

Below are screenshots of a few CSV files to merge.


  1. Using a text editor, create a script or batch file. In the editor, type in the tail command to create a new merged file with a header.

    tail -n +1 -q green.csv > merge.csv

    Note: -n +1 means to start from the first line.
    A single > means to output to a new file

  2. Type in the commands to append lines from subsequent files without headers to the output file.

    tail -n +2 -q orange.csv >> merge.csv
    tail -n +2 -q transfer.csv >> merge.csv
    #...etc...
    Note: -n +2 means to start from the 2nd line
    >> means to append to the output file

  3. Run the script or batch file in a Command Prompt.

    The CSV files are merged.

Sunday, May 14, 2017

Resolving Python Module 'numpy' has no 'xxxx' member error message in Visual Studio Code

While writing some Python code in Visual Studio Code that calls some Numpy classes, Pylint error messages appear complaining about non-existing Numpy members, e.g. the log2 method, and red wavy lines underline the offending code, as shown in the screenshot below.


One way to clear this up is to get Pylint to white-list Numpy using Visual Studio Code's Settings.

  1. In Visual Studio Code, select File | Preferences | Settings.

    The settings.json file is displayed in the editor.

  2. Click the Workspace Settings tab.

    The settings appear in the editor.

  3. In the editor, type in the following and save the file.

    { "python.linting.pylintArgs" : [ "--extension-pkg-whitelist=numpy" ] }



  4. Close and reopen the folder in Visual Studio Code.

    The missing member messages and the red wavy underlines no longer appear.

Monday, May 1, 2017

Create labels from multi-columns in QGIS

It is quite simple to create labels from one or more database columns in QGIS; just that you have to type in the correct syntax as shown below.

  1. Run QGIS. Display some layers, e.g. stations.


  2. In the Layer Panel, mouse right click on the layer to be labelled, e.g. stations.

    A pop up menu appears
    .

  3. Choose Properties.

    The Layer Properties dialog box appears.

  4. Choose Labels. In the combo box, select Show labels for this layer.


  5.  In the Label with field, click the Expression icon.

    The Expression dialog box appears.

  6. In the Expression tab, type in the syntax to form the multi-column label e.g.
    name || ' ' || code.

    Note: name and code are two columns from the layer database table, and || is the string concatenation operator and ' ' is a space character.
  7. Click OK. And Click OK to apply the changes.

    The labels appear.
  8. If multi-line labels are desired, then the newline character '\n' needs to be concatenated with the database fields in the Expression dialog box, as shown below.
    name || '\n' || code



    And the resultant map look like this.
Related Posts with Thumbnails