csvmaps

This is a command-line tool for manipulating map-like CSV data, where the first column is treated as a “key” for that row. Please see the README for the most up-to-date documentation: https://gitlab.com/transportengineering/csvmaps

Installation

Currently requires stack because of a dependency on an unreleased (or rather, also not on hackage) package cassava-streaming.

$ git clone git@gitlab.com:transportengineering/csvmaps.git
$ cd csvmaps
$ stack install

Usage

$ csvmaps --help
Usage: csvmaps [-i|--infiles STRING]... [-o|--outfile STRING]
               [--has-header BOOL] [--op MAPOP]
               [--expr MAPEXPR] [--save-labels BOOL]
               [--verbose BOOL]

Available options:
  -h,--help                Show this help text
  -i,--infiles STRING...   Files to combine. The first column of each csv will
                           be used as a key. For non-csv files ( iles that don't
                           end in .csv), each line will be treated as a key.
  -o,--outfile STRING      .csv file to write output to. If omitted, use stdout.
  --has-header BOOL        whether to ignore header in csv files
  --op MAPOP               The operation to use to combine csv files (default:
                           union)
  --expr MAPEXPR
                           The expression to use to combine csv files. Reference
                           the inputs with $N for the Nth document in the
                           "infiles". The operations available are: 
                           1. Union with the + operator; 
                           2. Difference with the - operator; 
                           3. Intersection with the * operator; 
                           4. Union combining all columns with the +. operator; 
                           5. Intersection combining all columns with the *. 
                              operator 
                           6. Labels for expressions with the syntax
                              '"LABEL": EXPR'.

Examples

Take the rows that exist in both v1, in addition (preferring v1) to the rows that exist in both v1 and v2 (preferring v2).

$ csvmaps -i v1.csv -i v2.csv --expr '(($1 *| $2) +| ($1 - $2))'

MAPEXPR syntax

$N

References the Nth document specified, starting at 1.

keys A

Discards the values of all rows.

nulls A

Discards the rows that have non-empty values.

non-nulls A

Discards the rows that have empty values.

A : "label"

Assigns a label to the expression A. When used with the --save-labels option, this will also result in a file called label.csv being created using A.

const ["Col_1","Col_2",...] A

Replaces all values of A with the columns given in the first argument.

pad N A

Ensures the number of columns after each key is at least N; fills ones that don’t exist with empty strings.

col N A

Takes the Nth column.

col [N1,N2,...] A

Uses the columns given.

Unions

A + B

Left-biased union of A and B. Same as A + B. When keys exist in both maps, the values at those keys are taken from the left operand.

A |+ B

Left-biased union of A and B. Same as A + B. When keys exist in both maps, the left operand’s values at those keys are used, unless they are empty.

A +| B

Right-biased union of A and B. Same as A +| B. When keys exist in both maps, the right operand’s values at those keys are used, unless they are empty.

A |+| B

Union of A and B that concatenates the values of all keys that both operands have in common.

Intersections

A * B

Left-biased intersection of A and B. Same as A * B. When keys exist in both maps, the values at those keys are taken from the left operand.

A |* B

Left-biased intersection of A and B. Same as A * B. When keys exist in both maps, the left operand’s values at those keys are used, unless they are empty.

A *| B

Right-biased intersection of A and B. Same as A *| B. When keys exist in both maps, the right operand’s values at those keys are used, unless they are empty.

A |*| B

Intersection of A and B that concatenates the values of all keys that both operands have in common.

Difference

A - B

Returns the rows of A whose keys are not in B.