I created a backend service for the Diamond Air Taxis Australian private/charter flights service, which provides possible flight itineraries for use on the search/booking pages, and then performs checking of booked itineraries until departure. If you want to see how those flights look, have a look at the YouTube channel, which Steve Boyd updates from time to time with recordings of his flights.

This is not a very high quality post: I don’t introduce anything that hasn’t been written about before already, and I don’t feel like I looked very deeply into the subject. It’s just an experience report, but hopefully that experience is useful to someone out there.

Pre-ramble

This is a brief rundown of the state of Cabal’s “public sublibrary” feature from the perspective of a me. I normally develop Haskell projects within a Nix shell. The level of reproducibility is extremely attractive. I recently span out a few libraries I’ve developed into their own open-source repositories; see the flurry of activity (not to be mistaken for productivity; mv did most of the work) on my GitLab.

For streaming-json I think the use-case of it is for web servers which produce large amounts of JSON. It’s structured in a way that makes it easier to do less allocation while producing said JSON, just writing to handles directly. There are a lot of web server libraries in the Haskell ecosystem and it’d be nice to support as many of them as I can.

The standard way to accomplish this is to just make separate Cabal packages for each interop library; they can all live in harmony in monorepo. When you have your Git repo cloned somewhere, you ultimately have a directory structure that looks like this:

~/Code/streaming-json/.git
~/Code/streaming-json/LICENSE
~/Code/streaming-json/README
~/Code/streaming-json/streaming-json/streaming-json.cabal
~/Code/streaming-json/streaming-json/src/Streaming/JSON.hs
~/Code/streaming-json/streaming-json/src/Streaming/JSON/Lines.hs
~/Code/streaming-json/streaming-json-servant/streaming-json-servant.cabal
~/Code/streaming-json/streaming-json-servant/src/Streaming/JSON/Servant.hs
~/Code/streaming-json/streaming-json-yesod/streaming-json-yesod.cabal
~/Code/streaming-json/streaming-json-yesod/src/Streaming/JSON/Yesod.hs
...
# haha, "streaming-json/streaming-json/streaming-json"

Anyway, it’s sorta crap. I don’t want to maintain 3 or more almost-identical .cabal files, in 3 or more different Hackage packages, in 3 or more almost-identical source trees, with the inability to see anything other than streaming-json in any of the paths at a glance ¹. It’s a bunch of minor irritations, but I just don’t like dealing with a bunch of minor irritations every time I switch files within a project in my editor.

In Cabal we don’t have feature flags like in Cargo. Feature flags undoubtedly do have their own potential pitfalls, discussed at some length across a few different GitHub issues, Reddit posts, and Discourse forums. Proponents such as myself are likely to think: well, Cargo does have them – and with very little actual restriction in terms of the foottrebuchets you can erect, and it seems to work quite nicely. But I could very well be blissfully ignorant of the pain points feature flags perhaps are creating for Rust/Cargo ecosystem maintainers. And definitely I am blissfully ignorant of the pain it would create for the Cabal project to actually implement.

Happily, Cabal does have another trick up its sleeve: public sublibraries. This was written about already by Kowainik back in 2019 – so how are things fairing almost 6 years later?

Recap: You can write sublibraries as ordinary library stanzas in your cabal file, just with an additional name, like library streaming-json-servant. Then set visibility: public within that stanza, and now you have a public sublibrary. If it’s in your available package set, then you should be able to then use that library from another package, like build-deps: streaming-json:streaming-json-servant.

This seems like a great way to implement optional features (with their own transitively-optional dependencies) in a library because unlike Cargo feature flags, which merely “should” be additive, Cabal sublibraries are additive ². So that footgun is avoided completely.

Getting to a REPL from various toolchain setup options

I’ve now tested out the feature across a few different “typical” Haskell development setups. The goal is to use a the streaming-json library and its streaming-json-servant sublibrary from another project, fetching streaming-json from its Git repo.

I set up a repo on my GitLab called sublib-test. Each branch has a different setup from which I try to pull in streaming-json and its sublibrary. There’s not any “actual” code there. (I feel no offence if readers just flick through the branches on that repo to see what’s different in each, rather than reading on; a repo speaks a thousand words.)

A few of the setups use Nix. You could set up a Nix project without using flakes at all, but use Nix flakes I have. I like the ergonomics of being able to nix flake update my dependencies, although you can definitely accomplish that with niv and its relatives too. In any case, the “real” difference between usual Nix setups for Haskell projects is the choice of using the default Haskell infra in Nixpkgs, or to use IOHK’s haskell.nix – in both style and substance they are quite different. The third option is using Nix just for getting a development shell going, which is fine, but nearly equivalent to using ghcup, except with better isolation of different toolchains across projects.

Basic scaffold

See the link in the heading. The branches on this repo are the different options below.

Option 1: Nix flake

1. Use nix flake init -t templates#haskell-hello to get the basic scaffold
2. Do some very minor finagling to get the flake.nix to support “extra deps” from outside Hackage.
3. Add the Git repo as a input to the flake.nix or use fetchFromGitLab. I put it as an input on the flake.nix (with flake = false), because it’s nice to be able to nix flake update a dependency.
4. nix develop (I would recommend Direnv’s use_flake in an actual project. You will likely get better editor integration [such as if your dev shell provides HLS, cabal-gild, fourmolu, etc.] that way.)
5. cabal repl
6. Realise that it doesn’t work and experience slight despair and confusion – what is a package anyway, as opposed to a component, or sublibrary? Are these distinctions the reason why sublibrary support is kinda broken and complex, because you need different code to handle all the different sorts of “components” that Cabal packages contain? Can you convince both Nix and Cabal of the existence of the sublibrary somehow? The answer to these questions are I don’t know; nor do I know if they are the right questions in the first place. ghc-pkg list seems to pick up the sublibrary.

Option 2: haskell-flake based on nix flake init -t templates#haskell-flake

Basically the same as plain but in a nice, declarative wrapper around plain Nix, which I’m definitely coming around to. My experience with nice declarative abstraction in Nix has sometimes involved having to completely break out of the nice declarative abstraction in order to do something though, so I’m slightly weary of it. Sublibrary support isn’t any different here from in the plain option.

Option 3: haskell-nix based on nix flake init -t templates#haskell-nix

This uses haskell.nix to parse the cabal.project file (see below). I almost gave up after 4 hours of being stuck on compiling generics-sop-lib-generics-sop-x86_64-w64-mingw32 until I realised it’s configured in nix/hix.nix where cross platform compilation support is configured (which is pretty cool to have! But not very useful for my development environment).

An initial cabal repl actually works within this environment – but only after cabal-install itself fetched streaming-json. For actual builds (as opposed to generating a dev shell) haskell.nix does actually fetch the repository, and it’s able to build and utilise the sublibrary. Huzzah – there’s at least one way for a Haskell Nix project to use Cabal sublibraries. I’m not sure how I feel about it overall, as it’s a quite complex abstraction, with its own nixpkgs even.

Option 4: ghcup + Cabal setup

This is what newcomers to Haskell would likely be using. The experience here was actually very straightforward.

1. Use ghcup to set up GHC and cabal-install.
2. Add a cabal.project which has the Git repo for streaming-json at the right revision.
3. cabal update
4. cabal repl
5. >>> import Streaming.JSON.Servant - yay!

Option 5: Stack

This was also very straightforward and a likely option that Haskell newcomers would use. Stack makes a bit more effort to have helpful (and searchable) error messages which was nice.

1. Use ghcup to acquire Stack.
2. Add a stack.yaml
3. stack repl
4. >>> import Streaming.JSON.Servant - yay!

Documentation

This is where sublibrary support still sucks: Haddock doesn’t generate documentation for sublibraries by default, but you can get some documentation out of it, by explicitly targetting all the components you want docs for, like cabal haddock streaming-json streaming-json:streaming-json-servant. The sublibrary documentation will be placed under an enigmatically-named directory l/sublibrary-name. I don’t think Hackage will do this for you at the moment though, so any sublibraries actually on there won’t get rendered docs AFAIK.

Conclusion

So there you have it. Support for public sublibraries is good for cabal-install, Stack, and Haskell.nix users. It’s sorta shit in “plain” Nix. This post was generated using the help of a keyboard and my tired fingers. Hopefully it is helpful to someone out there.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

Not wanting to let the 6+ gigabytes of storage required for my haskell.nix-provided GHC 9.2.1 shell to go to waste, I immediately set to work using one of the new goodies it provides; GHC 9.2.1 introduced 3 new language extensions, 2 of which (OverloadedRecordDot, and NoFieldSelectors) are of interest here.

NoFieldSelectors is great because we can finally put automatically generated record field accessors in the bin forever. The upshot is much less clutter with variable and record field names. Hurray!

In addition we have a very powerful extension in OverloadedRecordDot. With this you can now access fields on some data like in other – less civilised – languages, where accessing a field involves merely using the name of the field after a dot after the so-called object. There’s a huge advantage here, in that the namespace for what comes after the dot is only as big as the number of public fields on the object. We Haskellers are currently accustomed to suffering through qualified imports of record fields, or strangely prefixed record field names, or awkward label syntax and extra composition operators.

But there is a drawback. What happens to all my code that’s been using lens or optics? It’s fairly idiomatic in the lens extended universe to write field compositions as just field1.field2.field3, but now with OverloadedRecordDot enabled, . used without spaces around it is no longer the (.) operator from Prelude, but a special field accessing operator introduced by OverloadedRecordDot. obj.field is equivalent now to getField @"field" obj; obj.a.b would be getField @"b" (getField @"a" obj) – so we can see how this doesn’t really make sense with optics, as the fields are no longer really “first-class” objects but depend on whatever we are accessing.

You can get lenses out of a stone

We can still make this work. Let’s see what the definition for HasField is:

>>> :i getField
type HasField :: forall {k}. k -> * -> * -> Constraint
class HasField x r a | x r -> a where
  getField :: r -> a
    -- Defined in `GHC.Records'

We can define our own instances for this class that GHC will happily accept – it isn’t a class where only it is permitted to provide the instances automatically (a la Coercible).

The same idea has been implemented before in JavaScript/TypeScript land, where there is a Proxy object that lets you override the absolute anything out of almost everything. OverloadedRecordDot enables many of the same (pretty wild) possibilities.

Some prior art using Proxy for optics:

• https://github.com/aynik/proxy-lens
• https://github.com/hatashiro/lens.ts
• https://github.com/yelouafi/focused

So what follows is pretty much that, but for Haskell, using lens or optics, and OverloadedRecordDot.

First, we can define the type that will encapsulate our new “first class field” type. It’ll just be a unit type carrying around a type-level list of symbols, morally representing a bunch of fields composed with each other.

-- I recommend to put GHC extensions all on one line so that it's harder to
-- count how many extensions your file requires

{-# LANGUAGE AllowAmbiguousTypes, DataKinds, OverloadedRecordDot, ScopedTypeVariables, TypeApplications, UndecidableInstances, DeriveGeneric, TypeFamilies #-}

data L (fields :: [Symbol]) = L deriving (Show)

ltail :: L (x ': xs) -> L xs
ltail _ = L

Now, note that OverloadedRecordDot won’t work on data constructors as GHC will see them as modules first. So, define a lowercase synonym for L as well. A nice bonus is we can fix it to be of the empty list type instead of any old type unifying with L xs.

l :: L '[]
l = L

Defining HasField is easy:

instance HasField x (L xs) (L (x ': xs)) where
  getField _ = L

Which gives us some pretty nifty capabilities:

>>> :t l.asdf
l.asdf :: L '["asdf"]
>>> :t l.foo.bar
l.foo.bar :: L '["bar", "foo"]
>>> :t l.foo.bar.baz
l.foo.bar.baz :: L '["baz", "bar", "foo"]

Now, basically, all that’s left is a way to turn a L into an optic. This will work with both lens and optics style optics. The trick is to utilise some code previously written that was intended for use with the OverloadedLabels but works perfectly here as well. For lens, generic-lens gives us that ability to turn a type-level Symbol into a lens, and optics has the same functionality built-in through its LabelOptic typeclass.

lens version

class LToLens xs s t a b where
  asLens :: L xs -> Lens s t a b

instance (a ~ s, b ~ t) => LToLens '[] s t a b where
  asLens _ = id

instance
  (
    Field x u v a b,
    LToLens xs s t u v
  ) => LToLens (x ': xs) s t a b where
  asLens l =
    asLens (ltail l) .
    fieldLens @x @u @v @a @b

optics version

class LToOptic xs k s t a b where
  asOptic :: L xs -> Optic k NoIx s t a b

-- Unsure how needed the A_Lens constraint here is or if there's a better alternative
instance (a ~ s, b ~ t, k ~ A_Lens) => LToOptic '[] k s t a b where
  asOptic _ = Optic id

instance
  (
    LabelOptic x k u v a b,
    LToOptic xs l s t u v,
    JoinKinds l k m
  ) => LToOptic (x ': xs) m s t a b where
  asOptic l =
    asOptic @xs @l @s @t @u @v (ltail l) %
    labelOptic @x @k @u @v @a @b

Putting it to use

data A = A { a :: B } deriving (Show, Generic)
data B = B { b :: C } deriving (Show, Generic)
data C = C { c :: Int } deriving (Show, Generic)

test :: A
test = A (B (C 42))

test1 = test ^. asLens l.a.b.c
-- test1 = 42

test43 = (test & asLens l.a.b.c .~ 43) ^. asLens l.a.b.c
-- test43 = 43

-- basic type-changing update
data Test a = Test { test :: a }
  deriving (Generic, Show)

typechanging0 :: Test Int
typechanging0 = Test 23

typechanged :: Test ()
typechanged = typechanging0 & asLens l.test .~ ()

nested :: Test (Test (Test Char))
nested = Test (Test (Test ())) & asLens l.test.test.test .~ 'a'

Going further

Do some type-level parsing in the GHC.Records.HasField instance to enable:

• Prisms, i.e. fields beginning with /_[A-Z]/. (Note actual fields can do that as well, but realistically, who cares?)
• An escape hatch out of the L type. That is, accessing, for instance, l.lens, could have the same effect as calling asLens. Alternatively, have an escape hatch straight into getters and setters, or even a Lens “object” that supports many different operations on it. You could really go wild with syntactically approximating OOP with this extension.

Should anyone use this? Honestly, I dunno. There’s already so many different options for records in Haskell that yet another is maybe … perfectly ok. I haven’t packaged this up because I can’t use GHC 9.2.1 in production anyway due to previous changes in the 9.x series holding back many packages that now need various changes. GHC 8.10.7 is, basically, good enough for any production use, and this sort of stuff sadly falls in the “nice to have” basket.

All images © Michael Ledger, all rights reserved.

This is a TypeScript library making extreme use of type-level computation to create a really nice routing library (a la react-router), for SolidJS, and backed by a router5 router internally. See documentation on GitHub.

All images © Michael Ledger, all rights reserved.

Available on GitHub

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

Respecify was a novel web-based requirements authoring tool developed by me for Ricardo Rail. It aids requirements engineers by applying a simple English grammar that ensures.

1. Basic syntactic correctness.
2. All terms within a specification are defined and have at most 1 meaning.
3. The relationships specified do not contradict eachother.
4. Requirements meet some basic quality checks.

A tool demonstration was published for RE’17.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

This is a command-line tool for manipulating map-like CSV data, where the first column is treated as a “key” for that row. Please see the README for the most up-to-date documentation here.

Installation

Currently requires stack because of a dependency on an unreleased (or rather, also not on hackage) package cassava-streaming.

$ git clone git@gitlab.com:transportengineering/csvmaps.git
$ cd csvmaps
$ stack install

Usage

$ csvmaps --help
Usage: csvmaps [-i|--infiles STRING]... [-o|--outfile STRING]
               [--has-header BOOL] [--op MAPOP]
               [--expr MAPEXPR] [--save-labels BOOL]
               [--verbose BOOL]

Available options:
  -h,--help                Show this help text
  -i,--infiles STRING...   Files to combine. The first column of each csv will
                           be used as a key. For non-csv files ( iles that don't
                           end in .csv), each line will be treated as a key.
  -o,--outfile STRING      .csv file to write output to. If omitted, use stdout.
  --has-header BOOL        whether to ignore header in csv files
  --op MAPOP               The operation to use to combine csv files (default:
                           union)
  --expr MAPEXPR
                           The expression to use to combine csv files. Reference
                           the inputs with $N for the Nth document in the
                           "infiles". The operations available are: 
                           1. Union with the + operator; 
                           2. Difference with the - operator; 
                           3. Intersection with the * operator; 
                           4. Union combining all columns with the +. operator; 
                           5. Intersection combining all columns with the *. 
                              operator 
                           6. Labels for expressions with the syntax
                              '"LABEL": EXPR'.

Examples

Take the rows that exist in both v1, in addition (preferring v1) to the rows that exist in both v1 and v2 (preferring v2).

$ csvmaps -i v1.csv -i v2.csv --expr '(($1 *| $2) +| ($1 - $2))'

MAPEXPR syntax

$N

References the Nth document specified, starting at 1.

keys A

Discards the values of all rows.

nulls A

Discards the rows that have non-empty values.

non-nulls A

Discards the rows that have empty values.

A : "label"

Assigns a label to the expression A. When used with the --save-labels option, this will also result in a file called label.csv being created using A.

const ["Col_1","Col_2",...] A

Replaces all values of A with the columns given in the first argument.

pad N A

Ensures the number of columns after each key is at least N; fills ones that don’t exist with empty strings.

col N A

Takes the Nth column.

col [N1,N2,...] A

Uses the columns given.

Unions

A + B

Left-biased union of A and B. Same as A + B. When keys exist in both maps, the values at those keys are taken from the left operand.

A |+ B

Left-biased union of A and B. Same as A + B. When keys exist in both maps, the left operand’s values at those keys are used, unless they are empty.

A +| B

Right-biased union of A and B. Same as A +| B. When keys exist in both maps, the right operand’s values at those keys are used, unless they are empty.

A |+| B

Union of A and B that concatenates the values of all keys that both operands have in common.

Intersections

A * B

Left-biased intersection of A and B. Same as A * B. When keys exist in both maps, the values at those keys are taken from the left operand.

A |* B

Left-biased intersection of A and B. Same as A * B. When keys exist in both maps, the left operand’s values at those keys are used, unless they are empty.

A *| B

Right-biased intersection of A and B. Same as A *| B. When keys exist in both maps, the right operand’s values at those keys are used, unless they are empty.

A |*| B

Intersection of A and B that concatenates the values of all keys that both operands have in common.

Difference

A - B

Returns the rows of A whose keys are not in B.

I love music. I have a rather large digital collection of it, hoarded over many years, and I intend to grow it indefinitely. There are a wide variety of formats, from FLAC, to Ogg Theora, to MP3, to (very poor) WMA.

Converting all the music at once, preserving the exact directory structure, is more of a challenge than it ought to be – ideally, a command should exist, like rsync, that intelligently decides what to convert and what to just copy (converting 96kbps mp3 files to a different lossy format will not do it many favours).

That’s what shrinkmusic is. It takes an input directory, an output directory, computes a plan – what to convert, what to keep, what to skip due to already existing in the output directory, and what to ignore due to a user flag – and then executes it, in parallel, using as many jobs as you’d like.

Features

• Add new music without re-converting an entire library
• Decides (albeit by pretty dumb heuristics) what is worth converting, and what is OK to just copy
• Splits FILE.flac and FILE.cue pairs automatically
• Parallel execution
• Dry run mode
• Can ignore specific files/directories (e.g., I don’t care to copy hoarded podcasts – readily available on the internet anyway – over)

Install

Probably easiest to install using Stack or Nix.

Stack

$ git pull https://github.com/mikeplus64/shrinkmusic.git
$ cd shrinkmusic
$ stack install

Nix

$ nix-env -f ./default.nix -i

Usage

Usage: shrinkmusic (-i|--input INPUT) (-o|--output OUTPUT)
                   (-b|--bitrate BITRATE) (-j|--jobs JOBS) [-z|--ignore IGNORE]
                   [-d|--dry-run]

Available options:
  -h,--help                Show this help text
  -i,--input INPUT         Input directory
  -o,--output OUTPUT       Output directory
  -b,--bitrate BITRATE     bitrate for audio
  -j,--jobs JOBS           Number of jobs to use
  -z,--ignore IGNORE       Ignore this file
  -d,--dry-run             Do nothing; just output the plan

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

I’ve been working on and off on a HTML templating library for a few months. It was originally intended for use in Respecify, but that evolved into a single-page application. Thus a library for fast server-side HTML generation wasn’t necessary any more. (Though you could just as well use nice-html on a frontend through GHCJS.)

The biggest difference to BlazeHtml and Lucid and is that templates are compiled, so as much HTML is “rendered” (i.e., as many strings are concatenated as possible ☺) ahead-of-time as possible. This isn’t an extremely clever optimisation, but it can make a pretty big difference in render times.

To make it possible to still “inject” data into a template, a type parameter is given to the markup (compiled FastMarkup and non-compiled Markup) that gives templates “holes”. e.g., a template that requires the description of a person to render could be typed tpl :: FastMarkup (Person -> Text). On the other hand, a template with absolutely no dynamic data can be typed tpl :: forall a. FastMarkup a — or just tpl :: FastMarkup Void.

There are a few functions for rendering templates; take your pick. The easiest-to-use, and most magical, is r :: Render a m => a -> m Builder. This allows you to render any FastMarkup, constraining m to be a ReaderT monad when the parameter to FastMarkup has an arrow. A simpler and less magical alternative renderM :: Monad m => (a -> m Builder) -> FastMarkup a -> m Builder is also provided.

That is, given a simple template like:

data Params = Params { param1 :: Text }
tpl :: FastMarkup (Params -> Text)
tpl = compile $ do -- note compile :: Markup t a -> FastMarkup t
  p_ $ do
    "param1 is:" 
    dynamic param1

Rendering can be achieved with:

-- using the Reader interface:
r tpl `runReaderT` Params{param1 = "i am param1!!!"} :: Monad m => m Builder

-- alternatively using (:$) :: FastMarkup (a -> b) -> a -> a :$ b
r (tpl :$ Param{param1 = "i am param1!!!"}) :: Monad m => m Builder

-- alternatively using 
-- renderM :: Monad m => (a -> m Builder) -> FastMarkup a -> m Builder
-- for shunners of magic
renderM (\f -> return (fromText (f Param1{param1 = "i am param1!!!"}))) tpl 
  :: Monad m => m Builder

The biggest drawback of this approach is that it’s more difficult to write templates in this style, despite whatever benefits it confers. The primitives that you need to be aware of for inserting “dynamic” data are:

1. dynamic :: p -> Markup p () inserts a hole that will be escaped.

2. dynamicRaw :: p -> Markup p () inserts a hole that won’t be escaped, e.g. for a chunk of HTML.

3. stream :: Foldable f => Markup (a -> n) r -> Markup (f a -> FastMarkup n) r inserts a hole for any old Foldable – e.g. [Text], [Article], Vector TodoItem etc. This is admittedly a bit of a misnomer, since most sane Foldable s won’t ever actually stream. Except for String s produced by Prelude.readFile, Prelude.getContents, etc., but these functions are increasingly taboo.

4. sub :: Markup n a -> Markup (FastMarkup n) a puts templates in your templates.

A complete example

{-# LANGUAGE OverloadedStrings #-}
module TodoList where
import           Data.Text                   (Text)
import           Text.Html.Nice              ((:$) (..), Attr (..), Builder,
                                              FastMarkup, Render (..))
import           Text.Html.Nice.Writer
import           Text.Html.Nice.Writer.Html5

data Todo = Todo
  { todoDate :: Text
  , todoText :: Text
  }

todos :: [Todo]
todos =
  [ Todo "october 25 2017" "write todo list <html>asdf</html>" -- escaped
  , Todo "october 26 2017" "write another todo list"
  ]

template :: FastMarkup ([Todo] -> FastMarkup Text)
template = compile $ do
  doctype_
  html_ $ do
    head_ $ title_ "Todo list"
    body_ $ do
      h1_ "Todo list"
      stream $ div_ ! "class" := "todo-item" $ do
        text "\n<script></script>\n" -- this gets escaped
        b_ (dynamic todoText)
        " ("
        dynamic todoDate
        ")"

test :: Monad m => m Builder
test = r (template :$ todos)

Performance

These benchmarks derive from blaze-markup’s “bigtable” benchmark; but aren’t exactly the same. They’ve been shamelessly altered – I added some static markup to the front and end of the templates, since that’s more realistic than a table on its own – to highlight the strengths of nice-html.

The benchmark itself generates a table of N rows, and measures the time taken to render, as well as the amount of memory used, using the venerable criterion and the underrated weigh. The nice-html version looks (and by looks, I mean “is”) like:

{-# LANGUAGE OverloadedStrings #-}
-- derived from https://github.com/jaspervdj/blaze-markup/blob/master/benchmarks/bigtable/html.hs
module BigTable.Nice where
import           Control.Monad.Trans.Reader (runReader)
import           Criterion.Main             (Benchmark, bench, nf)
import           Data.Text.Lazy             (Text)
import           Data.Text.Lazy.Builder     (Builder, toLazyText)
import           Data.Text.Lazy.Builder.Int (decimal)
import           Weigh                      (Weigh, func)

import           Text.Html.Nice

rows :: FastMarkup ([[Int]] -> FastMarkup (FastMarkup Builder))
rows = compile $ do
  h1_ "i am a real big old table\n"
  p_ "i am good at lots of static data\n"
  p_ "i am glab at lots of static data\n"
  p_ "i am glob at lots of static data\n"
  p_ "i am glib at lots of static data\n"
  p_ "i am glub at lots of static data\n"
  p_ "i am glom at lots of static data\n"
  p_ "i am glof at lots of static data\n"
  p_ "i am gref at lots of static data\n"
  p_ "i am greg at lots of static data\n"
  table_ $ do
    thead_ . tr_ . mapM_ (th_ . builder . decimal) $ [1..10 :: Int]
    tbody_ . stream . tr_ . stream . td_ $ do
      p_ "hi!\n"
      dynamic decimal
      p_ "hello!\n"
  p_ "i am good at lots of static data\n"
  p_ "i am glab at lots of static data\n"
  p_ "i am glob at lots of static data\n"
  p_ "i am glib at lots of static data\n"
  p_ "i am glub at lots of static data\n"
  p_ "i am glom at lots of static data\n"
  p_ "i am glof at lots of static data\n"
  p_ "i am gref at lots of static data\n"
  p_ "i am greg at lots of static data\n"

bigTable :: [[Int]] -> Text
bigTable table = toLazyText (r rows `runReader` table)

benchmark :: [[Int]] -> Benchmark
benchmark t = bench "nice" (bigTable `nf` t)

weight :: [[Int]] -> Weigh ()
weight i = func (show (length i) ++ "/nice") bigTable i

Runtime

Abridged: blaze is fast; lucid is faster; nice-html is fasterer.

Benchmark perf: RUNNING...
benchmarking 10/blaze
time                 91.73 μs   (91.10 μs .. 92.33 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 92.64 μs   (92.25 μs .. 93.03 μs)
std dev              1.358 μs   (1.073 μs .. 1.807 μs)

benchmarking 10/nice
time                 35.76 μs   (35.52 μs .. 36.00 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 35.50 μs   (35.28 μs .. 35.67 μs)
std dev              626.9 ns   (467.4 ns .. 811.9 ns)
variance introduced by outliers: 14% (moderately inflated)

benchmarking 10/lucid
time                 57.08 μs   (56.91 μs .. 57.27 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 57.20 μs   (56.94 μs .. 57.36 μs)
std dev              711.5 ns   (531.2 ns .. 1.126 μs)

benchmarking 100/blaze
time                 762.7 μs   (760.5 μs .. 764.2 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 762.0 μs   (759.5 μs .. 763.9 μs)
std dev              7.546 μs   (5.949 μs .. 9.589 μs)

benchmarking 100/nice
time                 344.2 μs   (342.9 μs .. 345.4 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 343.5 μs   (342.4 μs .. 344.5 μs)
std dev              3.498 μs   (2.939 μs .. 4.304 μs)

benchmarking 100/lucid
time                 486.5 μs   (485.2 μs .. 487.8 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 485.5 μs   (483.9 μs .. 486.6 μs)
std dev              4.137 μs   (2.838 μs .. 7.064 μs)

benchmarking 1000/blaze
time                 7.243 ms   (7.183 ms .. 7.310 ms)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 7.298 ms   (7.246 ms .. 7.347 ms)
std dev              147.5 μs   (125.5 μs .. 178.1 μs)

benchmarking 1000/nice
time                 3.422 ms   (3.387 ms .. 3.465 ms)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 3.420 ms   (3.402 ms .. 3.436 ms)
std dev              56.16 μs   (46.34 μs .. 69.55 μs)

benchmarking 1000/lucid
time                 4.689 ms   (4.661 ms .. 4.714 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 4.685 ms   (4.667 ms .. 4.698 ms)
std dev              48.05 μs   (38.33 μs .. 62.37 μs)

Benchmark perf: FINISH

Memory use, including compilation overhead

Benchmark mem: RUNNING...

Case         Allocated  GCs
10/blaze       597,808    1
10/nice      3,062,248    5
10/lucid       247,008    0
100/blaze    4,556,200    8
100/nice     5,716,888   11
100/lucid    1,735,160    3
1000/blaze  44,138,200   85
1000/nice   32,264,800   62
1000/lucid  16,582,944   29
Benchmark mem: FINISH

Environment info

• packages pulled from Stackage’s lts-8.13 resolver.
• nice-html-0.3.0
• lucid-2.9.8.1
• blaze-html-0.8.1.3 and blaze-markup-0.7.1.1

Roadmap

1. A more honestly streaming stream using e.g. streaming or pipes or conduit – or maybe all 3 at once, just to stick it to the zealots of each ☺ – shouldn’t be too hard to implement.

2. Rewrite Text.Html.Nice.Writer to just use a plain-old State or Writer monad internally.

3. Have a virtual-DOM-esque rerender function that (somehow) only re-renders what is likely (i.e, if a parameter has changed) to change, and some JavaScript glue code to enable a client to replace “old” HTML. I’ve scratched at the surface of this with note, which just gives nodes a unique id attribute, but it would be really neat to achieve this – my eventual dream is to be able to write fast server-side single-page-applications (especially if updates are facilitated over e.g. a WebSocket) entirely in Haskell without needing GHCJS. jsaddle might already do this but I haven’t seriously looked into it.

Quick links

1. Hackage
2. GitHub
3. type-of-html is another fairly recent library that “compiles” HTML (and is a Haskell EDSL), but it does so at the type-level.

Enjoy!

All images © Michael Ledger, all rights reserved.

This post is about a command-line tool for manipulating map-like CSV data, where the first column is treated as a “key” for that row. This post is largely the same as the README available at https://gitlab.com/transportengineering/csvmaps#unions; except that it’ll (extremely slowly) get buried under other posts I make. I also self-plagiarized the same README again on the “project” page for csvmaps, also on this website.

Installation

Currently requires stack because of a dependency on an unreleased (or rather, also not on hackage) package cassava-streaming.

$ git clone git@gitlab.com:transportengineering/csvmaps.git
$ cd csvmaps
$ stack install

Motivation

The use case that motivated it was to manipulate a dictionary of terms from a project that we, Transport Engineering were tasked to clean up, with very little time to actually do it.

The dictionary was simply all the title-case(*) terms that we could automatically pull out of a particular version of the project. This lead to there being thousands of spurious terms (e.g., from words at the beginning of a sentence), and terms that were only partially detected, and a proportional amount of manual work to cull or edit them.

The tool allowed me to:

1. Delete the same terms deleted in version A in version A+1

2. For terms that were edited rather than deleted in version A, replace the non-edited version from A+1 (if it existed) with the version from A

3. Detect entirely new terms

4. Combine the official project dictionaries with the title-case one I’d produced

(*): It also included some joining words like “and”, “or”, “for”, “for” etc.

Usage

$ csvmaps --help
Usage: csvmaps [-i|--infiles STRING]... [-o|--outfile STRING]
               [--has-header BOOL] [--op MAPOP]
               [--expr MAPEXPR] [--save-labels BOOL]
               [--verbose BOOL]

Available options:
  -h,--help                Show this help text
  -i,--infiles STRING...   Files to combine. The first column of each csv will
                           be used as a key. For non-csv files ( iles that don't
                           end in .csv), each line will be treated as a key.
  -o,--outfile STRING      .csv file to write output to. If omitted, use stdout.
  --has-header BOOL        whether to ignore header in csv files
  --op MAPOP               The operation to use to combine csv files (default:
                           union)
  --expr MAPEXPR
                           The expression to use to combine csv files. Reference
                           the inputs with $N for the Nth document in the
                           "infiles". The operations available are: 
                           1. Union with the + operator; 
                           2. Difference with the - operator; 
                           3. Intersection with the * operator; 
                           4. Union combining all columns with the +. operator; 
                           5. Intersection combining all columns with the *. 
                              operator 
                           6. Labels for expressions with the syntax
                              '"LABEL": EXPR'.

Examples

Take the rows that exist in both v1, in addition (preferring v1) to the rows that exist in both v1 and v2 (preferring v2).

$ csvmaps -i v1.csv -i v2.csv --expr '(($1 *| $2) +| ($1 - $2))'

MAPEXPR syntax

$N

References the Nth document specified, starting at 1.

keys A

Discards the values of all rows.

nulls A

Discards the rows that have non-empty values.

non-nulls A

Discards the rows that have empty values.

A : "label"

Assigns a label to the expression A. When used with the --save-labels option, this will also result in a file called label.csv being created using A.

const ["Col_1","Col_2",...] A

Replaces all values of A with the columns given in the first argument.

pad N A

Ensures the number of columns after each key is at least N; fills ones that don’t exist with empty strings.

col N A

Takes the Nth column.

col [N1,N2,...] A

Uses the columns given.

Unions

A + B

Left-biased union of A and B. Same as A + B. When keys exist in both maps, the values at those keys are taken from the left operand.

A |+ B

Left-biased union of A and B. Same as A + B. When keys exist in both maps, the left operand’s values at those keys are used, unless they are empty.

A +| B

Right-biased union of A and B. Same as A +| B. When keys exist in both maps, the right operand’s values at those keys are used, unless they are empty.

A |+| B

Union of A and B that concatenates the values of all keys that both operands have in common.

Intersections

A * B

Left-biased intersection of A and B. Same as A * B. When keys exist in both maps, the values at those keys are taken from the left operand.

A |* B

Left-biased intersection of A and B. Same as A * B. When keys exist in both maps, the left operand’s values at those keys are used, unless they are empty.

A *| B

Right-biased intersection of A and B. Same as A *| B. When keys exist in both maps, the right operand’s values at those keys are used, unless they are empty.

A |*| B

Intersection of A and B that concatenates the values of all keys that both operands have in common.

Difference

A - B

Returns the rows of A whose keys are not in B.

This is just a brief post about radixtree, which is a library that:

1. Produces radix (alternatively, prefix?) trees from Text values
2. Provides a generic parser suitable for use with attoparsec, trifecta, parsec, or anything with a CharParsing (from parsers)

Background

I’m developing a requirements authoring tool called Respecify, at Transport Engineering. One of its core features is a nifty (if I do say so myself) parser generator that I’ve used to create an extremely constrained English grammar for, which then parses requirements, and enables some other core features of Respecify.

The parser has to be able to quickly parse hundreds of requirements at once, each containing terms from a fixed (-ish) dictionary of user-specified nouns and verbs. The number of terms in these dictionaries can easily number in the thousands, so it’s critical that this is fast. See this short project page for more information about Respecify.

Foreground

Radix trees are a much-beloved data structure useful for parsing terms from large dictionaries with lots of similar prefixes (e.g., English). In this context, each edge from a specific node is labelled with the text needed to advance a parser to the next node, and each node is marked with whether or not the parser can terminate successfully there, returning whatever datum is at that node.

Performance

The motivation is simply speed. Parsing large corpuses can be much faster with this approach. Benchmark results on a corpus of around 800 English terms demonstrate this:

Benchmark radixtree-parsing: RUNNING...
benchmarking attoparsec/radix
time                 5.032 μs   (5.025 μs .. 5.041 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 5.035 μs   (5.027 μs .. 5.047 μs)
std dev              31.24 ns   (23.48 ns .. 45.45 ns)

benchmarking attoparsec/radix compressed
time                 5.041 μs   (5.031 μs .. 5.053 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 5.060 μs   (5.045 μs .. 5.084 μs)
std dev              59.94 ns   (42.56 ns .. 82.04 ns)

benchmarking attoparsec/naiive
time                 69.32 μs   (69.16 μs .. 69.47 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 69.54 μs   (69.28 μs .. 69.89 μs)
std dev              1.028 μs   (778.0 ns .. 1.251 μs)

Benchmark radixtree-parsing: FINISH

(Here, “naiive” refers to, basically, choice . map text . sortOn (negate . length).)

I don’t know and haven’t measured if, for Respecify’s use-case, radixtree is actually faster than e.g., tokenising and using a hashtable / HashMap, though that approach also has its own unique downsides. Namely, that you actually have to define what a token is: even though a user might want the term “IEEE 754” in their references dictionary, the parser would have to lookup “IEEE”, then “IEEE 754”. But if the user just wants to ignore the parse error because “IEEE 754” isn’t in their reference list yet, the parser will have to just keep trying with each new token, e.g. “IEEE 754 is”, “IEEE 754 is great”, etc.

Memory use

The trade-off is that a RadixTree uses much more memory than a simple list of terms, though this is slightly mitigated with a clever (i.e., probably horribly dangerous – though it hasn’t done anything evil to me yet, and the test suite seems to validate it) variant named CompressedRadixTree, which just relies on a single Text array for all of its edges and leafs.

Example code

TODOs

1. Become less afraid of CompressedRadixTree
2. Allow arbitrary data to be accepted by RadixTree, so they can be used as an alternative to HashMap Text.

Links

1. Hackage
2. GitLab

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

GHC.TypeLits gives us some very powerful type-level functionality (with some caveats that will hopefully be worked out in future) that has not yet seen very widespread use. With some mildly ugly leg-work, I’ve made a small library that provides a n-dimensional statically bounded index type, and associated functions for their application with as minimal overhead as possible.

Motivation

Making plissken, I was unhappy with the state of linear algebra libraries on Hackage. The venerable hmatrix is excellent - once your input sizes outweigh the constant factors involved. For the matrices that my game engine lived and survived on – 4x4 matrices and 4-vectors – hmatrix was easily outperformed in simple 4x4 matrix multiplication by linear, which is a naiive Haskell implementation of the sort of “small” linear algebra that I need here. I ended up using hmatrix though, since it being built on Storable made for very easy interaction with OpenGL.

A trick I found in order to inline away “static recursion” was to abstract the use of the recursion parameter into a typeclass whose function is inlined. So long as you remember the all-powerful -O2 switch, GHC will happily inline away every recursive call, which will hopefully give way to further compile-time evaluation.

Here’s a little demonstration of such a class:

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE KindSignatures #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE Undecidableinstances #-} -- our use of this should be fine.
module Playaround where
import Control.Applicative
import GHC.TypeLits

data Peano = Succ Peano | Zero

class For (n :: Peano) where
  for :: Applicative m => Proxy n -> (Integer -> m ()) -> m () -> m ()

instance For Zero where
  {-# INLINE for #-}
  for _ _ acc = acc

instance (KnownNat (1+FromPeano n), For n) => For (Succ n) where
  {-# INLINE for #-}
  for p f acc = for (next p) f (acc *> f (peanoVal p))

next :: Proxy (Succ n) -> Proxy n
next _ = Proxy

type family FromPeano (n :: Peano) :: Nat where
  FromPeano Zero     = 0
  FromPeano (Succ n) = 1 + FromPeano n

type family ToPeano (n :: Nat) :: Peano where
  ToPeano 0 = Zero
  ToPeano n = Succ (ToPeano (n-1))

fromPeano :: Proxy (n :: Peano) -> Proxy (FromPeano n)
fromPeano _ = Proxy

toPeano :: Proxy (n :: Nat) -> Proxy (ToPeano n)
toPeano _ = Proxy

peanoVal :: KnownNat (FromPeano n) => Proxy (n :: Peano) -> Integer
peanoVal = fromInteger . natVal . fromPeano

That’s pretty gross. We can improve it by wrapping it in a small helper:

niceFor :: (For (ToPeano n), Applicative m)
        => Proxy (n :: Nat) -> (Integer -> m ()) -> m ()
niceFor p f = for (toPeano p) f (pure ())

Unfortunately GHC doesn’t see as we can intuitively, that since all Nat has an instance for ToPeano, For has the instance For (ToPeano (n :: Nat)), so we have to add that constraint.

Let’s try it out:

{-# LANGUAGE DataKinds #-}
module Main where
import Playaround
import qualified Data.Vector.Mutable as VM

main :: IO ()
main = do
  vect <- M.new 16
  niceFor (Proxy :: Proxy 15) $ \ix -> 
    M.write vect ix ix

$ ghc-core Main.hs
    -dsuppress-idinfo \
    -dsuppress-coercions \
    -dsuppress-type-applications \
    -dsuppress-uniques \
    -dsuppress-module-prefixes

Some scrolling later…

main1 :: State# RealWorld -> (# State# RealWorld, () #)
main1 =
  \ (eta :: State# RealWorld) ->
    case newArray# 16 (uninitialised) (eta `cast` ...)
    of _ { (# ipv, ipv1 #) ->
    case writeArray# ipv1 15 (I# 15) ipv of s'# { __DEFAULT ->
    case writeArray# ipv1 14 (I# 14) s'# of s'#1 { __DEFAULT ->
    case writeArray# ipv1 13 (I# 13) s'#1 of s'#2 { __DEFAULT ->
    case writeArray# ipv1 12 (I# 12) s'#2 of s'#3 { __DEFAULT ->
    case writeArray# ipv1 11 (I# 11) s'#3 of s'#4 { __DEFAULT ->
    case writeArray# ipv1 10 (I# 10) s'#4 of s'#5 { __DEFAULT ->
    case writeArray# ipv1 9 (I# 9) s'#5 of s'#6 { __DEFAULT ->
    case writeArray# ipv1 8 (I# 8) s'#6 of s'#7 { __DEFAULT ->
    case writeArray# ipv1 7 (I# 7) s'#7 of s'#8 { __DEFAULT ->
    case writeArray# ipv1 6 (I# 6) s'#8 of s'#9 { __DEFAULT ->
    case writeArray# ipv1 5 (I# 5) s'#9 of s'#10 { __DEFAULT ->
    case writeArray# ipv1 4 (I# 4) s'#10 of s'#11 { __DEFAULT ->
    case writeArray# ipv1 3 (I# 3) s'#11 of s'#12 { __DEFAULT ->
    case writeArray# ipv1 2 (I# 2) s'#12 of s'#13 { __DEFAULT ->
    case writeArray# ipv1 1 (I# 1) s'#13 of s'#14 { __DEFAULT ->
    (# s'#14, () #) `cast` ...
    } } } } } } } } } } } } } } } }

Great! The loop is unrolled rather nicely.

Enter indices

indices provides a multi-dimensional (Int based) index type, with use for array-heavy code in mind. I hope Soon® to release on Hackage the package static (patches welcome and appreciated) which provides a ForiegnPtr based array type that leverages indices’s, ahem, indices, just about everywhere.

Indices in indices are these two types:

data (a :: Nat) :. b = !Int :. b
data Z = Z

This is similar to the design seen in repa, except the “bound” of an index is its type. That’s crippling for code with arbitrarily-bounded arrays, but very nice otherwise. For instance, an index into a 4x4 matrix is 0:.0:.Z :: 4:.4:.Z .

For now, you can use indices for array-based code by leveraging its Ix instance. This is a little confusing due to the design of Ix, but it’s fairly simple: the type is always the upper bound, and zero is always the lower bound, not a value you give. That means that arrays constructed by array (_, _ :: t) are bounded by [0,t). As an aside, I’m leaning towards reworking static to simply use the array types found in arrays to simplify it greatly into the small linear-algebra package it yearns to be. Input is appreciated here. (At the moment I’m irked at having to store two superfluous lower/upper bounds – linked lists of Int – in the array constructors).

Here’s a demonstration, implementing vector dot product with a static, unrolled loop:

{-# LANGUAGE DataKinds     #-}
{-# LANGUAGE PolyKinds     #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE FlexibleContexts #-}
module MM where
import Data.Array.Unboxed
import Data.Index

type Vector m = UArray (m:.Z)

sizeV :: Vector m a -> Proxy (m:.Z)
sizeV _ = Proxy

vector :: (IArray UArray a, Dim (m:.Z)) => Proxy (m:.Z) -> [a] -> Vector m a
vector b = listArray (zero, maxBound `asProxyTypeOf` b)

dot a b =
  sfoldlRange
    (sizeV a `asTypeOf` sizeV b)
    (\sum ix -> sum + a!ix * b!ix)
    0

v4 x y z w = vector (Proxy :: Proxy (4:.Z)) [x,y,z,w]

main :: IO ()
main = do
  print (dot (v4 1 2 3 4) (v4 2 3 4 5) :: Double)
  print (sum (zipWith (*) [1,2,3,4] [2,3,4,5]) :: Double)

Note the slyness in me not writing the type signature to dot… Well, the important thing is that GHC happily infers the type without needing any hints. Right… guys?

You can contribute to indices on GitHub, find its documentation on Hackage, and install it with cabal.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

All images © Michael Ledger, all rights reserved.

Warning: this code is horribly broken outside of use in Zsh.

As such, all occurences of “sh” are referring to POSIX shells ie bash and /bin/sh, (which is usually symlinked to bash, minus some bash-only features).

I find myself trying to do things I can do in GHCi more each passing day in my innocent /bin/sh. I find myself seeking a good balance between the numerous layers of hacks that compose shell script and the purely functional wonder of a Haskell program. Such efforts already exist, most recently in Shelly.hs, but I believe this to be tackling the problem from the wrong end. Instead of hacking together a library to make Haskell code reminiscent of the beloved /bin/sh, the problem should be tackled with extreme prejudice by hacking together a sh script to mimic the best of Haskell, and retain the beauty of shell scripts.

The beauty of shell scripts?

• Piping
• (almost) Painless concurrency
• Small overhead (this problem can be alleviated by my efforts, however)
• Easy syntax
• Simplest type system in the world: everything is a string!
  • If it’s not a string, it’s a dirty lie.
  • $(( A dirty lie. ))

The horror of shell scripts

• The type system makes Visual Basic 6 cry.
• The type system makes a Haskell programmer spend days in solitude contemplating the purpose of human life.
• sh arrays suck
  • They are strings
  • Sometimes
• sh syntax quickly piles up; leaning pipe syndrome
• sh is slow (thanks zsh and dash!)

Complexity and fun: sh vs Haskell

When we are compare such vastly different languages as these, you have to remember that they both have extremely different purposes and real world uses. Haskell is an extremely feature rich general purpose programming language. sh is very convenient for simple and mundane tools that are best made by composing other programs together via piping. For those of us who have no idea of what they’re doing 90% of the time when they are using bash (that would be me), there is:

Enter fun.sh

fun.sh adds some tools seen and used mainly by functional programmers including take, drop, scanl, foldl, map and lambdas. Because sh has no real type system, the convention is to use pipes as you would use lists in Haskell, which allows fun.sh functions to be easily composable via piping.

Anonymous functions in fun.sh are deceptively dumb. fun.sh will simply consume and remember every argument given up until a “.”, “->”, “→” or “:” is found, then reading each argument before evaluating the body of the function.

A few demos

# sum of a bash array
boring_sum() {
    array="$@"
    s=0
    for i in $array; do
        s=$(($i + $s))
    done
    echo $s
}

# sum of a bash array, piping the array in
boring_sum_pipe() {
    s=0
    while read i; do
        s=$(($s + $i))
    done
    echo $s
}

# now, with fun.sh

$ sum(){ foldl λ a b . 'echo $(($a + $b))' } 
$ list {1..100} | sum
5050

# and to show off, 

$ product(){ foldl λ a b . 'echo $(($a * $b))' }
$ list {1..20} | product
2432902008176640000

$ factorial(){ list {1..$1} | product }
$ factorial 5
120

$ foobar(){ product | λ l . 'list {1..$l}' | sum | md5sum }
$ list {1,2,3} | foobar
fe9d26c3e620eeb69bd166c8be89fb8f  -

$ id(){ λ x . '$x' }
$ id <<< 'echo :)'
:)

# Oh no, the boss wants me to calculate the sum of every integer between 400 and 500! Whatever shall I do?!
$ list {400..500} | foldl λ x y . 'echo $(($x + $y))'
45450
# Thanks fun.sh, I don't know where I would be without you.

GitHub project.

GitHub project

Level 0 is a Snake II (think old-ish Nokia phones) clone written in Haskell, using SDL. cloc tells me it’s 301 LOC, in addition to 49 comments which I’m happy with.

Features

• it works
• it’s fast
• readable code (it’s readable to me!)
• map loading
• map editing
• map saving
• scoreboard

Prerequisites

• GHC (tested with 7.0.3 and 7.4.1)
• SDL from Hackage
• SDL-ttf from Hackage
• a font (by default tries to get /usr/share/fonts/TTF/TerminusBold.ttf)

Installation / usage

$ make

$ bin/level_0 [ms between frames [path to map file]]

eg

$ bin/level_0 16 map

I don’t know if it’s buildable on Windows.

A map is a plain text file, the first 32 characters on the first 32 lines are read, and when there is an ‘x’, you will have a wall that kills your snake when hit.