tmck-code.github.io

KSF Surf Maps: Data Visualisation

2026-04-15T00:00:00+00:00

20260415 KSF Surf Maps: Data Visualisation

Python Repo Practices

2026-02-04T00:00:00+00:00

20260204 Python Repo Practices

Setting up a new Python repo
Examples
- Basic structure

Setting up a new Python repo

Create the repository via the Github web UI
- This allows you to add a README.md, and a .gitignore for Python (super handy!)
- Add a license if desired (BSD-3-Clause is a good default)
- Clone the repo to your local machine
Install uv on your local machine
- Follow the instructions at https://docs.astral.sh/uv/#installation
Create a pyproject.toml file
- Define some basic dependencies in it
Run uv sync to create the uv.lock file
Create your dirs

ops/ - for Dockerfiles, CI/CD configs, etc.
test/ - for your unit tests
<your_package_name>/ - for your main package code
- make sure it’s underscored, not hyphenated
- and matches the name in pyproject.toml

Add a Dockerfile in ops/ if desired
- Copy one of the templates from this repo: https://github.com/astral-sh/uv-docker-example
Add a Makefile for common tasks
- e.g., make build, make test, make lint, etc.
Make sure there is an __init__.py file in
- the test/ directory
  - This ensures that the tests can be discovered and run properly
- the main package directory
  - This ensures that the package is recognized as a module
Commit and push your changes to GitHub

Examples

Basic structure

An example from one of my projects:

.
├── LICENSE
├── Makefile
├── README.md
├── abn_lookup_service
│   └── lookup.py
├── ops
│   ├── Dockerfile
├── pyproject.toml
├── test
│   ├── __init__.py
│   └── test_lookup.py
└── uv.lock

ANSI to UTF8 Part 1: Intro

2026-01-23T00:00:00+00:00

20260123 ANSI to UTF8 Part 1: Intro

Introduction

I love using the terminal in Linux. Over the years I’ve customised it so that it’s a joy for me to use, and I always relish in discovering something new that improves my experience.

Over the last few months I have fallen in love with ANSI art - pixel art from the 90s that first appeared in BBS (Bulletin Board Systems).

This art looks amazing when displayed in the Linux terminal, however there are many challenged, caveats & edge cases when doing this.

This is a part series on the topic of converting this art to UTF-8.

1. SAUCE metadata

2. Colour codes

3. ISO-8859-1, CP437 & UTF-8 characters

5. Non-Printing, c0 & Cursor codes

4. Fonts

5. Terminal colours

F-String number formatting

2025-06-23T00:00:00+00:00

20250623 F-String number formatting

The official docs (somewhat hard to find!): https://docs.python.org/3/library/string.html#format-string-syntax

When you need to log a number as a string, either when

writing data to file, or
formatting a log

doing it via f-strings is the most flexible way, and often the fastest!

20250623 F-String number formatting

For all examples that use benchmarking, I used ipython so that I could use the %timeit magic command.

Rounding floats

Usually you might consider using the round() function, but it might surprise you to know that using an f-string is faster!
First, let’s take a float with many decimal places, and round them it to two decimal places:

n = 4.255857

str(round(n, 2))
# '4.26'

f"{n:.2f}"
# '4.26'

Notice that the f-string has rounded the .255 to .26, rather than just truncating it to .25 as you might expact.

Now let’s benchmark the two methods:

%timeit str(round(n, 2))
# 305 ns ± 1.64 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

%timeit f'{n:.2f}'
# 115 ns ± 0.164 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

The f-string is more than twice as fast as the round() function!

0-padding numbers

Decimal padding

Whenever you need to vertically align logs or print output for easier readability, you can use padding zeroes instead of spaces to align your numbers.

With f-strings, you can add padding zeroes to decimal numbers, as well as the integer part of the number. For a list of given floats, we can try to:

ensure that each float has 2 decimal places
- add padding zeros if shorter than 2
- round to 2 decimal places if longer than 2

n = [ 4.0, 44.2, 55.25525]

for i in n:
    print(f'{i:>10} > {i:.02f}')

#       4.0 > 4.00
#      44.2 > 44.20
#  55.25525 > 55.26

Integer padding

You can also use the 0 padding to align the integer part of the number. This is a little unintuitive, as you might expect that you would define the number of digits to the left of the decimal point, e.g. to print a consistent-width percentage like this:

00%
55%
09%
64%
18%
73%
27%
82%
37%
91%
46%
00%

Let’s give it a try:

 n = list(range(0, 50, 9))
# [0, 9, 18, 27, 36, 45]

for i in n:
    print(f'{i:02.02f}%')

# 0.00%
# 9.00%
# 18.00%
# 27.00%
# 36.00%
# 45.00%

That didn’t work! This is because the padding needs to be the total width of the entire number, including the decimal place. In this case, this number is 5

2 characters to the left of the decimal point
1 character for the decimal point
and 2 characters for the decimal places

n = list(range(0, 50, 9))
# [0, 9, 18, 27, 36, 45]

for i in n:
    print(f'{i:05.02f}%')

# 00.00%
# 09.00%
# 18.00%
# 27.00%
# 36.00%
# 45.00%

Decimal formatting

Separators

You can use a separator like a comma to separate thousands etc, which is a nice way to split up large numbers for readability.

n = list(range(10**8, 10**11, 10**10*2))
#  [100000000, 20100000000, 40100000000, 60100000000, 80100000000]

for i in n:
    print(f'{i:,d}')

# 100,000,000
# 20,100,000,000
# 40,100,000,000
# 60,100,000,000
# 80,100,000,000

You can right-align the numbers when using a separator, provided that you know the desired width

for i in n:
    print(f'{i:14,d}')

    10,000,000
10,010,000,000
20,010,000,000
30,010,000,000
40,010,000,000
50,010,000,000
60,010,000,000
70,010,000,000
80,010,000,000
90,010,000,000

Get unix time FAST with pure bash

2025-05-19T00:00:00+00:00

20250519 Get unix time FAST with pure bash

printf '%(%s)T\n' -1

Beware! There are a few different versions of printf that could be installed on your machine:

☯ type -a printf
printf is a shell builtin
printf is /usr/bin/printf
printf is /bin/printf

The speed of this time command comes from the fact that it’s a shell builtin and doesn’t require forking a new process.

I’m too dumb to figure out how to benchmark this inline using hyperfine, so I just created 2 files with the commands and made them executable with chmod +x:

 ☯ cat date.sh
#!/bin/bash
date +%s

 ☯ cat printf.sh
#!/bin/bash
printf '%(%s)T\n' -1

Results

On the same machine, it takes 0.9 seconds to run a file that is completely empty (apart from the shebang), which leaves the printf command as being almost instant 🚀

Dictionary slice in Python

2025-05-06T00:00:00+00:00

20250506 Dictionary slice in Python

20250506 Dictionary slice in Python

One of my favourite methods in ruby is Hash.slice. It returns a new hash containing only the specified keys from the original hash - very handy!

It works like this:

h = {a: 1, b: 2, c: 3, d: 4}

h.slice(:a, :b)
# => {a: 1, b: 2}

TL;DR, this is the basis of my python equivalent:

import operator as op

def slice_dict(d: dict, keys: list) -> dict:
    return dict(zip(keys, op.itemgetter(*keys)(d)))

# however, you should just use dict comprehension:

def slice_dict(d: dict, keys: list) -> dict:
    return {k: d[k] for k in keys if k in d}

Recreating `slice` in Python

We can definitely use a dictionary comprehension to achieve the same result:

def slice_dict(d: dict, keys: list) -> dict:
    return {k: d[k] for k in keys if k in d}

d = {'a': 1, 'b': 2, 'c': 3, 'd': 4}

slice_dict(d, ('a', 'b'))
# {'a': 1, 'b': 2}

A better way?

We can use the dict constructor to create a new dictionary from a list of tuples, which is a more efficient way to slice a dictionary:

def slice_dict(d: dict, keys: list) -> dict:
    return dict((k, d[k]) for k in keys if k in d)

d = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
slice_dict(d, ('a', 'b'))
# {'a': 1, 'b': 2}

Using `operator.itemgetter`

We can use the [operator.itemgetter](https://docs.python.org/3/library/operator.html#operator.itemgetter) function to create a callable that retrieves the specified keys from the dictionary.

First, a demonstration of how itemgetter works:

import operator as op
d = {'a': 1, 'b': 2, 'c': 3, 'd': 4}

getter = op.itemgetter('a', 'b')

# now we can use the getter to retrieve the values from a dictionary
getter(d)
# (1, 2)

We can then create a new dictionary by zipping the returned values with the keys:

keys = getter(d)
# (1, 2)

dict(zip(('a', 'b'), keys))
# {'a': 1, 'b': 2}

Combining these two steps, we can create a function that slices a dictionary:

import operator as op

def slice_dict(d: dict, keys: list) -> dict:
    return dict(zip(keys, op.itemgetter(*keys)(d)))

d = {'a': 1, 'b': 2, 'c': 3, 'd': 4}

slice_dict(d, ('a', 'b'))
# {'a': 1, 'b': 2}

Gotchas

Non-existent keys

If a key does not exist, then this will raise a KeyError! e.g.

d = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
slice_dict(d, ('a', 'b', 'e'))
# KeyError: 'e'

To guard against this, we can use set operations to filter out any keys that don’t exist in the dictionary:

def slice_dict(d: dict, keys: list) -> dict:
    keys = d.keys() & keys
    return dict(zip(keys, op.itemgetter(*keys)(d)))

slice_dict(d, ('b', 'e', 'a'))
# {'a': 1, 'b': 2}

Order of keys

You might notice that the previous example returns the keys in the order they appear in the original dictionary, rather than the order defined in the keys arg.

To preserve the order of the keys arg, we can use a list comprehension to filter the keys instead:

def slice_dict(d: dict, keys: list) -> dict:
    keys = [k for k in keys if k in d]
    return dict(zip(keys, op.itemgetter(*keys)(d)))

Benchmarking

Here are some quick benchmarks to compare the three methods:

I ran this in ipython so that I can use the %timeit magic function

import operator as op

def slice_dict_comp(d: dict, keys: list) -> dict:
    return {k: d[k] for k in keys if k in d}

def slice_dict_gen(d: dict, keys: list) -> dict:
    return dict((k, d[k]) for k in keys if k in d)

def slice_dict_op(d: dict, keys: list) -> dict:
    return dict(zip(keys, op.itemgetter(*keys)(d)))

def slice_dict_op_safe(d: dict, keys: list) -> dict:
    keys = [k for k in keys if k in d]
    return dict(zip(keys, op.itemgetter(*keys)(d)))

for fn in [slice_dict_comp, slice_dict_gen, slice_dict_op, slice_dict_op_safe]:
    print(fn.__name__, fn(d, keys))
    %timeit fn(d, keys)

Results

function	time
slice_dict_comp	147 ns
slice_dict_gen	410 ns
slice_dict_op	350 ns
slice_dict_op_safe	600 ns

OSX dark mode shortcut

2025-01-02T00:00:00+00:00

20250102 OSX dark mode shortcut

Update 20250924: As of the release of macOS Tahoe, you can also use the new “Shortcuts” app to achieve this! These shortcuts can’t be assigned to keyboard shortcuts yet, but you can run them via Spotlight.

20250102 OSX dark mode shortcut
- Automator
- Shortcuts

Automator

Using the power of OSX Automator!

Automator offers many handy actions in OSX. After creating an “automation” in Automator, you can save it as a “Quick Action” and assign a keyboard shortcut to it in System Preferences.

Documentation: https://support.apple.com/en-au/guide/automator/welcome/mac

1. Launch Automator

2. Create “Quick Action”

3. Search for “appearance”

4. Drag “Change System Appearance” to the workflow

If you like, you can click “Run” in the top-right corner to test the action.

5. Save the Quick Action

6. Assign a Keyboard Shortcut

Open System Preferences > Keyboard > Shortcuts > Services > General and assign a keyboard shortcut to the Quick Action.

Done! Enjoy your new shortcut 🤓🙈

Shortcuts

1. Launch Shortcuts

2. Add a new shortcut

3. Create the shortcut logic

4. Name the shortcut

After creating the logic, close the window to save your shortcut. You’ll then be able to rename it.

4. Use via Spotlight

Despite the name, it doesn’t seem like it’s currently possible to assign a keyboard shortcut to a shortcut created in the Shortcuts app.

Instead, you can run it via Spotlight (Cmd + Space) by typing the name of the shortcut. This can be made easier by “adding quick keys” to the Spotlight search, in this example I’ve chosen the letter "t"

Christmas Tree Lights with Arduino

2024-12-10T00:00:00+00:00

20241210 Christmas Tree Lights with Arduino

20241210 Christmas Tree Lights with Arduino
- Ingredients

Ingredients

An Arduino board

I used an Arduino Uno Rev3

A string of “WS8211” LEDs

Some patience

Make sure you use the correct wires and ports! I struggled to find a wiring diagram on the internet, please accept this rough photo of my setup 😬:

Some Code!

via: https://github.com/tmck-code/ardy-pardy/blob/main/projects/xmas/xmas.ino

This uses the library Adafruit_NeoPixel as a dependency, which you can install via the Arduino IDE’s Library Manager.

// A basic everyday NeoPixel strip test program.

#include <Adafruit_NeoPixel.h>
#ifdef __AVR__
#endif

#define LED_PIN   13
#define LED_COUNT 50
#define SLEEP     250
int PIXEL_OFFSET = 0;

#define WHITE  0
#define YELLOW 1
#define GREEN  2
#define RED    3

Adafruit_NeoPixel strip(LED_COUNT, LED_PIN, NEO_GRB + NEO_KHZ800);

// Green, Yellow
uint32_t COLOURS[] = {
  strip.Color(255, 255, 255),
  strip.Color(255, 255, 0),
  strip.Color(0,   255, 0),
  strip.Color(255,   0, 0)
};

int N_COLOURS = sizeof(COLOURS)/sizeof(COLOURS[0]);
uint32_t XMAS_DUO[2] = {RED, GREEN};
uint32_t XMAS_TRIO[3] = {RED, GREEN, YELLOW};

void setup() {
  Serial.begin(9600);

  strip.begin();
  strip.setBrightness(50);
  strip.show();

  Serial.println("Starting XMAS display with " + String(N_COLOURS) + " colours:");
}

void loop() {
  for (int i=0; i<N_COLOURS-1; i++) {
    colorWipe(COLOURS[i], SLEEP);
    if (i % 2 == 0) {
      xmas(XMAS_DUO, sizeof(XMAS_DUO)/sizeof(XMAS_DUO[0]), YELLOW);
    } else {
      xmas(XMAS_TRIO, sizeof(XMAS_TRIO)/sizeof(XMAS_TRIO[0]), WHITE);
    }
  }
}

// Fill strip pixels one after another with a color, and delay in ms.
// Strip is NOT cleared first; anything there will be covered pixel by pixel
void colorWipe(uint32_t color, int wait) {
  for(int i=0; i<strip.numPixels(); i++) {
    strip.setPixelColor(i, color);
    strip.show();
    delay(wait);
  }
}

void xmas(uint32_t repeat_colours[], int n_colours, int highlight_colour) {
  for (int i=1; i<strip.numPixels(); i++) {
    strip.setPixelColor(i, COLOURS[highlight_colour]);
    strip.setPixelColor(i-1, COLOURS[repeat_colours[i % n_colours]]);
    strip.show();
    delay(SLEEP);
  }
}

Comparing Lists or Matrices

2024-11-28T00:00:00+00:00

20241128 Comparing Lists or Matrices

Recently I needed to compare matrices (or “lists of lists”), counting the number of different items. I challenged myself to use the standard library instead of numpy.

These matrices have 2 differing values:

m1 = [(0,0),
      (0,1),]

m2 = [(0,1),
      (0,0),]

Firstly, it would be easiest (for me) if I somehow transformed each matrix into a single sneuence of numbers. This can be done using chain from the itertools standard library tools, which is one of my favourite and probably overused hacks.

from itertools import chain

list(chain.from_iterable(m1))
# [0, 0, 0, 1]

list(chain.from_iterable(m2))
# [0, 1, 0, 0]

Using zip, we can iterate through both lists and compare each pair of elements:

list(zip(
    chain.from_iterable(m1),
    chain.from_iterable(m2)
))
# [(0, 0), (0, 1), (0, 0), (1, 0)]

Next, we need to compare each pair of elements, and I thought of using the operator functions, as there is ne which could be used to compare elements.

The documentation for ne is:

ne(a, b, /) Same as a != b.

I want to avoid using a lambda as it is both icky and slow, e.g.

list(map(
    lambda x: x[0]==x[1],
    zip(
        chain.from_iterable(m1),
        chain.from_iterable(m2)
    )
))
# [True, False, True, False]

I can use starmap to take each pair, and pass it to operator.eq as *args (hence the name “starmap”! very cool 😎)

from itertools import starmap

list(starmap(
    operator.ne,
    zip(
        chain.from_iterable(m1),
        chain.from_iterable(m2)
    )
))
# [True, False, True, False]

And finally, sum can be used to count the number of True/truthy results

sum(starmap(
    operator.ne,
    zip(
        chain.from_iterable(m1),
        chain.from_iterable(m2)
    )
))
# 2

Note: counting identical elements could be done easily by using operator.eq instead of operator.ne

Using JQ with style

2024-08-29T00:00:00+00:00

20240829 Using JQ with style

Note: this article is intended for people that already love JQ!

Did you know that you can configure the colours of jq’s output? This is a great way to make your output more readable in a way that you find visually appealing.

This is detailed in the official docs here

There are only 2 things to know!

1. The ENV var

The JQ_COLORS ENV var is used to configure the colours. The default jq style is expressed as:

JQ_COLORS="0;90:0;37:0;37:0;37:0;32:1;37:1;37:1;34".

Using an ENV var means that configuring this in your shell rc files is easy, and doesn’t slow anything down.

The ENV var contains a colon-separated list of ANSI colour codes, in the following order:

color for null
color for false
color for true
color for numbers
color for strings
color for arrays
color for objects
color for object keys

2. The colours

Part 1 - Style

Part 2 - Colour

value	style
1	bright
2	dim
4	underscore
5	blink
7	reverse
8	hidden

value	color
30	black
31	red
32	green
33	yellow
34	blue
35	magenta
36	cyan
37	white

Examples

Now that we have the required pieces, we can create some different themes!

In order to make this process easier, I like to organise all of the options into named variables so that it’s easier to experiment.

(For my shell, this lives in my .bash_aliases)

_JQ_REGULAR=0
_JQ_BRIGHT=1
_JQ_DIM=2
_JQ_UNDERSCORE=4
_JQ_BLINK=5
_JQ_REVERSE=7
_JQ_HIDDEN=8

_JQ_BLACK=30
_JQ_RED=31
_JQ_GREEN=32
_JQ_YELLOW=33
_JQ_BLUE=34
_JQ_MAGENTA=35
_JQ_CYAN=36
_JQ_WHITE=37

JQ_NULL="$_JQ_DIM;$_JQ_YELLOW"
JQ_TRUE="$_JQ_DIM;$_JQ_YELLOW"
JQ_FALSE="$_JQ_DIM;$_JQ_YELLOW"
JQ_NUMBERS="$_JQ_REGULAR;$_JQ_CYAN"
JQ_STRINGS="$_JQ_REGULAR;$_JQ_WHITE"
JQ_ARRAYS="$_JQ_REGULAR;$_JQ_MAGENTA"
JQ_OBJECTS="$_JQ_BRIGHT;$_JQ_RED"
JQ_OBJECT_KEYS="$_JQ_BRIGHT;$_JQ_YELLOW"

export JQ_COLORS="${JQ_NULL}:${JQ_FALSE}:${JQ_TRUE}:${JQ_NUMBERS}:${JQ_STRINGS}:${JQ_ARRAYS}:${JQ_OBJECTS}:${JQ_OBJECT_KEYS}"

	default
	This theme is generally balanced, with a focus on everything that is not a string value JQ_NULL="$_JQ_UNDERSCORE;$_JQ_WHITE" JQ_TRUE="$_JQ_BRIGHT;$_JQ_GREEN" JQ_FALSE="$_JQ_BRIGHT;$_JQ_RED" JQ_NUMBERS="$_JQ_REGULAR;$_JQ_CYAN" JQ_STRINGS="$_JQ_REGULAR;$_JQ_WHITE" JQ_ARRAYS="$_JQ_REGULAR;$_JQ_BLUE" JQ_OBJECTS="$_JQ_BRIGHT;$_JQ_RED" JQ_OBJECT_KEYS="$_JQ_BRIGHT;$_JQ_YELLOW"
	Another balanced theme, this time highlighting string values a little more, and keys a little less JQ_NULL="$_JQ_UNDERSCORE;$_JQ_WHITE" JQ_TRUE="$_JQ_BRIGHT;$_JQ_YELLOW" JQ_FALSE="$_JQ_BRIGHT;$_JQ_RED" JQ_NUMBERS="$_JQ_REGULAR;$_JQ_CYAN" JQ_STRINGS="$_JQ_REGULAR;$_JQ_YELLOW" JQ_ARRAYS="$_JQ_REGULAR;$_JQ_BLUE" JQ_OBJECTS="$_JQ_BRIGHT;$_JQ_MAGENTA" JQ_OBJECT_KEYS="$_JQ_REGULAR;$_JQ_GREEN"
	This theme has a strong focus on clearly displaying "string keys", and the open/close braces of objects `JQ_NULL="$_JQ_BRIGHT;$_JQ_MAGENTA" JQ_TRUE="$_JQ_REGULAR;$_JQ_GREEN" JQ_FALSE="$_JQ_REGULAR;$_JQ_RED" JQ_NUMBERS="$_JQ_REGULAR;$_JQ_CYAN" JQ_STRINGS="$_JQ_REGULAR;$_JQ_WHITE" JQ_ARRAYS="$_JQ_REGULAR;$_JQ_BLUE" JQ_OBJECTS="$_JQ_BRIGHT;$_JQ_WHITE" JQ_OBJECT_KEYS="$_JQ_BRIGHT;$_JQ_YELLOW"`
	This theme is designed to be helpful when when searching for keys and null values JQ_NULL="$_JQ_REVERSE;$_JQ_RED" JQ_TRUE="$_JQ_DIM;$_JQ_GREEN" JQ_FALSE="$_JQ_DIM;$_JQ_RED" JQ_NUMBERS="$_JQ_UNDERSCORE;$_JQ_CYAN" JQ_STRINGS="$_JQ_DIM;$_JQ_WHITE" JQ_ARRAYS="$_JQ_REGULAR;$_JQ_BLUE" JQ_OBJECTS="$_JQ_BRIGHT;$_JQ_WHITE" JQ_OBJECT_KEYS="$_JQ_REVERSE;$_JQ_GREEN"

Filtering Nones from a Python list

2024-08-28T00:00:00+00:00

20240828 Filtering Nones from a Python list

Recently I found myself wondering the best way to strip out None items from a python list. As you may be aware, list comprehension in python is almost always the fastest way to do something, but doesn’t look so stylish!

These are various approaches that I came up with that use a “functional” style using the builtin filter function

20240828 Filtering Nones from a Python list

0. Checking if things are None

I thought of a few ways to do this:

is not - the “most pythonic”
```
 x is not None
```
!= - comparison
```
 x != None
```
isinstance - comparing types
```
 x not isinstance(None)
```

We can benchmark these different approaches in ipython to see which comparison is best to use

x = 1

%timeit x is not None
# -> 12.4 ns ± 0.0851 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)

%timeit x != None
# -> 19.1 ns ± 1.24 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

%timeit not isinstance(x, type(None))
# -> 49.3 ns ± 0.0683 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

%timeit not isinstance(x, None.__class__)
# -> 48.2 ns ± 0.496 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

There is a clear winner: x is not None.

1. The list comprehension

def list_comprehension(l):
    return [x for x in l if x is not None]

2. lambda

Python has 2 filter functions, filter and filterfalse, which work identically, but for True/False values respectively.

from itertools import filterfalse

lambda_is     = lambda x: x is None
lambda_is_not = lambda x: x is not None

def filter_lambda_is(l):
    return list(filterfalse(lambda x: x is None, l))

def filter_lambda_is_not(l):
    return list(filter(lambda x: x is not None, l))

def filterfalse_predef_lambda(l):
    return list(filterfalse(lambda_is, l))

def filter_predef_lambda(l):
    return list(filter(lambda_is_not, l))

3. Partial

In the same vein as lambda, we can create a partial function using operator.is_

import operator

operator.is_(1, None)
# -> False

from functools import partial
from itertools import filterfalse

def filter_partial(l):
    return list(filter(partial(operator.is_not, None), l))

def filterfalse_partial(l):
    return list(filterfalse(partial(operator.is_, None), l))

p_is     = partial(operator.is_, None)
p_is_not = partial(operator.is_not, None)

def filter_predef_partial(l):
    return list(filter(p_is_not, l))

def filterfalse_predef_partial(l):
    return list(filterfalse(p_is, l))

4. Inner function

from functools import partial
from itertools import filterfalse

def filter_inner(l):
    p = partial(operator.is_not, None)
    def inner(x):
        return list(filter(p, x))
    return inner(l)

def filterfalse_inner(l):
    p = partial(operator.is_, None)
    def inner(x):
        return list(filterfalse(p, x))
    return inner(l)

Benchmarking!

We can use the following tests:

For testing, I used the following list, and ran each function 100,000 times

x = [0,1,2,3,None]*100

function	total time	median time	x slower than fastest
`filter_predef_partial`	664.819 𝑚s	6.676 𝜇s	x1.00
`filter_inner`	718.506 𝑚s	7.153 𝜇s	↓ x1.08
`filterfalse_predef_partial`	741.668 𝑚s	7.391 𝜇s	↓ x1.12
`filterfalse_partial`	752.076 𝑚s	7.391 𝜇s	↓ x1.13
`filterfalse_inner`	764.408 𝑚s	7.629 𝜇s	↓ x1.15
`list_comprehension`	809.320 𝑚s	8.106 𝜇s	↓ x1.22
`filterfalse_partial_cached`	1.301 s	12.875 𝜇s	↓ x1.96
`filter_lambda_is_not`	1.386 s	13.828 𝜇s	↓ x2.08
`filter_lambda_is`	1.386 s	13.828 𝜇s	↓ x2.09
`filter_predef_lambda`	1.413 s	14.067 𝜇s	↓ x2.13
`filterfalse_predef_lambda`	2.453 s	24.557 𝜇s	↓ x3.69

Conclusions

list comprehension is not always the fastest!
filter is faster than filterfalse
partial/inner functions, when predefined, are the fastest
- predefining is faster than defining inline
- and both are faster than lambdas

Join and Coalesce in Pyspark

2024-04-11T00:00:00+00:00

20240411 Join and Coalesce in Pyspark

20240411 Join and Coalesce in Pyspark

Scenario

I had the need to join two dataframes in pyspark and coalesce in order to capture all possible values for each ID key.

I noticed an answer on stack overflow https://stackoverflow.com/a/68534723/4431563, and was inspired to see if I could improve it slightly.

both of the dataframes have the same key column
records may exist in either dataframe, or both
the dataframes have some shared columns, and some unique columns
the shared columns may have null values in either dataframe

Example Dataframes

df1:

id	var1.2	var2	var4
1	null	aa	yy
2	a	null	yy
3	b	11	yy
4	h	22	yy

df2:

id	var1.2	var2	var3
1	f	Ba	xx
2	a	bb	xx
3	b	null	xx

The Solution

First, let’s create some dataframes to work with. See my pyspark-fu article for associated tips and tricks for local pyspark development.

df1 has an extra column var4
df2 has an extra column var3
both dataframes
- have the same key column id
- and shared columns var1.2 and var2

df1 =  spark.createDataFrame([
        (1,  None, 'aa', 'yy'),
        (2 , 'a',  None, 'yy'),
        (3 , 'b',  None, 'yy'),
        (4 , 'h',  None, 'yy'),
    ],
    'id int, `var1.2` string, var2 string, var4 string',
)

df2 =  spark.createDataFrame([
        (1,  'f', 'Ba',  'xx'),
        (2 , 'a', 'bb',  'xx'),
        (3 , 'b',  None, 'xx'),
    ],
    'id int, `var1.2` string, var2 string, var3 string',
)

The solution itself is fairly straightforward

do a full outer join, so that all records are included
just select the columns that are unique to each dataframe
for the shared columns, use F.coalesce to select the non-null value

Note: when coalescing, the order of the dataframes matters. Values from df1 will be used if they has a non-null value, which will ignore any valid values in df2
Ensure that your “preferred” dataframe is df1!

def join_coalesce(how, key_columns, df1, df2):
    shared_columns = (set(df1.columns) & set(df2.columns)) - set(key_columns)
    unique_columns = (set(df1.columns) ^ set(df2.columns)) - set(key_columns)
    print(f'{key_columns=}', f'{shared_columns=}', f'{unique_columns=}', sep='\n')

    return (
        df1
        .join(df2, on=key_columns, how=how)
        .select(
            *[F.col(f'`{c}`') for c in sorted(key_columns)],
            *[F.coalesce(df1[f'`{i}`'], df2[f'`{i}`']).alias(i) for i in sorted(shared_columns)],
            *[F.col(f'`{c}`') for c in sorted(unique_columns)],
        )
    )

result = join_coalesce('outer', ['id'], df1, df2)
result.show()

The Results

Behold!

key_columns=['id']
shared_columns={'var2', 'var1.2'}
unique_columns={'var3', 'var4'}

id	var1.2	var2	var3	var4
1	f	aa	xx	yy
2	a	bb	xx	yy
3	b	null	xx	yy
4	h	null	null	yy

Allow flatpak to use secondary hard drives

2024-04-03T00:00:00+00:00

20240403 Allow flatpak to use secondary hard drives

I run Steam on linux using the Flatpak version, which I find has less driver/configuration issues than the DEB/RPM equivalent.

Every time I install it on a new OS, I need to ensure that it can access my shared SSD with my steam library directory on it.

One note is that you’ll need to ensure that your SSD partition is mounted in the correct place before running your flatpak program.
I’d recommend configuring this to happen at startup with FSTAB (see my handy fstab article)

In this example, the directory is at /mnt/external/. My steam library is at /mnt/external/SteamLibrary

flatpak override --user --filesystem=/mnt/external/ com.valvesoftware.Steam

Now, I can restart steam and add this library location!

My Favourite Surf Maps

2024-03-10T00:00:00+00:00

20240310 My Favourite Surf Maps

20240310 My Favourite Surf Maps

All maps

map	tier	surf style	aesthetic
`surf_calcyate2`	1	smooth	concrete/foliage
`surf_nuclear`	1	easy	concrete bunker
`surf_school`	1	“learn to surf”	sand, sea & ramps
`surf_ivory`	2	smooth	marble architecture/foliage
`surf_calzone`	2	smooth	stained glass/large concrete spaces

Tier 1

`surf_calcyate2`

category	value
surf style	smooth
aesthetic	concrete/foliage

`surf_nuclear`

category	value
surf style	easy
aesthetic	concrete bunker

`surf_school`

category	value
surf style	“learn to surf”
aesthetic	sand/sea

Tier 2

`surf_ivory`

category	value
surf style	smooth
aesthetic	marble architecture/foliage

`surf_calzone`

category	value
surf style	smooth
aesthetic	stained glass/large concrete spaces

TODO

surf_palm (tier 2 linear)
surf_guitar_hi (tier 2 linear)

CS CFG tricks #1

2024-01-29T00:00:00+00:00

20240129 CS CFG tricks #1

Execute multiple commands with a single keybind

This article details (very briefly) how to trigger 1+ commands with a single keybind in Counter Strike (GO/2).

Simply add a semicolon (;) between each of the commands that you’d like the keybind to execute

Example 1

One of the most common uses I have found for this trick is to add a sound to a keybind action.

// set teleport location
bind q "sm_saveloc; playvol buttons\blip1 0.5"

CS Surf - Toggle all saveloc keybinds

2024-01-26T00:00:00+00:00

20240126 CS Surf - Toggle all saveloc keybinds

20240126 CS Surf - Toggle all saveloc keybinds

What is saveloc?

When surfing in CS Source/GO/2, the saveloc keybinds are essential!

This is the name given to the checkpoint system used to

add a checkpoint as you progress through the map
teleport to that checkpoint at any time to practice the map

If you’ve ever played Trackmania (or any other racing game), you’ll be familiar with such a system - if you drive off the track or wipe out, then you can teleport to a previous checkpoint and then try the same section again.

In surf this system is the same, except that you have to manually add the checkpoint yourself while surfing! (TODO: do an article in future to implement something automatic?)

There is a quick YouTube video that you can check outt for an overview of this plugin if you are new to it:

How do I use saveloc?

I use saveloc the way that most people do, and only use a handful of keybinds

save checkpoint
teleport to checkpoint
switch to previous checkpoint
switch to next checkpoint

In addition, I’m also including 2 other teleport commands in this article which are

teleport to beginning of map
teleport to beginning of stage

What do I need?

I find that I need to be very careful whenever I use saveloc, as accidentally hitting the “teleport to checkpoint” keybind means that you enter “practice” mode and your map time can no longer be counted as a leaderboard entry.

This means that accidentally hitting the sm_tele keybind is catastrophic.
Putting this keybind on a hard-to-reach key means that it would be much harder to actually practice as each teleport is harder to press

So, my aim is

Toggle all teleport & saveloc keybinds on and off with a single key
i.e.

saveloc is enabled by default

press ‘.’ to disable all teleport/saveloc keybinds (“leaderboard mode”)

press ‘.’ to re-enable all teleport/saveloc keybinds (“practice mode”)

The CFGs

My solution relies on “one weird trick™”, but first it’s easier to build up the required cfg files for the enable/disable actions.

surf_savelog.cfg

  // This file contains all SurfTimer saveloc keybinds

  // set teleport location
  bind q "sm_saveloc";
  // teleport to location
  bind e "sm_tele";
  bind mwheeldown "sm_tele";

  // change to previous/next teleport location
  bind 1 "sm_teleprev";
  bind p "sm_teleprev";
  bind 2 "sm_telenext";

  say "saveloc enabled";

surf_nosaveloc.cfg

  // This file _disables_ all SurfTimer saveloc keybinds

  // set teleport location
  unbind q;
  // teleport to location
  unbind e;
  unbind mwheeldown;

  // change to previous/next teleport location
  unbind 1;
  unbind p;
  unbind 2;

  say "saveloc disabled";

The Magic

The alias command points one cfg command -> another one.

Using this, we can create 2+ state commands, and then one main command that is “aliased” to one of the state commands by default, e.g.

one command to “bind all saveloc keybinds”
one command to “unbind all the saveloc keybinds”
and a third that “toggles” the binds, that initially defaults to “bind”

The bind/unbind commands need to include a 2nd command that assigns a new alias! This allows the bind/unbind commands to be called, and then set the shared alias to point to the other command.

This can be used for a simple toggle like I’ve done here, but can also be used to increment/cycle through 2+ states (e.g. I’ve used this to iterate through host_timescale on my home server in 5 stages).

After creating the two CFG files above^, I have the following section in my surf_online.cfg config file

// saveloc (bind by default)
exec surf_saveloc.cfg;
alias savelocon "exec surf_saveloc.cfg; alias saveloc savelocoff";
alias savelocoff "exec surf_nosaveloc.cfg; alias saveloc savelocon";
alias saveloc "savelocoff";
// toggle saveloc
bind . "saveloc";

This means that I can press the . button toggle between practice/attempt modes easily! Hope you enjoy :)

CS2 Surf

2023-11-05T00:00:00+00:00

20231105 CS2 Surf

20231105 CS2 Surf

Workshop CFG file

sv_cheats 1;               // allow some restricted commands e.g. r_drawviewmodel 1

//-- time
mp_roundtime 60;           // set round time to 60 minutes (max value)
mp_timelimit 99999999;     // set game time to "unlimited" (huge)
mp_roundtime_deployment 0; // disable "bomb defusal game mode" round time
mp_freezetime 0;           // remove countdown before game starts
mp_round_restart_delay 0;  // remove countdown before
mp_match_restart_delay 0;  // remove countdown before

mp_drop_knife_enable       // allow players to drop their knives.
r_drawviewmodel 0;         // hide the knife, but leave the crosshair

//-- The main options controlling surf speed
sv_airaccelerate 150;      // acceleration while in air (the main option - must be 150)
sv_accelerate 10;          // acceleration when walking
sv_maxvelocity 10000;      // uncap max velocity

//-- bunny hopping
sv_enablebunnyhopping 1;   // enable bunny hopping
sv_autobunnyhopping 1;     // enable automatic bunny hopping while holding space

//-- disable stamina
sv_staminamax 0;           // the max stamina penalty
sv_staminajumpcost 0;      // disable stamina cost from jumping
sv_staminalandcost 0;      // disable stamina cost from landing
sv_staminarecoveryrate 0;  // disable stamina recovery
sv_accelerate_use_weapon_speed 0; // don't change acceleration based on player weapon
sv_falldamage_scale 0;     // disable fall damage

//-- keybinds
bind mouse1 +turnleft;     // turn left with left mouse click
bind mouse2 +turnright;    // turn right with right mouse click

// This command will restart the game, allowing the CFG settings to be applied
mp_restartgame 0

CFG directory location

You’ll need to write this into a file in a specific “cfg” directory of your CS2 installation.

$STEAM_HOME/steamapps/common/Counter-Strike Global Offensive/game/csgo/cfg/

Differences from previous CSGO CFG files

The vast majority of the console commands seem to be the same! 🎉 This is a log of any differences as I find them.

Mouse turnbinds have changed slightly

bind	csgo command	cs2 command
turn left	bind mouse1 +left	bind mouse1 +turnleft
turn right	bind mouse1 +right	bind mouse1 +turnright

Parsing BOMs in Python

2023-09-19T00:00:00+00:00

20230919 Parsing BOMs in Python

Introduction
Show the BOM
Create a UTF8 file
- Reading the file in Python
UTF-16

Introduction

The “byte-order mark” or BOM is a special char that appears at the very beginning of UTF8 and UTF16 files.
This marker is there to be our friend, however many languages and libraries don’t generally deal with this marker by default, and python is no exception

You’ll usually encounter these files if you work with data that came from Windows programs, otherwise it’s usually rare to see.

Show the BOM

The UTF-8 BOM character is: U+FEFF

In linux, it’s possible to easily create a test file so that you can play around.

I like to use cat -A to check for non-printing characters, you can pipe anything to cat by using the - character, e.g.

☯ ~ echo -e '\xEF\xBB\xBF' | cat -A -
M-oM-;M-?$

☯ ~ printf '\ufeff\n' | cat -A -
M-oM-;M-?$

Create a UTF8 file

To create a UTF8 file, use the BOM character from above and add some extra text, and save it to a file.

☯ ~ printf '\ufeffhello world\n' > test.csv

# check the file using the `file` command
 ☯ ~ file test.csv
test.csv: Unicode text, UTF-8 (with BOM) text, with no line terminators

# check the file using cat -A
 ☯ ~ cat -A test.csv
M-oM-;M-?hello world

Reading the file in Python

When opening files, python will not remove the BOM character.

with open('test.csv') as istream:
    s = istream.read()

s
# '\ufeffhello world'

However, this can be easily fixed by using the utf-8-sig encoding!
The following info is buried within the python codec documentation:

On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file. On decoding utf-8-sig will skip those three bytes if they appear as the first three bytes in the file. In UTF-8, the use of the BOM is discouraged and should generally be avoided.

with open('test.csv', encoding='utf-8-sig') as istream:
    s = istream.read()

s
# 'hello world'

Now, you can see that the BOM character has been removed automatically! The same thing can be done with writing - automatically adding the BOM character by using the utf-8-sig encoding.

with open('test.csv', 'w', encoding='utf-8-sig') as ostream:
    print('hello world', file=ostream)

UTF-16

For UTF-16 files, the BOM character comes in 2 flavors, big-endian and little-endian. Python doesn’t offer a handy encoding for these, so you’ll have to do it manually.

UTF-16 BE: U+FEFF
UTF-16 LE: U+FFFE

To help out - let’s write a file with a BOM16 character and some text.

with open('test16.csv', 'wb') as ostream:
    ostream.write(codecs.BOM_UTF16)
    ostream.write(b'hello world\n')

☯ ~ file test16.csv
test16.csv: Unicode text, UTF-16, little-endian text, with no line terminators

☯ ~ cat -A test16.csv
M-^?M-~hello world$

The codecs package

The standard library has a codecs package that contains a few handy constants for the BOM characters.

import codecs

codecs.BOM_UTF16_LE
# b'\xff\xfe'
codecs.BOM_UTF16_BE
# b'\xfe\xff'

BOM detection

Using these constants, we can make a function that will detect a BOM character at the start of a file, and return the correct encoding.

import codecs

CODECS = {
    "utf-8-sig": [codecs.BOM_UTF8],
    "utf-16": [
        codecs.BOM_UTF16,
        codecs.BOM_UTF16_BE,
        codecs.BOM_UTF16_LE,
    ]
}

def detect_encoding(fpath: str) -> str:
    # open the file in bytes mode
    with open(fpath, 'rb') as istream:
        # read the first 3 bytes (the UTF-8 BOM is 3 chars, the UTF-16 BOM is 2)
        data = istream.read(3)
        # iterate over the codecs and return the encoding if the BOM is found
        for encoding, boms in CODECS.items():
            if any(data.startswith(bom) for bom in boms):
                return encoding
    return 'utf-8'

detect_encoding('test.csv')
# 'utf-8-sig'
detect_encoding('test16.csv')
# 'utf-16'

Demo

Finally, you could use this encoding detection inline when reading a file! For this test, I used a UTF16 file that I found in this repo: https://github.com/stain/encoding-test-files

with open(fpath, 'r', encoding=detect_encoding(fpath)) as istream:
    s = istream.read()

s
# 'première is first\npremière is slightly different\nКириллица is Cyrillic\n𐐀 am Deseret\n'

Jupyter Cell Wrappers

2023-07-04T00:00:00+00:00

20230704 Jupyter Cell Wrappers

import IPython.core.events as events

import json
import time

class VarWatcher(object):
    def __init__(self, ip):
        self.shell = ip
        self.timestamp = time.time()

    def pre_run_cell(self, info):
        'This runs before each cell'

        self.timestamp = time.time()
        print('time before run', self.timestamp)

    def _last_cell_id(self):
        last_key = None
        for cell_id, vs in self.shell.user_ns.items():
            if not cell_id.startswith('_i'):
                continue
            last_key = cell_id
        return last_key

    def post_run_cell(self, info):
        'This runs after each cell'

        duration = time.time() - self.timestamp
        print('time after run', time.time())
        info = {
            'execution_count': info.execution_count,
            'result': info.result,
            'duration': duration,
            'CONFIG': self.shell.user_ns.get('CONFIG', None),
            'cell': self._last_cell_id(),
            'cell_vars': self.shell.user_ns.get(self._last_cell_id()),
        }
        print(json.dumps(info, indent=2, default=str))

def load_ipython_extension(ip):
    vw = VarWatcher(ip)
    ip.events.register('pre_run_cell', vw.pre_run_cell)
    ip.events.register('post_run_cell', vw.post_run_cell)

# register the event callbacks for the first time

load_ipython_extension(get_ipython())

time after run 1689061674.859367
{
  "execution_count": 3,
  "result": null,
  "duration": 0.00021696090698242188,
  "CONFIG": null,
  "cell": "_i3",
  "cell_vars": "# register the event callbacks for the first time\n\nload_ipython_extension(get_ipython())"
}

# a cell with a var

x=1

time before run 1689061675.896075
time after run 1689061675.896435
{
  "execution_count": 4,
  "result": null,
  "duration": 0.00035881996154785156,
  "CONFIG": null,
  "cell": "_i4",
  "cell_vars": "# a cell with a var\n\nx=1"
}

# a cell with a var that's being watched by the callback

CONFIG = {'hello': 'WORLD'}

time before run 1689061676.494325
time after run 1689061676.49473
{
  "execution_count": 5,
  "result": null,
  "duration": 0.00040411949157714844,
  "CONFIG": {
    "hello": "WORLD"
  },
  "cell": "_i5",
  "cell_vars": "# a cell with a var that's being watched by the callback\n\nCONFIG = {'hello': 'WORLD'}"
}

PySpark Fu

2023-06-05T00:00:00+00:00

Pyspark Fu

Greetings fellow traveller! This is a loose collection of some of the useful Pyspark “pro-tips” that I’ve found while working with the framework, hopefully it will help someone else with these common headaches

Pyspark Fu

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession

CONF = {
    'spark.ui.showConsoleProgress':       'false',
    'spark.ui.dagGraph.retainedRootRDDs': '1',
    'spark.ui.retainedJobs':              '1',
    'spark.ui.retainedStages':            '1',
    'spark.ui.retainedTasks':             '1',
    'spark.sql.ui.retainedExecutions':    '1',
    'spark.worker.ui.retainedExecutors':  '1',
    'spark.worker.ui.retainedDrivers':    '1',
    'spark.executor.instances':           '1',
}

def spark_session() -> SparkSession:
    '''
    - set a bunch of spark config variables that help lighten the load
    - local[1] locks the spark runtime to a single core
    - silence noisy warning logs
    '''
    conf = SparkConf().setAll([(k,v) for k,v in CONF.items()])

    sc = SparkSession.builder.master('local[1]').config(conf=conf).getOrCreate()
    sc.sparkContext.setLogLevel('ERROR')
    return sc

spark = spark_session()

2. Create a simple dataframe for debugging

The pyspark official docs don’t often “create” the dataframe that the code examples refer to

df = spark.createDataFrame([
    {'a': 'b', 'n': {'a': 'b'}},
    {'a': 'c', 'n': {'z': 'x', 'y': 'b'}},
    {'a': 'd', 'n': {'o': None, 't': 'a', '2': 3}}
])

df.show(truncate=False)

+---+---------------------------+
|a  |n                          |
+---+---------------------------+
|b  |{a -> b}                   |
|c  |{y -> b, z -> x}           |
|d  |{2 -> 3, t -> a, o -> null}|
+---+---------------------------+

3. Joins

3.1. Avoid duplicate column names

# Let's construct two dataframes that share a column to join on

df1 = spark.createDataFrame([
    {'id': '123', 'name': 'pikachu'},
    {'id': '999', 'name': 'evee'},
    {'id': '007', 'name': 'charizard'},
])
df2 = spark.createDataFrame([
    {'id': '123', 'name': 'ash'},
    {'id': '999', 'name': 'chloe'},
    {'id': '007', 'name': 'ash'},
])

df1.show(), df2.show()

+---+---------+
| id|     name|
+---+---------+
|123|  pikachu|
|999|     evee|
|007|charizard|
+---+---------+

+---+-----+
| id| name|
+---+-----+
|123|  ash|
|999|chloe|
|007|  ash|
+---+-----+






(None, None)

# Now, lets join them together into a combined pokemon-and-trainer table
joined = df1.join(
    df2,
    on=df1['id'] == df2['id'],
    how='inner',
)
joined.show()

+---+---------+---+-----+
| id|     name| id| name|
+---+---------+---+-----+
|007|charizard|007|  ash|
|123|  pikachu|123|  ash|
|999|     evee|999|chloe|
+---+---------+---+-----+

This seems fine initially, but spark blows up as soon as you try and use the ‘id’ column in an expression

This example will produce the error:

[AMBIGUOUS_REFERENCE] Reference id is ambiguous, could be: [id, id].

This can be particularly annoying as the error will only appear when you attempt to use the columns, but will go undetected if this doesn’t happen

import pyspark.sql.utils
from pyspark.sql import DataFrame
from typing import List

def try_select(df: DataFrame, cols: List[str]):
    try:
        df.select(*cols).show()

    except pyspark.sql.utils.AnalysisException as e:
        print('select failed!', e)

try_select(joined, ['id', 'name', 'trainer'])

select failed! [AMBIGUOUS_REFERENCE] Reference `id` is ambiguous, could be: [`id`, `id`].

The solution: use a different parameter for the on columns

3.1.2 Join using list of names

joined = df1.join(
    df2,
    on=['id'],
    how='inner',
)
joined.show()

# Now let's try that same select again
try_select(joined, ['id', 'name', 'trainer'])

+---+---------+-----+
| id|     name| name|
+---+---------+-----+
|007|charizard|  ash|
|123|  pikachu|  ash|
|999|     evee|chloe|
+---+---------+-----+

select failed! [AMBIGUOUS_REFERENCE] Reference `name` is ambiguous, could be: [`name`, `name`].

3.1.3 Dataframe aliasing is a bit weird

df1.alias('pokemon').select('*').show()

+---+---------+
| id|     name|
+---+---------+
|123|  pikachu|
|999|     evee|
|007|charizard|
+---+---------+

import pyspark.sql.functions as F

joined = df1.alias('pokemon').join(
    df2.alias('trainers'),
    on=F.col('pokemon.id') == F.col('trainers.id'),
    how='inner',
)
joined.show()
joined.columns

+---+---------+---+-----+
| id|     name| id| name|
+---+---------+---+-----+
|007|charizard|007|  ash|
|123|  pikachu|123|  ash|
|999|     evee|999|chloe|
+---+---------+---+-----+

['id', 'name', 'id', 'name']

Now, our error message is much better, as it contains the dataframe aliases identifying which table the duplicate column name is from

try_select(joined, ['id'])

select failed! [AMBIGUOUS_REFERENCE] Reference `id` is ambiguous, could be: [`pokemon`.`id`, `trainers`.`id`].

Confusingly, using Dataframe.columns does not show the aliases, but they are usable when selecting

print(joined.columns)

try_select(joined, ['pokemon.id'])

['id', 'name', 'id', 'name']
+---+
| id|
+---+
|007|
|123|
|999|
+---+

4. Default empty DataFrames

Sometimes it’s handy to be able to instantiate an “empty” dataframe in the case that a file/some source data is missing

# This will result in an AnalysisException complaining that 
# the file did not exist
from pyspark.errors.exceptions.captured import AnalysisException

try:
    spark.read.json('optional_source.json')
except AnalysisException as e:
    print(e)

[PATH_NOT_FOUND] Path does not exist: file:/home/jovyan/work/articles/20230605_pyspark_fu/optional_source.json.

We can mitigate this by catching the exception, and creating a dataframe that matches the schema, but has 0 rows.

This ensures that any queries on the dataframe will still work, as all the columns will exist with the correct type.

This requires that we know the schema of the optional file

The easiest way to create a schema is usually to create a single-line file containing a valid line that matches the expected schema. Then, read that file into a dataframe and capture the schema for re-use (read: copy/paste)

import json

with open('not_there.json', 'w') as ostream:
    ostream.write(json.dumps({
        'id': 123, 'key': 'yolo', 'attrs': {'a': 'b'}
    }))

spark.read.json('not_there.json').schema.json()

'{"fields":[{"metadata":{},"name":"attrs","nullable":true,"type":{"fields":[{"metadata":{},"name":"a","nullable":true,"type":"string"}],"type":"struct"}},{"metadata":{},"name":"id","nullable":true,"type":"long"},{"metadata":{},"name":"key","nullable":true,"type":"string"}],"type":"struct"}'

I’ve never found a way (using StringIO or similar) to achieve this without writing a file - if you find a way then let me know!

Let’s bundle this up into a method that tidies up after itself:

import json
import os

def guess_schema(row: dict, tmp_fpath: str = 'tmp.json') -> dict:
    with open(tmp_fpath, 'w') as ostream:
        ostream.write(json.dumps({
            'id': 123, 'key': 'yolo', 'attrs': {'a': 'b'}
        }))    
    schema = json.loads(spark.read.json('not_there.json').schema.json())
    os.remove(tmp_fpath)

    return schema

schema = guess_schema(
    {'id': 123, 'key': 'yolo', 'attrs': {'a': 'b'}}
)
print(json.dumps(schema, indent=2))

{
  "fields": [
    {
      "metadata": {},
      "name": "attrs",
      "nullable": true,
      "type": {
        "fields": [
          {
            "metadata": {},
            "name": "a",
            "nullable": true,
            "type": "string"
          }
        ],
        "type": "struct"
      }
    },
    {
      "metadata": {},
      "name": "id",
      "nullable": true,
      "type": "long"
    },
    {
      "metadata": {},
      "name": "key",
      "nullable": true,
      "type": "string"
    }
  ],
  "type": "struct"
}

As you can see from this quick demo, it isn’t quick to craft pyspark schemas from hand! In my experience it’s prone to much human error and frustrating debugging, especially as schemas can grow large very quickly!

Now, we can tie this into the method to safely load/create a dataframe

from pyspark.errors.exceptions.captured import AnalysisException
import pyspark.sql.types as T

def safe_load(fpath: str, schema: dict):
    try:
        return spark.read.json(fpath)
    except AnalysisException as e:
        print(e)
        return spark.createDataFrame([], schema=T.StructType.fromJson(schema))

Side note: the method to convert a dict to a StructType (schema) is confusingly named fromJson despite the fact that the method accepts a dict, not a JSON string

df = safe_load('not_there.json', schema)

df.show()

+-----+---+----+
|attrs| id| key|
+-----+---+----+
|  {b}|123|yolo|
+-----+---+----+

After the initial generation, the schema can be stored in a file and loaded or just defined directly in the code, rather than “guessed” every time

Installing Github Copilot

2023-04-06T00:00:00+00:00

20230406 Installing Github Copilot

I’ve recently begun using Github Copilot, and have installed it for all my OSs for vim, neovim and vscode

I made a bootstrap script for Debian Bullseye:

it will install github copilot for these 3 IDE’s
and install an up-to-date node version

If you are running a different OS, then the first 3 commands will still work, and you will have to separately ensure that you have node >= v16 installed.

#!/bin/bash

set -euo pipefail

if [ ! -d ~/.vim/pack/github/start/copilot.vim ]; then
  echo "- installing for vim"
  git clone https://github.com/github/copilot.vim.git \
    ~/.vim/pack/github/start/copilot.vim
fi

if [ ! -d ~/.config/nvim/pack/github/start/copilot.vim ]; then
  echo "- installing for neovim"
  git clone https://github.com/github/copilot.vim.git \
    ~/.config/nvim/pack/github/start/copilot.vim
fi

echo "- installing for vscode"
code --install-extension GitHub.copilot

echo "- installing nodejs"
# nodejs needs to be version v16+, debian bullseye only ships with v12
curl -fsSL https://deb.nodesource.com/setup_19.x | sudo -E bash - && \
  sudo apt install -y nodejs

echo "! run :Copilot setup & :Copilot enable when in vim/nvim"

Now,

you can open vscode and click the popup button to log into github.
you can open vim, and run :Copilot setup, follow the instructions to login, before finally running :Copilot enable.

Done! Happy coding :)

Dictionary Paths

2023-03-22T00:00:00+00:00

20230322 Dictionary Paths

Traversing through a nested/multi-layer data structure is commonplace when writing code that handles data, and is also a fun coding problem!

20230322 Dictionary Paths

1. The first attempt

I find that its nice to start with a very simple solution that works, before thinking of other more difficult solutions

def fetch(data: dict, path: tuple):
	current = data
	for k in path:
		current = current[k]
	return current

Let’s create a dictionary, and then try to get a nested value

In [2]: d = {'a': {'b': {'c': 'd', 'x': 'y'}}}

In [3]: fetch(d, ('a', 'b', 'x'))
Out[3]: 'y'

2. reduce!

from functools import reduce

def fetch(data: dict, path: tuple):
    return reduce(lambda d, key: d.get(key), path, data)

However, this method doesn’t perform so nicely when the provided path isn’t 100% correct

def fetch(data: dict, path: tuple):
    return reduce(lambda d, key: d.get(key) if isinstance(d, dict) else None, path, data)

Now our line is too long! We could extract out the value grabbing to improve this

def fetch(data: dict, path: tuple):
	def safe_fetch(obj, key):
		if isinstance(obj, dict):
			return obj.get(key)

	return reduce(lambda d, key: safe_fetch(d, key), path, data)

This method returns None if the path is incorrect, or if any element along the path was not a dictionary

In [6]: fetch(d, ('z', 'b'))

In [7]: fetch(d, ('a', 'z'))

# Let's add some array brackets around the innermost dict
In [8]: d = {'a': {'b': [{'c': 'd', 'x': 'y'}]}}

In [9]: fetch(d, ('a', 'b', 'c'))

3. jq!

JQ is a great tool for manipulating or extracting JSON data on the command line, and it works in python too!

This allows us to make a path for any value, including support for lists/arrays, e.g.

`.a.b[0].x

In [1]: import jq

In [2]: d = {'a': {'b': [{'c': 'd', 'x': 'y'}]}}

In [3]: jq.compile('.a.b[0].x').input(d).first()
Out[3]: 'y'

4. Recursive

No article on nested dictionary traveral would be complete without a recursive implementation!

def fetch(data, i, path):
    print(f'{path=}, {i=}, {data=}')

    if i == len(path):
	    return data
	elif isinstance(data, list)
		if not isinstance(path[i], int):
			raise KeyError(
				f'Key for nested list must be an int! '
				f'key: "{path[i]}, nested element: "{data}"'
			)
	elif isinstance(data, (dict, list)):
		return fetch(data[path[i]], i+1, path)
	else:
		raise ValueError(f'unable to recurse past "{path[i]}"')

Let’s add some more robust error handling

if a nested dictionary key doesn’t exist
if the supplied path is too long
if a nested list index doesn’t exist
if a nested list key is not an integer

def fetch(data, path, i=0, debug=False):
    if i == len(path):
        return data

    key = path[i]
    if debug:
        print(f'{path=}, {i=}, {key=}, {data=}')

    elif isinstance(data, list):
        if not isinstance(key, int):
            raise KeyError(
                f'list index must be an int: {key=}, {path=}'
            )
        elif key >= len(data):
            raise IndexError(f'list index out of range: {key=}, {path=}, {data=}')
        return fetch(data[key], path, i+1, debug)

    elif isinstance(data, dict):
        if key not in data:
            raise KeyError(f'nested key not found: {key=}, {path=}')
        return fetch(data[key], path, i+1, debug)
    else:
        raise ValueError(f'unable to fetch next key in path: {key=}, {path=}')

Testing

In [17]: path_tests = {
    ...:     'valid1':              ('a', 'b', 0, 'c'),
    ...:     'valid2':              ('a', 'b', 0, 'x'),
    ...:     'dict_key_error1':     ('z', 'b'),
    ...:     'dict_key_error2':     ('a', 'b', 0, '!'),
    ...:     'list_index_error1':   ('a', 'b', 1),
    ...:     'list_index_error2':   ('a', 'b', 's'),
    ...:     'path_too_long_error': ('a', 'b', 0, 'c', 'another'),
    ...: }

In [18]: for k, path in path_tests.items():
    ...:     try:
    ...:         print(f'\ntesting "{k}"')
    ...:         result = fetch(d, path)
    ...:         print(f'- success! {result=}')
    ...:     except Exception as e:
    ...:         print(f'- failed! {e=}')
    ...:

testing "valid1"
- success! result='d'

testing "valid2"
- success! result='y'

testing "dict_key_error"
- failed! e=KeyError("nested key not found: key='z', path=('z', 'b')")

testing "nested_dict_key_error"
- failed! e=KeyError("nested key not found: key='!', path=('a', 'b', 0, '!')")

testing "nested_list_index_error1"
- failed! e=IndexError("list index out of range: key=1, path=('a', 'b', 1), data=[{'c': 'd', 'x': 'y'}]")

testing "nested_list_index_error2"
- failed! e=KeyError("list index must be an int: key='s', path=('a', 'b', 's')")

testing "path_too_long_error"
- failed! e=ValueError("unable to fetch next key in path: key='another', path=('a', 'b', 0, 'c', 'another')")

Conclusion

In conclusion, use jq! It’s a powerful tool and langauage that you can use with most other languages and environments

https://stedolan.github.io/jq/

Simplify with islice

2023-03-06T00:00:00+00:00

Simplify with islice

I was recently asked if I could help to make a bit of python code that would generate a range of column names for Google Sheet, given the number of columns.

e.g.

In [2]: column_names(10)
Out[2]: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

In [3]: column_names(30)
Out[3]: ['A', 'B', ... 'Y', 'Z', 'AA', 'AB', 'AC', 'AD']

and so on! Every time the end of the sequence is reached, the pattern starts again with one more letter (e.g. ZY, ZZ, AAA)

In practice, we solved this by just generating a large list of column names so that we are very unlikely to request one that is out of range.

Thanks to good ol’ Stack Overflow, I realised that the itertools.product method was perfect for this problem

https://docs.python.org/3/library/itertools.html#itertools.product

In [4]: from itertools import product

In [4]: list(product(('-', 'O', 'X'), repeat=2))
Out[4]:
[('-', '-'),
 ('-', 'O'),
 ('-', 'X'),
 ('O', '-'),
 ('O', 'O'),
 ('O', 'X'),
 ('X', '-'),
 ('X', 'O'),
 ('X', 'X')]

The initial solution

This method creates the “cartesian product” of the input set, which you can research if you are unfamiliar, but is essentially all the combinations of the input items, in order. By using the repeat function argument, we can control the length of the string that we’d like to create

The last standard-lib function that we can take advantage of is string.ascii_uppercase, which gives us the alphabet as uppercase characters

In [5]: import string

In [6]: def column_names(n):
   ...:     names = list(string.ascii_uppercase) + [''.join(el) for el in product(string.ascii_uppercase, repeat=2)]
   ...:     return names[:n]

In [7]: column_names(10)
Out[7]: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

In [8]: column_names(30)[24:]
Out[8]: ['Y', 'Z', 'AA', 'AB', 'AC', 'AD']

This function works fine for our use-case as it can provide up to 702 column names which is far more than we need, however I started thinking about how I’d make the function better.

Using a generator function

The solution has a few issues with it, namely

We can only generate a max of 702 column names
The 702 column names are always generated, most times only needing to return a small subset of that

In order to combine the first single-char alphabet, and keep expanding the width when we reach the end of a set (e.g. ZZ -> AAA), we can ust the count method from itertools, which creates an unbounded loop, giving the current index each iteration

https://docs.python.org/3/library/itertools.html#itertools.count

First, let’s make a function that will generate a list of the requested size

In [20]: def column_names(n):
    ...:     names = []
    ...:     i = 0
    ...:     for j in count(1):
    ...:         for el in product(string.ascii_uppercase, repeat=j):
    ...:             names.append(''.join(el))
    ...:             i += 1
    ...:             if i >= n:
    ...:                 return names
    
In [21]: column_names(10)
Out[21]: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

In [22]: column_names(30)[24:]
Out[22]: ['Y', 'Z', 'AA', 'AB', 'AC', 'AD']

This attempt works pretty well - it only creates the array of the correct size. However, this can be refactored to eliminate the need for the i variable to check if the sequence is finished.

Firstly, let’s convert the function to yield items instead of appending to a list. (This does require, for now, that we wrap our method call in a list constructor so that the list is actually materialised).

In [23]: def column_names(n):
    ...:     i = 0
    ...:     for j in count(1):
    ...:         for el in product(string.ascii_uppercase, repeat=j):
    ...:             yield ''.join(el)
    ...:             i += 1
    ...:             if i == n:
    ...:                 return
    ...:

In [24]: list(column_names(10))
Out[24]: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

In [25]: list(column_names(30))[24:]
Out[25]: ['Y', 'Z', 'AA', 'AB', 'AC', 'AD']

Now that we are yielding items, we can take advantage of a 3rd (and one of the best) itertools method, islice. islice takes an iterable, and a “stop” index number, meaning that it will iterate the number of times requested, then end the function.
This means that we can just write a function that infinitely generates new column names, and remove all references to the i variable!

In [26]: def column_names():
    ...:     for j in count(1):
    ...:         for el in product(string.ascii_uppercase, repeat=j):
    ...:             yield ''.join(el)
    ...:

In [27]: list(islice(column_names(), 10))
Out[27]: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

In [28]: list(islice(column_names(), 30))[24:]
Out[28]: ['Y', 'Z', 'AA', 'AB', 'AC', 'AD']

Our algorithm has now been simplified considerably thanks to using islice, as we can distill the function down to its purest form.
If you like, you can go one step further in shortening the code (just because it’s python!). You can use a map function to apply the ''.join to each element, meaning that the method becomes only 2 LOC!)

In [38]: def column_names():
    ...:     for j in count(1):
    ...:         yield from map(''.join, product(string.ascii_uppercase, repeat=j)
    ...: )
    ...:

In [39]: list(islice(column_names(), 10))
Out[39]: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

In [40]: list(islice(column_names(), 30))[24:]
Out[40]: ['Y', 'Z', 'AA', 'AB', 'AC', 'AD']

Now, as the final step, we can pack the list/islice method calls into our method, making it a nicer for our users to call :)

In [41]: def column_names(n):
    ...:     def _column_names():
    ...:         for j in count(1):
    ...:             yield from map(''.join, product(string.ascii_uppercase, repeat=j)
    ...:     return list(islice(_column_names(), n))
    ...:

In [42]: column_names(10)
Out[42]: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

In [43]: column_names(30)[24:]
Out[43]: ['Y', 'Z', 'AA', 'AB', 'AC', 'AD']

CSGO Surf

2022-07-18T00:00:00+00:00

20220718 CSGO surf

This article contains all of my .cfg files that I use when playing CSGO Surf

20220718 CSGO surf
- Installation
  - Online CFG

Installation

You’ll need to write this into a file in the “cfg” directory of your CSGO installation.

# In most cases this is something like:
STEAM_HOME="$HOME/.local/share/Steam"
# In my case, I use a shared partition to store my steam games, so for me its:
STEAM_HOME="/mnt/X/SteamLibrary"

After setting the $STEAM_HOME variable in your shell, you can use the following command to write the config file.

# Now, you can 
vim "$STEAM_HOME/steamapps/common/Counter-Strike Global Offensive/csgo/cfg/surf.cfg"

Online CFG

I load this file when playing surf on public online servers (e.g. Surf Heaven AU)

see my toggle keybind article for an explanation of how the CFG toggle keybind works!

surf_online.cfg

  // macros ---------------------------------------

  // saveloc (bind by default)
  exec surf_saveloc_on.cfg;
  alias saveloc_on "exec surf_saveloc_on.cfg; alias saveloc_toggle saveloc_off";
  alias saveloc_off "exec surf_saveloc_off.cfg; alias saveloc_toggle saveloc_on";
  alias saveloc_toggle "saveloc_off";

  // hud visibility (on by default)
  alias hud_off "cl_draw_only_deathnotices 1; alias hud_toggle hud_on";
  alias hud_on "cl_draw_only_deathnotices 0; alias hud_toggle hud_off";
  alias hud_toggle "hud_off"

  // keybinds -------------------------------------

  // turnbinds
  bind mouse1 +left;  // turn left with left mouse click
  bind mouse2 +right; // turn right with right mouse click

  // stage teleports
  bind t "say !rs";  // teleport to start of map
  bind z "say !r";   // teleport to start of stage

  // toggles
  bind . "saveloc_toggle"; // toggle saveloc
  bind l "hud_toggle" // toggle hud visibility

  // info/debug commands
  bind 5 "say !tier" // show current map info
  bind 6 "say !pb"   // show personal best time

  // map voting
  bind , "say !nominate" // nominate next map
  bind h "say /rtv"      // trigger vote for next map

  // force window to 1440p resolution
  mat_setvideomode 2560 1440 1

surf_saveloc_on.cfg

  // This file contains all SurfTimer saveloc keybinds

  // set teleport location, play beep sound
  bind q "sm_saveloc ; playvol buttons\blip1 0.5";
  // teleport to location
  bind e "sm_tele";
  bind mwheeldown "sm_tele";

  // change to previous/next teleport location
  bind 1 "sm_teleprev";
  bind p "sm_teleprev";
  bind 2 "sm_telenext";

  // enable back arrow key
  bind s +back;

  say "saveloc enabled";

surf_saveloc_off.cfg

  // This file _disables_ all SurfTimer saveloc keybinds

  // set teleport location
  unbind q;
  // teleport to location
  unbind e;
  unbind mwheeldown;

  // change to previous/next teleport location
  unbind 1;
  unbind p;
  unbind 2;
  // disable back arrow key
  unbind s;

  say "saveloc disabled";

FSTAB

2022-07-03T00:00:00+00:00

20220703 FSTAB

Note: This applies to Linux systems only!

What is /etc/fstab?

In order to mount a drive automatically in linux, you’ll have to ensure that it’s added to the /etc/fstab file.

Whenever I bootstrap a new linux server or desktop, this is one of the setup stages that I always do, to ensure that all of my drives are mounted correctly.

As I bootstrap automatically (mostly), I have quite a few configuration files for a heap of different apps & services that reference those drives, usually by the /mnt/<drive> path.
This means that I need to ensure that no matter which linux OS I’m using or reinstalling, that each drive will still be mounted to the same place

Help

See some more documentation by using man fstab in your terminal

The file format

I’ve modified the usual format and vertically-aligned all of the setting headings & values so that it’s easier to read (both for you and for me!). Usually, the options/dump/pass sections are all crammed together separated by a single space.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>                          <mount point>  <type>  <options>         <dump> <pass>
UUID=8e519e43-880b-4a9f-83b0-e83899493ca6 /              ext4    defaults,noatime 0      1
UUID=18A7-14C2                            /boot/efi      vfat    umask=0077       0      2

Some very basic explanations for some of the more mysterious fields

dump
- legacy option for backing up drives using the dump command
- always set this to 0
pass: indicates if a drive should be checked for errors on startup.
- 0 (default) - don’t check this partition
- 1 = this is the root partition, check it
- 2 = this is a seconndary partition, check it
- I’d recommend to always use 2 unless you have a specific reason not to
options
- Define behaviour
- Super-hard to find documentation on
- The most important option for secondary partitions is errors=remount-ro
  - This means that the drive will mount in read-only mode if any problems occur

Making an entry

Let’s make an entry!

For our example, let’s imagine that we want to create an entry that mounts the music partition in the /mnt/music dir (which, hypothetically, we’ve already created).
(To make things more interesting, this partition is in ntfs format, aka a partition usually created in/for Windows)

Find the partition information

I use one command to get most of the information that I need to create the entry:

sudo lsblk -o name,size,label,fstype,partlabel,uuid

 ☯ ~ sudo lsblk -o name,size,label,fstype,partlabel,uuid
 NAME          SIZE LABEL  FSTYPE   PARTLABEL    UUID
 nvme0n1     931.5G
 ├─nvme0n1p3 341.8G        ext4     root         0bbaa1ff-b625-4972-8099-b59dc15004ee
 ├─nvme0n1p4   334G        ext4     root         15cd5735-4977-430f-ab46-e1202e6ff854
 └─nvme0n1p5   7.8G swap   swap                  088a2190-6ed0-4f76-88ca-70803aada85f
 nvme1n1     931.5G
 ├─nvme1n1p1   512M        vfat                  1a5a2317-3a77
 └─nvme1n1p4 637.1G X      ntfs     music        d2320a36-68e6-4ea1-b5da-6374efa4b355

Ensure that the mount directory exists

You’ll have to ensure that the directory that the drive will be mounted at already exists! You can do this via the following commands:
```
 # Make the directory
 ☯ ~ sudo mkdir /mnt/music
 # Make your user the owner instead of root
 ☯ ~ sudo chown $USER:$USER /mnt/music
```

Create the entry

 UUID=d2320a36-68e6-4ea1-b5da-6374efa4b355  /mnt/music  ntfs  errors=remount-ro  0  2

Run!

You can restart your computer, or run

☯ ~ sudo mount -a

If there were no errors, you won’t see any output.

Then, check the contents of the mounted directory to see if it has been mounted as expected

☯ ~ ls -alh /mnt/music

awscli / boto3 snippets

2022-06-10T00:00:00+00:00

20220610 awscli / boto3 snippets

Get all logs from a cloudwatch log stream

Includes full pagination support, so that you actually fetch all of the logs

Usage

# Usage:
# ./get_all_logs.py <log_group> <log_stream> <output_file_path>

# e.g.
# ./get_all_logs.py /syslog/myinstance my_log_stream output.log

Snippet

#!/usr/bin/env python3

'Consumes all logs from a cloudwatch log stream'

import json
import sys
from datetime import datetime
from itertools import count

import boto3

client = boto3.client('logs')

logGroupName = sys.argv[1]
logStreamName = sys.argv[2]
output = sys.argv[3]

time_start = datetime.now()

total = 0
page_args = {}
with open(output, 'w', encoding='utf-8') as ostream:
    for i in count():
        response = client.get_log_events(
            **{
                'logGroupName': logGroupName,
                'logStreamName': logStreamName,
                'startFromHead': True,
            },
            **page_args
        )
        ostream.write('\n'.join([json.dumps(event) for event in response['events']]))
        total += len(response['events'])

        if response['nextForwardToken'] == page_args.get('nextToken', ''):
            break

        page_args = {'nextToken': response['nextForwardToken']}
        print(json.dumps(
            {'duration': str(datetime.now()-time_start), 'total': total, 'next_page': page_args}
        ))

print('- Finished')

Minecraft

2022-05-18T00:00:00+00:00

20220518 Minecraft

This article contains some handy tips and tricks around running Minecraft clients and servers, particularly in Linux

20220518 Minecraft

Client

Installing Java for client

Linux

Download the openjdk jre

 # debian-based
 ☯ ~ sudo apt install default-jre default-jre-headless
 # arch-based
 ☯ ~ sudo pacman -S jre-openjdk jre-openjdk-headless

Make sure $JAVA_HOME is set in your environment

For linux, this can be set in your /etc/environment file

  ☯ ~ echo "/usr/lib/jvm/java-17-openjdk-amd64/" | sudo tee -a /etc/environment

If you installed the openjdk jre, you should be able to find your $JAVA_HOME with this command

  ☯ ~ find /usr/lib/jvm/ -iname java-1?-openjdk | sort | tail -1
  /usr/lib/jvm/java-17-openjdk-amd64

You can combine the two commands to make this fully automatic, but it’s usually good to check things as you go when doing for the first time

  echo $(find /usr/lib/jvm/ -iname java-1?-openjdk | sort | tail -1) | sudo tee -a /etc/environment

M1 OSX

Download the openjdk jre

 # m1 osx
 ☯ ~ arch -arm64 brew install openjdk

Set up Java in your environment

This command comes from the output of the homebrew install, and seems to be needed to make minecraft work
```
 # For the system Java wrappers to find this JDK, symlink it with
 ☯ ~ sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk
```
Make sure that the openjdk bin directory is in your $PATH
You may use .zshrc/.bashrc or whatever else in place of the .bash_profile if you prefer
```
 ☯ ~ echo 'export PATH="/opt/homebrew/opt/openjdk/bin:$PATH"' >> $HOME/.bash_profile
```

Locations

$HOME/.minecraft/shaderpacks/1.18.2/
$HOME/.minecraft/mods/--

Mods & Shaders

There isn’t a way to fully automate the download of mods & shaders, as curseforge.com the site that normally hosts the files doesn’t host files at static URLS.

One bit of advice would be - for each major version of the minecraft client that you use, download all of the jars and store them in a “backup” location that you can use later if needed.

Mods - Recommendations

These links will trigger the “latest” download for each tool

The Fabric API is a tool that loads mods for your minecraft client. This means that you’ll need to use “fabric” mods if you look into installing others

Generic install link pages

Fabric Loader
Fabric API
Litematica (creates/loads schematic overlays)
- MaLiLib (dependency of Litematica)
MiniHUD (customise the F3 screen)
Tweakeroo (item placement)
Starlight (light engine optimisations)
OptiFabric (allows you to run OptiFine & Fabric together)

Latest install links (1.19.2)

Fabric Loader
Fabric API
Litematica (creates/loads schematic overlays)
- MaLiLib (dependency of Litematica)
MiniHUD (customise the F3 screen)
Tweakeroo (item placement)
Starlight (light engine optimisations)
OptiFabric (allows you to run OptiFine & Fabric together)

If you want to get fancy, you can use your browser to download these all simultaneously

replace firefox with your browser if you need
add/remove links as you like
paste all lines into your terminal and hit enter!

firefox https://maven.fabricmc.net/net/fabricmc/fabric-installer/0.11.0/fabric-installer-0.11.0.jar &
firefox https://www.curseforge.com/minecraft/mc-mods/fabric-api/download/3921681 &
firefox https://www.curseforge.com/minecraft/mc-mods/litematica/download &
firefox https://www.curseforge.com/minecraft/mc-mods/malilib/download &
firefox https://www.curseforge.com/minecraft/mc-mods/minihud/download &
firefox https://www.curseforge.com/minecraft/mc-mods/tweakeroo/download &
firefox https://www.curseforge.com/minecraft/mc-mods/starlight/download &
firefox https://www.curseforge.com/minecraft/mc-mods/optifabric/download &

wait

Mods - Installation

Move all of the jars into your mods folder

 cd $HOME/Downloads
 mv -v \
   fabric-api*jar litematica*jar malilib*jar minihud*jar tweakeroo*jar starlight*jar optifabric*jar \
   $HOME/.minecraft/mods

Install fabric API

 cd $HOME/Downloads
 java -jar fabric-installer-*.jar

If you have optifine, you can install that too using java

cd $HOME/Downloads
java -jar preview_OptiFine_1.19_HD_U_H8_pre1.jar

Notes

In order for OptiFabric to work, you’ll need to move the optifine jar file to ~/.minecraft/mods after installing it. This isn’t usually required for optifine to work, but is required by OptiFabric

Recommendations - Shaders

Server

Docker

use itzg/minecraft-server

Using this awesome docker image, you can specify the VERSION of your server, and also optionally use FABRIC for installing server-side datapacks/mods

docker pull itzg/minecraft-server:latest

docker run \
  -d \
  --name mc \
  --restart=unless-stopped \
  -p 25565:25565 \
  -P 25575:25575 \
  -e VERSION=1.19 \
  -e TYPE=FABRIC \
  -e EULA=TRUE \
  -v $HOME/.minecraft-server/:/data \
  itzg/minecraft-server:latest

mcrcon

mcrcon is a cli tool that you can use for admin control over your server

https://github.com/Tiiffi/mcrcon

Install using:

git clone https://github.com/Tiiffi/mcrcon.git
cd mcrcon
make
sudo make install

You can then connect a mcrcon session using these parameters:

0.0.0.0 for the hostname of the docker container
25575 as the mcrcon port
minecraft as the default server password

 ☯ ~ mcrcon -H 0.0.0.0 -P 25575 -p minecraft
Logged in. Type 'quit' or 'exit' to quit.
>worldborder get
The world border is currently 59999968 blocks wide

Finding the world seed

via the Docker server

You can use mcrcon, just add your command that you want to run after you’ve specified your host/port/password

 ☯ ~ mcrcon -H 0.0.0.0 -P 25575 -p minecraft seed
Seed: [-2679236495807353480]

Bash Practices

2022-05-11T00:00:00+00:00

20220511 Bash Practices

20220511 Bash Practices
- 1. Halting execution after an error
  - Normal behaviour
- Using set & pipefail
  - 2. Checking variables
  - 3. Without printing

1. Halting execution after an error

Note: echo $? gives you the exit status of the previous command. 0 == success, 1 == failure.

set -euxo pipefail

Normal behaviour

 ☯ cat test.sh
ls 1
echo "finished"

 ☯ bash test.sh ; echo $?
ls: cannot access '1': No such file or directory
finished
0

Using set & pipefail

 ☯ cat test.sh
set -euxo pipefail
ls 1
echo "finished"

 ☯ ~/d/tmck-code.github.io bash test.sh ; echo $?
+ ls 1
ls: cannot access '1': No such file or directory

2. Checking variables

 ☯ ~ cat test.sh
set -euxo pipefail
echo "${1}"
echo "finished"

 ☯ ~ bash test.sh ; echo $?
test.sh: line 2: 1: unbound variable
1

To get more control over unbound variables, we can give the variable a default empty value, and then do an explicit check to see if the variable is empty

 ☯ ~ cat test.sh
set -euxo pipefail
[ -z "${1:-}" ] && echo "\$1 must be given!" && exit 1
echo "finished"

 ☯ ~ bash test.sh ; echo $?
+ '[' -z '' ']'
+ echo '$1 must be given!'
$1 must be given!
+ exit 1
1

3. Without printing

You might want to suppress the debug output, for brevity or to protect secret credentials from being logged.

You can either

omit the x character
or toggle with
1. set -x to enable debug printing
2. set +x to disable debug printing

 ☯ ~ cat test.sh
set -euxo pipefail
set +x
echo "without debug"
set -x

[ -z "${1:-}" ] && echo "\$1 must be given!" && exit 1
echo "finished"

 ☯ ~ bash test.sh ; echo $?
+ set +x
without debug
+ '[' -z '' ']'
+ echo '$1 must be given!'
$1 must be given!
+ exit 1
1

Convert PDF To PNG

2022-03-30T00:00:00+00:00

20220330 Convert PDF To PNG

If you have a Mac, you’re in luck, you can use the free included Preview to open a PDF, and then “export” it as a PNG, even letting you set the resolution.

https://support.apple.com/en-au/guide/preview/prvw1012/mac

Fresh Python Repo

2022-03-08T00:00:00+00:00

20220308 Fresh Python Repo

This is a basic method for initialising a python-based repository that aims to have as few boilerplate files and dirs as possible, and to be portable so that you can develop on different OS’s/architectures.

This blog entry is a WIP, more steps will be added over time

Create an empty repository in Github, and clone it to your local computer.

 cd demo-repo

 mkdir ops test demo-repo
 # Create boilerplate files
 touch Makefile README.md ops/Dockerfile requirements.txt setup.py
 # Create __init__.py files to make modules
 touch demo-repo/__init__.py test/__init__.py

Create a dummy “hello world” file (this will be removed later)

 ☯ ~/demo-repo cat demo-repo/demo.py
 print("hello world")

Now your repo should look like this:

 ☯ ~/demo-repo tree
 .
 ├── demo-repo
 │   └── demo.py
 ├── Makefile
 ├── ops
 │   └── Dockerfile
 ├── README.md
 ├── requirements.txt
 ├── setup.py
 └── test

 3 directories, 5 files

Fill in the Dockerfile with the minimum boilerplate

Inherit from the official python docker image with a major version release tag
upgrade apt packages
upgrade pip and install requirements.txt

 FROM python:3.9

 WORKDIR /code
 RUN apt update && apt upgrade -y

 ADD requirements.txt .

 RUN python3 -m pip install --upgrade pip && \
     python3 -m pip install -r requirements.txt

 ADD . .

tmck-code.github.io

KSF Surf Maps: Data Visualisation

20260415 KSF Surf Maps: Data Visualisation

Python Repo Practices

20260204 Python Repo Practices

Setting up a new Python repo

Examples

Basic structure

ANSI to UTF8 Part 1: Intro

20260123 ANSI to UTF8 Part 1: Intro

Introduction

1. SAUCE metadata

2. Colour codes

3. ISO-8859-1, CP437 & UTF-8 characters

5. Non-Printing, c0 & Cursor codes

4. Fonts

5. Terminal colours

F-String number formatting

20250623 F-String number formatting

Rounding floats

0-padding numbers

Decimal padding

Integer padding

Decimal formatting

Separators

Get unix time FAST with pure bash

20250519 Get unix time FAST with pure bash

Results

Dictionary slice in Python

20250506 Dictionary slice in Python

Recreating slice in Python

A better way?

Using operator.itemgetter

Gotchas

Non-existent keys

Order of keys

Benchmarking

Results

OSX dark mode shortcut

20250102 OSX dark mode shortcut

Automator

1. Launch Automator

2. Create “Quick Action”

3. Search for “appearance”

4. Drag “Change System Appearance” to the workflow

5. Save the Quick Action

6. Assign a Keyboard Shortcut

Shortcuts

1. Launch Shortcuts

2. Add a new shortcut

3. Create the shortcut logic

4. Name the shortcut

4. Use via Spotlight

Christmas Tree Lights with Arduino

20241210 Christmas Tree Lights with Arduino

Ingredients

An Arduino board

A string of “WS8211” LEDs

Some patience

Some Code!

Comparing Lists or Matrices

20241128 Comparing Lists or Matrices

Using JQ with style

20240829 Using JQ with style

1. The ENV var

2. The colours

Examples

Filtering Nones from a Python list

20240828 Filtering Nones from a Python list

0. Checking if things are None

1. The list comprehension

2. lambda

3. Partial

4. Inner function

Benchmarking!

Conclusions

Join and Coalesce in Pyspark

20240411 Join and Coalesce in Pyspark

Scenario

Example Dataframes

Recreating `slice` in Python

Using `operator.itemgetter`

`surf_calcyate2`

`surf_nuclear`

`surf_school`

`surf_ivory`

`surf_calzone`