blog

Categories     Timeline

Concept2 PM5 with LaymansHex

For the fun of it, I decided to reimplement a basic version of pm5conv in LaymansHex and a bit of Bash - to parse workout data of a Concept2 PM5 monitor.

I basically took the data format description and translated it to LaymansHex definitions and Bash glue:

% for i in `ls | grep -E "(layhex|bash)"`; do echo "==> $i\n"; cat $i; echo; done
==> LogDataAccessTbl.bin.layhex

little endian
            : byte[offset*32]
Magic       : byte[1]
WorkoutType : uint8
            : byte[10]
NoSplits    : uint16
            : byte[2]
Offset      : uint16
            : byte[6]
Size        : uint16
Index       : uint16
            : byte[4]
Marker      : byte[32]

==> LogDataStorage.bin-Workout1.layhex

big endian
                    : byte[offset]
                    : byte[1]
                    : byte[1]
                    : byte[2]
                    : byte[4]
Timestamp           : byte[4]
UserID              : uint16
                    : byte[4]
RecordID            : uint8
                    : byte[3]
TotalDuration       : uint16
TotalDistance       : uint32
SPM                 : uint8
                    : byte[1]
SplitSize           : uint16
                    : byte[18]
                    : byte[splitNo*32]
SplitDistance       : uint16
SplitHeartRate      : uint8
SplitSPM            : uint8
                    : byte[28]

==> pm5conv.bash

#!/bin/bash

cmd="laymanshex"

accTable="LogDataAccessTbl.bin"
storage="LogDataStorage.bin"

# uses evil eval

function printNonEmpty {
  varname='$'"$1"
  eval "val=$varname"
  if [ "$val" != "" ]; then
      echo "$1=$val"
  fi
}

function printTimestamp {
  timestamp="0x$1"
  year="$(( ((timestamp & 0xFE000000) >> 25) + 2000 ))"
  day="$((   (timestamp & 0x01F00000) >> 20 ))"
  month="$(( (timestamp & 0x000F0000) >> 16 ))"
  hour="$((  (timestamp & 0x0000FF00) >> 8  ))"
  minute="$(( timestamp & 0x000000FF ))"
  printf "Date=%d-%02d-%02d %02d:%02d\n" $year $month $day $hour $minute
}

function printSplit {
  outputSplit="$($cmd -nopadding -fvar=offset=$1,splitNo=$2 $storage-Workout$3.layhex $storage 2> /dev/null)"
  status=$?
  if [ $status -eq 0 ]; then
    eval "$outputSplit"
    echo
    echo "Split $(($2+1))"
    echo "----------"
    printNonEmpty "SplitDuration"
    printNonEmpty "SplitDistance"
    printNonEmpty "SplitHeartRate"
    printNonEmpty "SplitSPM"
  fi
}

function printWorkout {
  echo "Workout $1"
  echo "============="
  outputHeader="$($cmd -nopadding -fvar=offset=$2,splitNo=0 $storage-Workout$3.layhex $storage 2> /dev/null)"
  status=$?
  if [ $status -eq 0 ]; then
    eval "$outputHeader"
    printTimestamp "$Timestamp"
    printNonEmpty "TotalDuration"
    printNonEmpty "TotalDistance"
    printNonEmpty "SplitSize"
    printNonEmpty "SPM"
    j=0
    while [ "$j" -lt "$NoSplits" ]; do
      printSplit $2 $j $3
      j=$((j+1))
    done
    echo
  fi
}

function printAll {
  i=0
  while : ; do
    offset=$(($i))
    output="$($cmd -nopadding -fvar=offset=$offset $accTable.layhex $accTable 2>/dev/null)"
    status=$?    
    if [ $status -ne 0 ]; then
      break
    else
      eval "$output"    
      printWorkout $Index $Offset $WorkoutType
    fi
    i=$((i+1))
  done
}

printAll

Output of pm5conv.bash:

% ./pm5conv.bash
Workout 1
=============
Date=2018-02-27 21:00
TotalDuration=962
TotalDistance=363
SplitSize=3000
SPM=29

Split 1
----------
SplitDistance=363
SplitHeartRate=0
SplitSPM=27

Workout 2
=============
Date=2018-02-28 20:41
TotalDuration=4208
TotalDistance=1559
SplitSize=3000
SPM=27

Split 1
----------
SplitDistance=1121
SplitHeartRate=0
SplitSPM=29

Split 2
----------
SplitDistance=438
SplitHeartRate=0
SplitSPM=26

Workout 3
=============
Date=2018-03-01 18:47
TotalDuration=12058
TotalDistance=4166
SplitSize=3000
SPM=28

Split 1
----------
SplitDistance=1056
SplitHeartRate=0
SplitSPM=29

Split 2
----------
SplitDistance=1060
SplitHeartRate=0
SplitSPM=28

Split 3
----------
SplitDistance=1007
SplitHeartRate=0
SplitSPM=28

Split 4
----------
SplitDistance=1024
SplitHeartRate=0
SplitSPM=29

Split 5
----------
SplitDistance=19
SplitHeartRate=0
SplitSPM=0

[...]

Reproduktionszahl

Mir leuchtet nicht recht ein, warum in der öffentlichen Diskussion immer wieder die (effektive) Reproduktionszahl verwendet wird - nur um sie anschließend in die Änderung der Anzahl neuer Fälle zu übersetzen. Für den Nicht-Virologen bedeutet sie doch im Prinzip nur: <1 gut, =1 naja, >1 schlecht.

Was mir fehlt ist eine Aufbereitung der neuen Fälle, geglättet um tägliches Rauschen und Wochentagsmeldemüdigkeit, um die Entwicklung besser verstehen zu können. Ich hab mich mal daran versucht und zu dem Blogpost über die aktiven Fälle Kurven mit gleitendem Mittelwert über die jeweils letzten 4 Tage und zusätzlich mit wochentagsabhängigem Faktor hinzugefügt.

Sequences in LaymansHex

I have spent some time thinking about handling sequences and in-file offsets with LaymansHex. I decided on two things:

  1. LaymansHex is not and should not be used as a “proper” binary parser. It is a quick-and-dirty tool to read and set some values.
  2. Binary sequences are implemented in multiple ways and adding every way to the file description format is tedious and makes the use complex.

Both points led me to believe, that it’s better to handle sequences and offsets with multiple calls to LaymansHex and variable byte field sizes in the format definition to allow for offsets and sliding windows. I implemented this in commit 279429d0bff0681d533b23c260249361078d0a78

I added a minimal example to the of the README.

theHunter COTW player file format

theHunter: Call of the Wild is a hunting simulator, which - unfortunately - has some characteristics of a typical first-person shooter, namely missions, rewards and locked equipment. Since this prevents actual hunting, I thought it would be interesting to analyse the file format that saves level, XP, cash and so on.

I wrote a fairly generic tool called laymanshex that takes a partial file description and a binary file as input and outputs the values. Values can be changed with the ‘-set’ argument.

For example:

% ./laymanshex thp_player_profile_adf.layhex thp_player_profile_adf
           Level = 19
              XP = 20235
     SkillPoints = 0
      PerkPoints = 0
SkillPointsSpent = 7
 PerkPointsSpent = 6
            Cash = 16640
      RifleLevel = 14
    HandgunLevel = 4
    ShotgunLevel = 4
        BowLevel = 1
      RifleScore = 1992
    HandgunScore = 382
    ShotgunScore = 417
        BowScore = 0

% ./laymanshex -set="Level=60,XP=100000" thp_player_profile_adf.layhex thp_player_profile_adf
Created backup thp_player_profile_adf.bak20200426231811

           Level = 60
              XP = 100000
     SkillPoints = 0
      PerkPoints = 0
SkillPointsSpent = 7
 PerkPointsSpent = 6
            Cash = 16640
      RifleLevel = 14
    HandgunLevel = 4
    ShotgunLevel = 4
        BowLevel = 1
      RifleScore = 1992
    HandgunScore = 382
    ShotgunScore = 417
        BowScore = 0

% ./laymanshex -set="Cash=0x7FFFFFFF" thp_player_profile_adf.layhex thp_player_profile_adf | grep Cash
            Cash = 2147483647

The partial file description for thp_player_profile_adf (COTW version 1.49) looks like this:

little endian
                 : byte[113]
Level            : int32
XP               : int32
SkillPoints      : int32
PerkPoints       : int32
SkillPointsSpent : int32
                 : byte[60]
PerkPointsSpent  : int32
                 : byte[60]
Cash             : int32
RifleLevel       : int32
HandgunLevel     : int32
ShotgunLevel     : int32
BowLevel         : int32
RifleScore       : int32
HandgunScore     : int32
ShotgunScore     : int32
BowScore         : int32

Releases of laymanshex can be found here, go sources in the laymanshex git repo and possible further file descriptions in the laymanshex-files git repo.

The file format of COTW saves has changed many times in the past and I assume it will again at some point in the future.

Tscrape twitter digest

[Update 2020-06-02: This does not work any more, because Twitter changed their site]

0 % cat fakenews                                                                                                                                                                                                                       master
#!/bin/bash

#########################################################
# fakenews                                             ##
#   create an html digest of watched twitter accounts  ##
#   with https://codemadness.org/git/tscrape/ and gawk ##
#########################################################

# keep one twitter account per line in here
accountsfile=".fakenews"
outfile="/tmp/fakenews_out.html"
numberOfTweets=5

echo '<html><head><title>News</title></head><body>' > $outfile

while read account; do
    echo "Fetching $account..."
    echo "<h3>$account</h3><dl>" >> $outfile
    curl -H 'User-Agent:' -s "https://twitter.com/$account" |
	tscrape |
	head -n $numberOfTweets |
	awk -F '\t' '{$1 = strftime("%d-%m-%Y %H:%M:%S",$1); print "<dt>" $6 "</dt><dd>" $4 " <br><small>" $1 "</small></dd>"}' |
	sed 's/https:[^ ]*/<a href="&">&<\/a>/g' >> $outfile
    echo "</dl>" >> $outfile
done < "$accountsfile"

echo '</body></html>' >> $outfile
xdg-open $outfile

<--Previous

Later-->