thaerious

JSON Serialization in Unity

JSONUtility (Unity Class)

Unity has it’s own JSON serialization package [ref] which is very close in usage to the native C# serialization. It works with public field, or private fields marked with the ‘SerializeField’ attribute. It does not work with class properties, or fields marked ‘readonly’. To otherwise skip a field mark it with the ‘NonSerialized’ attribute.

// Will serialize
public string name;
[SerializeField] private string value;

// Will not serialize
public readonly string name;
public string name {get; set;}
[NonSerialized] int age;

The serialization method does not require type information, it will simply create json text based on the public field values. This means you do not need to deserialize into the same object you serialized from. The class it’s self must be marked with the ‘Serializable’ attribute.

using UnityEngine;

[System.Serializable]
public class PlayerState : MonoBehaviour{
    public string name;
    public int age;

    public string SaveToString(){
        return JsonUtility.ToJson(this);
    }

    // Given:
    // name = "Dr Charles"
    // age = 33
    // SaveToString returns:
    // {"name":"Dr Charles","age":33}
}

Unlike serialization, deserialization requires a type. Any values found in the JSON text that are not fields in the provided type will be ignored.

PlayerState playerState = JsonUtility.FromJson<PlayerState>(string);

JSONEncoder (Helper Class)

In order to automatically decode a json object to the correct type I have included the type name in the JSON text object. The emitted JSON has two root fields: type & instance.

using UnityEngine;
using System;

public class JSONDetails<T>{
    public T instance;
    public string type;

    public JSONDetails(T instance) {
        this.instance = instance;
        this.type = instance.GetType().ToString();
    }    
}

public class JSONEncoder {
    public string type = "";
    public JSONEncoder() { }

    public static string Serialize<T>(T anObject) {
        return JsonUtility.ToJson(new JSONDetails<T>(anObject), true);
    }

    public static object Deserialize(string json) {
        JSONEncoder wrapper = JsonUtility.FromJson<JSONEncoder>(json);
        Type g = Type.GetType(wrapper.type);
        Type t = typeof(JSONDetails<>).MakeGenericType(new Type[] { g });
        var hydrated = JsonUtility.FromJson(json, t);   
        return t.GetField("instance").GetValue(hydrated);
    }
}

At times serialization may require type information to ensure the correct type is serialized. Deserialization returns an object which can be cast as necessary. Typically this should used to detect the presence of an interface and cast accordingly.

public void ReceiveEvent(string json, string source) {
    var isEvent = JSONEncoder.Deserialize(json);

    if (isEvent is IModelEvent<M> != false) {
        IModelEvent<M> modelEvent = (IModelEvent<M>)isEvent;
        this.model = modelEvent.UpdateModel(this.model);
    }

    if (isEvent is IModelEvent<N> != false) {
        this.Broadcast(isEvent, source);
    }
}

Google API Service Accounts With NodeJS

For a recent project I needed update to a single shared Google calendar. I decided to go with a Google service account to manage this. I’m going to post here how I did it.

What is a Service Account

A service account is used by an application as opposed to a real living person. It allows your application to make authorized API calls directly. A service account is identified by its email address, which is unique to the account.

  • Service accounts can not be logged into with browsers.
  • Authorization is done with a public-private RSA key.
  • Service accounts are not part of your Google domain. They do not share global resources like a live user.

Create a Service Account

At the very top choose the project drop down. This will open up the project window where you can select an ongoing project or create a new one. Each service account is located in a project. After you create a service account, you cannot move it to a different project.

In the Google Developers console (link) under the leftmost APIs & Services menu you will find the credentials option. At the top of the screen click the Create Credentials option. You will have a choice as to which type of credential you would like to create. Select service account. Most, though not all, API’s should work with service accounts.

Service Account Authentication

There are a number of ways to authenticate an application using a service account (link). If you are deployed on Google cloud you can use an attached service account. You can use a Workload Identity with Kubernetes pods. There is also the Workloads Identity Federation that works with other service providers. Here we will be using a service account key which allows us to deploy on our own cloud provider.

Create a Public-Private key.

Click on your service account name to bring up the management menu. There is a number of menus near the top of the screen; select keys. Press the add keys dropdown menu and select create new key. Use json unless you have a reason to do otherwise. Remember not to add this key to your git repository or put it anywhere the public can see it.

Create and Share a Calendar

Create a calendar in Google Calendars and look at it’s settings (3 little dots next to the name). Find “share the calendar with specific people” and add the email address of your service account. Give your service account the role “make changes to events”. Since the service account isn’t a real user, you don’t get a confirmation email, and the calendar won’t immediately show up using the Google Calendar API list method.

Find the calendar id under the integrate calendar heading on the settings page. It should look like an email address. Save this, we will be using it in a bit.

Getting Started

Install the google api package.

npm install googleapis

View the Google APIs documentation on github.

https://github.com/googleapis/google-api-nodejs-client

In the Google console, you will find “APIs and Services > Enabled APIs and Services” add the Calendar API to your project.

Code Pre-requisites

The following are three preliminary steps we need to preform before accessing the calendar API. After this we can start accessing the API methods. I will implement them in a class structure just to keep things clean.

  • Import the google api library
  • Create a new authentication object.
  • Obtain an instance of the calendar interface.
import { google } from "googleapis";
constructor() {
    const auth = new google.auth.GoogleAuth({
        keyFilename: GoogleCalendar.KEY_FILENAME,
        scopes: GoogleCalendar.SCOPES,
    });

    this.calendar = google.calendar({
        version: "v3",
        auth: auth,
    });
}

API Methods

Now that the environment is setup we will now go through a few of the available API calls. For a full list of see the Calendar API Documentation.

List

The list method allows you to view available calendars. It is found in the calendar.calendarList implementation. This is also where you find the create, get, and delete calendar methods. This is one of the simpler API calls.

list() {
    return new Promise((resolve, reject) => {
        this.calendar.calendarList.list((err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });
}

Insert

Newly added calendars won’t show up in the list until you have inserted them. We use the calendar identifier we saved above. In this case we also pass in an options object, containing a resource object, which in turn has the calendar id we want to add.

insert(id) {
    return new Promise((resolve, reject) => {
        const options = {
            resource: {
                id: id,
            },
        };

        this.calendar.calendarList.insert(options, (err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });
}

Delete

When you look at the API documentation for delete you notice that calendar id is in the url. This implies that we use a calendarId field in our options object. The resource field, as used above, goes in the body. This is something to watch out for when interpreting HTTP calls to NodeJS API calls.

remove(id) {
    return new Promise((resolve, reject) => {
        const options = {
            calendarId: id,
        };

        this.calendar.calendarList.delete(options, (err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });
}

Add Event

To add an event we access the events property of the calendar API. In this case we include both the calendar id in the URL as well as body properties.

addEvent(id, start, end, summary){
    return new Promise((resolve, reject) => {
        const options = {
            calendarId: id,
            resource: {
                start: {
                    date: "2022-05-21",
                },
                end: {
                    date: "2022-05-23",
                },
                summary: summary,
            },                
        };

        this.calendar.events.insert(options, (err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });        
}

References

Google Event Object

Google Calendar Create Event

Summation of a Partial Beatty Sequence

 \sum_{i=1}^{n}\mathcal{B}_{r\in\mathbb{Z}}^i = \sum_{i=1}^{n}\lfloor ir \rfloor

A partial Beatty sequence is the sum of all the Integer portions of positive multiples between 1 and n of an Irrational number.

Raleigh-Beatty Theorem

Rayleigh’s theorem, states that a sequence, consisting of all the positive integers that are not in the sequence, is itself a Beatty sequence generated by a different irrational number.

For any Beatty \mathcal{B}_r^n sequence where r > 1 there exists a complementary sequence \mathcal{B}_s^n where s>1 is defined as \frac{1}{r}+\frac{1}{s}=1. The union of both sequences is the set of all Integers from 1 to n.

We can find an equation for s in terms of r.

\begin{align}
\begin{gather*}
\frac{1}{s} + \frac{1}{r} = 1 \\ \downarrow \\
\frac{1}{s} = 1 - \frac{1}{r} \\ \downarrow \\
\frac{1}{s} = \frac{r}{r} - \frac{1}{r} \\ \downarrow \\
\frac{1}{s} = \frac{r-1}{r} \\ \downarrow \\
s = \frac{r}{r-1}
\end{gather*}
\end{align}

The complement consists of all integers not in the sequence. For any sequence dividing the value by the equation will give you the index of the value that is less than N = \mathcal{B}_r^n. This allows us to find the last index of the complement because the last value of the complement is lower than that of the sequence. \lfloor \frac{\mathcal{B}_r^i}{s} \rfloor.

\lfloor ir \rfloor = \mathcal{B}_r^i \rightarrow \lfloor \frac{\mathcal{B}_r^i}{r} \rfloor = i-1

\mathcal{B}_r^n is the largest term in the sequence, dividing that by ‘r’ would give you the index of the sequence .

For clarity we will express the largest value of the sequence as N = \mathcal{B}_r^i , and the upper index of the complement as m = \lfloor N/s \rfloor.

\begin{align}
\begin{gather*}
N = \mathcal{B}_r^n\\ \\
m = \lfloor N/s \rfloor\\ \\
\sum_{i=1}^{n}\lfloor ir \rfloor + 
\sum_{j=1}^m
\lfloor js \rfloor = 
\frac{N^2 + N}{2}
\end{gather*}
\end{align}

Building up to the Solution

Since the union of any Beatty sequence and its complement will cover all the positive integers up to n. This leads to the realization that any Integer between 1 and n can be expressed at either \lfloor ir \rfloor or \lfloor j(r+1) \rfloor.

The sum of the sequence in question:

\begin{align}
S(n, r) = \sum_{i=1}^{n}\mathcal{B}_r^i
\end{align}

The sum of the complementary sequence:

\begin{align}
S(m, s) = \sum_{j=1}^{m}\mathcal{B}_s^j | s = \frac{r}{r-1}
\end{align}

By way of the Raleigh-Beatty Theorem, the two sums are equivalent to the sum of the Integers from 1 to the highest Beatty sequence value (\mathcal{B}_r^n). This can be rearrange to say; the sum of a Beaty sequence is equal to the difference sum of all the encompassed Integers and the sequence’s complement.

\begin{align}
\begin{gather*}
S(n,r) + S(n,s) =
\sum_{i=1}^Ni \\
\downarrow \\
S(n,r) = 
\sum_{i=1}^Ni - 
S(n,s) 

\end{gather*}
\end{align}

At this point we want to express the sum of the complement in terms of sum the original sequence. We will do this by extracting \mathcal{B}_r^i from \mathcal{B}_i^s. So we need to define a value for s, we will use \sqrt{2}.

Expressing the Complement in Terms of the Original

Starting with r and s (1), we assign a value to r.

\begin{align}
r = \sqrt{2}, s= \frac{r}{r-1}
\end{align}

Substitute r for \sqrt{2}

\begin{align}
s= \frac{\sqrt{2}}{\sqrt{2}-1}
\end{align}

Factor out \sqrt{2}

\begin{align}
\begin{gather*}
s= 1-\frac{\sqrt{2}(\sqrt{2}+1)}{(\sqrt{2}-1)(\sqrt{2}+1)} \\
\downarrow\\
s= \frac{2+\sqrt{2}}{1} \\
\downarrow\\
 s=2+\sqrt{2}
\end{gather*}
\end{align}

Whole integers can be brought out of the floor function.

\begin{align}
\begin{gather*}
\mathcal{B}_s^i = \lfloor is\rfloor=\lfloor i(2+\sqrt{2})\rfloor \\
\downarrow \\
\lfloor 2i+i\sqrt{2}\rfloor \\
\downarrow \\
2i+\lfloor i\sqrt{2}\rfloor \\
\end{gather*}
\end{align}

Express \mathcal{B}_s^i in terms of \mathcal{B}_r^i.

\begin{align}
\begin{gather*}
\text{since } \mathcal{B}_r^i=\lfloor i\sqrt{2}\rfloor \\ 
\downarrow\\
\mathcal{B}_s^i = \mathcal{B}_r^i + 2i
\end{gather*}
\end{align}

Recursive Function

Express the complement sum in terms of the original sum.

\begin{align}
\begin{gather*}
S(n,s)=
\sum_{i=1}^n \mathcal{B}_s^i\\
\downarrow\\
\sum_{i=1}^n(\mathcal{B}_r^i+2i) \\
\downarrow\\
\sum_{i=1}^n\mathcal{B}_r^i+\sum_{i=1}^n2i \\
\downarrow\\
\sum_{i=1}^n\mathcal{B}_r^i+2\sum_{i=1}^ni \\
\downarrow\\
S(n, r) + 2(\frac{n(n+1)}{2}) \\
\downarrow\\
S(n,s)=S(n, r) + n(n+1)

\end{gather*}
\end{align}

From (5) using (11)

\begin{align}
\begin{gather*}
S(n,r) =  
\sum_{i=1}^Ni -
S(m,s) \\
\downarrow \\
S(n,r) = 
\sum_{i=1}^Ni - S(m,r) - m(m-1)
\end{gather*}
\end{align}

Expressed as a recursive function.

\mathcal{S}(n, r)= 
\begin{cases}
0 \leq n \leq 1 & 1 \\
n > 1 & 
\sum_{i=1}^Ni - S(m,r) - m(m-1)
\end{cases}

External References

https://en.wikipedia.org/wiki/Beatty_sequence

WSL and Powershell Ports

Connecting to WSL from Windows

In WSL

$ nc -l 8080
hello world

In Powershell

$ ncat 127.0.0.1 8080
$ hello world

In WSL the nc -l 8080 command listens on port 8080 for any incoming data, then prints it to the screen. In Powershell ncat 127.0.0.1 8080 sends everything you type to port 8080. You should see what you type both in the Powershell terminal and in the WSL terminal. Alternately you can open your browser and enter http://127.0.0.1:8080/ into your URL bar, and the WSL terminal will print out the HTML request.

Connecting to Windows from WSL

Install windows version of ncat [direct link] from https://nmap.org/.

In Powershell

$ ncat -l 8080
hello world

In WSL

$ cat /etc/resolv.conf
nameserver 172.59.192.1

$ nc 172.59.192.1 8080
$ hello world

phpMyAdmin

About

The phpMyAdmin tool is an SQL database manager written in PHP. It resides in the web root on your server, making it easily accessed from a web browser.

There are a number important security considerations when using software like phpMyAdmin, since it:

  • Communicates directly with your MySQL installation.
  • Handles authentication using MySQL credentials.
  • Executes and returns results for arbitrary SQL queries.

The phpMyAdmin tool is available for a number of different operating systems. This article will focus solely on the Linux installation.

Manual Installation

While there are a number of ways to install phpMyAdmin, there is an easy quick install method. This involves downloading and extracting it directly to your web root directory. While you will need at least basic bash terminal knowledge, it is relatively trivial to set up. However, you will require sudo privileges or access to the web-user-account.

Following are the condensed steps for the quick install found in the phpMyAdmin documentation.

cd /www
sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.1.1/phpMyAdmin-5.1.1-english.tar.gz -O phpMyAdmin.tar.gz
sudo tar -xvf phpMyAdmin.tar.gz
sudo chown -R wp-user:wp-user phpMyAdmin-5.1.1-english/
sudo cp config.sample.inc.php config.inc.php
  1. Change directory to your web root.
  2. Download the tar file. You can choose from a number of options here. We rename it here with the -O flag
  3. Unpack the tar, it will be placed in it’s own directory.
  4. Give ownership of the directory to the web user account.
  5. Create the configuration file by copying the sample configuration file.

Usage

When a user logs into phpMyAdmin the username and password are sent directly to the SQL database. It is just an interface to the database, and any operation it does can be done on the command line. As such, all users must be valid database users.

Securing

You should use cookie mode authentication so that your user/password pair are not kept in the configuration file. The variable may be set in the example config file as:

$cfg['Servers'][$i]['auth_type'] = 'cookie';

You will need to also add a 'blowfish secret' value to the config file.

$cfg['blowfish_secret'] = 'anyrandomtextyouwant';

Deny access to the temp, libraries, and templates subdirectories. Put the following in the server directive of your nginx enabled sites file.

location /phpmyadmin/libraries { deny all; }
location /phpmyadmin/templates { deny all; }
location /phpmyadmin/tmp { deny all; }

References

https://www.phpmyadmin.net/

https://docs.phpmyadmin.net/en/latest

Installing NodeJS Manually on Linux

If you are installing NodeJS on a machine that you have admin privileges to, you can install NodeJS by downloading the binaries directly. This allows you to bypass any special setup that a version manager may require. Browse to https://nodejs.org/en/download/ and download the linux binaries. Alternatively use wget.

cd /opt/node
sudo wget https://nodejs.org/dist/v17.0.1/node-v17.0.1-linux-x64.tar.xz
sudo tar -xvf node-v17.0.1-linux-x64.tar.xz
sudo mv node-v17.0.1-linux-x64 17.0.1

You would of course replace the version numbers with the version you are interested in. Also, the move (mv) command is not required, I just prefer to keep the version numbers simple. If this is your first NodeJS install you will need to add sudo mkdir /opt/node.

After you have installed the binary files into the /opt directory you will want to create links so that you can execute them from anywhere.

sudo ln -s /opt/node/17.0.1/bin/node node
sudo ln -s /opt/node/17.0.1/bin/npm npm
sudo ln -s /opt/node/17.0.1/bin/npx npx

JS Array of Objects to Dictionary

We will examine what is going on with this snippet of code? But first, what is the goal. The files variable is an array of objects. Each object has two fields: ‘name’, and ‘fullpath’. We want to return a dictionary with ‘name’ as the key, and the ‘fullpath’ as the value. We will look at this snippet of code piecemeal.

const fileMap = Object.assign({}, ...files.map(x=>({[x.name] : x.fullpath})));

Spread Syntax

The JS spread syntax allows iterable expressions, such as an array, to be expanded into an argument list during a function call. A simplified example is shown below.

const A = [1, 3, 5];
function sum(a, b, c){
      return a + b + c;
}
console.log(sum(...A));
> 9

In the case of the Object.assign function the first parameter to the object to write to, all remaining arguments are source objects. Using the spread syntax on the map function (which returns an array of objects) passes each object into ‘assign’ treating them each as individual sources.

Array.prototype.map

The map function of Array, accepts a callback function and return a new array. It fills each element of the array by passing the source value into the callback function. You can achieve the same thing by iterating through the array.

const A = [1, 1, 3, 5, 8];
const B = [];
for (let i = 0; i < A.length; i++){
      B[i] = A[i] + i;
}

Equivalent operation using map:

const A = [1, 1, 3, 5, 8];
const B = A.map((x, i)=>x + i);

Lambda Functions

The snippet x=>({[x.name] : x.fullpath}) seems a little odd at first. Why are there parentheses around the curly braces. To understand this we need to look at the two ways JS allows lambda functions.

Implicit return statement:

x => {return x + 1}

Implied return statement:

x => x + 1

An implicit return statement has curly braces, while the implied does not. Because the object constructor uses the same braces as the function body, if you were to do this: x => {"name" : "apple"} JS would think you are trying to do an implicit-return lambda function. So we need to encapsulate the object declaration with parentheses to turn that statement into an implied-return lambda function: x => ({"name" : "apple"}).

ES2016 Computed Property Name Shorthand

The second oddity of the statement x=>({[x.name] : x.fullpath}) , is why is [x.name] in square brackets. This was introduced in ES1015 as shorthand to tell JS to use the value in the variable and not to treat it as a key value. This is a bit more obvious if we look at it outside of the context of a lambda function.

Given an object with two fields, you want to use one of the fields for the key of a new object. The vanilla JS way of things:

const key = "name";
const obj2 = {};
obj2[key] = "apple";
console.log(obj2);
> { name: 'apple' }

The shorthand way of doing this, which fits nicely into a compact lambda function, is as follows:

const key = "name";
const obj2 = {[key] : "apple"};

NodeJS Code Coverage With C8

Code coverage allows you to determine how well unit-tests cover your codebase. The standard code coverage suite for JavaScript is the NYC interface for Istanbul. However, this package doesn’t play well with NodeJS modules. Instead we are going to use the C8 package. C8 uses functionality native to NodeJS to create output that is compatible with Istanbul reporters.

Installation

npm i --save-dev c8

Execution

You can run c8 on any source file by simply passing in the appropriate node command.

$ npx c8 node src/index.js
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files  |   88.23 |      100 |      50 |   88.23 |
 MooYou.js |   84.61 |      100 |      50 |   84.61 | 9-10
 index.js  |     100 |      100 |     100 |     100 |
-----------|---------|----------|---------|---------|-------------------

You can also easily run it with your Mocha test files.

$ npx c8 mocha
  MooYou Class Test
    #who
      ✔ returns 'you'


  1 passing (3ms)

-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files  |   84.61 |      100 |      50 |   84.61 |
 MooYou.js |   84.61 |      100 |      50 |   84.61 | 9-10
-----------|---------|----------|---------|---------|-------------------

Configuration

C8 can be configured from the command line, in your package.json file, or in a configuration file in your project directory. In particular you can instruct C8 to report one files as opposed to just those in your tests (npx c8 --all mocha). You can also specify which files to include and/or ignore. See the configuration documentation for specific details.

Generating Reports

By default the coverage data will be located in the coverage/tmp directory. You can change this directory with the --temp-directory flag. Don’t forget to add this to your .gitignore file.

Run npx c8 report to regenerate reports after c8 has already been run. Use the -r flag to specify which reporter to use, and the -o flag to specify where to output the report. By default generated reports can be found in the coverage/ dirctory.

Vanilla C8 command with HTML report:

npx c8 -r html mocha

Generate C8 report after the fact:

npx c8 -r html report

Two-part C8 command with custom directories and HTML report:

npx c8 --temp-directory .c8 mocha
npx c8 report --temp-directory .c8 -o html_coverage -r html

In Code Markup

You can add comments into your source code that tells c8 to ignore certain portions of code.

Ignoring all lines until told to stop.

/* c8 ignore start */
function foo() {
  ...
}
/* c8 ignore stop */

Ignore next line.

/* c8 ignore next */

External References

https://www.npmjs.com/package/c8

https://github.com/bcoe/c8

GIT Branches Quick Tips

Delete a Remote Branch

git push -d [remote] [branch]

Create a New Branch From a Hash

git branch [branch] [sha]

Force a Push

Use with caution it can delete or overwrite existing commits.

git push -f

Push a New Branch

This creates an upstream tracking branch related to your local branch.

git push -u origin [branch]

Push All Branches

git push --all

Publish Tags to the Remote Repository

git push --tags

Git: Checkout a Single File from a Previous Commit

Often we alter or delete a file that we didn’t really want to. Sometimes this goes unnoticed for a few commits. It’s fairly easy in git to retrieve a specific file from a specific commit.

git checkout <commit> <path>
git checkout f08b32 ./src/main.c
git checkout HEAD~2 ./src/main.c