nodejs

Express – Attaching Data to a Proxy Router

Problem

Forward an HTTP request from one server to another while attaching data to the request. In the example the user will select a file from the Portal-Server and attach the file data on a request to the API-Server. We will also attach the user identification that was returned from the Authentication-Server.

Encoding Type

When you make a POST request, you have to encode the data that forms the body of the request in some way [1]. This encoding type is communicated in the header field enctype, which has one of three values.

  • application/x-www-form-urlencoded – A URL encoded form, this is the default.
  • multipart/form-data – Multipart form with fields separated by a unique value. Used for uploading files.
  • text/plain – Sends the data without any encoding.

We will only concern ourselves with the first two encoding types.

Encoding: application/x-www-form-urlencoded

Submit the POST with the parameters encoded in the body as a string of key-value pairs. Each pair is separated by an ‘&’ and the pair is seperated by ‘=’.

key1=value1&key2=value2

The key value pairs are inserted into the body and can be extracted by the server with the bodyParser package. The key-value pairs can also be directly attached to the URL request, in which case they are retrieved from the req.query field. See implementation.

Encoding: multipart/form-data

Generally used for submitting file, or binary data. We will highlight the differences by manipulating the first example. We will keep everything else the same except for the encoding type. To retrieve the data on the server we will use the multer package.

The only difference in the multipart request as opposed to the urlencoded request is the in the body.

------WebKitFormBoundaryB2AGWn1yYcOCi0T9
Content-Disposition: form-data; name="jobid"

0
------WebKitFormBoundaryB2AGWn1yYcOCi0T9
Content-Disposition: form-data; name="jobname"

sdf
------WebKitFormBoundaryB2AGWn1yYcOCi0T9--

Instead of the ampersand delimited string of parameters, the parameters are now delimited by the unique boundary string “——WebKitFormBoundaryB2AGWn1yYcOCi0T9“. The boundary string is chosen in such a way as it does not appear anywhere else in the body data. Normally the programmer does not need to worry themselves about it, it is generated automatically. The boundary string is send in the HTTP content-type request header, appended to the type by a semicolon. See implementation.

Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryB2AGWn1yYcOCi0T9

Attaching a file to a form

The simplest way to add a file is to include the file-input HTML tag on the form.

<input id="upload-dialog" type="file" name="fileupload">

The raw file is added to the body preceded by some file information. This is similar to how data fields get appended to the body.

------WebKitFormBoundaryvI9zg1SrRVBgB6yG
Content-Disposition: form-data; name="fileupload"; filename="iama_text_file.txt"
Content-Type: text/plain

In here is some simple text.
Two lines to fancy it up a bit.

------WebKitFormBoundaryvI9zg1SrRVBgB6yG--

To retrieve the file on the server we will again use the multer package, except this time we’ll use the ‘single‘ method. This will place the file in the ‘uploads‘ subdirectory. Note, the parameter we pass matches the name we gave to the upload-dialog above.

multer({ dest: 'uploads/' }).single('fileupload')

If you now examine the uploads/ subdirectory, you’ll find a file with a hash file for a name. This is because do not want to save raw filenames to your file-system. The uploaded file information is stored in the req.file field (see below). You can file the saved filename as well as the original filename here. You can sanitize the filename and move or copy the file as you see fit.

fieldname: 'fileupload',
originalname: 'iama_text_file.txt',
encoding: '7bit',
mimetype: 'text/plain',
destination: 'uploads/',
filename: '5f66c866e68790a2c494275829acd0b8',
path: 'uploads/5f66c866e68790a2c494275829acd0b8',
size: 61

Attaching File Data to a Proxy

Now that we understand the formatting of a file-upload, we want to manipulate a HTTP request to dynamically attach file data. The situation is that a file was previously uploaded and stored on the Portal-Server. Now we want to send that file to an API-Server with whatever other information is required. Doing this allows us to test the API-Server without special consideration, simply by using a web form (as we’ve been doing). Then when we deploy the API-Server we firewall it so that only the Portal-Server sees it. But, we proxy the call from the Portal-Server to the API-Server.

To facilitate proxies we are going to use the http-proxy-middleware package. To view the full code visit the github repository for this project.

Proxy Request

The basic proxy requires a options object. The one we’ll be using is presented below. There are many more options available in the documentation.

  • target – Target host name.
  • pathRewrite – Regex rewrite of the target’s url path.
  • onProxyReq – The function to manipulate the request.
{    
    target: "http://localhost:8000",
    pathRewrite: { 'forward': 'raw' },

    onProxyReq: appendFile
}

We will only be attaching the proxy to a single url. We call ‘multer’ first to extract existing parameters. Then we call the proxy, passing in the options object. This is all that is required for a simple proxy. What remains is attaching the file and data information to the request.

router.post("/forward",
    multer({}).none(),
    createProxyMiddleware(proxyOptions),
);

Attach Data to Request Body

The onProxyReq function accepts three parameters: proxyReq, req, and res. We will only be concerned with the first two.

function appendFile(proxyReq, req, res) {
    const args = {
        ...req.body,
        userid: "user@id"
    }

    const filepath = Path.join(".", "uploads", req.body.filename);
    const formData = new FormData();

    Object.keys(args).map(key => formData.append(key, args[key]));
    formData.append('fileupload', FS.readFileSync(filepath), 'file.txt');

    proxyReq.setHeader('content-type', `multipart/form-data; boundary=${formData.getBoundary()}`);
    proxyReq.setHeader('content-length', formData.getBuffer().length);

    proxyReq.write(formData.getBuffer().toString("utf-8"));
    proxyReq.end();
}

(8) The FormData class is a library to create readable "multipart/form-data" streams. Can be used to submit forms and file uploads to other web applications.

(10) We add all values to the FormData object. We preserve old values (3) and append new values (4). This is optional and can be omitted if you want to remove the previous parameters.

(11) The file data is added to the FormData object using the readFileSync function which returns a Buffer object.

(13) We manually set the ready on the proxy request particularly to specify the boundary string.

(14) The FormData object will also provide us with the buffer length.

(16) The buffered FormData object is written out to proxyReq.

External References

[1] https://www.w3.org/html/wg/spec/association-of-controls-and-forms.html#attr-fs-enctype

[2] https://nodejs.org/api/http.html#http_class_http_clientrequest

https://dev.to/sidthesloth92/understanding-html-form-encoding-url-encoded-and-multipart-forms-3lpa

https://stackoverflow.com/questions/37630419/how-to-handle-formdata-from-express-4

https://stackoverflow.com/questions/64803772/nodejs-fileupload-multer-vs-express-fileupload-which-to-use

https://stackoverflow.com/questions/72809300/how-to-add-user-data-in-http-proxy-middleware

Google API Service Accounts With NodeJS

For a recent project I needed update to a single shared Google calendar. I decided to go with a Google service account to manage this. I’m going to post here how I did it.

What is a Service Account

A service account is used by an application as opposed to a real living person. It allows your application to make authorized API calls directly. A service account is identified by its email address, which is unique to the account.

  • Service accounts can not be logged into with browsers.
  • Authorization is done with a public-private RSA key.
  • Service accounts are not part of your Google domain. They do not share global resources like a live user.

Create a Service Account

At the very top choose the project drop down. This will open up the project window where you can select an ongoing project or create a new one. Each service account is located in a project. After you create a service account, you cannot move it to a different project.

In the Google Developers console (link) under the leftmost APIs & Services menu you will find the credentials option. At the top of the screen click the Create Credentials option. You will have a choice as to which type of credential you would like to create. Select service account. Most, though not all, API’s should work with service accounts.

Service Account Authentication

There are a number of ways to authenticate an application using a service account (link). If you are deployed on Google cloud you can use an attached service account. You can use a Workload Identity with Kubernetes pods. There is also the Workloads Identity Federation that works with other service providers. Here we will be using a service account key which allows us to deploy on our own cloud provider.

Create a Public-Private key.

Click on your service account name to bring up the management menu. There is a number of menus near the top of the screen; select keys. Press the add keys dropdown menu and select create new key. Use json unless you have a reason to do otherwise. Remember not to add this key to your git repository or put it anywhere the public can see it.

Create and Share a Calendar

Create a calendar in Google Calendars and look at it’s settings (3 little dots next to the name). Find “share the calendar with specific people” and add the email address of your service account. Give your service account the role “make changes to events”. Since the service account isn’t a real user, you don’t get a confirmation email, and the calendar won’t immediately show up using the Google Calendar API list method.

Find the calendar id under the integrate calendar heading on the settings page. It should look like an email address. Save this, we will be using it in a bit.

Getting Started

Install the google api package.

npm install googleapis

View the Google APIs documentation on github.

https://github.com/googleapis/google-api-nodejs-client

In the Google console, you will find “APIs and Services > Enabled APIs and Services” add the Calendar API to your project.

Code Pre-requisites

The following are three preliminary steps we need to preform before accessing the calendar API. After this we can start accessing the API methods. I will implement them in a class structure just to keep things clean.

  • Import the google api library
  • Create a new authentication object.
  • Obtain an instance of the calendar interface.
import { google } from "googleapis";
constructor() {
    const auth = new google.auth.GoogleAuth({
        keyFilename: GoogleCalendar.KEY_FILENAME,
        scopes: GoogleCalendar.SCOPES,
    });

    this.calendar = google.calendar({
        version: "v3",
        auth: auth,
    });
}

API Methods

Now that the environment is setup we will now go through a few of the available API calls. For a full list of see the Calendar API Documentation.

List

The list method allows you to view available calendars. It is found in the calendar.calendarList implementation. This is also where you find the create, get, and delete calendar methods. This is one of the simpler API calls.

list() {
    return new Promise((resolve, reject) => {
        this.calendar.calendarList.list((err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });
}

Insert

Newly added calendars won’t show up in the list until you have inserted them. We use the calendar identifier we saved above. In this case we also pass in an options object, containing a resource object, which in turn has the calendar id we want to add.

insert(id) {
    return new Promise((resolve, reject) => {
        const options = {
            resource: {
                id: id,
            },
        };

        this.calendar.calendarList.insert(options, (err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });
}

Delete

When you look at the API documentation for delete you notice that calendar id is in the url. This implies that we use a calendarId field in our options object. The resource field, as used above, goes in the body. This is something to watch out for when interpreting HTTP calls to NodeJS API calls.

remove(id) {
    return new Promise((resolve, reject) => {
        const options = {
            calendarId: id,
        };

        this.calendar.calendarList.delete(options, (err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });
}

Add Event

To add an event we access the events property of the calendar API. In this case we include both the calendar id in the URL as well as body properties.

addEvent(id, start, end, summary){
    return new Promise((resolve, reject) => {
        const options = {
            calendarId: id,
            resource: {
                start: {
                    date: "2022-05-21",
                },
                end: {
                    date: "2022-05-23",
                },
                summary: summary,
            },                
        };

        this.calendar.events.insert(options, (err, res) => {
            if (err) reject(err);
            if (res) resolve(res.data);
        });
    });        
}

References

Google Event Object

Google Calendar Create Event

NodeJS Code Coverage With C8

Code coverage allows you to determine how well unit-tests cover your codebase. The standard code coverage suite for JavaScript is the NYC interface for Istanbul. However, this package doesn’t play well with NodeJS modules. Instead we are going to use the C8 package. C8 uses functionality native to NodeJS to create output that is compatible with Istanbul reporters.

Installation

npm i --save-dev c8

Execution

You can run c8 on any source file by simply passing in the appropriate node command.

$ npx c8 node src/index.js
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files  |   88.23 |      100 |      50 |   88.23 |
 MooYou.js |   84.61 |      100 |      50 |   84.61 | 9-10
 index.js  |     100 |      100 |     100 |     100 |
-----------|---------|----------|---------|---------|-------------------

You can also easily run it with your Mocha test files.

$ npx c8 mocha
  MooYou Class Test
    #who
      ✔ returns 'you'


  1 passing (3ms)

-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files  |   84.61 |      100 |      50 |   84.61 |
 MooYou.js |   84.61 |      100 |      50 |   84.61 | 9-10
-----------|---------|----------|---------|---------|-------------------

Configuration

C8 can be configured from the command line, in your package.json file, or in a configuration file in your project directory. In particular you can instruct C8 to report one files as opposed to just those in your tests (npx c8 --all mocha). You can also specify which files to include and/or ignore. See the configuration documentation for specific details.

Generating Reports

By default the coverage data will be located in the coverage/tmp directory. You can change this directory with the --temp-directory flag. Don’t forget to add this to your .gitignore file.

Run npx c8 report to regenerate reports after c8 has already been run. Use the -r flag to specify which reporter to use, and the -o flag to specify where to output the report. By default generated reports can be found in the coverage/ dirctory.

Vanilla C8 command with HTML report:

npx c8 -r html mocha

Generate C8 report after the fact:

npx c8 -r html report

Two-part C8 command with custom directories and HTML report:

npx c8 --temp-directory .c8 mocha
npx c8 report --temp-directory .c8 -o html_coverage -r html

In Code Markup

You can add comments into your source code that tells c8 to ignore certain portions of code.

Ignoring all lines until told to stop.

/* c8 ignore start */
function foo() {
  ...
}
/* c8 ignore stop */

Ignore next line.

/* c8 ignore next */

External References

https://www.npmjs.com/package/c8

https://github.com/bcoe/c8

Updating NPM Packages

Updating NPM Packages

  1. Navigate to the root directory of your project and ensure it contains a package.json file:
    • cd /path/to/project
  2. In your project root directory, run the outdated command to view outdated packages:
    • npm outdated
  3. In your project root directory, run the update command:
    • npm update

package.json

The way the wanted version is set in the package.json file may affect how updates are decided.

Version numbers are declared as a [major, minor, patch] tuple.

Caret Ranges ^1.2.3

Allows changes that do not modify the left-most non-zero element of the version specification. So 1.3.1 will update to 1.3.2 or 1.4.0. Though, 0.3.0 will update to 0.3.1 but not update to 0.4.0.

Version will update if:

  • ^1.2.3 : 1.2.3 < version < 2.0.0
  • ^0.3.0 : 0.3.0 < version < 0.4.0

X-Ranges

Allows ‘X’, ‘x’, or ‘*’ to stand in for a number in the version specification.

Version will update if:

  • “” : any version
    • same as “*” or “x.x.x”
  • 1.x.x : 1.0.0 < version < 2.0.0
    • same as “1”
  • 1.2.x : 1.2.0 < version < 1.3.0
    • same as “1.2”