Node JS

Express – Attaching Data to a Proxy Router

Problem

Forward an HTTP request from one server to another while attaching data to the request. In the example the user will select a file from the Portal-Server and attach the file data on a request to the API-Server. We will also attach the user identification that was returned from the Authentication-Server.

Encoding Type

When you make a POST request, you have to encode the data that forms the body of the request in some way [1]. This encoding type is communicated in the header field enctype, which has one of three values.

  • application/x-www-form-urlencoded – A URL encoded form, this is the default.
  • multipart/form-data – Multipart form with fields separated by a unique value. Used for uploading files.
  • text/plain – Sends the data without any encoding.

We will only concern ourselves with the first two encoding types.

Encoding: application/x-www-form-urlencoded

Submit the POST with the parameters encoded in the body as a string of key-value pairs. Each pair is separated by an ‘&’ and the pair is seperated by ‘=’.

key1=value1&key2=value2

The key value pairs are inserted into the body and can be extracted by the server with the bodyParser package. The key-value pairs can also be directly attached to the URL request, in which case they are retrieved from the req.query field. See implementation.

Encoding: multipart/form-data

Generally used for submitting file, or binary data. We will highlight the differences by manipulating the first example. We will keep everything else the same except for the encoding type. To retrieve the data on the server we will use the multer package.

The only difference in the multipart request as opposed to the urlencoded request is the in the body.

------WebKitFormBoundaryB2AGWn1yYcOCi0T9
Content-Disposition: form-data; name="jobid"

0
------WebKitFormBoundaryB2AGWn1yYcOCi0T9
Content-Disposition: form-data; name="jobname"

sdf
------WebKitFormBoundaryB2AGWn1yYcOCi0T9--

Instead of the ampersand delimited string of parameters, the parameters are now delimited by the unique boundary string “——WebKitFormBoundaryB2AGWn1yYcOCi0T9“. The boundary string is chosen in such a way as it does not appear anywhere else in the body data. Normally the programmer does not need to worry themselves about it, it is generated automatically. The boundary string is send in the HTTP content-type request header, appended to the type by a semicolon. See implementation.

Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryB2AGWn1yYcOCi0T9

Attaching a file to a form

The simplest way to add a file is to include the file-input HTML tag on the form.

<input id="upload-dialog" type="file" name="fileupload">

The raw file is added to the body preceded by some file information. This is similar to how data fields get appended to the body.

------WebKitFormBoundaryvI9zg1SrRVBgB6yG
Content-Disposition: form-data; name="fileupload"; filename="iama_text_file.txt"
Content-Type: text/plain

In here is some simple text.
Two lines to fancy it up a bit.

------WebKitFormBoundaryvI9zg1SrRVBgB6yG--

To retrieve the file on the server we will again use the multer package, except this time we’ll use the ‘single‘ method. This will place the file in the ‘uploads‘ subdirectory. Note, the parameter we pass matches the name we gave to the upload-dialog above.

multer({ dest: 'uploads/' }).single('fileupload')

If you now examine the uploads/ subdirectory, you’ll find a file with a hash file for a name. This is because do not want to save raw filenames to your file-system. The uploaded file information is stored in the req.file field (see below). You can file the saved filename as well as the original filename here. You can sanitize the filename and move or copy the file as you see fit.

fieldname: 'fileupload',
originalname: 'iama_text_file.txt',
encoding: '7bit',
mimetype: 'text/plain',
destination: 'uploads/',
filename: '5f66c866e68790a2c494275829acd0b8',
path: 'uploads/5f66c866e68790a2c494275829acd0b8',
size: 61

Attaching File Data to a Proxy

Now that we understand the formatting of a file-upload, we want to manipulate a HTTP request to dynamically attach file data. The situation is that a file was previously uploaded and stored on the Portal-Server. Now we want to send that file to an API-Server with whatever other information is required. Doing this allows us to test the API-Server without special consideration, simply by using a web form (as we’ve been doing). Then when we deploy the API-Server we firewall it so that only the Portal-Server sees it. But, we proxy the call from the Portal-Server to the API-Server.

To facilitate proxies we are going to use the http-proxy-middleware package. To view the full code visit the github repository for this project.

Proxy Request

The basic proxy requires a options object. The one we’ll be using is presented below. There are many more options available in the documentation.

  • target – Target host name.
  • pathRewrite – Regex rewrite of the target’s url path.
  • onProxyReq – The function to manipulate the request.
{    
    target: "http://localhost:8000",
    pathRewrite: { 'forward': 'raw' },

    onProxyReq: appendFile
}

We will only be attaching the proxy to a single url. We call ‘multer’ first to extract existing parameters. Then we call the proxy, passing in the options object. This is all that is required for a simple proxy. What remains is attaching the file and data information to the request.

router.post("/forward",
    multer({}).none(),
    createProxyMiddleware(proxyOptions),
);

Attach Data to Request Body

The onProxyReq function accepts three parameters: proxyReq, req, and res. We will only be concerned with the first two.

function appendFile(proxyReq, req, res) {
    const args = {
        ...req.body,
        userid: "user@id"
    }

    const filepath = Path.join(".", "uploads", req.body.filename);
    const formData = new FormData();

    Object.keys(args).map(key => formData.append(key, args[key]));
    formData.append('fileupload', FS.readFileSync(filepath), 'file.txt');

    proxyReq.setHeader('content-type', `multipart/form-data; boundary=${formData.getBoundary()}`);
    proxyReq.setHeader('content-length', formData.getBuffer().length);

    proxyReq.write(formData.getBuffer().toString("utf-8"));
    proxyReq.end();
}

(8) The FormData class is a library to create readable "multipart/form-data" streams. Can be used to submit forms and file uploads to other web applications.

(10) We add all values to the FormData object. We preserve old values (3) and append new values (4). This is optional and can be omitted if you want to remove the previous parameters.

(11) The file data is added to the FormData object using the readFileSync function which returns a Buffer object.

(13) We manually set the ready on the proxy request particularly to specify the boundary string.

(14) The FormData object will also provide us with the buffer length.

(16) The buffered FormData object is written out to proxyReq.

External References

[1] https://www.w3.org/html/wg/spec/association-of-controls-and-forms.html#attr-fs-enctype

[2] https://nodejs.org/api/http.html#http_class_http_clientrequest

https://dev.to/sidthesloth92/understanding-html-form-encoding-url-encoded-and-multipart-forms-3lpa

https://stackoverflow.com/questions/37630419/how-to-handle-formdata-from-express-4

https://stackoverflow.com/questions/64803772/nodejs-fileupload-multer-vs-express-fileupload-which-to-use

https://stackoverflow.com/questions/72809300/how-to-add-user-data-in-http-proxy-middleware

NodeJS Code Coverage With C8

Code coverage allows you to determine how well unit-tests cover your codebase. The standard code coverage suite for JavaScript is the NYC interface for Istanbul. However, this package doesn’t play well with NodeJS modules. Instead we are going to use the C8 package. C8 uses functionality native to NodeJS to create output that is compatible with Istanbul reporters.

Installation

npm i --save-dev c8

Execution

You can run c8 on any source file by simply passing in the appropriate node command.

$ npx c8 node src/index.js
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files  |   88.23 |      100 |      50 |   88.23 |
 MooYou.js |   84.61 |      100 |      50 |   84.61 | 9-10
 index.js  |     100 |      100 |     100 |     100 |
-----------|---------|----------|---------|---------|-------------------

You can also easily run it with your Mocha test files.

$ npx c8 mocha
  MooYou Class Test
    #who
      ✔ returns 'you'


  1 passing (3ms)

-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files  |   84.61 |      100 |      50 |   84.61 |
 MooYou.js |   84.61 |      100 |      50 |   84.61 | 9-10
-----------|---------|----------|---------|---------|-------------------

Configuration

C8 can be configured from the command line, in your package.json file, or in a configuration file in your project directory. In particular you can instruct C8 to report one files as opposed to just those in your tests (npx c8 --all mocha). You can also specify which files to include and/or ignore. See the configuration documentation for specific details.

Generating Reports

By default the coverage data will be located in the coverage/tmp directory. You can change this directory with the --temp-directory flag. Don’t forget to add this to your .gitignore file.

Run npx c8 report to regenerate reports after c8 has already been run. Use the -r flag to specify which reporter to use, and the -o flag to specify where to output the report. By default generated reports can be found in the coverage/ dirctory.

Vanilla C8 command with HTML report:

npx c8 -r html mocha

Generate C8 report after the fact:

npx c8 -r html report

Two-part C8 command with custom directories and HTML report:

npx c8 --temp-directory .c8 mocha
npx c8 report --temp-directory .c8 -o html_coverage -r html

In Code Markup

You can add comments into your source code that tells c8 to ignore certain portions of code.

Ignoring all lines until told to stop.

/* c8 ignore start */
function foo() {
  ...
}
/* c8 ignore stop */

Ignore next line.

/* c8 ignore next */

External References

https://www.npmjs.com/package/c8

https://github.com/bcoe/c8

Updating NPM Packages

Updating NPM Packages

  1. Navigate to the root directory of your project and ensure it contains a package.json file:
    • cd /path/to/project
  2. In your project root directory, run the outdated command to view outdated packages:
    • npm outdated
  3. In your project root directory, run the update command:
    • npm update

package.json

The way the wanted version is set in the package.json file may affect how updates are decided.

Version numbers are declared as a [major, minor, patch] tuple.

Caret Ranges ^1.2.3

Allows changes that do not modify the left-most non-zero element of the version specification. So 1.3.1 will update to 1.3.2 or 1.4.0. Though, 0.3.0 will update to 0.3.1 but not update to 0.4.0.

Version will update if:

  • ^1.2.3 : 1.2.3 < version < 2.0.0
  • ^0.3.0 : 0.3.0 < version < 0.4.0

X-Ranges

Allows ‘X’, ‘x’, or ‘*’ to stand in for a number in the version specification.

Version will update if:

  • “” : any version
    • same as “*” or “x.x.x”
  • 1.x.x : 1.0.0 < version < 2.0.0
    • same as “1”
  • 1.2.x : 1.2.0 < version < 1.3.0
    • same as “1.2”