Frar Tech Blog

Problem

Forward an HTTP request from one server to another while attaching data to the request. In the example the user will select a file from the Portal-Server and attach the file data on a request to the API-Server. We will also attach the user identification that was returned from the Authentication-Server.

Encoding Type

When you make a POST request, you have to encode the data that forms the body of the request in some way [1]. This encoding type is communicated in the header field enctype, which has one of three values.

• application/x-www-form-urlencoded – A URL encoded form, this is the default.
• multipart/form-data – Multipart form with fields separated by a unique value. Used for uploading files.
• text/plain – Sends the data without any encoding.

We will only concern ourselves with the first two encoding types.

Encoding: application/x-www-form-urlencoded

Submit the POST with the parameters encoded in the body as a string of key-value pairs. Each pair is separated by an ‘&’ and the pair is seperated by ‘=’.

key1=value1&key2=value2

The key value pairs are inserted into the body and can be extracted by the server with the bodyParser package. The key-value pairs can also be directly attached to the URL request, in which case they are retrieved from the req.query field. See implementation.

Encoding: multipart/form-data

Generally used for submitting file, or binary data. We will highlight the differences by manipulating the first example. We will keep everything else the same except for the encoding type. To retrieve the data on the server we will use the multer package.

The only difference in the multipart request as opposed to the urlencoded request is the in the body.

------WebKitFormBoundaryB2AGWn1yYcOCi0T9
Content-Disposition: form-data; name="jobid"

0
------WebKitFormBoundaryB2AGWn1yYcOCi0T9
Content-Disposition: form-data; name="jobname"

sdf
------WebKitFormBoundaryB2AGWn1yYcOCi0T9--

Instead of the ampersand delimited string of parameters, the parameters are now delimited by the unique boundary string “——WebKitFormBoundaryB2AGWn1yYcOCi0T9“. The boundary string is chosen in such a way as it does not appear anywhere else in the body data. Normally the programmer does not need to worry themselves about it, it is generated automatically. The boundary string is send in the HTTP content-type request header, appended to the type by a semicolon. See implementation.

Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryB2AGWn1yYcOCi0T9

Attaching a file to a form

The simplest way to add a file is to include the file-input HTML tag on the form.

<input id="upload-dialog" type="file" name="fileupload">

The raw file is added to the body preceded by some file information. This is similar to how data fields get appended to the body.

------WebKitFormBoundaryvI9zg1SrRVBgB6yG
Content-Disposition: form-data; name="fileupload"; filename="iama_text_file.txt"
Content-Type: text/plain

In here is some simple text.
Two lines to fancy it up a bit.

------WebKitFormBoundaryvI9zg1SrRVBgB6yG--

To retrieve the file on the server we will again use the multer package, except this time we’ll use the ‘single‘ method. This will place the file in the ‘uploads‘ subdirectory. Note, the parameter we pass matches the name we gave to the upload-dialog above.

multer({ dest: 'uploads/' }).single('fileupload')

If you now examine the uploads/ subdirectory, you’ll find a file with a hash file for a name. This is because do not want to save raw filenames to your file-system. The uploaded file information is stored in the req.file field (see below). You can file the saved filename as well as the original filename here. You can sanitize the filename and move or copy the file as you see fit.

fieldname: 'fileupload',
originalname: 'iama_text_file.txt',
encoding: '7bit',
mimetype: 'text/plain',
destination: 'uploads/',
filename: '5f66c866e68790a2c494275829acd0b8',
path: 'uploads/5f66c866e68790a2c494275829acd0b8',
size: 61

Attaching File Data to a Proxy

Now that we understand the formatting of a file-upload, we want to manipulate a HTTP request to dynamically attach file data. The situation is that a file was previously uploaded and stored on the Portal-Server. Now we want to send that file to an API-Server with whatever other information is required. Doing this allows us to test the API-Server without special consideration, simply by using a web form (as we’ve been doing). Then when we deploy the API-Server we firewall it so that only the Portal-Server sees it. But, we proxy the call from the Portal-Server to the API-Server.

To facilitate proxies we are going to use the http-proxy-middleware package. To view the full code visit the github repository for this project.

Proxy Request

The basic proxy requires a options object. The one we’ll be using is presented below. There are many more options available in the documentation.

• target – Target host name.
• pathRewrite – Regex rewrite of the target’s url path.
• onProxyReq – The function to manipulate the request.

{    
    target: "http://localhost:8000",
    pathRewrite: { 'forward': 'raw' },

    onProxyReq: appendFile
}

We will only be attaching the proxy to a single url. We call ‘multer’ first to extract existing parameters. Then we call the proxy, passing in the options object. This is all that is required for a simple proxy. What remains is attaching the file and data information to the request.

router.post("/forward",
    multer({}).none(),
    createProxyMiddleware(proxyOptions),
);

Attach Data to Request Body

The onProxyReq function accepts three parameters: proxyReq, req, and res. We will only be concerned with the first two.

function appendFile(proxyReq, req, res) {
    const args = {
        ...req.body,
        userid: "user@id"
    }

    const filepath = Path.join(".", "uploads", req.body.filename);
    const formData = new FormData();

    Object.keys(args).map(key => formData.append(key, args[key]));
    formData.append('fileupload', FS.readFileSync(filepath), 'file.txt');

    proxyReq.setHeader('content-type', `multipart/form-data; boundary=${formData.getBoundary()}`);
    proxyReq.setHeader('content-length', formData.getBuffer().length);

    proxyReq.write(formData.getBuffer().toString("utf-8"));
    proxyReq.end();
}

(8) The FormData class is a library to create readable "multipart/form-data" streams. Can be used to submit forms and file uploads to other web applications.

(10) We add all values to the FormData object. We preserve old values (3) and append new values (4). This is optional and can be omitted if you want to remove the previous parameters.

(11) The file data is added to the FormData object using the readFileSync function which returns a Buffer object.

(13) We manually set the ready on the proxy request particularly to specify the boundary string.

(14) The FormData object will also provide us with the buffer length.

(16) The buffered FormData object is written out to proxyReq.

External References

[1] https://www.w3.org/html/wg/spec/association-of-controls-and-forms.html#attr-fs-enctype

[2] https://nodejs.org/api/http.html#http_class_http_clientrequest

https://dev.to/sidthesloth92/understanding-html-form-encoding-url-encoded-and-multipart-forms-3lpa

https://stackoverflow.com/questions/37630419/how-to-handle-formdata-from-express-4

https://stackoverflow.com/questions/64803772/nodejs-fileupload-multer-vs-express-fileupload-which-to-use

https://stackoverflow.com/questions/72809300/how-to-add-user-data-in-http-proxy-middleware