Home > Connector Builder > Connector Builder > Multipart Form-Data Upload

NOTE This will be referring to Box’s “Upload File” API endpoint. Documented here.

Anatomy of a Multipart Form-Data Request

Before we dissect what a form-data request looks like, here’s necessary setup:

  • Run nc -l -p 8000 to create a netcat server listening on localhost:8000. This will display the body of our multipart request. If you don’t have nc installed, just look up how to install the GNU netcat utilities onto your system (sorry windows).
  • Create a .txt file called ‘stuff.txt’ with ‘asdf’ as the content.

  • Write this to a .js file (I’m using stuff.js because I’m lazy):

    var request = require("request"),
fs = require("fs");

request({
url: "http://localhost:8000/",
formData: {
key: "value"
}
}, function (err, res, body) {
if (err) console.log("ERROR", err);
else console.log("BODY", body);
process.exit();
});

Now that we have our setup complete, let’s try running node stuff.js. If you look over at the terminal running nc -l -p 8000, you should see something that looks like this:

POST / HTTP/1.1
host: localhost:8000
content-type: multipart/form-data; boundary=--------------------------180610939475792226718191
content-length: 163
Connection: close

----------------------------180610939475792226718191
Content-Disposition: form-data; name="key"

value
----------------------------180610939475792226718191--

Let’s note the following:

  • Lines above Connection: close are headers sent with this HTTP request. The good stuff starts 2 lines after this.
  • ----------------------------180610939475792226718191 is what’s referred to as the boundary of this request; it’s used to deliminate each key/value pair of the form-data request
  • Underneath that line should be something that looks like the above headers, but the request module in node doesn’t send any headers with each piece. We could see ‘content-type’, ‘content-length’, or any set of headers.
  • After this follows the disposition parameters, which are used to describe the chunk of data for this key/value pair. For example, we see name=“key”, which is the key part of our form-data chunk. We could also see data such as ‘filename’ (which is used when sending local files over the wire).
  • Finally comes the form-data value, where we see “value”. If we sent over another form-data key/value pair, we’d see the structure as above but with a different ‘name’ and value section.
  • To terminate this request, a multipart EOF line is sent (----------------------------180610939475792226718191--).

If you want to see what disposition parameters look like in the request, change formData to:

{
key: fs.createReadStream("stuff.txt")
}

From there, just rerun nc -l -p 8000 and node stuff.js.

And thus follows the anatomy of a multipart form-data request! Now, let’s get to how the FS structures it’s multipart form-data job format to accomodate for each piece.

General Structure

So, how do we apply this to the connector builders file module? First, let’s take a look at our File.Push module.

{
"brick": "file.push",
"id": "IDHERE",
"inputs": {
"id": {
"_type": "string",
"_array": false,
"_value": null
},
"body": {
"_type": "object",
"_array": false,
"_value": null
},
"query": {
"_type": "object",
"_array": false,
"_value": null
},
"headers": {
"_type": "object",
"_array": false,
"_value": null
}
},
"outputs": {}
}

Quickly going over each of these inputs, since this is a newer module:
* id: The platform file system ID that our system will use to push the right file into the external service (in this case, Box)
* body: The body, or job, that will be the main content that we push up to the service. This represents the actual request that would go to the external service. All other inputs effect how our FS behaves with that request.
* query: The query we would need to send to the FS. Not needed for this example.
* headers: The headers we would send to the FS. Not needed for this example. Reminder Any headers you need in your request, would be passed into your body.

As you can see, body, is the most important piece of this equation, so let’s take a look at what components we need inside of it, normally:

{
url: URL string (required),
protocol: HTTP (required)
method: HTTP method (required),
qs: query string (or JSON object representing query string),
headers: HTTP headers (as a JSON object)
}

However, things are a bit different with multipart requests. This multipart form-data request involves sending separate key/value pairs over an HTTP connection, with FS data being treated as just another piece of the request body. To accommodate for this semantic, the body of a request being sent through the file.push kernel module looks like this:

{
url: URL string (required),
protocol: HTTP(required)
method: HTTP method (required),
qs: query string (or JSON object representing query string),
headers: HTTP headers (as a JSON object),
form-data: form data information (as a string or JSON object),
form-loc: form data location for FS data
}

There are two new additions here: form-data and form-loc. See below for more details.

form-data
This describes form-data key/value pairs that don’t contain FS data. Form-data would include things that the request needs to utilize the file correctly (Box examples = parent folder ID in the service, the name it should be saved under, etc). This value can take two forms:

  • object: Contains the following structure:

    {
[key name]: {
content: raw data to be sent under this key as a string. Can be implied when params/headers are empty.
params: disposition parameters, e.g. 'filename' (optional, JSON object),
headers: extra headers to sent with this piece of form data, e.g. 'content-type' (optional, JSON object)
},
// other form-data key/value pairs ...
}

  • string: If a string, form-loc (inside job content) is ignored and this data operates as the key/value pair sent with FS data. Treated as exactly the following:

    {
[form-data value]: {
content: FS data (attached internally),
params: {},
headers: {}
}
}

As an implementation note, the FS sends these key/value pairs before sending over the internal FS data.

We noted above that Box requests stringified attributes for it’s implementation of file upload, so let’s see what that looks like in this format (in javascript):

var formData = {
"attributes": JSON.stringify({
name: "file_name.txt",
parent: {
id: 0 // if we want to insert this file under the root directory
}
}),
// if Box required more form-data parts ...
};

form-loc

Describes the form-data key/value pair containing FS data. This value can take two formats:

  • object: Contains the following structure:

    {
name: name of key to place FS data under ([key name] in 'form-data', string),
params: disposition parameters, e.g. 'filename' (optional, JSON object),
headers: extra headers to send with this piece of form data, e.g. 'content-type' (required, if params are present, JSON object)
}

  • string: If a string, treated as exactly follows:

    {
name: [form-loc value],
params: {},
headers: {}
}

Continuing with our Box example, here’s what this part of the job structure looks like, in Javascript:

var formLoc = {
name: "file",
params: {
"filename": "\"local_file.txt\"" // Box requires a local file name be attached to file content
},
headers : {}
};

Note Box requires a local file name be attached to the actual file data being sent with the upload request (and hence why it goes under the “form-loc” key, it’s FS data) and we add it as the “filename” disposition parameter.
Note We also stringify the local file name as well; form-data requests require the filename parameter be stringified.

Implementation in Javascript
Taking all these parts together, we can now construct the entire body of the file.push kernel module, in Javascript, to show how the pieces work together:

var token = "imatoken";
var formLoc = {
name: "file",
params: {
"filename": "\"local_file.txt\"" // Box requires a local file name be attached to file content
}
};
var formData = {
"attributes": JSON.stringify({
name: "file_name.txt",
parent: {
id: 0 // if we want to insert this file under the root directory
}
})
};

var job = {
url: "https://upload.box.com/api/2.0/files/content",
method: "POST",
headers: {
Authorization: "Bearer " + token
},
"form-data": formData,
"form-loc": formLoc
};