File System

NOTE This will be referring to Box’s “Upload File” API endpoint. Documented here.

Anatomy of a Multipart Form-Data Request

Before we dissect what a form-data request looks like, here’s necessary setup:

• Run nc -l -p 8000 to create a netcat server listening on localhost:8000. This will display the body of our multipart request. If you don’t have nc installed, just look up how to install the GNU netcat utilities onto your system (sorry windows).
• Create a .txt file called ‘stuff.txt’ with ‘asdf’ as the content.

• Write this to a .js file (I’m using stuff.js because I’m lazy):

  var request = require("request"),
  fs = require("fs");
  
  request({
  url: "http://localhost:8000/",
  formData: {
  key: "value"
  }
  }, function (err, res, body) {
  if (err) console.log("ERROR", err);
  else console.log("BODY", body);
  process.exit();
  });

Now that we have our setup complete, let’s try running node stuff.js. If you look over at the terminal running nc -l -p 8000, you should see something that looks like this:

POST / HTTP/1.1
host: localhost:8000
content-type: multipart/form-data; boundary=--------------------------180610939475792226718191
content-length: 163
Connection: close

----------------------------180610939475792226718191
Content-Disposition: form-data; name="key"

value
----------------------------180610939475792226718191--

Let’s note the following:

• Lines above Connection: close are headers sent with this HTTP request. The good stuff starts 2 lines after this.
• ----------------------------180610939475792226718191 is what’s referred to as the boundary of this request; it’s used to deliminate each key/value pair of the form-data request
• Underneath that line should be something that looks like the above headers, but the request module in node doesn’t send any headers with each piece. We could see ‘content-type’, ‘content-length’, or any set of headers.
• After this follows the disposition parameters, which are used to describe the chunk of data for this key/value pair. For example, we see name=“key”, which is the key part of our form-data chunk. We could also see data such as ‘filename’ (which is used when sending local files over the wire).
• Finally comes the form-data value, where we see “value”. If we sent over another form-data key/value pair, we’d see the structure as above but with a different ‘name’ and value section.
• To terminate this request, a multipart EOF line is sent (----------------------------180610939475792226718191--).

If you want to see what disposition parameters look like in the request, change formData to:

{
key: fs.createReadStream("stuff.txt")
}

From there, just rerun nc -l -p 8000 and node stuff.js.

And thus follows the anatomy of a multipart form-data request! Now, let’s get to how the FS structures it’s multipart form-data job format to accomodate for each piece.

General Structure

So, how do we apply this to the connector builders file module? First, let’s take a look at our File.Push module.

{
"brick": "file.push",
"id": "IDHERE",
"inputs": {
"id": {
"_type": "string",
"_array": false,
"_value": null
},
"body": {
"_type": "object",
"_array": false,
"_value": null
},
"query": {
"_type": "object",
"_array": false,
"_value": null
},
"headers": {
"_type": "object",
"_array": false,
"_value": null
}
},
"outputs": {}
}

Quickly going over each of these inputs, since this is a newer module:
* id: The platform file system ID that our system will use to push the right file into the external service (in this case, Box)
* body: The body, or job, that will be the main content that we push up to the service. This represents the actual request that would go to the external service. All other inputs effect how our FS behaves with that request.
* query: The query we would need to send to the FS. Not needed for this example.
* headers: The headers we would send to the FS. Not needed for this example. Reminder Any headers you need in your request, would be passed into your body.

As you can see, body, is the most important piece of this equation, so let’s take a look at what components we need inside of it, normally:

{
url: URL string (required),
protocol: HTTP (required)
method: HTTP method (required),
qs: query string (or JSON object representing query string),
headers: HTTP headers (as a JSON object)
}

However, things are a bit different with multipart requests. This multipart form-data request involves sending separate key/value pairs over an HTTP connection, with FS data being treated as just another piece of the request body. To accommodate for this semantic, the body of a request being sent through the file.push kernel module looks like this:

{
url: URL string (required),
protocol: HTTP(required)
method: HTTP method (required),
qs: query string (or JSON object representing query string),
headers: HTTP headers (as a JSON object),
form-data: form data information (as a string or JSON object),
form-loc: form data location for FS data
}

There are two new additions here: form-data and form-loc. See below for more details.

form-data
This describes form-data key/value pairs that don’t contain FS data. Form-data would include things that the request needs to utilize the file correctly (Box examples = parent folder ID in the service, the name it should be saved under, etc). This value can take two forms:

• object: Contains the following structure:

  {
  [key name]: {
  content: raw data to be sent under this key as a string. Can be implied when params/headers are empty.
  params: disposition parameters, e.g. 'filename' (optional, JSON object),
  headers: extra headers to sent with this piece of form data, e.g. 'content-type' (optional, JSON object)
  },
  // other form-data key/value pairs ...
  }

• string: If a string, form-loc (inside job content) is ignored and this data operates as the key/value pair sent with FS data. Treated as exactly the following:

  {
  [form-data value]: {
  content: FS data (attached internally),
  params: {},
  headers: {}
  }
  }

As an implementation note, the FS sends these key/value pairs before sending over the internal FS data.

We noted above that Box requests stringified attributes for it’s implementation of file upload, so let’s see what that looks like in this format (in javascript):

var formData = {
"attributes": JSON.stringify({
name: "file_name.txt",
parent: {
id: 0 // if we want to insert this file under the root directory
}
}),
// if Box required more form-data parts ...
};

form-loc

Describes the form-data key/value pair containing FS data. This value can take two formats:

• object: Contains the following structure:

  {
  name: name of key to place FS data under ([key name] in 'form-data', string),
  params: disposition parameters, e.g. 'filename' (optional, JSON object),
  headers: extra headers to send with this piece of form data, e.g. 'content-type' (required, if params are present, JSON object)
  }

• string: If a string, treated as exactly follows:

  {
  name: [form-loc value],
  params: {},
  headers: {}
  }

Continuing with our Box example, here’s what this part of the job structure looks like, in Javascript:

var formLoc = {
name: "file",
params: {
"filename": "\"local_file.txt\"" // Box requires a local file name be attached to file content
},
headers : {}
};

Note Box requires a local file name be attached to the actual file data being sent with the upload request (and hence why it goes under the “form-loc” key, it’s FS data) and we add it as the “filename” disposition parameter.
Note We also stringify the local file name as well; form-data requests require the filename parameter be stringified.

Implementation in Javascript
Taking all these parts together, we can now construct the entire body of the file.push kernel module, in Javascript, to show how the pieces work together:

var token = "imatoken";
var formLoc = {
name: "file",
params: {
"filename": "\"local_file.txt\"" // Box requires a local file name be attached to file content
}
};
var formData = {
"attributes": JSON.stringify({
name: "file_name.txt",
parent: {
id: 0 // if we want to insert this file under the root directory
}
})
};

var job = {
url: "https://upload.box.com/api/2.0/files/content",
method: "POST",
headers: {
Authorization: "Bearer " + token
},
"form-data": formData,
"form-loc": formLoc
};

Notes:

• If you get a file ID, it is only valid for the environment you’re testing in (alpha ID’s don’t work in Beta, etc)
• With File.Push, your file will automatically be deleted after the transaction is complete. You must get a new File ID every time you test if it completes the File.Push action
• Errors:
• If you are getting an error message called “enoent” this means that your file ID is incorrect. It may be because it’s already been deleted after a successful push, or it was never a real ID to start with
• If you are getting an error message called “enostream” that means that the way you are trying to use the File.Push / Pull module is incorrect.

Now that we’ve seen what the general code expectations of this would be, we can have idea of how to approach this in the connector builder. Using Box as an example, we know that we need to:

1. Build our form-data object
2. Build our form-loc object
3. Build the body of our upload request
4. Implement the File.Push module correctly
5. Render this data to our method’s UI

Looking into each one a bit more:

1.Looking at Box’s documentation, we know that we need a single key called “attributes” that is an object containing two items, a “name” field that will be used to save the file inside of their service, and a “parent” object that contains an “id” that tells Box which parent folder to save the file under.

• To do this, I created an object.construct that looked like so:

  {
  "brick": "object.construct",
  "id": "ATTRIBUTES",
  "inputs": {
  "name": "{{input.File.Name}}",
  "parent": {
  "id": "{{input.File.Parent Folder ID}}"
  }
  },
  "outputs": {
  "output": {
  "_type": "object",
  "_array": false
  }
  }
  }

• Then I ran it through JSON.Stringify, to stringify that object

• Then I did a final Object.Construct to build out “attributes” correctly. NOTE I did not need to strictly follow the pattern mentioned above because this “attributes” key does not need headers or params.

  {
  "brick": "object.construct",
  "id": "FORM-DATA",
  "inputs": {
  "attributes": "{{STRINGIFIED ATTRIBUTES.output}}"
  },
  "outputs": {
  "output": {
  "_type": "object",
  "_array": false
  }
  }
  }

2.To build form-loc, Box requires both a “name” field set to “file” and a parameter set to the local file name, which is stringified.

• I first used JSON.Stringify to stringify the innermost object.

• Then I used object.construct to build the form-loc’s entire object. NOTE: You must explicitly define your headers. You cannot set it equal to "headers" : {}.

  {
  "brick": "object.construct",
  "id": "FORM-LOC",
  "inputs": {
  "name": "file",
  "params": {
  "filename": "{{STRINGIFIED FILENAME.output}}"
  },
  "headers": {
  "_type": "object",
  "_value": {}
  }
  },
  "outputs": {
  "output": {
  "_type": "object",
  "_array": false
  }
  }
  }

3.Then we must build the body to send as a request to Box. This matches the format defined inside of “General Structure”

{
"brick": "object.construct",
"id": "BODY",
"inputs": {
"url": "https://upload.box.com/api/2.0/files/content",
"protocol": "HTTP",
"method": "POST",
"headers": {
"Authorization": "Bearer {{auth.access_token}}"
},
"form-data": "{{FORM-DATA.output}}",
"form-loc": "{{FORM-LOC.output}}"
},
"outputs": {
"output": {
"_type": "object",
"_array": false
}
}
}

4.Now we need to set up our File.Push module correctly. NOTE 1: input.File.File Content is the ID the user passes in, that represents the ID inside of the platform’s file system (surfaced to them in other cards as “File Content”). NOTE 2: You must add in the body object yourself, because the template is currently broken. There will be a fix for this end of next week, hopefully. This represents the response from the external service about your request.

{
"brick": "file.push",
"id": "REQUEST",
"inputs": {
"id": {
"_type": "string",
"_array": false,
"_value": "{{input.File.File Content}}"
},
"body": {
"_type": "object",
"_array": false,
"_value": "{{BODY.output}}"
},
"query": {
"_type": "object",
"_array": false,
"_value": {}
},
"headers": {
"_type": "object",
"_array": false,
"_value": {}
}
},
"outputs": {
"body": {
"_type": "object",
"_array": false
}
}
}

5.Finally, I did a simple Object.Construct to render my data correctly. Like so:

{
"brick": "object.construct",
"id": "OUTPUT",
"inputs": {
"File": {
"File ID": "{{prevData.body.pathName}}"
}
},
"outputs": {
"output": {
"_type": "object",
"_array": false
}
}
}