JSON data

Linux has no built in way to handle JSON properly, yet many APIs will return JSON output. Here’s how to parse and use JSON data inside your shell scripts with a simple utility.

Parsing JSON With jq

jq is a utility that can handle everything you may want to do with JSON. It’s super simple to install; simply download the binary and move it to your a folder on your PATH, usually /usr/local/bin/. It has no dependencies, so it’s just one binary to download. There are downloads for macOS and Windows as well.

To start, you can pipe JSON to it to pretty-print it. Some APIs will minify their responses to save space, leading to this awful mess when you curl something in the console:

curl in the console

Pipe it to jq like so:

curl https://api.example.com/ | jq

And it’s now readable, with syntax highlighting to boot:

Readable text with syntax highlighting

If you give jq a command, it will use it to select data out of the JSON, much like how sed operates on text. We’ll use this JSONPlaceholder as an example API to try out commands. For simplicity’s sake, we’ll leave out the curl command and pipe to keep everything readable. You can also curl to a file and pipe cat to jq, if you don’t want to see the curl output every time you test a new command.

A simple period ‘.‘ represents the data piped into jq. If you simply want to get a key from the object, you can do:

jq '.status'

You can nest this as well; for example, '.data.geo.host' would return a value farther down the list. Optionally, you can add a question mark for error checking like “.data.geo?.host“, which specifies that the property is optional.

Working with arrays is where jq gets interesting. You can use square brackets to get a specific array element. For example, you could get the name of the first entry with:

jq '.[0].name'

You can slice arrays using [2:5], and use the same question mark syntax for optional properties. Though keep in mind that if an array is empty, it doesn’t mean it’s an invalid property.

However, you’ll likely want to fetch a specific entry based on the data within it, not on the order of the array. To do that, pass the whole array to the select utility:

jq '.[] | select(.name | contains("Leanne"))'

This filters the array to only include items that pass the test; in this case, does the name field contain a certain name? This can be any boolean expression, even arithmetic ones like select(.count >= 2).

Reformatting JSON

jq can construct JSON as well, which means you can use it to reformat JSON input. You can make objects simply by wrapping them in curly brackets, and pass jq statements as the values. jq will output the new object. For example:

cat json | jq '.[0] | {name: .name, company: .company}'

Outputs nicely to the command line:

jq outputs the new object

While this is useful for stripping out only the information you want, you can also construct new objects using jq commands.

For full reference on all of jq‘s syntax, you can consult its online manual.

Alternatives to jq

If you don’t want to use another program, you could use a few alternatives. The easiest method is to forego doing it in bash, and instead use a scripting language like Python or JavaScript (with node) to do the actual processing. You would instead pipe the JSON command output to a file:

curl https://api/example.com/ > json.txt

And then load it into a variable within the script for use.

You could also use simple text selection utilities; for example, if you just need a specific key out of a response, you could use grep alongside a regular expression to select the value:

grep "name" | sed 's/"name": "(.*)"/1/'

However, this can (and probably will) break very easily with any changes in the JSON. Passing in the users API returned the name of each user, but also the username (which grep matched as well), as well as the name of the company (which had the exact same key). Unless you plan to account for all of this, you should use something that is aware of the JSON’s structure, rather than treating it as text.

Profile Photo for Anthony Heddings Anthony Heddings
Anthony Heddings is the resident cloud engineer for LifeSavvy Media, a technical writer, programmer, and an expert at Amazon's AWS platform. He's written hundreds of articles for How-To Geek and CloudSavvy IT that have been read millions of times.
Read Full Bio »