Property Data with Node.js and CSV

For this guide, we're going to assume you're interested in using Datafiniti's property data to do some marketing analysis on homes in the US. Let's say you're a data scientist that's been tasked with the following:

  1. Collect data on homes.
  2. Sort the data by state.
  3. Find which states have the most expensive homes.

Your environment and data needs:

  1. You're working with Node.js.
  2. You want to work with CSV data.

Here are the steps we'll take:

1. Install the request module for Node

In your terminal, run the following to install the request module for Node:

npm install request

2. Get your API token

The next thing you'll need is your API token. The API token lets you authenticate with Datafiniti API and tells it who you are, what you have access to, and so on. Without it, you can't use the API.

To get your API token, go the Datafiniti Web Portal (https://portal.datafiniti.co), login, and click on your account name and the top-right. From there, you'll see a link to the "My Account" page, which will take you to a page showing your token. Your API token will be a long string of letters and numbers. Copy the API token or store it somewhere you can easily reference.

For security reasons, your API token will be automatically changed whenever you change your password.

📘

For the rest of this document, we'll use AAAXXXXXXXXXXXX as a substitute example for your actual API token when showing example API calls.

.

3. Run your first search

The first thing we'll do is do a test search that will give us a sense for what sort of data might be available. Eventually we'll refine our search so that we get back the most relevant data.

Since we want homes in the US, let's try a simple search that will just give us online listings for US-based properties.

Write the following code in your code editor (replace the dummy API token with your real API token):

/*
  Illustrates an API call to Datafiniti's Property Database.
*/
var request = require('request');

// Set your API parameters here.
var API_token = 'AAAXXXXXXXX';
var format = 'JSON';
var query = 'country:US';
var num_records = 1;
var download = false;

var request_options = {
  url: 'https://api.datafiniti.co/v4/properties/search',
  method: 'POST',
  json: {
    'query': query,
    'format': format,
    'num_records': num_records,
    'download': download
  },
  headers: {
    'Authorization': 'Bearer ' + API_token,
    'Content-Type': 'application/json'
  }
}

console.log(request_options);

// Make the API call.
request(request_options, function(error, response, body) {
  if (error) {
    console.log(error);
    console.log(response);
  } else {
    console.log(body);
  }
});

You should get a response similar to this:

{
  "num_found": 7983205,
  "total_cost": 1,
  "records": [
    {
      "address": "711 Kent Ave",
      "brokers": [
        {
          "agent": "Raj Singh",
          "company": "YOUR REALTY INC.",
          "dateSeen": [
            "2016-06-06T18:09:28Z"
          ],
        }
      ],
      "city": "Catonsville",
      "country": "US",
      "dateAdded": "2016-06-06T18:09:28Z",
      "features": [
        {
          "key": "Air Conditioning",
          "value": [
            "Heat Pumps"
          ]
        },
        {
          "key": "Sewer Type",
          "value": [
            "Public"
          ]
        }
      ],
      "latitude": "39.284462",
      "listingName": "711 Kent Ave, Catonsville, Md 21228",
      "longitude": "-76.734069",
      "lotSizeValue": 0.16,
      "lotSizeUnit": "Acres",
      "mlsNumber": "BC9677283",
      "numBathroom": 2,
      "numBedroom": 4,
      "postalCode": "21228",
      "prices": [
        {
          "amountMax": 199900,
          "amountMin": 199900,
          "currency": "USD",
          "dateSeen": [
            "2016-08-08T00:00:00Z",
            "2016-08-03T00:00:00Z"
          ],
          "isSale": "false",
        },
        {
          "amountMax": 212000,
          "amountMin": 212000,
          "currency": "USD",
          "dateSeen": [
            "2016-06-06T00:00:00Z"
          ],
          "isSale": "false",
        }
      ],
      "propertyTaxes": [
        {
          "amount": 3195,
          "currency": "USD",
          "dateSeen": [
            "2016-06-06T18:09:28Z"
          ],
        }
      ],
      "propertyType": "Single Family Dwelling",
      "province": "MD",
      "statuses": [
        {
          "dateSeen": [
            "2016-08-09T09:16:10Z"
          ],
          "isUnderContract": "false",
          "type": "For Sale"
        }
      ],
      "id": "AV9WzHyO_RWkykBuv11F"
    }
  ]
]

Let's break down each of the parameters we sent in our request:

API Call Component

Description

"query": "country:US"

query tells the API what you want to search. In this case, you're telling the API you want to search by country. Any property in the US will be returned.

"num_records": 1

num_records tells the API how many records to return in its response. In this case, you just want to see 1 matching record.

Now let's dive through the response the API returned:

Response Field

Description

"num_found"

The total number of available records in the database that match your query. If you end up downloading the entire data set, this is how many records you'll use.

"total_cost"

The number of credits this request has cost you. Property records only cost 1 credit per record.

"records"

The first available matches to your query. If there are no matches, this field will be empty.

Within each record returned, you'll see multiple fields shown. This is the data for each record.

Within the records field, you'll see a single business returned with multiple fields and the values associated with that business. The JSON response will show all fields that have a value. It won't show any fields that don't have a value.

Each business record will have multiple fields associated with it. You can see a full list of available fields in our Business Data Schema.

4. Refine your search

If you think about the original query we made, you'll realize we didn't really specify we only wanted homes for sale. There are several other types of properties (e.g., commercial, rentals) that may also be in the data. Since we only want homes for sale, we should narrow our search appropriately. Modify your code to look like this:

/*
  Illustrates an API call to Datafiniti's Business Database for hotels.
*/
var request = require('request');

// Set your API parameters here.
var API_token = 'AAAXXXXXXXX';
var format = 'JSON';
var query = 'country:US AND propertyType:"Single Family Dwelling"';
var num_records = 10;
var download = false;

var request_options = {
  url: 'https://api.datafiniti.co/v4/properties/search',
  method: 'POST',
  json: {
    'query': query,
    'format': format,
    'num_records': num_records,
    'download': download
  },
  headers: {
    'Authorization': 'Bearer ' + API_token,
    'Content-Type': 'application/json'
  }
}

console.log(request_options);

// Make the API call.
request(request_options, function(error, response, body) {
  if (error) {
    console.log(error);
    console.log(response);
  } else {
    console.log(body);
  }
});

This code is different in a couple ways:

  1. It adds AND propertyType:"Single Family Dwelling" to narrow down results to just US hotels.
  2. It changes records=1 to records=10 so we can look at more sample matches.

Datafiniti lets you construct very refined boolean queries. If you wanted to do more complicated searches, you could use OR operations, negation, and more.

5. Initiate a download of the data

Once we like what we see from the sample matches, it's time to download a larger data set! To do this, we're going to update our code a fair bit (an explanation follows):

/*
  Illustrates an API call to Datafiniti's Business Database for hotels.
*/
var request = require('request');
var fs = require('fs');

// Set your API parameters here.
var API_token = 'AAAXXXXXXXX';
var view_name = 'property_flat_prices';
var format = 'CSV';
var query = 'country:US AND propertyType:"Single Family Dwelling"';
var num_records = 50;
var download = true;

var request_options = {
  url: 'https://api.datafiniti.co/v4/properties/search',
  method: 'POST',
  json: {
    'query': query,
    'num_records': num_records,
    'view': view_name,
    'format': format,
    'download': download
  },
  headers: {
    'Authorization': 'Bearer ' + API_token,
    'Content-Type': 'application/json'
  }
}

console.log(request_options);

// A function to check if a download request has completed
function checkDownloadUntilComplete(options, callback) {
  var download_id = options.download_id;
  var download_options = {
    url: 'https://api.datafiniti.co/v4/downloads/' + download_id,
    method: 'GET',
    headers: {
      'Authorization': 'Bearer ' + API_token,
      'Content-Type': 'application/json'
    }
  }

  request(download_options, function(error, response, body) {
    var num_files_downloaded = 0;
    var download_response = JSON.parse(body);
    console.log('Records downloaded: ' + download_response.num_downloaded);
    if (download_response.status !== 'completed') {
      // NEED A SLEEP FUNCTION HERE!
      checkDownloadUntilComplete(options, callback);
    } else {
      var results = download_response.results;
      console.log(results);
      for (var i = 0; i < results.length; i++) {
        var filename = download_id + '_' + i + '.' + format;
        var file = fs.createWriteStream(filename);
        request(results[i]).pipe(file).on('end', function() {
          console.log('File ' + (i+1) + ' out of ' + results.length + ' saved: ' + filename);
          num_files_downloaded++;
          if (num_files_downloaded === results.length) process.exit();
        });
      }
    }
  });
}

// Initiate the download request.
request(request_options, function (error, response, body) {
    var request_response = body;
    var download_id = request_response.id;

    // Check on status of the download request.
    checkDownloadUntilComplete ({download_id : download_id}, function (error, response) {});
  }
);

A few things to pay attention to in the above code:

  1. We changed format from JSON to CSV. We'll want to view the files in CSV format, so it will be easier to look at.
  2. We changed num_records from 10 to 50. This will download the first 50 matching records. If we wanted to download all matching records, we would remove num_records. num_records will tell the API to default to all available records.
  3. We set view to property_flat_prices. Using property_flat_prices will nest fields like features into a single cell, while splitting each price for every home into multiple rows.
  4. We changed download from false to true.

🚧

If num_records is not specified, ALL of the records matching the query will be downloaded.

Since we've handled multiple steps of the download process in this code, we won't go into the details here, but we do recommend you familiarize yourself with those steps. Checking them out in our Property Data with Postman and CSV guide.

🚧

When using the API, you will not receive any warning if you are going past your monthly record limit. Keep a track on how many records you have left by checking your account. You are responsible for any overage fees if you go past your monthly limit.

If you would like to narrow your search to just exact matches you can place the search term in quotation marks.

6. Open the result file(s) in Excel

The download code will save one or more result files to your project folder. Open one of those files in Excel. It will look something like:

27362736

7. Analyze the results

Using Excel, we can easily count the total number of hotels on a state-by-state basis. The province column gives us where each hotel is located, so we can use it to tally up the numbers.