How to generate OAuth signature using HMAC SHA256 in Python

Most companies use an ERM tool these days to manage their finances and core business processes, and Oracle’s NetSuite has been one of the business leaders in ERM tools for the past few decades. However, these tools are used and managed by a few select individuals while the data stored can be of use to multiple teams. Which is why it makes sense to have an automated pipeline to ingest data from these tools using their native APIs to the primary databases/data warehouses, which can be later piped into different BI tools.

NetSuite has inbuilt support for SOAP APIs and RESTlet APIs. At Funding Societies, we decided to go ahead with the newly introduced RESTlet API solely because our codebase was in Python and were very quickly blocked and surprised with the lack of documentation and examples of implementation of NetSuite Restlet API with Python.

NetSuite provides many types of REST mechanisms to connect to NetSuite.

1. Usual Rest WebService call wherein Input parameter is in form of JSON.
2. Query Language: We can also send Native Query as an input parameter.
3. Restlet: Here we can write SuiteScripts in NetSuite, Deploy inside NetSuite and call this Script via RESTLET.

Here we will discuss the end-to-end implementation of NetSuite Query Language (NQL) and Saved Search capabilities and extracting data from it using RESTlet API using Python.

Our Use Case

The finance team regularly posts data of the company’s earnings and expenditure on NetSuite using the UI tool. These expenditures can have multiple reasons such as vendor bill payments, rent, salaries, business operation expenses, etc. The data team was approached with the use case of fetching this data for end-of-month and end-of-year calculations to be done in the data warehouse and presenting it to C-level executives using a dashboard.

Our Proposed Solution

After some research, the data engineering team proposed a solution that we can ingest the data from NetSuite using regular API requests and ingest it to restricted access tables in our Snowflake Data Cloud warehouse from where we could build dashboards in Periscope, our BI tool, and share the dashboard.

Challenges We Faced

After spending some time working on the solution and trying to find relevant documentation and surfing several stackoverflow pages, we realized that this implementation is not as straightforward as it seems and will require more work. To summarize, these are the challenges we faced when implementing this integration:

1. OAuth Authentication
2. Querying relevant tables
3. Creating and fetching data from saved search
4. Handling API limitations

In this article, we will be discussing the first challenge. OAuth Authentication.

Steps to follow before implementing OAuth Authentication

1. Enable the Token-based Authentication Feature

To enable the token-based authentication feature:

Go to Setup -> Company -> Enable Features.

Click the SuiteCloud subtab.

In the SuiteScript section, check the following boxes:

Client SuiteScript. Click I Agree on the SuiteCloud Terms of Service page.

Server SuiteScript. Click I Agree on the SuiteCloud Terms of Service page.

Note: Enabling both the Client SuiteScript and Server SuiteScript features is required to use RESTlets with token-based authentication.

In the Manage Authentication section, check the Token-based Authentication box. Click I Agree on the SuiteCloud Terms of Service page.

Click Save.

2. Set Up Token-based Authentication Roles and Token-based Authentication (TBA) Permissions

We have created a new Role dedicated for Token Based Permissions.

Then we have added token-based authentication permissions to this role then assign users to this role.

3. Assign Users to Token-based Authentication Roles

4. Create Integration Records for Applications to Use TBA

• Go to Setup > Integration > Manage Integrations > New
• Enter a Name for your application.
• Enter a Description, if desired.
• The application State is Enabled by default. (The other option available for selection is Blocked.) The value of this field is always specific to one NetSuite account.
• Enter a Note, if desired. The value of this field is always specific to one NetSuite account.
• On the Authentication tab, check (or clear) the appropriate boxes for your application.

In some cases, more than one method of authentication can be specified on an integration record.
Important: You should transition from user credentials to another method of authentication. Specifying more than one method on a record can be useful while making the transition from user credentials to token-based authentication (TBA).

• For accessing SOAP web services, both the Token-based Authentication (TBA) and the User Credentials boxes can be checked.
• For accessing REST web services, both the Token-based Authentication (TBA) and the OAuth 2.0 boxes can be checked.
• For accessing RESTlets, the Token-based Authentication (TBA), the OAuth 2.0, and the User Credentials boxes can be checked

We need to fetch the following values from NetSuite.

1. NetSuite Account ID: This value can be found from the NetSuite URL. It will look something like this: https://<accountID>.app.netsuite.com/app/login
2. Consumer Key & Consumer Secret: These values can be found by logging into NetSuite and going to Setup -> Integration -> Manage Integrations -> New. Once here, a new application needs to be created for the integration. It is here that we can manage permissions for the user to different aspects of the ERM tool. Check USER CREDENTIALS and TOKEN-BASED AUTHENTICATION. Click on Save and then the consumer key and consumer secret can be seen on the screen. It is important to save these 2 credentials as they will never be found again and if their credentials are not saved, an entirely new application needs to be created.

3. Token ID & Token Secret: Once consumer details are stored, we need to create the tokens for the application. The user with the above credentials needs to login on NetSuite and go to Manage Access Tokens link available on the home dashboard under settings. Create a new token and select the Application Name that corresponds to the associated integration record created earlier. Once done, the Token ID and Token Secret will be provided. Just like the previous step, these tokens will not be visible later, so it is important to store them.

4. Role ID: Go to Setup -> Integration -> Web Services Preferences and copy the Account ID. Select the username from the drop-down menu and select the role of the corresponding account. This is the Role ID.

Now that we have all our required credentials, we can start with the creation of the authentication process.

Implementing the authentication was perhaps the most difficult and time consuming aspect of the entire project. We found and tested various python libraries and packages (some of which seemed to have been created especially for NetSuite authentication) but none of them worked for us. This led to us taking a more hands on approach and deciding to implement the full OAuth structure by code. NetSuite’s OAuth requires the following steps:

1. Creating a cryptographic nonce.
2. Add UNIX timestamp
3. Using the nonce and timestamp from Steps 1 and 2 create the Authentication Signature
4. Create Authentication header using the Signature

Creating a cryptographic nonce

A cryptographic nonce is an arbitrary number that can be used just once in a cryptographic communication. It is used to make a request unique.

NetSuite uses an 11 character alpha-numeric nonce, which needs to be created in a random fashion. For our case, we have used the random and math packages to implement the nonce.

def getAuthNonce():
 nonce_text = “”
 length = 11
 possible= ‘ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789’
 for i in range(length):
  nonce_text += possible[math.floor(random.uniform(0, 1) * len(possible))]
  return nonce_text

Creating a UNIX Timestamp

This is perhaps the most straightforward part of the entire signature creation process. It can be easily implemented using python’s inbuilt time package.

import time
TIME_STAMP = round(time.time())

Create the Authentication Signature

In this step, we will require all the values and tokens we previously collected. Another value that we will need in this step is the SIGNATURE METHOD value which can be hardcoded to HMAC-SHA256. We also need the value of the API request method we are going to use. If getting data from saved search, then use GET or else if getting data from NQL (NetSuite Query Language [SuiteSQL]) then POST

This step requires multiple rounds of encoding and hashing. For this step, we preferred using all basic libraries and packages by python since a specific url encoding of a special package was producing the wrong signature which caused an error in our authentication process.

Create a simple data string having all values fetched above in the following order:

1. If using Saved Search then Deployment ID, [format: deploy=<deployment_id>], if not then this can be skipped
2. Consumer Key
3. Authentication Nonce
4. Signature Method (hardcoded to ‘HMAC-SHA256’)
5. Time Stamp
6. Token ID
7. OAuth Version (hardcoded to ‘1.0)
8. If using Saved Search then Script ID [format: script=<script_id>]. If not then this can be skipped
9. Maintaining this order is important as any other order will create an incorrect signature.

For the next step, we will need the BASE URL of the API call, which can be one of the following:

In case of Saved Search: https://<ACCOUNT_ID>.restlets.api.netsuite.com/app/site/hosting/restlet.nlIn case of SuiteSQL: https://<ACCOUNT_ID>.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql

In order to create the main signature, we will need to create the Signature Value and the Signature Key to encrypt the said value with.

Signature Value: This value needs to be encoded to be understood by URLs. Python’s urllib package can help with that. Concat the API Request Method with the encoded base url and finally encoded data string created above.

Signature Key: For this, we need to concatenate encoded Consumer Secret and Token Secret.

When the above steps are completed, we need to hash the binary version of the above key and value data and then encode the result with base64 encoding and decode it back from binary to normal string. Once done, we need to encode again using urllib. This is how the signature is created.

When fetching data using NetSuite SQL:

def getSignature(BASE_URL, HTTP_METHOD, OAUTH_NONCE, TIME_STAMP):
 data = 'oauth_consumer_key' + "=" + CONSUMER_KEY + "&"
 data += 'oauth_nonce' + "=" + OAUTH_NONCE + "&"
 data += 'oauth_signature_method' + "=" + SIGN_METHOD + "&"
 data += 'oauth_timestamp' + "=" + str(TIME_STAMP) + "&"
 data += 'oauth_token' + "=" + TOKEN_ID + "&"
 data += 'oauth_version' + "=" + OAUTH_VERSION
 signatureValue = HTTP_METHOD + '&' + urllib.parse.quote(BASE_URL, safe='~()*!.\'') + '&' + urllib.parse.quote(data,safe='~()*!.\'')
 
 signatureKey = urllib.parse.quote(CONSUMER_SECRET, safe='~()*!.\'') + '&' + urllib.parse.quote(TOKEN_SECRET,safe='~()*!.\'')
 
 signatureValue = bytes(signatureValue, 'utf-8')
 signatureKey = bytes(signatureKey, 'utf-8')
 shaData = hmac.new(signatureKey, signatureValue, digestmod=hashlib.sha256).digest()
 
 base64EncodedData = base64.b64encode(shaData)
 oauth_signature = base64EncodedData.decode('utf-8')
 oauth_signature = urllib.parse.quote(oauth_signature, safe='~()*!.\'')
 return oauth_signature

When fetching data using Saved Search:

def getSignature(BASE_URL, HTTP_METHOD, OAUTH_NONCE, TIME_STAMP):
 data = 'deploy=' +  + str(deployment_id) + "&"
 data += 'oauth_consumer_key' + "=" + CONSUMER_KEY + "&"
 data += 'oauth_nonce' + "=" + OAUTH_NONCE + "&"
 data += 'oauth_signature_method' + "=" + SIGN_METHOD + "&"
 data += 'oauth_timestamp' + "=" + str(TIME_STAMP) + "&"
 data += 'oauth_token' + "=" + TOKEN_ID + "&"
 data += 'oauth_version' + "=" + OAUTH_VERSION + "&"
 data += 'script=' + str(script_id)
 signatureValue = HTTP_METHOD + '&' + urllib.parse.quote(BASE_URL, safe='~()*!.\'') + '&' + urllib.parse.quote(data,safe='~()*!.\'') signatureKey = urllib.parse.quote(CONSUMER_SECRET, safe='~()*!.\'') + '&' + urllib.parse.quote(TOKEN_SECRET,safe='~()*!.\'') signatureValue = bytes(signatureValue, 'utf-8')
 signatureKey = bytes(signatureKey, 'utf-8')
 shaData = hmac.new(signatureKey, signatureValue, digestmod=hashlib.sha256).digest()
 
 base64EncodedData = base64.b64encode(shaData)
 oauth_signature = base64EncodedData.decode('utf-8')
 oauth_signature = urllib.parse.quote(oauth_signature, safe='~()*!.\'')
 return oauth_signature

Finally, we need to create the authentication header to send to the API request. For this, we need to follow the following steps:

1. Create Nonce
2. Create Timestamp
3. Define API Request Method
4. Create Signature (It needs to be done in this step, since the Nonce and Timestamp value needs to remain the same for Authentication Header and Signature)
5. To create the final header, we need to add the following details keeping the order intact:

• “OAuth”
• NetSuite Account ID
• Token ID
• Consumer Key
• Nonce
• Timestamp
• Signature Method
• OAuth Version
• Signature

The Authentication Header for both, SuiteSQL and Saved Search, will be the same. Only the signature creation will be different.

def createHeader():
 BASE_URL = 'https://<ACCOUNT_ID>.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql'
 OAUTH_NONCE = getAuthNonce()
 TIME_STAMP = round(time.time())
 HTTP_METHOD = "POST"
 oauth_signature = getSignature(BASE_URL, HTTP_METHOD, OAUTH_NONCE, TIME_STAMP) OAuthHeader = 'OAuth '
 OAuthHeader += 'realm="' + NETSUITE_ACCOUNT_ID + '",'
 OAuthHeader += 'oauth_token="' + TOKEN_ID + '",'
 OAuthHeader += 'oauth_consumer_key="' + CONSUMER_KEY + '",'
 OAuthHeader += 'oauth_nonce="' + OAUTH_NONCE + '",'
 OAuthHeader += 'oauth_timestamp="' + str(TIME_STAMP) + '",'
 OAuthHeader += 'oauth_signature_method="' + SIGN_METHOD + '",'
 OAuthHeader += 'oauth_version="1.0",'
 OAuthHeader += 'oauth_signature="' + oauth_signature + '"'
 
 headers = {
  "Authorization": OAuthHeader,
  "prefer": "transient",
  "Cookie": "NS_ROUTING_VERSION=LAGGING"
 }
 return headers

Finally once this is done, we are ready to call the Saved Search or query the tables of NetSuite to fetch the required data.

I hope that this was beneficial in understanding the creation of the OAuth steps and procedure for NetSuite. This authentication can be used for various types of integrations with NetSuite.

Stay tuned for more chapters of this series. In the next chapter we will see how to find and query relevant tables from NetSuite’s database.

Huge thanks to my co-author Sankar A for working with me tirelessly to put this detailed guide together!

How to generate HMAC SHA256 signature in Python?

What is HMAC SHA256 signature?

What is HMAC in Python?