Burp Automation | Automating Burp Scanning Via Rest API & Robot Framework Using Python3

Manual security scanning is very time consuming and we can leverage headless Burp Suite to perform the scanning and get the results uploaded in the Google Drive directly via Pydrive Module in Python.

Burp Automation

This automation uses Burp Suite Pro along with Robot Framework and REST API using Python3. Hence called as Burp`O`Mation.

Github: https://github.com/justmorpheus/burp-automation

It can be also used in Jenkins to perform automated UI tests. This will initiate an automated spider and crawler by leveraging the power of the Burp Scanner along with the burp extender.
Once the scan is complete the report is generated in HTML & CSV which is automatically uploaded in the GDrive Folder.

This is a step-by-step guide to integrate Burp Suite with CI/CD Pipeline for automated API Fuzzing. This tool uses bash script for one click install of all the commands using Burp Rest API using python and Robot Framework for automated testing.

It allows you to do the following quite effectively

• One-click run using Bash installs all the dependencies with verbose prerequisites.
• Uses python3 and robot framework which is easy to automate.
• User Burp Suite Rest API and runs Burp Suite Professional in the headless mode along with multiple Burp Suite extensions like additional-scanner-checks, BurpJSLinkFinder, and active-scan-plus-plus.
• Automatically performs pentest of API/Web endpoint including scope addition and deletion using robot script.
• Slack integration for real-time alerts along with uploading reports to GDrive in YYYY-MM-DD format.

About The Framework

Robot Framework is a generic test automation framework for Acceptance Testing and Test-Driven Development (ATDD). It has easy-to-use tabular test data syntax and utilizes the keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java. Users can also create new higher-level keywords from existing ones, with the same syntax used for creating test cases.

Prerequisites to automate Burp using Rest API with Robot Framework

• Windows/Linux OS
• Burp Suite Pro with license installed
• Burp Suite Rest API (Vmware)
• Python 2.x ( Available in Most Linux Distribution)
• Pip
• Robot Framework
• Robot2Burp package

(This step-by-step guide will include installing the prerequisites as well as pre-setup steps required for running this automation via one-click)

Prerequisites

1. Debian OS(Ubuntu).
2. Activated Burp Suite Pro (burpsuite_pro_v2021.6.2.jar) with prefs.xml .
3. Vmware Burp Suite Rest API (It will be installed via automated script).
4. Python3 & Pip (It will be installed via automated script).
5. Robot Framework (It will be installed via automated script).
6. Update custom_lib_2.py for scope and time for the scan(current time is 3 minutes).
7. Setting up G Drive authentication for PyDrive
8. Upload credentials (client_secrets.json & mycreds.txt) to GitHub repository.
9. Docker for using dockerfile in CentOS.

Sample client_secrets.json, mycreds.txt and prefs.xml is already provided in the repository. Update the files with working configs.

Note: Update the time.sleep in custom_lib_2.py if want to run the scan for longer time.

Pre-Setup

1. Generate G Drive credentials.

• Go to APIs Console and make your own project.
• Search for ‘Google Drive API’, select the entry, and click ‘Enable’.
• Select ‘Credentials’ from the left menu, click ‘Create Credentials’, select ‘OAuth client ID’.

Oauth Client

• Save the credentials by clicking on ‘Download JSON’ on the right side of Client ID to download client_secret_<really long ID>.json. and rename as client_secrets.json

This will generate client_secrets.json.

client_secrets.json

• Now, the product name and consent screen need to be set -> click ‘Configure consent screen’ and follow the instructions.
  
  Once finished:
  -Select ‘Application type’ to be Web application.
  - Enter an appropriate name.
  - Input http://localhost:8080 for ‘Authorized JavaScript origins’.
  - Input http://localhost:8080/ for ‘Authorized redirect URIs’.

Add Web App

• Click Save & Continue.
• Now in the Test Users, add the user who will be used for G drive authentication.

Add user of GDrive Folder

• Finally, check the summary and go back to the dashboard. This is complete.

Check Summary

• To generate the mycreds.txt using client_secrets.json .

Pydrive requires authentication over the browser. Hence we can generate the credentials locally and upload the required file in the repository. Sample files have been provided in the repository. Replace them with your own file.
This needs to be done once.

• Run the following commands on the local system (Browser is required for the first-time authentication).

# wget https://gist.githubusercontent.com/justmorpheus/3c0f1f1034bc9532afa45fd27fe0dcb4/raw/12ced55441772692960bb9e19de2ccdc73c1910f/oauth-quickstart.py
#python3 -m pip install pydrive

Quick-start script for generating mycreds.txt

• Run the command.

#python oauth-quickstart.py

Python quick-start

• Follow the default steps and continue. Once authentication is complete close the browser.

This will generate mycreds.txt

Demo Google Drive Authentication

• Final files (client_secrets.json & mycreds.txt) to be uploaded on repository.

Do not share this folder to public as this contains secret. This will be uploaded to repo.

$ find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"
.
 |-mycreds.txt
 |-oauth-quickstart.py
 |-client_secrets.json

2. Generate parent folder id for report upload.

• Create a shared drive in Google Drive and copy the folder id (parent id) for uploading reports.

Parent id for uploading reports.

3. Generate the incoming Webhook URL for Slack.

• Incoming URL can be generated by creating the App and enabling the webhook for the App.

Slack Webhook

4. Modify custom_lib_2.py and update the Scope, Slack Webhook for alert and parent folder id of Google Drive (From step 2.)

This can also be automated, please do it yourself.

Scope update

2. Activating Burp via CLI mode.

Activate Burp Suite on your system. Burp Suite Pro saves all the activation details, settings, and user preferences at ~/.java/.userPrefs/burp/prefs.xml file. Upload the prefs.xml to the repository.

3. Place mycreds.txt, client_secrets.json & prefs.xml in the same directory of GitHub.

4. Now we are good to go and perform the installation steps for automation.

Automating Burp Suite Scan

1. Create a folder in the home directory.

#mkdir automation

2. Change directory to automation.

#cd automation

3. Download the automate.sh bash script.

#wget https://raw.githubusercontent.com/justmorpheus/burp-automation/master/automation.sh

4. Provide the execute permission to the automate.sh

chmod +x automation.sh

5. Run the script. Also, provide an argument as the GitHub repository in which automation script is kept.

#bash automation.sh https://github.com/justmorpheus/burp-automation.git

6. Rest everything will be done by automation script.

Automation via Bash

7. Bash script will run the python robot command (script will run this command).

# python3 -m robot -d output fuzzing.robot

Output should be similar to the below mentioned image: Status: Headless Burp API Running.
In case of Status: Headless Burp API Failed To Start. Run the above command manually and troubleshoot the error.

8. Once all the test cases are complete, the report will be uploaded to the Google Drive and the link can be copied via console.

Report format: CSV, HTML & sitemap for dev.

Final Report

9. Check the slack channel for the Gdrive Link.

10. Additional Support For Cent-OS.

• Download and run this docker file.
• Modify the dockerfile accordingly.

Dockerfile For CentOS(DIY)

• Commands to run docker file.

#docker build --no-cache -t burp-automate .
#docker run --name burpomation -d burp-automate

• This will build and run the image.

Since initial thought process was to run this on the Jenkins pipeline as part of automated scanning, there is no retention of any data because the workspace should be cleaned before running Jenkins.

• Run the log command for burpomation logs.

#docker logs burpomation -f

Docker logs for status of the script.

• Docker's final result should be PASS.

Final Docker Result: Pass

Reference

- https://we45.com/blog/automating-burp-with-jenkins/
- https://burpsuite.guide/blog/activate-burpsuite-inside-docker-container/
- https://portswigger.net/burp
- https://github.com/vmware/burp-rest-api
- https://doyensec.com/

Disclaimer: Do not use Burp Suite Pro in production for the CI/CD pipeline instead use Burp Suite enterprise. All information provided is for educational purposes only. The author does not keep responsibility for any illegal activities performed by anyone.

Thanks
justm0rph3u5