9.3.2. PLUGIN VARIABLE: GET_INTERNET_DATA¶

The functions to run for this plugin when “get_internet_data” mode is enabled set as a list variable GET_INTERNET_DATA.

This list should be strings that reference function names that live inside the same plugin file.

The referenced functions must each be defined in the plugin file itself and must have the following signature:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

NAME = "plugin_name"

GET_INTERNET_DATA = [
    "referenced_function_name",
]

# where "referenced_function_name" would be the string added to the GET_INTERNET_DATA list in this plugin
def referenced_function_name(wequests, pkgs, **kwargs):
    # get something from the internet
    url = "http://some/url/here"
    r = wequests.request(url=url)

    # use beautiful soup from pkgs to parse the response
    soup = pkgs.BeautifulSoup(r.content, "lxml")

    # get a value to return using soup
    # this would theoretically get an integer value from the HTML
    # by looking for a single div that has class assigned to it named "some_value"
    ret = soup.find("div", {"id": "some_value"})

    # return the value to Tanium HAT's plugin system for saving to CSV
    return ret

The plugin system passes two important arguments to each get_internet_data function:

• wequests: This is a wrapper object around the requests package, and is pre-configured to use whatever proxies were established in the configuration file (if any). This class can be used to perform get an HTML response.
• pkgs: This is a dictionary like object that exposes a number of packages to the get_internet_data function:
  • tanium_hat.constants - Tanium HAT constants
  • os - standard python module
  • sys - standard python module
  • math - standard python module
  • json - standard python module
  • datetime - standard python module
  • logging - standard python module
  • pandas - external python module for data manipulation
  • numpy (https://docs.scipy.org/doc/numpy/reference/) - external python module for data manipulation
  • bs4.beautifulsoup (https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - class from external python module for html parsing
  • tanium_kit - python toolkit from libs_tanium

Values returned from all GET_INTERNET_DATA functions in all plugins will be written to a CSV file called “internet_data.csv” under config[data_path]. This CSV file can then be used during mode ANALYZE_DATA to perform calculations. Example CSV file:

1
2

"function_name2","function_name2","function_name3"
"123","456","whatever"