URL Normalization

The Threat Intelligence Service transforms received URLs into a normalized format, so that all URLs match the formats utilized by a wide variety of log sources. For example, Check Point firewalls include full URLs, whereas Palo Alto firewalls only provide the fully qualified domain name on the URL.

1. Go to the file path specified in the Threat Intelligence Service configuration. The default path is C:\Program Files\LogRhythm\LogRhythm Threat Intelligence Service\config. For more information about configuring the Threat Intelligence Service connection, see Configure the Connection to LogRhythm.
2. Open the URLNormalizationList.json file in a text editor (e.g., Notepad) to view the URL normalization rule.
3. The following parameters are defined by default:
   • Enabled: false
   • IncludeOriginal: false
   • RuleDescription: "TestRule"
   • MatchRegularExpression: "https?:\\/\\/(www\\.)?([-a-zA-Z0-9@:%._\\+~#=]{2,256}\\.[a-z]{2,6})\\b([-a-zA-Z0-9@:%_\\+.~#?&//=]*)"
   • SubstitutionRegularExpression: [{SubstituteExpression: "$2$3"}]
4. Configure the desired parameters and save the URLNormalizationList.json file.

The URL normalization rule matches any URL that starts with http:// and https://, and removes the http(s):// and www from the URL. The output from the Threat Intelligence Service is any URLs that does not match the pattern, as well as the matching URLs with http(s):// and www removed.

To include matching URLs with http(s):// and www in the URL, change the IncludeOriginal parameter to true.