Rule Engine v2

Rule Engine Overview*

If you're not familiar with the Rule-Engine library, it is a Python project to provide filtering expressions for arbitrary Python objects in a secure manner. The primary use-case is to allow developers to expose filter functionality to end users for matching structured two-dimensional data.

The following is an example rule to match large log files (>1MiB) owned by the user "alice" to demonstrate the principals.

size >= 2**20 and owner == 'alice' and path =~~ '\.log$'

For anyone familiar with Python, the expression should be relatively easy to understand with the notable exception of the fuzzy-comparison operator =~~. This operator performs a regular expression (right operand) search operation against a string expression (left operand) and evaluates to true when there is a match. The logic can be inverted by using the inverse operator !~~ or applied more strictly using use an exact match by using the =~ operator (which also has an inverted counterpart).

The Array Data Type*

The new ARRAY data type allows Python sequences such as tuples and lists to be used with rules. These values are then compatible with equality and membership checks (using the new reserved keyword in ). For example with this rule writers can now check if name in people or target not in banned.

The following is an example showing the Python API usage along with a rule to check if an event that Alice donated to is today.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# first define the rule to use for matching / filtering
rule = rule_engine.Rule('date == $today and "Alice" in donors')
# next define the event object with it's attributes which will be symbols
event = {
    'date': datetime.date.today(),
    'donors': ['Alice', 'Bob'],  # new ARRAY type
    'title': 'Annual Fundraiser'
}
# apply the rule on the event object
rule.matches(event) # => True

Data Attributes*

The new attribute system will by default resolve attributes recursively on objects. This is useful when a rule is being applied to a compound data type such as a dictionary where some values are also dictionaries, or an object with subobjects as attributes.

In addition to recursively resolving objects, some data types have built in attributes. For example, STRING objects have a .length attribute which can be used to check the length of a string value and a .as_lower attribute which will convert the string to using all lowercase letters.

The following example illustrates this new functionality with another Python API example showing a rule which matches network traffic. The rule used uses both the recursive attribute resolver for dst.port and the builtin attributes defined for the new ARRAY type for seen.length.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# check for 3 or more events of tcp traffic going to port 4444
rule = rule_engine.Rule(
    'protocol == "tcp" and dst.port == 4444 and seen.length >= 3'
)
# evaluate the
rule.matches({
    'dst': {'addr': '172.16.50.81', 'port': 4444},
    'src': {'addr': '192.168.1.20', 'port': 53718},
    'protocol': 'tcp',
    'seen': [
        datetime.date(2019, 7, 6),
        datetime.date(2019, 8, 17),
        datetime.date(2019, 9, 29)
    ]
}) # => True

Thanks for reading and happy filtering!