Upsilon query syntax

Upsilon is the query language used in the Have I Been Squatted Rules Engine. It expresses detection rules that match domain permutations using signals and patterns.

Query Syntax Overview

Upsilon uses a Lucene-like query syntax that’s intuitive and easy to read. Queries are composed of field comparisons combined with logical operators.

Basic Structure

field:value
field:>value
field:*pattern*

Escaping and quoting

Upsilon tokenizes unquoted values on whitespace. When a value contains spaces, use one of two forms depending on the operator:

quoted strings (exact match + list membership):

title:"Google Maps"
technologies:["Google Analytics" "Google Tag Manager" "Nginx"]

escape spaces for wildcard / array operators (*...*, @..., @*...*, @@..., @@*...*):

technologies:@*Google\ Maps*
http_banner:*nginx\ 1.18*

You can also escape delimiter characters in unquoted tokens using a backslash (e.g. $, $, \[, \], \,, \\).

Logical Operators

Combine conditions using AND, OR, and NOT:

field1:value1 AND field2:value2
field1:value1 OR field2:value2
NOT field:value
(field1:value1 OR field2:value2) AND field3:value3

Very long chains of AND/OR can exceed the parser nesting limit (on the order of one thousand binary operators). Prefer shorter clauses and explicit parentheses.

Operators by Data Type

String Fields

String fields support various matching patterns:

Exact Match

permutation:amazon.com
kind:typosquatting

Wildcard Matching

Contains - Match strings containing a substring:

permutation:*amazon*
http_banner:*nginx*

Starts With - Match strings beginning with a prefix:

permutation:amazon*
dns_ns:ns1*

Ends With - Match strings ending with a suffix:

permutation:*.com
dns_mx:*mail.protection.outlook.com

Existence Check

Check if a field exists (has any value):

_exists_:whois
_exists_:origin_x509.subject_dn

Shorthand for the same existence check on a normal field:

dns_a:*

Check if a field does not exist:

NOT _exists_:dns_a

Inequality and slash form

Not equal (same shape as other comparisons):

kind:!=typosquatting

Slash-delimited pattern (parsed separately from wildcards; the rule matcher currently treats the inner text as a substring match, not full regular expression semantics):

http_banner:/nginx/

Number Fields

Numeric fields (scores, edit distance, autonomous system numbers in geolocation.asn.number, and similar) support comparison operators:

Comparison Operators

levenshtein_distance:>5          # greater than
levenshtein_distance:>=5         # greater than or equal
levenshtein_distance:<10         # less than
levenshtein_distance:<=10        # less than or equal
levenshtein_distance:=7          # equal to

Practical Examples

# High phishing score
classification.phishing:>0.8

# ASN in specific range
geolocation.asn.number:>=10000 AND geolocation.asn.number:<=20000

# Low edit distance
levenshtein_distance:<3

Boolean Fields

Boolean fields can be matched directly:

origin_x509.is_ca:true
origin_x509.is_ca:false
certificate_transparency.is_precert:true

Date Fields

Date fields support comparison operators and special time-based functions:

Date Comparisons

# RFC3339 format dates
registration_metadata.registration_date:>"2024-01-01T00:00:00Z"
origin_x509.not_after:<"2025-12-31T23:59:59Z"

Days Since/Until

Calculate days from current time:

# Registered within last 30 days
registration_metadata.registration_date.days_since:<=30

# Expires in more than 90 days
registration_metadata.expiration_date.days_until:>90

# Certificate valid for at least 365 days
origin_x509.not_after.days_until:>=365

# Certificate issued within last 7 days
origin_x509.not_before.days_since:<=7

Suffix comparisons require a numeric right-hand side (days). An explicit operator may be omitted on suffix forms to mean equality (for example field.days_since:30).

Array Fields

Array fields represent lists of values (e.g., DNS records, technologies).

Contains (Any Element Matches)

Use @ prefix for “any element contains”:

# Any DNS A record contains "192.168"
dns_a:@*192.168*

# Any technology contains "Google Maps"
technologies:@*Google\ Maps*

# Any technology starts with "Apache"
technologies:@Apache*

# Any nameserver ends with "cloudflare.com"
dns_ns:@*cloudflare.com

The same meaning without explicit wildcards in the token (still substring match per element):

technologies:@Google
dns_a:@192.168

All Elements Match

Use @@ prefix to require all elements match:

# All nameservers contain "cloudflare"
dns_ns:@@*cloudflare*

# All MX records end with ".google.com"
dns_mx:@@*.google.com

Without explicit wildcards:

dns_ns:@@cloudflare

Array Aggregations

Query array properties:

# Array has more than 5 elements
technologies.len:>5

# Check minimum/maximum values in numeric arrays
dns_a.len:>=2
scores.min:>=0.1
scores.max:<=0.9

There is no .avg: suffix in the Lucene parser; do not use field.avg: in Upsilon.

Identifier fields

The identifiers field matches one exact, case-sensitive public web identifier pair. Write the identifier kind and value as one quoted kind=value string:

identifiers:"google_tag_manager=GTM-ABC123"
identifiers:"meta_pixel=123456789012345"

The first = separates the kind from the value, so an identifier value may contain additional = characters. The kind and value must occur in the same identifier object. Syntactically valid future kinds remain queryable even when they are not listed in current product examples.

Check whether the field is present or negate one exact pair:

_exists_:identifiers
NOT identifiers:"google_tag_manager=GTM-ABC123"

Identifier fields do not support wildcards, partial matching, ranges, lists, or the identifiers:* existence shorthand. Use _exists_:identifiers for an existence check.

An identifier match can link sites that reuse the same public deployment identifier. It is investigation evidence, not proof of maliciousness.

IP Address Fields (inet)

IP address fields support special operators for network matching:

Exact IP Match

Use # prefix for exact IP address matching:

# Match exact IPv4 address
dns_a:#192.168.1.1

# Match exact IPv6 address
dns_aaaa:#2001:db8::1

CIDR Range Matching

Use # prefix with CIDR notation to match IP addresses within a subnet:

# Match any IP in the 192.168.0.0/16 subnet
dns_a:#192.168.0.0/16

# Match IPv6 subnet
dns_aaaa:#2001:db8::/32

# Match private IP ranges
dns_a:#10.0.0.0/8 OR dns_a:#172.16.0.0/12 OR dns_a:#192.168.0.0/16

Combining IP Queries

# Domain resolves to AWS IP space
dns_a:#54.0.0.0/8 OR dns_a:#52.0.0.0/8

# Certificate SAN includes localhost
origin_x509.san_ip:#127.0.0.1

List Membership

Check if a field value is in a list of values:

# ASN is one of several suspicious providers
geolocation.asn.number:[12345 67890 13335]

# Country is in high-risk regions
geolocation.country:[CN RU KP]

# Status is active or pending
registration_metadata.status_codes:[clientTransferProhibited serverTransferProhibited]

For array-typed fields, list membership is evaluated as “any element equals any listed value.”

Field references

Compare against another field’s runtime value using a $ prefix on the right-hand side (both names must exist in the schema):

some_numeric_field:>$other_numeric_field

Web Enrichment Queries

Use crawl-derived fields to detect for-sale pages, shell sites, and reused visual assets:

# For-sale parking page with marketplace branding
sitemap.title:*for\ sale* OR business_intel.company_names:Spaceship.com

# Social-link shell site
sitemap.external_link.target_host:[facebook.com instagram.com twitter.com tiktok.com]

# Reused favicon fingerprint
favicon.sha256:18617a981991607982022c0a36a9c958935e4e614e5343f9f1b528844d939aed

# Placeholder-heavy landing pages
sitemap.broken_link.kind:placeholder_anchor AND sitemap.broken_link_count:>0

These selector-backed fields behave like regular string and number fields even though the raw export stores them inside nested objects and arrays:

# Any crawled page title matches
sitemap.title:*login*

# Any extracted company name matches
business_intel.company_names:*holdings*

# Any open-port finding matches
ports.port:[80 443 8443]

Port Scan Queries

Use port scan fields for exposed services and infrastructure triage:

# Mail services exposed on the same host
ports.port:[25 110 143 587 993]

# More than two distinct services exposed
ports.unique_port_count:>2

# Scan reached multiple IPs
ports.unique_address_count:>1

Complete Examples

Infrastructure Detection

Detect domains hosted on suspicious infrastructure:

# Suspicious ASN with high phishing score
geolocation.asn.number:16509 AND classification.phishing:>0.7

# Domains outside expected geographic regions
NOT geolocation.country:[US CA GB FR DE] AND levenshtein_distance:<5

# Hosted on Cloudflare with suspicious technology
dns_ns:@*cloudflare* AND technologies:@*phishing*

Registration Patterns

Detect suspicious domain registration characteristics:

# Recently registered with suspicious registrar
registration_metadata.registration_date.days_since:<=30 AND
registration_metadata.registrar:*namecheap*

# Domain expires soon (potential abandonment)
registration_metadata.expiration_date.days_until:<=90

# Suspicious status codes
registration_metadata.status_codes:*clientHold* OR
registration_metadata.status_codes:*pendingDelete*

Certificate Analysis

Analyze TLS certificate properties:

# Certificate with many Subject Alternative Names
origin_x509.san_dns_count:>50

# Self-signed or suspicious issuer
origin_x509.is_ca:true OR origin_x509.issuer_dn:*self-signed*

# Recently issued certificate
origin_x509.not_before.days_since:<=7

# Certificate includes wildcard DNS entry
origin_x509.san_dns:@**

Certificate transparency (CT)

Query certificate transparency (CT) log data:

# Domain seen frequently in certificate transparency logs
certificate_transparency.occurrences_count:>10

# Precertificate entry
certificate_transparency.is_precert:true

# Recently seen in certificate transparency logs (Unix timestamp)
certificate_transparency.last_seen_ts:>1704067200

DNS Patterns

Detect suspicious DNS configurations:

# Uses Google Public DNS
dns_ns:@*google.com

# No MX records (suspicious for business domains)
NOT _exists_:dns_mx AND levenshtein_distance:<3

# Resolves to private IP space
dns_a:#10.0.0.0/8 OR dns_a:#172.16.0.0/12 OR dns_a:#192.168.0.0/16

Multi-Signal Detection

Combine multiple signals for high-confidence detection:

# High-confidence phishing detection
(
  permutation:*login* OR
  permutation:*account* OR
  permutation:*verify*
) AND
registration_metadata.registration_date.days_since:<=14 AND
classification.phishing:>0.8 AND
NOT geolocation.country:[US CA GB]

# Suspicious technology stack
technologies:@*WordPress* AND
technologies:@*PHP* AND
origin_x509.issuer_dn:*Let's\ Encrypt* AND
registration_metadata.registration_date.days_since:<=7

# Parked typo domain with marketplace evidence
permutation:*microsoft* AND
(
  sitemap.title:*for\ sale* OR
  business_intel.company_names:Spaceship.com OR
  sitemap.external_link.target_host:www.spaceship.com
)

# Brand-like shell site with social outbound links and extracted business details
permutation:*microboft* AND
sitemap.external_link.target_host:[facebook.com instagram.com] AND
business_intel.company_name_count:>0

# External email discovery observed through Microsoft
metadata.origin:external_email AND
metadata.sources.provider:microsoft AND
metadata.sources.kind:external_email

Operator Precedence

Operators are evaluated in the following order (highest to lowest):

Parentheses ()
NOT
AND
OR

Always use parentheses to make complex queries explicit:

# Explicit precedence (recommended)
(field1:value1 OR field2:value2) AND field3:value3

# Implicit precedence (same as above, but less clear)
field1:value1 OR field2:value2 AND field3:value3

Best Practices

1. Be Specific

Use precise field comparisons rather than broad wildcards:

# Good - specific pattern
permutation:*-login.com

# Less optimal - too broad
permutation:*login*

2. Combine Multiple Signals

Create robust rules by combining multiple indicators:

# Strong signal combination
registration_metadata.registration_date.days_since:<=30 AND
classification.phishing:>0.7 AND
NOT geolocation.asn.number:[15169 13335 16509]

3. Use Appropriate Operators

Choose the right operator for the field type:

# Good - uses CIDR for IP ranges
dns_a:#192.168.0.0/16

# Less optimal - uses wildcards for IPs
dns_a:*192.168*

4. Test rules

Start with conservative conditions and refine based on results:

# Start conservative
classification.phishing:>0.9 AND registration_metadata.registration_date.days_since:<=7

# Refine based on false positives/negatives
classification.phishing:>0.8 AND registration_metadata.registration_date.days_since:<=14

Field Reference

For a complete list of available fields and their types, see the Signals Reference.

Field Type Quick Reference

Type	Examples	Operators
`string`	`permutation`, `http_banner`	`:`, `!=`, `:`, `:`, `:`, `_exists_:`, `field:*`, `/.../` (substring in matcher)
`number`	`levenshtein_distance`, `geolocation.asn.number`	`:>`, `:>=`, `:<`, `:<=`, `:=`, `!=`
`boolean`	`origin_x509.is_ca`, `certificate_transparency.is_precert`	`:true`, `:false`
`date`	`registration_metadata.registration_date`	`:>`, `:<`, `.days_since:`, `.days_until:`
`array<string>`	`dns_ns`, `technologies`	`:@`, `:@@`, wildcards, `.len:`, `.min:`, `.max:`
`array<inet>`	`dns_a`, `dns_aaaa`	`:#`, `:#cidr`, `:@` (text-style containment on serialized values)
`identifier`	`identifiers`	exact `:"kind=value"`, `_exists_:`, `NOT` exact match
any	(n/a)	RHS `$otherField`

Error Handling

Common Errors

Field not found

unknown_field:value
# Error: Field 'unknown_field' does not exist

Invalid operator for type

permutation:>value
# Error: String fields don't support '>' operator

Missing required field

# Rule references field that doesn't exist in event
# Error: Field 'dns_a' not found in event

Validation

Rules are validated before execution to catch errors early:

Field names must exist in the schema
Operators must be appropriate for the field type
Syntax must be valid

Additional Resources

Rules Engine Guide - How to create and manage rules
Signals Reference - Complete field documentation
API Documentation - Programmatic access to rules