Expressions

Building expressions

An expression has four kinds of nodes:

• Col reads a column: col('price').
• Lit holds a literal value: lit(3), lit('north'), a date, or a list of values for in.
• Call applies a registered function to arguments.
• Desc marks a sort key or ranking as descending.

Methods on any expression build operator calls. Arguments that are not expressions are wrapped in lit().

Functions in the Phibble\Expr namespace build the other calls. Options are named arguments, as in mean(col('x'), na_rm: true) or lag(col('x'), n: 2, default: 0). var_() stands for var(), which is a reserved word in PHP. call('name', ...) builds a call to any registered function by name.

columns() lists the columns an expression reads, and toArray() returns the tree as plain arrays:

$heavy = col('body_mass')->gt(5000)->and(col('species')->isIn(['Gentoo', 'Chinstrap']));

echo json_encode($heavy->toArray(), JSON_PRETTY_PRINT), "\n";
echo json_encode($heavy->columns()), "\n";

{
    "type": "call",
    "fn": "&",
    "args": [
        {
            "type": "call",
            "fn": ">",
            "args": [
                {
                    "type": "col",
                    "name": "body_mass"
                },
                {
                    "type": "lit",
                    "value": 5000
                }
            ]
        },
        {
            "type": "call",
            "fn": "in",
            "args": [
                {
                    "type": "col",
                    "name": "species"
                },
                {
                    "type": "lit",
                    "value": [
                        "Gentoo",
                        "Chinstrap"
                    ]
                }
            ]
        }
    ]
}
["body_mass","species"]

Using expressions with verbs

Each verb evaluates expressions per group:

• filter() keeps rows where the expression is true. NA counts as false.
• mutate() adds a column with one value per row. An aggregate such as mean() gives one value per group, which is repeated for each row of the group.
• summarize() needs one value per group. Anything else throws a LengthException.
• arrange() sorts by expressions over the whole table, ignoring groups. Wrap a key in desc() to sort it in descending order.

$penguins = DataFrame::fromCsv('penguins.csv');

$out = $penguins
    ->filter(is_na(col('body_mass'))->not())
    ->groupBy(['species'])
    ->mutate([
        'z' => col('body_mass')->sub(mean(col('body_mass')))->div(sd(col('body_mass'))),
        'rank' => min_rank(desc(col('body_mass'))),
        'size' => case_when(
            col('body_mass')->ge(5000), 'large',
            col('body_mass')->ge(3500), 'medium',
            default: 'small',
        ),
    ])
    ->filter(col('rank')->le(2))
    ->select(['species', 'body_mass', 'z', 'rank', 'size']);

echo $out;

# A tibble: 6 × 5
# Groups:   species [3]
  species   body_mass     z  rank size  
  <chr>         <int> <dbl> <int> <chr> 
1 Adelie         4725  2.23     2 medium
2 Adelie         4775  2.34     1 medium
3 Gentoo         6300  2.43     1 large 
4 Gentoo         6050  1.93     2 large 
5 Chinstrap      4550  2.13     2 medium
6 Chinstrap      4800  2.78     1 medium

$penguins = DataFrame::fromCsv('penguins.csv');

echo $penguins
    ->groupBy(['island'])
    ->summarize([
        'n' => n(),
        'mass' => mean(col('body_mass')),
        'mass_rm' => mean(col('body_mass'), na_rm: true),
        'q90' => quantile(col('body_mass'), 0.9, na_rm: true),
        'species' => n_distinct(col('species')),
        'any_heavy' => any(col('body_mass')->gt(6000), na_rm: true),
    ]);

# A tibble: 3 × 7
  island        n  mass mass_rm   q90 species any_heavy
  <chr>     <int> <dbl>   <dbl> <dbl>   <int> <lgl>    
1 Biscoe      168   NA    4716.  5670       2 TRUE     
2 Dream       124 3713.   3713.  4300       2 FALSE    
3 Torgersen    52   NA    3706.  4300       1 FALSE

Functions

The registry in Phibble\Expr\Functions::REGISTRY lists every function an expression may call, with its arguments and options. Calling anything else throws when the expression is built.

$sales = tribble(
    ['store', 'week', 'units'],
    ['a', 1, 10], ['a', 2, 14], ['a', 3, 9],
    ['b', 1, 7], ['b', 2, 7], ['b', 3, 12],
);

echo $sales
    ->groupBy(['store'])
    ->mutate([
        'prev' => lag(col('units')),
        'change' => col('units')->sub(lag(col('units'))),
        'running' => cumsum(col('units')),
        'rank' => min_rank(desc(col('units'))),
        'tile' => ntile(col('units'), n: 2),
    ]);

# A tibble: 6 × 8
# Groups:   store [2]
  store  week units  prev change running  rank  tile
  <chr> <int> <int> <int>  <int>   <int> <int> <int>
1 a         1    10    NA     NA      10     2     1
2 a         2    14    10      4      24     1     2
3 a         3     9    14     -5      33     3     1
4 b         1     7    NA     NA       7     2     1
5 b         2     7     7      0      14     2     1
6 b         3    12     7      5      26     1     2

NA and arithmetic

Expressions follow R’s rules for missing values and arithmetic:

• Arithmetic and comparisons with NA give NA.
• & and | use three-valued logic. FALSE & NA is false and TRUE | NA is true.
• / and ^ always return floats. Division by zero gives Inf, -Inf, or NaN.
• %% takes the sign of the divisor, so -7 %% 3 is 2.
• round() takes, of the two candidates, the one closer to the stored value, and an exact tie goes to the even digit. round(2.5) is 2, and round(2.675, digits: 2) is 2.67 because 2.675 is stored as slightly less than 2.675.
• Text compares and sorts in byte order.

$df = tibble(
    a: [true, false, null, null],
    b: [null, null, true, false],
    x: [2.5, 0.5, -1.5, 2.675],
    y: [1, null, 3, 4],
);

echo $df->mutate([
    'and' => col('a')->and(col('b')),
    'or' => col('a')->or(col('b')),
    'sum' => col('y')->add(1),
    'round' => Phibble\Expr\round(col('x')),
    'round2' => Phibble\Expr\round(col('x'), digits: 2),
    'mod' => lit(-7)->mod(3),
    'cumsum' => cumsum(col('y')),
]);

# A tibble: 4 × 11
  a     b         x     y and   or      sum round round2   mod cumsum
  <lgl> <lgl> <dbl> <int> <lgl> <lgl> <int> <dbl>  <dbl> <int>  <int>
1 TRUE  NA     2.5      1 NA    TRUE      2     2   2.5      2      1
2 FALSE NA     0.5     NA FALSE NA       NA     0   0.5      2     NA
3 NA    TRUE  -1.5      3 NA    TRUE      4    -2  -1.5      2     NA
4 NA    FALSE  2.68     4 FALSE NA        5     3   2.67     2     NA

Serialization

toArray() returns a JSON-friendly array, and Expr::fromArray() rebuilds the expression. Each node has a type key: col, lit, call, or desc. A call lists its positional arguments under args and its options under named. A date literal carries dtype (date or datetime).

$expr = lag(col('score'), n: 2, default: 0, order_by: col('week'));
$json = json_encode($expr->toArray());

echo $json, "\n";

$back = Phibble\Expr\Expr::fromArray(json_decode($json, true));
var_dump($back == $expr);

{"type":"call","fn":"lag","args":[{"type":"col","name":"score"}],"named":{"n":{"type":"lit","value":2},"default":{"type":"lit","value":0},"order_by":{"type":"col","name":"week"}}}
bool(true)

The test suite checks the in-memory evaluator against dplyr. The script tests/r/expr.R evaluates the same expressions in R and stores the results in tests/fixtures/expr/expected.json, which other backends can reuse.