Bridging Laravel Scout to Eloquent

Laravel Scout provides an interface to external search engines such as Algolia and Meilisearch. These are typically remote indexes of the same data that you’re storing locally, but with far faster and more powerful searching and ranking capabilities. However, they have some limitations on how you can express the search itself (allowing only a single string value), and what you can do with the search result (because it comes back as a Laravel Collection and not a Query Builder). This makes it difficult to do things like perform a complex search remotely, and then filter the result further via Eloquent operations, perhaps involving different parts of your database. A typical Scout search might be:

use App\Models\Order;
$orders = Order::search('Star Trek')->get();Code language: PHP (php)

This gives us a bunch of Order instances, but we had no opportunity to ask it to do things like load relations or filter the results while it was doing it.

Fortunately it’s not difficult to bridge these two worlds. The resulting Collection contains model instances, so we can extract their IDs and use them to construct an Eloquent search that selects the same records, but locally this time (Scout already did the heavy lifting of figuring out which ones we wanted):

$builder = Order::query()->whereIn('id', $orders->keys());Code language: PHP (php)

This isn’t ideal (because it will fetch those records a second time), but it will be reasonably efficient because the IDs it searches on are exact matches for primary keys in the database.

We now have a builder that will select the same records as the Scout search did, but we can continue adding to it before requesting the final results.

$result = $builder->where('orders.name', 'like', 'a%')
    ->with(['orderItems', 'customer'])
    ->get();Code language: PHP (php)

So that’s how we can get to use Eloquent features on top of a Scout search.


After finding out that this bridging wasn’t built-in, I submitted a PR to add it, but sadly it was rejected. With the PR code in place, the syntax would have looked like this:

use App\Models\Order;
$orders = Order::search('Star Trek')
    ->toEloquent()
    ->where('orders.name', 'like', 'a%')
    ->with(['orderItems', 'customer'])
    ->get();Code language: PHP (php)

To be fair, this isn’t much of a saving in the external syntax, but it is more efficient because it can get the record IDs directly from Scout without having to load the models from the database. I don’t think that efficiency gain can be obtained from outside Scout’s own code.

I hope that helps someone!

Git & SSH sitting in a tree…

I do work for a bunch of different clients, who variously use GitLab and GitHub. For many years I put up with the incessant problem of accidentally signing my commits as the wrong user. It’s just so easy to forget to set the right GPG key and email address when you just want to get on with a project. It’s not the end of the world, but it’s annoying.

Often the same thing goes for the SSH keys you use to push and pull from your git repo; it’s a bit too easy to be lazy and use the same SSH key across multiple clients, when a little isolation would be a good idea from a security perspective.

What if you could work some magic so that identities and GPG and SSH keys are set to the right values right from the start, for every project for each of your clients? Read on…

This whole setup reminds me very much of a post I wrote in 2009 (13 years ago!) on the “holy trinity” of DNS, TLS, and virtual host wildcards that allow you to dynamically host vast numbers of previously undefined sites without having to touch your web server config at all, a classic example of convention over configuration.

First of all let me introduce you to .gitconfig. This file usually sits in your home directory, so for me on macOS that’s /Users/marcus/.gitconfig. This file contains your global git defaults, and is an easy-to-read config file in an “ini” style (and no, those are not real values!):

[user]
    name = Marcus Bointon
    email = marcus@example.com
    signingkey = AC34DF5B434BB76
[github]
    user = Synchro
    token = f693251e52043a23fe5fbd955cff56ff
...

You’ll find lots of other sections in here, which you can read about in the git config docs. But we are only really interested in one option: includeIf. This directive conditionally includes another git config file into your settings, and one of the things you can make it conditional upon is the path to your project. This is useful. I typically set up my client’s projects in the macOS default Sites folder within my home directory. Each client gets a folder, and each of their projects lives within that. This provides a tidy location to put a separate .gitconfig file that can be applied to all of their projects. It ends up like this:

~/.gitconfig
~/Sites/
    client1/
        .gitconfig
        project1/
        project2/
    client2/
        .gitconfig
        project1/
        project2/

Each .gitconfig file only needs to include the differences from the defaults that are set in the primary config file that lives in your home dir. To set up the GPG signing key and email for all of their projects, the file would contain this:

[user]
    email = remotedev1@client1.example.net
    signingkey = 434BB76AC34DF5B

Back in our primary file, we would add this conditional statement to automatically pull in this extra config whenever git is operating in this folder:

[includeIf "gitdir:~/Sites/client1/"]
    path = ~/Sites/client1/.gitconfig

And that’s it as far as GPG goes – commits will now be signed with the key and email address that are specific to this client, so when you set up your next project for them, you won’t have to do anything to set it up; it’ll Just Work™.

But what about SSH? The chances are that your client will have asked you for an SSH public key to add to their repo to provide you with sufficient access, but setting the GPG key doesn’t do anything towards selecting an SSH key for that purpose. You could do that using environment variables (which can be quite annoying) before, but fortunately, git 2.10.0 added the core.sshCommand config option that allows us to specify the SSH command that git uses for file transfer operations, and that can include a -i parameter to select an SSH identity (and -C to use compression for a possible speed boost). Add this to your client’s .gitconfig file, using the path to your client-specific identity file (not the public key which has a .pub suffix) like this:

[core]
    sshCommand = "ssh -i ~/.ssh/id_ed25519_client1 -F /dev/null"

Side note: I do hope you’re using Ed25519 keys for SSH; they’re newer, smaller, stronger, and faster than RSA keys, and they’ve been supported in OpenSSH since version 6.5 in 2014, so if your server doesn’t support them, you probably have bigger problems, or maybe you’re just running RHEL… I hope you’ve seen the post-quantum features of OpenSSH 9.0 too. The SSH client config file (usually found in ~/.ssh/config) is also really useful for twiddling per-directory or per-server configs that you can just set and forget.

Once you’ve done that, your commits will now be signed using your clients’ GPG key, and pushed to their repo using their specific ssh key, and you won’t have to change anything when you start new projects for them, so long as you put them in the same folder.

“What about my IDE?”, I hear you ask. Not to worry, most IDEs use your system’s git and ssh configs, so all this should work just fine with PHPStorm, VSCode, etc.

While I’m sure some bright spark can make this even more dynamic to automate this across clients, I find new clients are rare, but projects turn over fast enough for this to be a real win for getting that first commit signed and pushed correctly, first time.