Bridging Laravel Scout to Eloquent

Laravel Scout provides an interface to external search engines such as Algolia and Meilisearch. These are typically remote indexes of the same data that you’re storing locally, but with far faster and more powerful searching and ranking capabilities. However, they have some limitations on how you can express the search itself (allowing only a single string value), and what you can do with the search result (because it comes back as a Laravel Collection and not a Query Builder). This makes it difficult to do things like perform a complex search remotely, and then filter the result further via Eloquent operations, perhaps involving different parts of your database. A typical Scout search might be:

use App\Models\Order; $orders = Order::search('Star Trek')->get();
Code language: PHP (php)

This gives us a bunch of Order instances, but we had no opportunity to ask it to do things like load relations or filter the results while it was doing it.

Fortunately it’s not difficult to bridge these two worlds. The resulting Collection contains model instances, so we can extract their IDs and use them to construct an Eloquent search that selects the same records, but locally this time (Scout already did the heavy lifting of figuring out which ones we wanted):

$builder = Order::query()->whereIn('id', $orders->keys());
Code language: PHP (php)

This isn’t ideal (because it will fetch those records a second time), but it will be reasonably efficient because the IDs it searches on are exact matches for primary keys in the database.

We now have a builder that will select the same records as the Scout search did, but we can continue adding to it before requesting the final results.

$result = $builder->where('orders.name', 'like', 'a%') ->with(['orderItems', 'customer']) ->get();
Code language: PHP (php)

So that’s how we can get to use Eloquent features on top of a Scout search.


After finding out that this bridging wasn’t built-in, I submitted a PR to add it, but sadly it was rejected. With the PR code in place, the syntax would have looked like this:

use App\Models\Order; $orders = Order::search('Star Trek') ->toEloquent() ->where('orders.name', 'like', 'a%') ->with(['orderItems', 'customer']) ->get();
Code language: PHP (php)

To be fair, this isn’t much of a saving in the external syntax, but it is more efficient because it can get the record IDs directly from Scout without having to load the models from the database. I don’t think that efficiency gain can be obtained from outside Scout’s own code.

I hope that helps someone!

The Good Ship Laravel

I like writing songs about open source, but I’ve never actually released any or posted them publicly, mainly because my singing is fairly terrible, and trying to find others will to sing about these things seems hard! I really liked “the Wellerman” sea shanty craze of 2021, I had a thought that I should make use of the the nautical theme that runs through a lot of Laravel’s nomenclature to write a shanty of my own, that wasn’t just another cover of the Wellerman. It also occurred to me that I could semi-speak the words (in a pirate voice of course!) instead of outright singing, and that made it feel a bit less daunting. I wrote the intro first, and I liked the storytelling aspect, though as intros go it’s quite long. The first verse came quite easily as I built a list of words and kind of ticked them off the list. The timing and rhyming structure is straight Limerick, which makes things very easy. The main melody was just the result of doodling on the keyboard for a bit. I was pleased with the sailor/Taylor rhyme for the chorus, but it took me ages to come up with the rest of it. I had several failing attempts at a melody for the chorus, eventually just singing something that the words fitted, and then turning that into an accordion line, then building out everything else around it.

The instrumentation was very simple – it’s a sea shanty so we need simple folk instruments – accordion and cajon, and then a plucked upright bass to fill it out. In the final chorus I threw in some lovely blatty brass and a bit of piccolo, since the top end was kind of empty.

The Logic Pro arrangement

I recorded the vocals for the intro in July 2021, but re-recorded them later for consistency. I sang the first (lowest) line of the chorus vocals, and then did something I’ve done before – copy the track, and then use Logic’s Flex Pitch editor to shift notes around to make harmonies, generally upwards, since I’d sung a low line to start with. Having found harmonies that worked, I then re-sang the new line, as a heavily edited one doesn’t sound quite right, especially when pitch shifts are quite large. I then repeated the process for a second time, giving me a three-part harmony for the chorus. Flex Pitch let me correct pitch, but also timing – the harmonies sound so much better when they line up in time too. The low line was only possible for me to sing because at the time I was recovering from COVID and a very nasty sore throat, so while I was feeling much better, my voice was much deeper then usual, and I could hit much lower notes! Overall I found the singing much easier than the other things I’ve tried to sing because it was pitched much more comfortably for my voice.

Software & Hardware

  • Apple Logic Pro X
  • Behringer UMC404HD USB audio interface
  • Aston Element dynamic microphone
  • Adam Audio TR5V monitors
  • KRK RP10S subwoofer
  • Arturia Minilab Mk II MIDI keyboard
  • Behringer DSP8024 Ultra-Curve Pro (room correction)
  • Mackie Big Knob passive volume control

Instruments & effects

  • Accordion, cajon, upright bass, piccolo, seagulls and waves from Logic’s standard sample library
  • Brass section from Logic’s Studio Horns instrument
  • Rowing boat sample I found from some ancient soundfonts collection
  • iZotope RX7 noise reduction
  • SSL Channel Strip (EQ, compression)
  • Logic standard compressor, EQ, de-esser
  • Logic “Space Designer” reverb
  • One of Logic’s default mastering configs for final output
Intro:
I was cast adift in development seas
a shiver of bugs a’circlin’ me
Naught but a pair of oars and my IDE
to keep my app from drownin’

I spied at last a distant sail
I signalled for ‘elp to that caravel
As she hove to I made out her name
’twas the good ship Laravel!

Verse 1:
Gather ye round my developers
and I’ll spin you a yarn most eloquent
a tale of passport and breeze,
socialite and jetstream
a cloud full of vapor and elegance

We’ve resources and models and more
controllers and actions galore
Fortified with some rum,
and a sack of enums
we’ll build an app clients will adore


Chorus:
Train your telescope on that far horizon
Don’t get marooned on development island
We’re gonna build an app so well
On the good ship Laravel

Get on board now, every sailor
dance to the tune of cap’n Taylor
You’ve never built an app so well as
On the good ship Laravel

Verse 2:

With livewire on top of your scripts
and laracasts dishing out tips
We’ve got the best pest
to chase the rats from your tests
and artisan helping you ship

The framework’s the star, that’s for sure
but there’s packages of treasure to explore
but the best bit’s the crew,
and you can join too –
everyone’s welcome aboard

Chorus2:
Train your telescope on that far horizon
Don’t get marooned on development island
We’re gonna build an app so well
On the good ship Laravel

Get on board now, every sailor
dance to the tune of cap’n Taylor
You’ve never built an app so well as
On the good ship Laravel
The good ship Laravel

An explanation for non Laravel folk!

A shiver is the collective noun for sharks. An IDE is an integrated development environment such as PHPStorm or VS Code; think MS Word, but for programming. An app, in this context, is a web application built in PHP. Sail is the name of a Laravel feature for managing local development environments. A caravel is a 15th century Portuguese sailing boat, exactly the kind of vessel that a stranded pirate might encounter, and also the word that gave inspiration for Laravel‘s name. “Hove to” is a sailing manoeuvre used to more or less stop a boat by pointing the sails in opposing directions, very useful when picking up castaways. A yarn is a story, often nautical, and a thread, but it’s also the name of a Javascript package manager. Eloquent is the name of Laravel’s database abstraction layer. Passport, Breeze, Socialite, and Jetstream are all Laravel features for building authentication workflows. Real clouds are made of vapor, but Laravel’s serverless service is called Vapor, and runs in the cloud. Elegance? Well, it mostly rhymes with eloquent, and is something that any framework aspires to. Models, controllers, actions, and resources are all important parts of a typical web app built in an object-oriented style; I was planning to have a line about “plundering” to go with resources, but that didn’t make it. Fortify is another Laravel authentication feature. Enums are a common programming language feature, but notable because they were added natively to PHP 8.1 recently. Telescope is an in-app debugging utility. Horizon is a queue monitoring extension. Cap’n Taylor is of course Taylor Otwell, the creator of Laravel. Livewire is a toolkit for building dynamic, interactive web interfaces for Laravel apps. Laracasts provides an amazing library of training material for Laravel and related technologies, and also a great forum. Pest is a relatively new system for building automated tests that Laravel uses. Artisan is a command line tool that helps automate numerous development tasks. The crew is Laravel’s development team, but also the enormous and diverse community of developers that make Laravel far greater than a typical framework – it’s home for many of us!

Postman pre-request script for Laravel registration

A common way to register in a Laravel API is to send a POST request to /users containing a username, password, any other info, and also an HMAC signature using a server-side secret. In this example, it’s validated on the server by this class:

public const HASH_ALGORITHM = 'sha256'; protected const REQUEST_KEYS = [ 'email', 'name', ]; private $secret; public function __construct(string $secret = null) { if (! $secret) { throw new \InvalidArgumentException('The registration secret must be provided'); } $this->secret = $secret; } public function verify(Request $request): bool { return rescue( function () use ($request) { $hash = hash_hmac( self::HASH_ALGORITHM, json_encode($request->only(self::REQUEST_KEYS)), $this->secret ); return hash_equals( base64_encode($hash), $request->get('signature', '') ); }, false );
Code language: PHP (php)

So we can see that it’s expecting a Base64-encoded HMAC-SHA256 signature of a JSON array containing the email and name properties.

If you’re trying to make this request in Postman, you obviously need to calculate this same signature or it won’t work. Fortunately Postman has pre-request scripts that can inspect bits of your request and environment and generate new elements before your request is sent, and that’s what we need to use.

We don’t want the secret to be saved in our request collection, so we keep it in an environment, and pull it out dynamically when the request is made. Postman includes the Crypto-js package, which includes the necessary signature and encoding functions we need. Coming from PHP, the syntax for these operations feels very convoluted, but it goes like this:

const signature_string = '{"email":"' + request.data.email + '","name":"' + request.data.name + '"}'; const hmac = CryptoJS.HmacSHA256(signature_string, pm.environment.get('REGISTRATION_SECRET')); const b64 = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(hmac)); pm.environment.set("REGISTRATION_SIGNATURE", b64);
Code language: JavaScript (javascript)

This builds the string to sign from the request elements we need, calculates the HMAC of it using our secret, and then base64-encodes it before saving it in the Postman environment.

You can then add the signature into your request by adding the REGISTRATION_SIGNATURE variable to the body:

Hope that helps someone!

Minimal jQuery and extensions in Laravel

I recently needed to add a simple tag editor to a form input. I initially thought that Vue might help with this. That was a big mistake! Vue is seemingly pretty hopeless for this kind of thing – you can’t really use it for “sprinkling” JS elements, and progressive enhancement is a complete non-starter; that’s really more the domain of old-school jQuery, though I hear good things about alpine.js for this kind of enhancement. I learned a lot about Vue that I hadn’t planned on doing, but the overall impression was that it was simply the wrong tool for the job. I am not interested in building an SPA, and I’m certainly not out to rebuild my entire site just for sake of a single widget! My analogy for Laravel and Vue is like oil and vinegar in a salad dressing; they will happily coexist in the same bottle and taste great together, but they really like to be separate.

So, back to jQuery I go. jQuery was disabled by default in Laravel 6, but it’s just commented out in resources/js/bootstrap.js. It’s already included in the npm packages.json config file, so running npm install loads it into node_modules. I was really unclear about how additional JS modules should be added to Laravel using Mix. This was also a bit confusing because there isn’t really much you can point at and say “that’s mix”; it really comes down to what you put in webpack.mix.js in the project root and the npm run devscript that builds assets. However, the main thing that this does is load app.js, but then I found that things like this should be added in bootstrap.js (which is loaded by app.js) rather than adding them to the the mix config.

So the aim of the exercise was to add a tagging UI widget that takes a standard text input full of comma-delimited terms, and turns it into a pretty clickable tag element. Out of many worthy options, I settled on Tagify. After adding it with npm i @yaireo/tagify --save I had trouble figuring out where to actually load it from, but eventually I got it working by adding:

require('@yaireo/tagify/dist/jQuery.tagify.min.js');

to my bootstrap.js file, and evidently this looks in node_modules/to find the file without any additional help. After this I created a script in public/js/editthing.js (because inline scripts are worth avoiding if you’re serious about your CSP) that turned the standard input into a tag widget:

$('#tags').tagify();

and this script was loaded from blade using Laravel’s asset helper, which creates appropriate base URLs for public resources:

<script src="{{ asset('js/editthing.js') }}"></script>

I sometimes find Laravel docs and articles infuriating – there is often both too much and too little information at the same time, or they say things like “just add it to mix”. Many articles on simple subjects like this subvert Laravel’s features to shortcut to “working” (but ultimately counterproductive) solutions, and I wanted to approach this “the Laravel way”. For that reason I thought I’d make detailed notes for these very simple steps that I couldn’t find written down together elsewhere!

HTH