Developing a Full-Text Search Enabled Meteor App

I was looking for a simple way to add full-text search to the Meteor app I’m currently developing. And. Found nothing. There is an item for this on the Meteor roadmap. Originally this was planned for later as part of Meteor core. Recently it was moved to “important but not part of core”.

Since I have acquired some admittedly rather theoretical knowledge about information retrieval during my master’s program I decided to roll my own search and contribute it to the Meteor community. I named it Spomet (a contraction of Spotting Meteors).

In this post I will build a search-enabled Meteor app, and along the way show you how to integrate Spomet, and how to index and search some content. For the impatient, check out the final product hosted at meteor.com and the full sources at GitHub. Additionally I recently published a post, focused solely on Spomet and it’s API.

Basic Setup

For this tutorial, I’m assuming that you have some basic CoffeeScript knowledge and that you have already downloaded and installed Meteor. In case you lack one of those requirements you should check here for the basics about CoffeeScript and here for Meteor’s installation instructions.

The first step is using the console to create a basic Meteor app as a starting point. The following command creates an ideas folder and puts three visible files (ideas.jsideas.html and ideas.css) in it. Actually you have a basic Meteor app by now.

meteor create ideas

You might want to try it.

cd ideas

meteor

Navigate your favourite browser to:

http://localhost:3000

And you should see something like this:

Basic Meteor app after meteor create
Basic Meteor app after meteor create

To stop the app switch back to the console and hit [STRG]+C.

Setup Continued

Create a packages folder.

mkdir packages

mkdir packages/spomet

Get the code. It’s hosted on GitHub.

cd ..

git clone https://github.com/Crenshinibon/spomet-pkg.git

cp -r spomet-pkg/* ideas/packages/spomet/

Since version 0.6.5 of Meteor you have to explicitly add custom packages to your application:

cd ideas

meteor add spomet

Alternatively you can use Meteorite to add the package to your app. You have to install Meteorite first and then you can use the mrt command and add Spomet to the app:

npm install -g meteorite

mrt add spomet

To enable Meteor’s support for CoffeeScript you have to add another package:

meteor add coffeescript

Add Twitter Bootstrap, for simple and nice user interfaces; jQuery, for DOM navigation and manipulation; and Meteors account management system, that provides quite potent user management out of the box. We will use those later on.

meteor add bootstrap

meteor add jquery

meteor add accounts-base

meteor add accounts-password

meteor add accounts-ui

We are removing the autopublish package, because we are going to leverage Meteor’s pub/sub mechanism to filter and sort the ideas.

meteor remove autopublish

The Example App

To put the search functionality to any use, we need to develop a small example application.  I have chosen an application for ideation management. It is basically a facility to enter and examine ideas. Real ideation management systems (like Spigit or Hype for example) are used by bigger companies to collect ideas from their employees or customers. Ultimately leveraging the wisdom of the crowds. Here are the basic requirements, I’m going to develop in this example application:

  • Let employees/customers create user accounts
  • Authenticated users might insert ideas, with a title and a description
  • Let authenticated users vote for ideas.
  • Show employees existing similar ideas to the idea they are currently looking at.
  • Let employees search for existing ideas
  • Display existing ideas, ordered by newness or votes or by search relevancy

UI Layout

Here is the user interface mockup I came up with. I used, again, the fabulous moqups.com to create it. It’s nothing fancy and should be somewhat self-explanatory.

UI design made with Moqups
UI draft made with Moqups

File Structure

Every Meteor example that I came across groups it’s files roughly in the Ruby on Rails structure of things. At the first level the structure is driven by technological aspects (models, templates, client, server). I don’t really like this approach, because it doesn’t fit the way I approach software development. I iteratively add functionality. Doing this by modifying many different files in many different directories feels awkward. Coherence is lacking, an OOP guru might argue.

Meteor is actually very relaxed regarding where you put the files, and I will  leverage this here and propose grouping by functional area, then split by technological aspects as needed.

The automatic file loading of Meteor might make things difficult. So a good idea might even be, to encapsulate functionalities into Meteor packages.

A First Version

We are going to provide an entry point for our application in main.html. Therefore we are renaming ideas.html to main.html and open the file in our favourite editor. TextMate in my case.

mv ideas.html main.html

mate main.html

We are making heavy use of Bootstrap’s facilities to layout and style our application. In fact, I will try to circumvent any custom CSS definitions. There will be some custom CSS classes, but only to ease identifying HTML elements.

To get you started, I will briefly explain, what the certain Bootstrap classes do and how the Handlebars templates, that help you in structuring your app, work in Meteor. But first let’s have a look at the completed main.html.

There isn’t much going on in here. Besides the obvious html enclosing tag, we define a HTML head area with a title (this gets displayed in the browser’s window title). The remaining things that make up a decent HTML5 boilerplate are handled safely by Meteor.

Inside the body tag, we create a basic structure for our app. We wrap everything into a div and style it with the container-fluid class from Bootstrap. This class gives us a 12-column layout with pretty size adjustments on browser resizing.

Inside this container we define two rows with div tags. The row-fluid class let’s the browser render the contained elements in a new row, spanning the whole container. With {{loginButtons}}, the first element provides access to Meteor’s built-in user management system.

But we are going to configure it to use a username and make the eMail field optional. To achieve this we create a main.coffee file in our apps root folder:

touch main.coffee

mate main.coffee

And put the following code into it:

Check out the Meteor documentation for more options configuring the accounts package.

Back to main.html.  The second row is split in half. This is achieved through the span6 class. Bootstrap gives you span1 to span12 classes to control the size of child elements in each of it’s rows. For predictable behaviour the sum of spans in each row shouldn’t exceed twelve.

You can use offsetX classes to force spacing between the elements of each row.

{{> ideas}} and {{> similar}} include Meteor templates into the calling template. Those templates will be defined in different files. But first we have to improve the project’s folder structure.

To group the files delivering a functionality we create a folder for each:

mkdir ideas-list

mkdir similar-list

mkdir idea-details

We create a file: ideas-list.html in the first folder and open it with our editor.

touch ideas-list/ideas-list.html

mate ideas-list/ideas-list.html

Here comes the code we put into that file. It’s much more code than before. But don’t panic it’s quite simple and someone more tidy than me, would have split the content into more files, but that wouldn’t reduce the amount of it.

The first thing to mention is the template tag. With it’s name attribute it gives you a handle to include it into any other template as you might have guessed already.

The template we reference from main.html is named ideas. The very first in idea-list.html. In it we define yet another container, which gives us rows with 12 columns, again. We define three rows, one to hold the header, one for the controls for the list and another one to present the ideas.

The next template in the file simply displays a header (“Ideas”) and in case the user is logged in, a button to create new ideas. We use {{#if currentUser}} to conditionally display the button. #if is called a block helper and currentUser is a convenient way to ask Meteor which, if any, user is currently logged in. We give the button the Bootstrap class btn, which makes it visually better suiting.

In the first column of the ideasListControls template we are accessing the search field template provided by Spomet. We are delivering some custom layout information with searchFieldConf, through calling a template helper here. Basically we are reaching out to the controller, if you will.

Since we don’t have the controller yet, we create it with:

touch ideas-list/ideas-list.coffee

mate ideas-list/ideas-list.coffee

And put the following code into that file.

With Meteor.isClient,  we make sure, that the code gets only executed on the client. Then we define the helper, by using the templates name and the same name we are referencing in the template. We return a simple object, that specifies the size of the search field and the text to be used on the search button.

Back to ideas-list.html and the ideasListControl template. After including the search box in the first column we are including another template in the second column, who’s job is to let the user specify the field to sort by and the sort-direction.

In the ideasListSort template we put a form tag and give it the Bootstrap class form-inline. Both let the sort controls appear neatly in one line with the search field. Inside the form we define a drop-down box using the select tag, define it’s size with the class input-medium and put some values into it with the option tags.

The button element is made a little bit prettier and colourful with the classes: btn, btn-small and btn-info. The button holds two icons, to indicate the actual sort direction. Those icons come with Bootstrap and are provide free-of-charge from Glyphicons.

The last template in this file: ideasList, is meant to display the actual list of ideas. It’s very basic. It iterates over the ideas and calls yet another template to display each of them. #each is a block helper provided by Handlebars. It tries to access the template helper ideas to get an array like representation to iterate over. Since we don’t have the template helper, yet, it shows nothing.

There is still one template missing we have referenced in main.html. We create it with:

touch similar-list/similar-list.html

mate similar-list/similar-list.html

And put the following code into it.

As this doesn’t show anything new and we are going to touch this file later again, I will spare you lengthy descriptions. Instead we are going to start our app for the second time and have a look at what we have achieved so far.

meteor

Now point your browser to http://localhost:3000 or reload the previous tab. You should see something similar to this:

Screen Shot 2013-08-09 at 10.52.39 AM

And when you create an account, it should look like this:

Screen Shot 2013-08-09 at 10.53.51 AM

Adding Behaviour

For now we can’t do much with our app. So let’s move on. At first we need something to store the ideas. Meteor gives us collections, those are pretty mighty. They do the heavy lifting of moving the data between the server and the clients, they make the data persistent on the server (using MongoDB), they provide a cache on the client (MiniMongo) for fast responses and finally they are reactive, which means, every change in the underlying data results in updates of the user interface referencing the data.

We are going to have only one collection by now, named Ideas. Since we need it in all parts of our application a natural place to put it, would be the project’s root folder. (Un-)fortunately Meteor automatically loads the files, but in a specific order. Deeper goes first. So our collection wouldn’t be available in the deeper folders if we define it at root level.

To circumvent this we are going to put the collection file into the lib folder, which get’s loaded first.

mkdir lib

touch lib/ideas-collection.coffee

mate lib/ideas-collection.coffee

We put the following code into the newly created file:

The first line alone creates the Ideas collection. Because of the prepended @-sign we can reference it from the other files of our application. The remaining code inserts test users to Meteor’s user management system and some test ideas to the Ideas collection. It should be possible to pretty much guess the purpose of the idea’s properties from the source.

I think only votes and votesCount need a short explanation. In votes we store the users, that have voted. This way, we are able to ensure that every user can only vote once. votesCount holds the number of votes, which is the same as the number of users in the votes array. So why this redundancy? Well. It makes life simpler. Especially when we later need to sort by number of votes.

Besides putting the example-ideas into the Ideas collection, title and description of each idea is added to Spomet. We call the method Spomet.add and provide the necessary attributes in a hash. Two attributes are generally mandatory, when adding documents to Spomet: the text, obviously and the base reference – the ID of an idea in our case. Because we want two attributes of our ideas, title and description, to be findable, we are using the path attribute as well. The type parameter could have been omitted, but might be helpful if we want to add other types of documents (comments for example) later.

Now let’s tackle the list of ideas. We are going to keep it simple for now. And change the ideas-list.coffee file into:

We have discussed the first part above already. New are the Meteor.subsribe call, the ideas template helper and the Meteor.publish call. The pub/sub methods provide access to all entries in the Ideas collection and the helper hands the result down to the template. Next we have to define the idea template, so that we can actually see ideas.

touch idea-details/idea-details.html

mate idea-details/idea-details.html

There, we define the idea template that we have referenced from the ideasList template in ideas-list.html. Remember: for every single idea anew. We provide two representations. The first is a short version, showing only the title, the change date and the votes for the idea. The second additionally displays the full description beneath the former values.

Only one idea can be expanded at any given time. We control this with a session variable. Meteor’s Session is a reactive datasource like the collections, so changes to a variable will result in a flawless update of the referencing parts of our app. Here is the full version of the ideas-details.html file:

You should be able to follow the basic flow by now. The {{#if expanded}} block helper distinguishes between header-only and the expanded version and redirects to the appropriate sub templates.

By the way, the Bootstrap class well gives us the nice looking border and background for each idea, no custom CSS magic needed here.

The ideaHeader template is divided into three regions. The first holds the idea’s name. The second the change date: prettyChangeDate? What the heck? …  Let me explain: we are going to include moment.js to get nicely formatted dates. So we are accessing a helper here, that delivers us a formatted version of the date. I will come back to this in a minute.

The third part of the header resides in yet another template. It provides the voting mechanism. Users are not allowed to vote on their own ideas, so we omit the voting-button for them. The span shows the number of votes. We let Bootstrap present it nicely by utilising it’s badge class.

The ideaDetails template loads the ideaHeader template we have discussed just above and provides beneath the area to view and edit the idea’s description. Only the author of the idea gets a textarea to edit the description. In case he changes anything to it, buttons are shown to cancel or submit the changes made.

I guess I will discuss the inclusion of a full-fledged wysiwyg editor in a later post. This one is really packed. Xing’s wysihtml5 in combination with bootstrap-wysihtml5 are looking promising but have some reload issues. This editor might yield better results but lacks support for older browsers.

The logic behind the idea-details templates resides in corresponding coffee file. Create and open it:

touch idea-details/idea-details.coffee

mate idea-details/idea-details.coffee

Here comes the code:

You are still with me! You are so brave!

The first two definitions handle the expansion of ideas. The very first is a helper, that accesses a session variable to find out, if the current idea should be expanded. Remember Session is a reactive data source, so in case the value changes, the UI is adjusted accordingly. No action on our side required.

The second thing looks a little bit different, than what we have seen so far. Here we don’t define a function. Instead we call the function: events of the idea template and provide an object, a so-called event-map, as the parameter. The object’s keys define the conditions when it’s function value should be called. Here the click event on a div tag with the class idea results in a call to the function specified.

The template’s context, the idea object in the actual case, is available inside the function as this (@ in CoffeeScript). We use it to acquire the unique _id attribute and store it in the session variable, that controls the expansion of an idea. The change in the Session variable automatically re-renders the dependent portions of the user interface.

Including moment.js

Next we provide a prettified version of the idea’s change date. This is achieved by using moment.js. Which is an extremely powerful library to work with JavaScript dates. Moment.js is not available as a native Meteor package. But it’s generally possible to include any third party JavaScript library into your project. Recent versions of Meteor made this even easier. Create the folders:

mkdir client/

mkdir client/compatibility

Download moment.min.js from their website and put it into the latter folder.

That’s it.

A call to moment() in the web browser’s JavaScript console, as well as our prettifier function should yield the expected results.

The helper function: Template.ideaVotes.userIsNotAuthor determines if the current user is not the author of the current idea. This controls the display of the voting-button.

Next we define an event-map for it. A click event on the button executes the function. Here we use e.stopPropagation() to prevent the event bubbling up the DOM tree. Otherwise the expanded state of the idea would be altered when we vote. Remember the click event we registered on the div-element wrapping the idea.

Work-In-Progress Description

After that, we are defining the helpers for the ideaDetails template. The first one determines if the current user is the ideas author. It controls if the textarea – to manipulate the idea’s description – is displayed or simply the current description.

The helper: transientDescription returns a work-in-progress version of the idea’s description. Which is the same as the stored one, by default. We store all transient descriptions in the session variable. This way the user doesn’t have to save before editing another idea.

When the user changes the description, this change is reflected in the transient version first. He has to explicitly save those changes to trigger an update of the stored version.

This whole session variable approach has some drawbacks. For example: if the user hits refresh all changes are lost. Another one is the clumsy handling. You have to get and reset the whole variable every time you wish to update a single value.

Another approach is using a local collection to store the unsaved values. This would result in easier handling, but the refresh problematic would still be valid. Yet another approach would include storing the unsaved description side-by-side with the saved description in the idea object. This would make the changes persistent, but the additional memory consumption might be regarded as a drawback here.

That said you are free to choose your own path. For the sake of our example app I will stick to the Session variable solution, though.

The descUpdated template helper is responsible to find out, if there are changes between the currently persistent version of the description and the work-in-progress version stored in Session. You might remember, that this helper is used to determine if the Cancel and Save buttons are visible.

The last part of the idea-details.coffee file is the event map for the ideaDetails template. We register a keyup event on the textarea, which updates the session variable holding the work-in-progress description.

A submit event on the form makes the work-in-progress version of the description persistent. Furthermore it updates Spomet to the actual version of the description by calling Spomet.replace. Spomet.replace takes the same hash we have used when we initially added documents to Spomet as it’s first argument. Spomet allows you to handle more than one version of a document. So, optionally you can use a second parameter, the version number of the document to remove after adding the submitted version. If you don’t specify anything, the last version get’s removed.

We are not going to use multiple versions, so we simply omit the last parameter and rely on the internal mechanism to replace the only current version with the new one.

Phew! That was a lot of code and a lot of text explaining it.

There are still many things to cover, though. Actually accessing the search results, for example. And what about creating new ideas? And finding similar ones?

You should consider a short fresh-air break.

Really.

Creating New Ideas

Before further looking at the search facility, by now we just added or updated documents, let’s handle the creation of new ideas first.

The New button resides in the ideas-list.html file, to be specific, it is defined in the ideasListHeader template. So the next step is to open up the ideas-list.coffee file. And now?

The first step is quite simple and should be obvious: add an event map on the ideasListHeader template which defines an click event on the button. But what should happen? Add a new idea straight away to the Ideas collection? What are the default values, then? Is this version going to be added to Spomet?

Using a local collection for unsaved Ideas and committing those to the real collection later might be another option? Flagging the newly created ideas to distinguish between their existing peers might work as well. This question proofs always difficult for me to decide.

For this example I will settle on a local collection. I took this approach in another project (I will report about it, in a post on it’s own) and it felt quite natural. But it requires a few things.

  • Define the local collection.
  • Display the local collection’s elements.
  • Provide some way to commit a new element to the real Ideas collection.
  • Provide a way to discard newly created ideas.

We define the local collection in ideas-collection.coffee by adding the following two lines:

We have to restrict it’s visibility. Only the clients can reasonably handle local collections.

There should be a place where the unsaved ideas are going to be displayed. For that purpose we extend the ideas-list.html file by another template.

This template is embedded just above the search and filter controls for the existing ideas.

The controller is extended with the following helper to hand the new ideas down to the template.

Now we need to define the templates and controller to display and handle the new ideas. We create a new folder.

mkdir new-idea

And put the following files in it.

touch new-idea/new-idea.html

touch new-idea/new-idea.coffee

Their contents should be fairly self-explanatory by now.

Accessing Search Results

We can create new ideas now, the next step is to actually access and display the results a search might yield. After that we are going to look into sorting the ideas. But first things first.

We are going to include a fork in the ideasList template (it’s in ideas-list.html). In case the user is currently searching not the idea template gets called but the to-be-created result template, instead.

In the controller we define two helpers. The first accesses the defaultSearch object (the one associated with Spomet’s search box) and asks for the current search query. If there is none, then one can conclude, that there wasn’t searched for anything. If there is a query, well the user obviously searched for something.

The second helper accesses the defaultSearch object as well and asks for the results. The returned value is a Meteor collection cursor. So we can return it safely to the block helper #each loop.

The documents in the result set have the following structure:

That shows us, that we can’t simply reuse an idea template, to display a result. There is some mapping necessary. And because we additionally want to display the search score (just to give visible feedback when sorting by it) we create a new template and put it inside it’s own files and folder.

mkdir result

touch result/result.html

touch result/result.coffee

In the first file we define the result template. It adds a row where the search score gets displayed. Then we use the block helper: #with to set the correct context and call the original idea template within.

The next file shows the corresponding controller. There is a helper to reduce the number of decimals of the score. And another helper to find and return the corresponding idea. The idea is picked up by #with and then injected into the idea template.

Searching for existing ideas should work now. The default order in which the ideas are displayed is defined by the result’s score. Let’s make this adjustable.

Sorting Ideas

At first we have to filter the current sort options to match the current situation (searching or not), because it doesn’t make sense to order by search score if the user isn’t currently searching. To achieve this we have to change the idea-list view and the corresponding controller.

We change the ideaListSort template so, that the select options are received from the controller in the #each block helper and include an option for every returned value. Furthermore we make the appearance (white or black) of the icons on the sort direction button controllable by the corresponding template helpers.

In the controller we define the template helpers that are expected, as well as several functions that are necessary to make the sorting work.

At first we define two session variables, one to store the sort field and one to store the sort direction. We use Session.setDefault here, so that the value is set only once. The next function (updateList) uses this information to update the actual list, by re-subscribing to the published collection. We distinguish between the list of searched and not-searched ideas, because there are different actions involved. In the first case we access the defaultSearch object and set the sort, the actual re-subscription is handled by the Search object. And in the latter case we call the function: reSubscribe, defined in idea-collection.coffee. We changed the earlier defined subscription accordingly:

The updateList function is called from the event callbacks associated with sort controls. Those are defined at the bottom of the prior gist. The function toggleSortDir is called from the click event of the sort direction button and does what its name implies.

The sortKeys hash is a simple helper to map from the string displayed in the sort field dropdown box to the corresponding database fields. Real world apps with multi-lingual user interfaces would have to take another route to achieve something similar (using HTML attributes on the options, for example).

The template helpers: sortAsc and sortDesc check for the current sort direction as stored in the session variable and deliver the CSS class string: icon-white in case of a match.

The template helper: options, delivers the available options for sorting the ideas, depending on the fact, that the user is currently searching or not. By looking at the sort-field session variable the selected state for each option is evaluated. Not-fitting options are replaced by default ones.

The last part is the afore mentioned event map. That should really be self-explanatory by now.

Sorting works. We are almost done. Let’s tackle the last part of our app. Searching for similar ideas.

Searching for Similar Ideas

We have already created a folder (similar-list) to store the related files and created the basic view (similar-list.html). Now we have to populate the similarsList template in a meaningful way.

We are reusing the result template earlier discussed. Next the controller has to be created and edited.

touch similar-list/similar-list.coffee

mate similar-list/similar-list.coffee

The controller creates a new Spomet.Search object called simSearch, makes it available throughout our app be prepending the @-sign and configures the search to only use the custom index. This is basically a performance consideration. The template helper simply delivers the results form simSearch.

By now nothing happens. We have to actually trigger the search in the desired circumstances. Therefor we update several other files. At first we activate the similar search in case we select an idea from the list. To achieve this, we append the event map for the idea template in idea-details.coffee. The changed event callback should look like this:

We create the search phrase by joining title and description of the selected idea. Then we exclude the selected idea from the search results by calling setExcludes with the idea’s _id wrapped in an array. After that we trigger the search by calling find with the phrase as the parameter. Clicking on a idea in the list triggers the similar search and similar ideas are listed on the right side.

Additionally in the idea-details.coffee file we extend the keyup event of the ideaDetails template. So that when the user edits the description of an idea he gets shown similar other ideas.

In new-idea.coffee we register two new events: one keyup event for the title text input as well as one for the description textarea. While typing the contents of new ideas the system constantly updates the similar list to display mutually related ideas.

That’s it.

A screenshot of the final version.
A screenshot of the final version.
Advertisements

One thought on “Developing a Full-Text Search Enabled Meteor App

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s