Podcast

On this episode, I am joined by Thomas Payet, the co-founder and COO of Meilisearch. From the Solspace side, I am joined by David Estrada, the developer on the Solspace team with the most Meilisearch experience.

Meilisearch is an open source API-based utility focused on excellent search speed and relevance of content within a given website. In this first episode of our two-part series with Thomas, we get into an overview of how Meilisearch came to be. We talk about the history of the search bar on websites and discuss how technologies evolved to support users as they try to search within a website for the content they desired.

We discuss why Meilisearch went in the open source direction and we learn about their plans for monetizing and generating revenue to support the overall effort. The prime revenue stream is expected to be their hosted cloud offering. They will handle scaling for customers, software updates, uptime and the like for a reasonable and predictable fee.

Full Transcript

[Music] Welcome to the Solspace podcast. Thanks for listening.

Mitchell: Welcome back everybody. Thank you for joining the Solspace Podcast again. I don't really know how it's possible for us to get the kind of guests we manage to get on the podcast.

I mean, it's a little tiny thing, small web development agency. It's not a big deal, but occasionally I get really cool people on here. So today, my guest is Thomas Payet from MeiliSearch.

Thomas, you're the co-founder and chief operating officer at MeiliSearch. Is that right?

Thomas: Yes, exactly.

Mitchell: Okay. And also on the podcast is David Estrada from the Solspace team. I asked David to join because he introduced Solspace to MeiliSearch as a possible solution, in particular on one client we just deployed it for.

And David did virtually all the work on that project, as well as some of the other uses that we've put it to. So I thought this would be a good group to have a conversation about MeiliSearch. So Thomas, can I ask you to just introduce, for someone who's never heard of MeiliSearch, has no idea what it is, can you talk a little bit about the very basic fundamentals of what this tool is?

Thomas: Yes, of course. MeiliSearch is an open source search API. That means that we are a tool for developers to use to implement a great search experience on their own websites, on their application.

So we are providing the best, trying to provide the best developer experience when it comes to implementing a search bar. So we are used today to use Google, Amazon, and they do provide a great search experience. But on a lot of different websites or applications, it's less than good.

And it's often difficult to use because you have to adapt to the search bar, you have to really understand how that works, if you want to find the things you're looking for. And so we are building the technology behind the search bar to make it as relevant and as fast as possible. And so developers just have to use our codes to make a great search experience on their own app.

Mitchell: Okay, you're making me want to ask about the history of this type of search capability on websites. So maybe backing up a decade or two, of course, the types of websites that Solspace works on, we're always doing CMS work. So there's a content management system underlying the website.

And in almost all cases, that CMS will have a native search capability built in. So you can have a search field, you know, put your keywords in, hit submit, it goes and hits the database for that CMS, and returns results through a templating system that's owned by the CMS. That's old school. Now we're moving into a more of a headless orientation these days with how these websites are built in the first place.

But the search aspect of all this, you know, you have that capability. But then back in olden times, you could also use the Google search appliance to bring search to bear on a given website, where the idea was, you just want to search that website content, just that website, you don't want to go anywhere else, just that specific website's content. So maybe we can talk about sort of the progression of search in web development from there.

And I wonder if you could speak about that.

Thomas: I think search is like one of the first feature or tool you would need when when you manage data. So when it comes to content management, STEM, you, you need a search system at some point. And search has always been existing search engine.

It's not something new. Google has been very good at popularizing what is a search engine today. But yeah, it's been existing for as long as the computers exist.

And I think that, yeah, today, there are a lot of search implementation that already works. And usually, it uses like the database that you already have, they do provide basic search feature. So they do manage to find the documents or the content you are looking for based on the query that you made.

But it's a whole job on its own to adapt and have a search experience that is intuitive, that gives the right document you're looking for, because depending on the context, depending on the type of document you are searching in, depending on the end user on the search bar, all of that makes that all search bars have different needs. And you might have different expectation from depending on where you are and what you're searching for. And so yeah, today, if you're building any websites, you would use like, basically, you would use like a like function on your SQL database.

So that would try to find the document that contains the query that you just wrote. But it takes some time because your database is not made for that kind of queries. So it has to understand what you're looking for, what are the mistakes you could have making the queries and stuff.

And yeah, that's difficult. And so I think 10 or 20 years ago, there is a very popular open source project created was it was Lucene. It is Lucene actually, yeah.

Mitchell: Oh, yeah, that's right. I remember that now.

Thomas: And I think it's one of the default option when it comes to build a search feature in any app today. Lucene is not that easy to use for developers today. From there, Elasticsearch was created 10 years ago now, maybe 15 years ago.

It's based on Lucene and it provides a better developer experience, I would say. There is an API, it scales, so there is a lot of features on top of Lucene. And now Elasticsearch is completely like the default option, the default tool that a developer would use to build a search bar within the app.

And Elastic is made for a lot of different use cases. And I think today, one of the main use cases is not to search from an app, like what we can imagine as end users today, but you can use Elastic to search in logs, to make complex queries in data analytics. So imagine that you have a whole data set of the population of a country and you want to know the average age of all the people called David.

This is something you could do with Elastic. And so today with Meilisearch, we are trying to focus on a subset of what you can do with Elastic. We are trying to do like front-end search.

So the search that we are all used to have on any app, and we are trying to make it as intuitive as possible. So there are a few algorithms that we use to make it intuitive. So we do manage typo tolerance.

So we make sure that, yeah, if you make a mistake in the words that you type, we can find the right documents. We manage the prefix search. So we know that if you begin to type something, we can already start to search for something and get it back to you.

So we are actually using today's technology to build the search that we are used to on Google and provide it to any developer. So it's evolving and there is still a lot much more that we can do. Today, if you are searching for something on Google, and I'm searching for the same keywords, we might not have the same results because Google is taking into account your search history, your context, actually all your metadata that you have on your Google account.

And this is something that we should be able to provide to any kind of app. When someone is searching for golf, depending on the context, you might want to return results about the car, the golf by Volkswagen, or the sports. And yeah, this is all the improvements we can continue to build in search today.

I don't know, have you used the search on Twitter? I think it's bad. It's always difficult to find the tweets of the person you want to follow.

And it's because it's actually very difficult to index all those tweets in real time to understand what the user is searching for. Is it a name? Is it a tweet?

Is it the content of a particular link? And so this is what we aim to build at Meilisearch is like an API that would empower developers to build a great search experience easily and fast. So out of the box, we do manage a lot of different stuff.

But then we make sure that we provide a lot of different rules that you can fine tune the search and the features because depending on the app that you're building, and the developer knows the business, how can I say, yeah, the use case that he wants to build a search for. So you need to fine tune it. And yeah, this is the compromise that we have to find at Meilisearch is making most of the things working out of the box, but still give control over what's really happening under the hood without making it too complex and difficult.

Mitchell: Yeah, you're always going to be fighting with that. I mean, that balance is always a problem that we're going to hand over to you. And see, the great thing about your approach, though, is you're so firmly rooted in the open source approach to software development, to product development, and to even running a company and having a revenue model.

So I was reading about some of your blog posts and getting caught up on that and what a commitment you've made to that. So because you are taking an open source approach, has that borne fruit for you to get the kind of feedback you need from the developer community so that they can help guide you where they need you to go? Is that one of the strengths?

Can you maybe talk about that a little bit?

Thomas: Yeah, when we started Meilisearch, we made it open source by default because we are developers and we didn't think about it. That just made sense. And later on, we realized that this is actually a different way of doing business and that open source is like a big way of getting users' attention and feedback very early on.

So while the code was open source from the beginning, it took us a year to have a proper open source project with good documentation, with an API that is usable. And from there, it's only been... So I'm sorry, when we started Meilisearch, we were working as software developers for an e-commerce company and we were working on this search program.

And building Meilisearch was a no-brainer between the offering that you could have with Elastic or Algolia. And coming from there and understanding the pain as a developer to implement a good search bar, I think we find it quite easily, the exact pain points that we had as developers. And we were not the only ones to have that kind of pain points.

And we built it for ourselves first and then we just had to think about how to make it, how to adapt it to more use cases in the end. So it was about documenting the code, having an intuitive behavior when it comes to interacting with an API. And very early, because all of that was on GitHub.

So GitHub is this platform for developers and this big social network for developers where you can see the code of others and comment and contribute. Very early, even if we didn't communicate that much about it, people who were following us saw that we were building Meilisearch. And they were trying it because it's free.

We made sure that you could try Meilisearch in less than five minutes. And that's what actually happens. Developers, they were ready to invest five minutes to try a new sort of solution.

Mitchell: This is what happened to us. That's exactly what happened with us. I mean, David showed me, why don't you try this?

And I thought, I have to make time for that. He said, just go fire it up on your own local machine. And he's totally right.

In just a matter of moments, I had an instance of Meilisearch running on my local machine. A few more minutes, I was hitting it with an API. A few more minutes, I was pushing into the index, creating documents.

So that's a big selling point. I mean, you already know that, but I can definitely confirm as a developer who's trying to encourage clients to adopt this as a solution on their site. That's the key, is getting that ease, like cutting the friction and letting us get in there really quickly and understand and validate it.

Thomas: I'm so glad that you had this experience. We quickly realized that explaining what a search API is can be complicated. Explaining the features, how we differentiate from Elastic or from Algolia.

On paper, using the features, it's something very complicated to write about and make sure people understand. So the bet was really to get Meilisearch running and up to speed with your own data as fast as possible. So very early on, we have this open source repository with the core project, so Meilisearch.

But we have all those integrations in different languages, for different frameworks, so that any developer can use his own toolset, if I can say so, to try Meilisearch with what he's used to work with. So it was an investment at the beginning, but today it really pays because if you are thinking about implementing a search bar and you have those different options, I think Meilisearch is the easiest one to get to start with. And from there, we've made the most difficult part of the job and the second would be to keep you.

Yeah, again, if you tried it and you're happy with the result, happy with the relevancy and the speed, this is always a huge win for us.

Mitchell: David, when you tried it, how did you first get started? I mean, I got started because you sent me some notes about like shortcuts on how to really get going with it quickly, but you were sort of the person who broke ground using Meilisearch at Solspace. What did you do?

David: Yeah, at the time, like a couple of years ago, I was mainly using Algolia to do a bunch of integrations. Then I come from Laravel, so PHP Laravel background. So I noticed that they started integrating that into Eloquent, which is the ORM in Laravel.

So that's how I find out about Meilisearch. Interesting. And then I started like, okay, this looks really awesome.

It's on par with Algolia in so many features. It provides the SDKs that I need in order to build things quickly, like you were saying. And then from there, I went into testing things here and there, just like Mitchell described, like installing in your machine was a big thing.

And the ability to start like sending documents to indexes here and there, start testing and seeing what the capabilities of this new thing that I found were. And then I started to preach that in the company and tried to convince Mitchell to, we should start offering this solution to this client that we had. Because we tried for a time to to try to get some stuff from Algolia, but it was difficult for them to do this.

So we started exploring this and tried to go into the Meli. I also checked your competitors, which is TypeSense. But interesting enough, like you were mentioning the documentation, and that was key for me, because I was the one that had to develop this thing.

So I needed to have really good documentation, or we needed to have really good documentation in order to to make this project successful. And that's why I really love about Meilisearch that they provide, or you guys provided us with really good documentation. So really good ways to go from point A to point B in no time.

Mitchell: So there are some factors in the sales process, Thomas, that's probably useful for you to get more feedback on. I'm sure you're sure you hear this kind of thing, but a little bit more is useful. We knew that we had a client who had a large data set, and they had the faceted search problem as opposed to the keyword search problem.

And I'm going to reserve a lot of that for the second episode we're going to have with you. But the ability to have a long array of checkboxes of a bunch of different facets of the data to allow their users to sort of filter and navigate and explore and browse the data set was critical. And that sort of screamed out for, at the time, Algolia.

So we managed to engage their sales team and talk to them about the problem. And the challenge is we could not get a clear picture on cost. We couldn't pin them down.

We couldn't get clear numbers on how much this was going to cost my client every month until we ripped it out and replaced it with something else. So we're going to spend a bunch of money to rebuild search on their site on Algolia, but we didn't know how much it was going to cost, if it was going to be too expensive. So along came Meilisearch, which offered the same set of solutions we needed, but because it was open source and we could run our own server version of it, the cost was completely predictable.

Within range, like, are we going to need to upgrade the server and get some more horsepower? We'll see, but we'll take the risk. So that was a really big factor for us to be able to predict the cost.

And I wonder, maybe you could talk a little bit about that as part of... And we haven't even gotten into the question of the cloud offering, the subscription cloud offering that you're in the process of rolling out. So maybe we could get into this a little bit.

It's pretty important.

Thomas: So we've been working for this open source project for two years and a half now. And now comes the problematic of how to make it sustainable. And from the beginning, we knew that offering Meilisearch as a service could be an option.

There are many ways to monetize on top of open source. But we decided that hosting Meilisearch, offering Meilisearch as a service would be like the first option we wanted to try for two main reasons. The first one being that developers and users of Meilisearch were asking us for a listed version.

And the second one is it's easier for us as a young company to build a product than to go and sell Meilisearch to bigger enterprises, if that makes sense. So we decided to take the cloud route. And this is what we've been building for the eight or 10 months.

So it's Meilisearch that we are hosting. We do the update. We do the upgrade.

We make sure that you don't have to think whether if it's up or not. And we do manage all of that for you. Today, we've been onboarding a few companies already.

So we might have up to 50 or 60 companies already using the Meilisearch clouds. It's closed based for now. Actually, we plan to make it generally available in a few weeks.

So it's in some matters of days.

Mitchell: That's great. Those are good numbers too.

Thomas: Our thinking behind Meilisearch cloud is we want to address business usage of Meilisearch. If you're a developer and you want to build anything with Meilisearch, any features, it should be in the open source. If you're a business and you're making money, then maybe we can get some value out of it.

And so we are building the clouds with businesses in mind. Whereas when we are building Meilisearch, we are trying to make the life of developers easier. And this is how we will differentiate the cloud from the open source.

And today, we don't have any special features on the cloud, except that we can manage the updates. You don't have to think about hosting and the size of your machine. This is all the thing that we manage.

Mitchell: So you scale, I don't have to worry about scale.

Thomas: But we still have a lot of questions around the pricing. Today, we know that a lot of people are using Algolia, are not happy with the way they manage the pricing. For many different things, it's changing regularly.

It's based on usage. But they are updating it. And I think they are doing it maybe a bit better than they used to.

And so on our side, we don't have any conviction yet. We know that people need to know how much they will pay by the end of the month. That's the first thing.

But from there, maybe there will be different pricing based on different usage, use cases. Today, Algolia's pricing really works when it comes to eCommerce. Because it's easy to prove that you have a big return on investment when you're making better search, giving better results.

And so it's easy for them to sell to big eCommerce companies. And this is also the reason I believe they are building a lot of eCommerce specific features. Today, people using many search, they have an eCommerce website, they have a content website.

Maybe it's a B2B app, a SaaS. And the pricing based on the number of documents or the number of searches does not make sense when the search is not centric to your app. And this is something we...

So today, the pricing that we have for our onboarded users is just that we manage up to a certain amount of documents, just to make sure that we are not able to scale indefinitely so far. So we manage up to a certain millions of documents, but it's all included. And we just...

If you want to update your main search version, you have to click on a button. Because today, we still might have some breaking changes, which won't be the case at the beginning of 2023. So there will be automatic updates.

And yeah, and we monitor all the servers. And because we developed many search and we know the technology behind it, we are able to optimize for the... Yeah.

Mitchell: Yeah, that's right. You're in a position to optimize things like indexing in a way that I couldn't. I mean, I can't get into that.

That's just not a coding language I work in. Yeah. So with the cloud offering, one question I have, you're reminding me of other types of cloud offerings.

People use Cloudflare pretty widely. We do. I talked to them on the podcast a couple of years ago.

Is it possible for the Meilisearch cloud offering to have a concept of replication? I mean, first of all, it's already incredibly fast. We'll get into the speed in the second episode.

It's just ridiculous. People think it's fake. When I show clients a Meilisearch instance, they think I put a fake demo together.

But aside from that, is there going to be some concept of replication so that if someone is hitting a website, hitting Meilisearch on a website from Japan versus from Latin America versus from wherever, is it possible to have instances that are closer to that user in nodes or clusters or what have you?

Thomas: This is the big subject for 2023. We might use Cloudflare in the end to make that kind of thing. I think we will be able to split in Meilisearch the indexing from the search part.

That would allow us to index from somewhere and have multiple search instances all around the world and deploy them like this. Meilisearch is built in Rust. I won't go into a lot of details, but it's a very modern stack that allows us to build on the Cloudflare workers at some point.

Mitchell: Oh, cool. I was about to ask.

Thomas: This is what we have in mind. We've made some proof of concept of having Meilisearch running on the Cloudflare workers. The Cloudflare workers are the little machine that runs on the CDN.

The CDN is like the content delivery network. It's all of those servers of the internet that are very close to our homes. A lot of things are replicated in the CDN at many different locations.

Cloudflare is offering to use those machines to have some compute power now. We made a proof of concept of having Meilisearch, the search, not the indexing, the search running in those workers. That works, actually.

We want to have something like this in the coming years.

Mitchell: Yeah, that's great. Now, it's a selling point, but honestly, in practice, it's probably the difference of fractions of seconds based on what I've seen with just the underlying performance of Meilisearch itself. To really be able to detect the speed change, I guess at enterprise level with millions of documents in an index, it may become something that matters.

Anyway, you got me curious about that, so I thought I would ask. How about if I ask you one more question, and then we call this an episode, and then we'll get back into a second episode where we talk in more depth about specific use cases for how people in your open source community, as well as my own team and people we know are using Meilisearch, and get into some of the different possibilities for it. Why is it called Meilisearch?

Thomas: Meili is the god of forgetfulness in the mythology. We thought about how to name our search engine, and we really liked Discord, you know, the tool to have discussion for video games and community. We really liked the idea of solving Discord with Discord.

We wanted the same, so Meilisearch is trying to solve forgetfulness with the search engine.

Mitchell: My oldest daughter's middle name is Meili. It's not because of any gods that you mentioned. It's because she was born in Hawaii.

In Hawaii, there's a special plant, a leaf of a special bush that they make the leis and the sort of headdresses out of. I think it's pronounced Meili, but we switched it. That's the only reason I've adopted your product is because you named it after my kid.

You failed on that answer, but it's okay. I'll forgive you. Thomas, thank you for joining on this episode. I'm asking the listeners to stay tuned. In another couple of weeks, we're going to put the second episode up where we're going to talk about use cases and how we can deploy this for different types of clients and different types of websites. This has been great.

I'm still thrilled that I'm talking to one of the co-founders of such a great project and product. You'll probably never talk to me again because you're going to get so huge, but I really appreciate you joining. Thank you.

Thomas: Thank you very much, Mitchell. Thank you, David.

[Music] You've been listening to The Solspace Podcast.