Comments on: Putting Freebase in a Star Schema

By: madtiger

madtiger — Fri, 13 Mar 2009 15:12:43 +0000

I've been parsing through this data for a few days and I stumbled across this page. I agree, do you have a copy of these scripts, or is it something proprietary to your business?

By: Josh Ribakoff

Josh Ribakoff — Fri, 27 Feb 2009 11:03:18 +0000

Nice work, this will be very interesting to anyone coding knowledge based agents for reasoning about the real world

By: John Sichi

John Sichi — Fri, 27 Feb 2009 02:50:49 +0000

Well, yeah, LucidDB (http://www.luciddb.org), but since I work on that, I'm biased.

Last OSCON, I actually loaded up a WEX dump into LucidDB in between sessions, but the bulk of it was semi-/un-structured text, which wasn't very interesting since LucidDB doesn't support lobs and fulltext/XML search yet.

Also in the open source camp: Infobright is the lowest learning curve for MySQL users; MonetDB has XQuery support and is very fast as long as everything fits in memory.

By: Paul Houle

Paul Houle — Fri, 27 Feb 2009 02:08:40 +0000

Got a specific one in mind?

By: Martin Gajdos

Martin Gajdos — Thu, 26 Feb 2009 08:07:38 +0000

It would be nice if you could post the scripts you have used for this. I'd love to take a look at those.

Thanks

By: John Sichi

John Sichi — Wed, 25 Feb 2009 21:47:37 +0000

If you're going to the effort to transform to a star, it's worth looking into a column store for the DB as well. Besides query acceleration and automatic compression, you can also get much faster load times using bulk load facilities which avoid row-at-a-time index updates. I won't mention the names of any of the contenders here

By: paul_houle

paul_houle — Wed, 25 Feb 2009 18:41:08 +0000

Sphinx looks like a nice product, but it looks like it addresses a different question: full-text search. Freebase is a semantic system where, instead of using an imprecise word like "jaguar", you can reference "jaguar the cat" or "jaguar the game console" by a guid. That said, full-text search can be a useful complement to this kind of system.

Years ago I worked on a project called the Global Performing Arts Database

http://www.glopad.org/

where text about an media item was distributed in literally hundreds of different tables, since the system coded statements like

"Picture A was taken during a production of Hamlet"
"Hamlet was written by Shakespeare"

in machine-readable (RDBMS) form. Of course we wanted Picture A to show up in a picture of Shakespeare, so we had to do a graph traversal of the RDBMS tables to collect any text that might be relevant to an item (careful not to follow paths that would lead to irrelevant results.) This way we'd build up a document vector which we'd index in a conventional full text system. Can Sphinx support that kind of thing easily?

By: L.G.

L.G. — Wed, 25 Feb 2009 18:18:44 +0000

Just use Sphinx:

sphinxsearch.com