Introduction from chemistry notes

Hello world!

With this post I would like to start tracking the progress of my Google Summer of Code project. The idea of the project is to integrate chemistry and biology knowledge into the KDE desktop. Think of (bio)chemical meta data extraction, indexing and search and this is where you meet Strigi.

Based on the powerful concept of Jstreams, Strigi is a high performance desktop search engine, which is now an inalienable part of KDE4. Strigi has the power to use different backends (clucene, sqlite3, ...) and a simple yet very powerful idea of pluggable stream analyzers. This architecture leads to a very small number of dependencies and places an emphasis on interfaces. The interface is of your choice: link directly, use sockets, dbus or even command line utilities. The main Strigi developers Jos van den Oever and Flavio Castelli already did a great job by providing a stable engine and now Strigi is moving towards the integration with Nepomuk semantic desktop project and Freedesktop.org Xesam specifications.

Nepomuk focuses on meta data ontologies and relations. Sebastian Trueg is the leader of KDE-Nepomuk project and there is also one GSoC student, Dmitriy Soloduhin, involved in it. And thanks to Phreedom (Evgeny Egorochkin), we now have Nepomuk ontologies in Strigi.

Xesam is providing unified api specifications for search and metadata services as a result of collaboration of Freedesktop.org with Strigi, Beagle, Tracker, Pinot, Recoll and Nepomuk-KDE projects.

Now back to chemistry. Blue Obelisk establishes interaction between open projects dealing with chemical systems and cultivates the standards, such as InChI and CML. Blue Obelisk has been born in the US at the ACS meeting, but has many of its roots in the University of Cambridge, group of Peter Murray Rust, and the University of Cologne (CUBIC). Christoph Steinbeck's group at CUBIC brought to life open projects such as CDK, Bioclipse and NMRShiftDB. I am happy that Egon Willighagen, who was the member of Steinbeck's group and is an active contributor to numerous open source projects, is now my mentor and supervisor in this GSoC project.

I was lucky to study Bioinformatics in CUBIC for the last year. I am very excited about my ion channel project which is now over, and I hope to stay with the topic during my PhD studies. By the way, if you have any open PhD positions for bioinformaticians, please let me know.

It is hard to resist the temptation to tell you some interesting facts on ion channels, but returning to the main topic I should tell you about the key projects that are very important for my GSoC project. These are BODR, Chemical MIME, OpenBabel, InChI, CML, chemical structures, Avogadro and Kalzium.

BODR stands for Blue Obelisk Data Repository and is a shared repository for many important chemoinformatics data.

Chemical MIME expands the list of standard MIME types with chemical file formats and provides example files for each format. Daniel Leidert maintains the chemical-mime-data database in Linux distributions. It conforms David Faure's specifications for MIME type databases in KDE4, the automagical type detection relies on it. The file extension is not enough to uniquely identify the MIME type: e.g. ".sdf" stands for SD chemical format and at the same time StarOffice Math Document.

OpenBabel is both a library and command line toolbox, which allow to manipulate chemical data in different formats. It fully supports Chemical MIME. Jerome Pansanel maintains KOpenBabel wrapper (Qt and KDE GUI for OpenBabel converter) and also a large set of molecules in CML format, called Chemical Structures 2.0.1. This is also very important, because CML (Chemical Markup Language) is an XML-based chemical format which is supposed to be the standard.

Some public databases, like PubChem and BODR Chemical Structures, implement InChI identifier, which is an IUPAC standard. InChI allows to represent a chemical structure in an unambiguous way. OpenBabel can generate InChIs from chemical structures. NCI and Kegg databases in CML with InChIs generated can be viewed at NCI and Kegg.

BKchem chemical drawing program by Beda Kosata can regenerate structures from InChIs. Other interesting chemical drawing programs, which at the moment can not import InChIs, are GChemPaint and Molsketch. BKchem uses Tk widgets, and GChemPaint is a part of GNOME desktop. Molsketch by Harm van Eersel is a molecular drawing tool for KDE. If supplied as a KPart, Molsketch can find a bright future in different KDE4 application.

Kalzium is a part of kdeedu, it started as the periodic table of the elements program by Carsten Niehaus and now is gaining momentum and attracting more hackers, who want a better chemistry support in KDE. Kalzium/Avogadro is a 3D molecular visualization library maintained by Benoît Jacob. It uses Eigen, a lightweight linear algebra C++ template library which is already a part of KDE4. Kalzium/Avogadro has acquired another GSoC student -- Marcus Hanwell. The leader of another interesting chemical KDE project KryoMol , Armando Navarro Vázquez, recently has sent the patch to separate Kalzium Molecular Viewer as a KPart.

Kfile-chemical is a project started by Egon and later supported by Jerome and Daniel. Initially it was a set of kfile plugins that allowed chemical meta data extraction. But with the initiative to port kfile plugins to Strigi, kfile-chemical now provides Strigi with chemistry aware stream analyzers. It is hosted in KDE SVN Playground, and since it is aimed to have a low number of dependencies it has the potential to become a part of kdeedu, for example.

Since kfile-chemical is where I make my first efforts, I'll briefly describe what I am doing now and what you can expect by the end of the project.

1. Make all kfile-chemical analyzers compatible with Strigi/KDE/Nepomuk chemical ontology. This means that there are chemical filed properties defined in Strigi and during the metadata extraction process stream analyzers are supposed to fill in the relevant fields. The chemical field properties at the moment are: chemistry.inchi, chemistry.molecular_formula, chemistry.molecular_weight, chemistry.pdbid, chemistry.xray_resolution. Other properties are supposed to be stored generic field properties, such as content.title and container.items;
   
2. Generate InChI (chemistry.inchi) for structures, which do not have it already, using OpenBabel library;
3. Provide a test suite for the analyzers to make sure nothing breaks when one of the libraries in this mixture is updated;
   
4. Expand the list of supported chemical file types to cover as many of Chemical MIMEs as possible.

If it goes smooth, I will try to integrate OSCAR3 to process plain text and create InChIs for molecules found in that text. This will allow indexing and semantic linking of the literature and the chemical files.

On our first meeting, Egon suggested that I should provide a KDE4 GUI chemical search tool, which could possibly be expanded to more generic purposes, like querying abstract field properties from the KDE-Nepomuk ontology. This is also great to test all the technologies and libraries involved. I won't bloat this post with the mockups and screenshots, because it is already quite long, but I will certainly come back to it later this week. So, the idea is to have the following workflow implemented:

1. While indexing, InChI string is extracted or generated with the help of libOpenBabel by one of the kfile-chemical analyzers;
2. InChI string is stored in chemistry.inchi field property in Strigi storage;
3. It can be queried directly by issuing a "chemistry.inchy:" query in strigiclient
4. The GUI tool can use Molsketch KPart to input the structure with the mouse. The structure is then converteed using OpenBabel to InChI and used as a search key;
   
5. The name of the compound, or the synonym, can be specified as a search key;
6. The search query is sent to Strigi via dbus and the search results received in response;
7. Search results are either sort of text documents, or the chemical structures. To visualize chemical structures Kalzium/Avogadro KPart can be used.
   

BTW, is it possible to do a substructure search using InChI, not talking about ignoring some InChI layers?

This project is a good powertesting for all Strigi technologies. But I also hope to be useful to Strigi by extending some functionality and writing unit testcases. And I am sure Jos wont let me go like that :-) also because my primary affiliation here is KDE/Strigi.

I am happy to have this opportunity to work side-by-side with very skilled open source developers and to teach myself a good style, and of course, to have this project right on the intersection of my interests: open source, Linux, KDE and Bio(Chemo)Informatics.

The initial project proposal can be found here.

Now I will tell few words about my current progress. While trying to get all the tools and libraries listed above working on my machine, I was surprised by a crash in KDE/Avogadro, which was caused by a bug in my radeon Mesa DRI drivers. Fortunately I was able to trace the problem and fixed this annoying crash in Avogadro OpenGL initialization. Then I switched to kfile-chemical and started with adapting CML stream analyzer to current standard and ontology. To have my testcases done, I had to introduce passing filters to the command line strigicmd tool. While working on the tests, involving all CML structures from the Chemical Structures 2.0.1, I realized that because the current CML metadata extraction is not XML aware I have to rewrite it using StreamSaxAnalyzer to make it work as desired. This is what I am doing at the moment.

My next steps would be: to generate InChI for CML lacking the identifier. Then I will improve all other available chemical analyzers and create tests for them. Then I will run some productivity tests involving the mirror of the PDB database. And, of course, I will start implementing the GUI chemical query tool.

Progress report and back on track from chemistry notes

Preamble.
I'm happy to get back to hacking this week.
Last two weeks have been almost completely lost due to some urgent reallife issues back in my home country. So I had nothing to do than just to shift my flight and to solve the problems. This was completely unplanned, and left me for two weeks without a single commit, thus making my supervisors nervous about the outcome of my project. Now I can be on the channel, commit daily and blog twice a week, as Egon recommended and as he now insists.

CML2 SAX streamanalyzer in kfile-chemical.
While reading the CML specifications, I thought that there is too much flexibility in it, hence making it hard to parse. To start I took few CML2 samples from Jerom's Chemical Structure 2.0 project which is a part of the BlueObelisk data repository. These files already contain the information, it just had to be extracted. I wrote the analyser based on streamsaxanalyzer. I used xmlindexer and strigicmd tools to see how the analyzer works. I will try to extend the analyzer to support the variety of CML's I can find in the wild. To disribute sample test files together with kfile-chemical I need them to be free/to have a proper license. I am not sure whether the test files from the Chemical MIME project can be included. Please comment on that if you have any clue.

Test suite in kfile-chemical.
I have added a python test suite and the first testcase of 20+ tests is for the CML analyzer. Strigi is intefaced via xmlindexer and strigicmd. I find these command line tools useful for testing, since they do not use any central storage or daemons to work. The test fixtures prepare a clean directory and the list of sample CML files, so that every test in the testcase is executed in a clean environment. For all the test I have used clucene backend. All tests run pretty fast, except for the valgrind test for memory leaks.

The analyzers which are not covered by tests and which are not compliant with current Strigi ontology fieldproperties have been temporary disabled. You can expect most of them to be fixed and enabled back later this week.

CML testcase showed that querying by InChI (chemical.inchi=...) gives me false positives. So there is a question now whether FieldRegister::stringType is suitable to handle exact identifiers like InChI or it is better to make it binary.

I was also wondering why chemistry.name (content.title is its parent) in xmlindexer is turned to content (exactly, not content.title) in strigicmd with clucene backend.

The search by content.version field returns no results and when querying a float molecular weight (chemistry.molecular_weight:58.1222) is gives me no results too.

This leaves me with 3/20 tests failing.

InChI generator.
InChI is uniquely identifying a chemical structure. That is why it is a good idea to have InChI's for all the analyzed chemical files, where possible. OpenBabel can convert any recognised format to InChI strings. I made a working example to see if it is easy and fast enough to generate InChI's in a Strigi streamanalyzer. It is called inchi-generator and it works for valid CML2 files only. I had to buffer the contents of the Strigi stream to pass it to OpenBabel convertor, but I feel there could be a more elegant solution, since OpenBabel works with streams as well, they are just not compatible with Strigi streams.

Linking to OpenBabel.
I had very strange problems with unresolved symbols in OpenBabel format plugins until
Geoffrey helped me. It's all about plugins! Strigi loads streamanalyzers with dlopen() on Linux and so does libopenbabel when it needs a format plugin. The solution was simple, to add RTLD_GLOBAL to code which loads libopenbabel. Since libopenbabel is linked to inchi-generator RTLD_GLOBAL had to be added to Strigi loader. I wonder if it can cause problems to other analyzers. Another solution would be to load libopenbabel from inchi-generator in runtime.

Openbabel 2.1 (SVN) Debian packages
The FindOpenBabel2.cmake script by Carsten is used in KOpenBabel, Kalzium and now in kfile-chemical. It requires --atleast-version=2.1.0. In Debian unstable you can only find version 2.0.2. Michael Banck, Debian maintainer, provides build rules in debichem repository. I do not know what could be the reason for the new version not to be available in SID. Probably it's related to the patches to provide a better version abstraction, e.g. to have two OpenBabel versions installed at the same time.
Anyway, as OB 2.1 is a requirement and can be packaged, I have put the x86 Debian libinchi, libopenbabel and openbabel packages here http://neksa.net/debian/.

Strigi chemical fieldproperties.
Talking to Phreedom in the very beginning of my project, I thought that the chemical fildproperties should represent the minimal set of metadata attributes, but in practice, taking into account the variety of chemical formats, it is hard to define the list once and for all. That is why I have added few other chemistry.fieldproperties. Among these are IUPAC Name, PubChem Compound ID, experimental method of structure elucidation, some physicochemical properties which are to my mind most queried in PDB and few more statistical counters. We better remove some unused properties later rather than keep of storing the metadata which could be extracted.

Further steps.
The test suite has to be expanded to cover all the formats currently existing in kfile-chemical. The analyzers need to be fixed to match current Strigi ontology. This could be done during this week.

Openbabel integration requires more attention, since there is no "magic" MIME detection. I will try to employ Chemical MIME patterns to do the detection. InChI generation is only possible if we know the source format. Strigi is stream-based, hence we can not look at the file extension in streamanalyzers.

And of course, and eye candy, a GUI chemical search tool is one of my deliverables.

I would also love to spend some time on Strigi, perhaps Jos will find the kfile-chemical testsuite good for testing the built-in analyzers.

Offtopic.
July 18th -- 25th I will be attending the annual conference of the Society for Computational Biology (ISCB) and the satellite meetings. This time in Vienna, Austria. If you happen to be there at the same time please contact me.

19-20 : 3DSig Structural Bioinformatics and Computational Biophysics meeting.
21 : 3-rd ISCB Student Council Symposium
21-25 : ISMB/ECCB conferences.

My submission has been accepted, so I will be presenting some of the results of my CUBIC project.

One good news from CUBIC. Thanks to the project, my final grade is now A = First Class Honours. I hope this will improve my chance to get a nice place for the PhD.

kfile-chemical/STRIGI -> strigi-chemical from chemistry notes

Kfile-chemical had three branches:

• KDE3, where all chemical analyzers were KFilePlugin's and provided KFileMetaInfo;
  
• KDE4, where nothing happened since it was branched;
• STRIGI, all the metadata extractors were Strigi StreamLineAnalyzers.
  

I have started my project in kfile-chemical, but it will end up in a different tree. As it was recently proposed by Egon (my GSoC mentor) and confirmed by Jerome (kfile-chemical maintainer) and Jos (Strigi core developer), the STRIGI branch was separated from kfile-chemical.

Now it is called strigi-chemical. The reason is that it has no KDE dependencies at the moment. Strigi-chemical also lives in playground /utils/strigi-chemical/.

The situation at the moment is:

• kfile-chemical is now what was previously kfile-chemical/KDE3 branch
• kfile-chemical/KDE4 branch removed
• strigi-chemical is now what was previously kfile-chemical/STRIGI branch