Speed up Verity DB indexing

I recently made a change that will be in the ColdFusion MX 7 updater that seems to give use a pretty nice improvement in the speed of indexing custom data using cfindex. I was trying to resolve a bug where records in our bug database that had XML in the description of the bug were not getting indexed.

Turns out that we were applying the "universal" filter to the content that we were indexing. This meant that Verity would examine the content (or the first few K) to see if it could ID the document type. This takes time and in our case was doing the wrong thing as it would see free form text with the XML declaration in it, pass it off the to XML parser, which would choke and the record would not get indexed. You can see this result by using the new STATUS attribute to assign the indexing status to a variable and dumping that after indexing.

Anyway, here is the fix. Find the file verity/Data/stylesets/ColdFusionK2/style.dft and comment out the /filter = universal line as follows:

  field: CF_BODY
# VDK will attempt to autorecognize BODY content if we use this
#    /filter="universal"

You can either recreate your collections after making this change, or you can shut down the CFMX Search Service and edit the style.dft file directly in your collections that you use to index custom data.

Enjoy!

Which English do you want?

Talking with Ray today about some Verity things, I was sharing some tidbits about the english locale and performance that I realized should be published more widely.

First, some history. The Verity 5.5 version in CFMX7 is very different that the verity 2.5 version we shipped in CF5/6/6.1. In particular we (and verity) defaulted to the "english" locale. Since then, Verity has acquired the linguistiX technology, which is the company that used to provide their international locales, but I believe they have since bought the company and/or technology and now they default to using "englishx". Where the "x" stands for linguistiX.

But ColdFusion MX 7 still defaults to english. Why? Two reasons.
1. To ensure compatibility with previous CF releases. Users expect to get the same results from indexing and searching when they upgrade to CFMX7.
2. Performance. The englishx locale is "better". According to Verity it is much better than the old english locale. Better means better search results. In order to get these results, it takes more time in the indexing of the data. Since we were trying to maintain indexing performance, we used the english locale as it means indexing will be faster.

Bottom line: If you want better search results, and indexing performance is not critical, create your collections using English (advanced) in the CF administrator or "englishx" when using the cfcollection tag.

Verity article published

I wrote an article about some of the new Verity features in ColdFusion MX 7 and it has been published on the ColdFusion DevNet site:
http://www.macromedia.com/devnet/mx/coldfusion/articles/search_cf7.html

I think that there is a lot of good stuff in the current release. I am not sure if enyone really gets the categories idea yet, but I know that pretty soon people will catch on.