SQLite library routine called out of sequence, with mod_python, apache, trac, and sqlite all installed by darwinports

[careo@…]

With apache2 2.2.0, pysqlite 2.0.7, python 2.4.2, sqlite3 3.3.4, and trac 0.9.4 (all installed through Darwinports) on OS X 10.4.5, I'm getting the following error (I've stripped out the date, ip and "PythonHandler trac.web.modpython_frontend" from the beginning of each line for readability):

Traceback (most recent call last):
  File "/opt/local/lib/python2.4/site-packages/mod_python/apache.py", line 299, inHandlerDispatch\n    result = object(req)
  File "/opt/local/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 199, in handler\n    env = get_environment(mpr, project_opts)
  File "/opt/local/lib/python2.4/site-packages/trac/web/main.py", line 335, in get_environment\n    return _open_environment(env_path, threaded)
  File "/opt/local/lib/python2.4/site-packages/trac/web/main.py", line 51, in _open_environment\n    env_cache[env_path] = open_environment(env_path)
  File "/opt/local/lib/python2.4/site-packages/trac/env.py", line 375, in open_environment\n    if env.needs_upgrade():
  File "/opt/local/lib/python2.4/site-packages/trac/env.py", line 279, in needs_upgrade\n    db = self.get_db_cnx()
  File "/opt/local/lib/python2.4/site-packages/trac/env.py", line 137, in get_db_cnx\n    return self.__cnx_pool.get_cnx()
  File "/opt/local/lib/python2.4/site-packages/trac/db.py", line 157, in get_cnx\n    cnx = self._cnx_class(**self._args)
  File "/opt/local/lib/python2.4/site-packages/trac/db.py", line 280, in __init__\n    timeout=timeout)
ProgrammingError: library routine called out of sequence

[jberry@…]

Note that this bug carries on from #2196.

[pbur@…]

I'm seeing a similar error on linux with all pieces compiled myself except for Apache. The of my stack looks like this, same line number in db.py:

SystemError: NULL result without error in PyObject_Call

[cboos]

You should try to upgrade PySqlite to version 2.2.2, as a lot of error code checking has been added during the process the integration of that library into Python 2.5.

See also pysqlite:ticket:146.

[edbrannin@…]

I have the same problem, with PySqlite v2.2.2 (on OS X 10.4.6, with DarwinPorts):

$ port installed
The following ports are currently installed:
  apache2 @2.2.2_1 (active)
  apr @1.2.7_0 (active)
  apr-util @1.2.7_0 (active)
  clearsilver @0.10.3_1 (active)
  coreutils @5.92_0 (active)
  db4 @4.3.29_0+darwin_8 (active)
  expat @2.0.0_1 (active)
  flex @2.5.33_0 (active)
  gettext @0.14.5_1 (active)
  libiconv @1.10_1+darwin_8 (active)
  mod_python @3.2.8_1 (active)
  neon @0.25.5_0 (active)
  openssl @0.9.8b_0+darwin_8 (active)
  pcre @6.6_0 (active)
  perl5.8 @5.8.8_0+darwin_8 (active)
  py-bsddb @2.4.3_0 (active)
  py-ipython @0.7.2_0 (active)
  py-readline @2.4.3_0 (active)
  py-sqlite @2.2.2_0 (active)
  python24 @2.4.3_1+darwin_8 (active)
  readline @5.1.004_0 (active)
  screen @4.0.2_1 (active)
  sqlite3 @3.3.5_0+darwin_8 (active)
  subversion @1.3.2_0+mod_dav_svn (active)
  subversion-pythonbindings @1.3.2_0 (active)
  trac @0.9.5_0 (active)
  zlib @1.2.3_0 (active)

[Russell Hind <rhind@…>]

I have the same problem (py-sqlite 2.3.1 and trac-0.9.5). Just done a fresh install of everything using DarwinPorts on OSX 10.4.6.

The following ports are currently installed:
  apache2 @2.2.2_1 (active)
  apr @1.2.7_0 (active)
  apr-util @1.2.7_0 (active)
  clearsilver @0.10.3_1 (active)
  db4 @4.3.29_0+darwin_8 (active)
  expat @2.0.0_1 (active)
  flex @2.5.33_0 (active)
  gettext @0.14.5_1 (active)
  libiconv @1.10_1+darwin_8 (active)
  mod_python @3.2.8_1 (active)
  neon @0.25.5_0 (active)
  openssl @0.9.8b_0+darwin_8 (active)
  pcre @6.6_0 (active)
  perl5.8 @5.8.8_0+darwin_8 (active)
  py-bsddb @2.4.3_0 (active)
  py-sqlite @2.3.1_0 (active)
  python24 @2.4.3_1+darwin_8 (active)
  readline @5.1.004_0 (active)
  sqlite3 @3.3.5_0+darwin_8 (active)
  subversion @1.3.2_0+mod_dav_svn (active)
  subversion-pythonbindings @1.3.2_0 (active)
  trac @0.9.5_0 (active)
  zlib @1.2.3_0 (active)

Has anyone got anywhere with this yet?

Thanks

Russell

[Russell Hind <rhind@…>]

I have now tried it under CGI and this works, so what is different when running trac under cgi as opposed to mod_python (apart from the speed). I will try and get fastcgi working to help solve this.

Thanks

Russell

[slianoglou@…]

The funny thing is that I have a complete DarwinPorts version installed on my MacBook Pro ... soup to nuts, everything is pulled down from DarwinPorts *except* trac. I'm using the latest svn/trunk of trac, and it works perfectly.

Anyway, it works with no hitch.

Then I've been trying to set it up in the same exact fasion on my PowerMac G4 but I'm getting this "out of sequence" error.

I peeked into the apache logs to see if I could uncover anything else, and found this:

[Tue Jun 27 10:24:14 2006] [crit] (70023)This function has not been implemented on this platform: DBD: driver for  not available
[Tue Jun 27 10:24:14 2006] [crit] (70023)This function has not been implemented on this platform: DBD: failed to initialise

Which I thought might be the problem, but when I looked back into the apache error_log on my MBP, it was there also!

Somhow an intel vs. !PowerPC issue? I guess I'll try to compile everything from source on my PowerMac and see if it's not that threaded issue in SQLite, but it'z bizzarre.

-steve

[anonymous]

Replying to slianoglou@gmail.com:

Just as an update, I compiled everything from source pretty much as outlined here: TracOnOsxNoFink and it all worked out.

The differences from that are that I'm using a version of apache2 (2.0.55) that I had previously compiled and that in addition to using Python 2.3.5 that comes with Tiger, I'm also using the sqlite3 that came with Tiger.

Aslo I pretty much disregarded notes that said something to the effect of not using the latest versions of the remainder of the required packages ... so far everything is feng shui.

[Russell Hind <rhind@…>]

Just to say I have now set up trac to use FastCGI rather than CGI and still all works fine, and appears to be as quick as mod_python (if not faster) so maybe this is an option for other everyone else (easier than re-compiling everything yourself!)

The differences I know about between the fink build that I got working and the DarwinPorts build I could were that fink uses Apache 2.0.55 vs DP's 2.2.2, python 2.3 in fink vs 2.4 in DP, and sqlite2 in fink vs sqlite3 in DP. Not sure about the mod_python program.

But FastCGi seems fine anyway so not worried about this now, will stick using that. FYI: My installed ports now are:

The following ports are currently installed:
  apache2 @2.2.2_1 (active)
  apr @1.2.7_0 (active)
  apr-util @1.2.7_0 (active)
  boehmgc @6.7_0+darwin_8 (active)
  clearsilver @0.10.3_1 (active)
  db4 @4.3.29_0+darwin_8 (active)
  expat @2.0.0_1 (active)
  flex @2.5.33_0 (active)
  gettext @0.14.5_1 (active)
  glib2 @2.10.3_0 (active)
  icu @3.4_0+darwin_8 (active)
  libiconv @1.10_1+darwin_8 (active)
  mod_fastcgi @2.4.2_1 (active)
  mod_python @3.2.8_1 (active)
  mono @1.1.15_0+darwin_8 (active)
  neon @0.25.5_0 (active)
  openssl @0.9.8b_0+darwin_8 (active)
  pcre @6.6_0 (active)
  perl5.8 @5.8.8_0+darwin_8 (active)
  pkgconfig @0.20_0 (active)
  py-bsddb @2.4.3_0 (active)
  py-sqlite @2.3.1_0 (active)
  python24 @2.4.3_1+darwin_8 (active)
  readline @5.1.004_0 (active)
  sqlite3 @3.3.5_0+darwin_8 (active)
  subversion @1.3.2_0+mod_dav_svn (active)
  subversion-pythonbindings @1.3.2_0 (active)
  trac @0.9.5_0 (active)
  zlib @1.2.3_0 (active)

[bahamut@…]

Russell, what's your FCGI configuration, I tried this solution and it is utterly slow compared to mod_python, and even prone to errors (like source code appearing on Trac pages above the pages themselves, sometimes).

In any case, going around the problem isn't a good strategy in my opinion. There is a cause to it and simply shrugging it off will just make it come bite us later.

[Russell Hind <rhind@…>]

Replying to bahamut@macstorm.org:

Russell, what's your FCGI configuration, I tried this solution and it is utterly slow compared to mod_python, and even prone to errors (like source code appearing on Trac pages above the pages themselves, sometimes).

I copied trac.fcgi to /opt/local/apache2/cgi-bin and added the following lines to tell it where the environment is:

import os
os.environ['TRAC_ENV'] = "/Users/Shared/trac"

This is the relevant section from my apache config. Hope this helps.

Russell

<IfModule mod_fastcgi.c>
        FastCgiIpcDir /opt/local/apache2/fastcgi
        AddHandler fastcgi-script .fcgi
</IfModule>

Alias /trac_htdocs/ "/opt/local/share/trac/htdocs/"

<Directory /opt/local/share/trac/htdocs>
        Options Indexes MultiViews
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>

ScriptAlias /trac /opt/local/apache2/cgi-bin/trac.fcgi

<Location /trac/login>
  AuthType Basic
  AuthName "Trac Wiki and Bug Tracking System"
  AuthUserFile /Users/Shared/.htpasswd
  Require valid-user
</Location>

[simon-code@…]

I installed using DarwinPorts on OSX 10.4.7, and have pretty much the same installed ports as mentioned in 14 - and received the same error -- ProgrammingError: library routine called out of sequence.

The simple workaround is to install the source distribution instead:

1. Uninstall the py-sqlite package only without dependencies

   sudo port -f uninstall py-sqlite
   

2. Find the source for your version from local DP distfiles /opt/local/var/db/dports/distfiles/..., and uncompress it - alternatively just grab the latest source distribution from pyslite.org.
3. Follow the simple build and install instructions at http://initd.org/pub/software/pysqlite/doc/install-source.html

   > cd ~/Desktop/pysqlite-2.3.1   (# or other relevant location)
   > /opt/local/bin/python setup.py build
   > sudo /opt/local/bin/python setup.py install
   

It is now installed as a regular Python package outside DarwinPorts, and the error message does not appear - and mod_python works flawlessly.

Also, I use a simple wxPython-based Python shell application for much of my interactive testing, and using that shell the same db error appeared when I instantiated any db connection interactively (through trac or directly). Same problem using the shell inside the SPE IDE. With source install of py-sqlite, the GUI apps works fine as well.

The problem didn't appear earlier using plain 'Terminal Python', or when testing with fast_cgi (as indicated from others).

Seems to indicate a DarwinPorts packaging/build/install issue as I managed to get the same DP distribution source to work just by using simple python setup.py... Likely threading is the common denominator between apache/mod_python and a wxPython GUI app?

[Russell Hind <rhind@…>]

I've now upgraded through darwinports to 0.10 and can now use mod_python again. Here is my list of installed ports and versions

Russell

  apache2 @2.2.3_0 (active)
  apr @1.2.7_0 (active)
  apr-util @1.2.7_0 (active)
  clearsilver @0.10.3_1 (active)
  db4 @4.3.29_0+darwin_8 (active)
  expat @2.0.0_1 (active)
  flex @2.5.33_0 (active)
  gettext @0.15_0 (active)
  glib2 @2.12.3_0 (active)
  icu @3.4.1_0+darwin_8 (active)
  libiconv @1.11_0+darwin_8 (active)
  lynx @2.8.5rel.2_0 (active)
  mod_fastcgi @2.4.2_1 (active)
  mod_python @3.2.8_1 (active)
  neon @0.25.5_0 (active)
  openssl @0.9.8c_0+darwin_8 (active)
  pcre @6.6_0 (active)
  perl5.8 @5.8.8_0+darwin_8 (active)
  pkgconfig @0.21_0 (active)
  py-bsddb @2.4.3_0 (active)
  py-sqlite @2.3.1_0 (active)
  python24 @2.4.3_1+darwin_8 (active)
  readline @5.1.004_0 (active)
  sqlite3 @3.3.7_0+darwin_8 (active)
  subversion @1.4.0_0+mod_dav_svn (active)
  subversion-pythonbindings @1.4.0_0 (active)
  trac @0.10_0 (active)
  zlib @1.2.3_0 (active)

[jashugan@…]

Based on Russell's comment, I have also got version 0.10 working with mod_python through darwin ports. There was one hiccup , but other than that everything runs fine.

[mateo@…]

I'm runing trac 0.10 with all of the same darwinports versions (other than a slightly more recent version of pcre) as Russel, but I'm still getting the "library routine called out of sequence" errors.

[anonymous]

Replying to mateo@plankdesign.com:

I'm runing trac 0.10 with all of the same darwinports versions (other than a slightly more recent version of pcre) as Russel, but I'm still getting the "library routine called out of sequence" errors.

Did you upgrade from 0.9.x? I had the problem with 0.9.x but when I upgraded to 0.10, I did a force install of everything to make darwinports re-build all ports and don't have a problem now. Maybe the force re-build of everything is necessary?

Cheers

Russell

[anonymous]

Replying to anonymous:

Did you upgrade from 0.9.x? I had the problem with 0.9.x but when I upgraded to 0.10, I did a force install of everything to make darwinports re-build all ports and don't have a problem now. Maybe the force re-build of everything is necessary? Cheers Russell

No, it was a fresh install. Here's my backtrace, if it might be helpful to anyone; it's not the same as the error ticket's original one, but that may be due to my using 0.10:

Traceback (most recent call last):
  File "/opt/local/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch\n    result = object(req)
  File "/opt/local/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 87, in handler\n    gateway.run(dispatch_request)
  File "/opt/local/lib/python2.4/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
  File "/opt/local/lib/python2.4/site-packages/trac/web/main.py", line 346, in dispatch_request\n    env = _open_environment(env_path, run_once=environ['wsgi.run_once'])
  File "/opt/local/lib/python2.4/site-packages/trac/web/main.py", line 57, in _open_environment\n    env_cache[env_path] = open_environment(env_path)
  File "/opt/local/lib/python2.4/site-packages/trac/env.py", line 434, in open_environment\n    if env.needs_upgrade():
  File "/opt/local/lib/python2.4/site-packages/trac/env.py", line 312, in needs_upgrade\n    db = self.get_db_cnx()
  File "/opt/local/lib/python2.4/site-packages/trac/env.py", line 182, in get_db_cnx\n    return DatabaseManager(self).get_connection()
  File "/opt/local/lib/python2.4/site-packages/trac/db/api.py", line 71, in get_connection\n    return self._cnx_pool.get_cnx()
  File "/opt/local/lib/python2.4/site-packages/trac/db/pool.py", line 83, in get_cnx\n    cnx = self._connector.get_connection(**self._kwargs)
  File "/opt/local/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line 113, in get_connection\n    return SQLiteConnection(path, params)
  File "/opt/local/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line 159, in __init__\n    timeout=timeout)
ProgrammingError: library routine called out of sequence

cheers,

mateo

[Russell Hind <rhind@…>]

Replying to anonymous:

No, it was a fresh install. Here's my backtrace, if it might be helpful to anyone; it's not the same as the error ticket's original one, but that may be due to my using 0.10:

I can't help with the error message being different, but out of interest, what version of XCode do you have installed to build the ports? I did a force re-build with darwinports because I also updated to XCode 2.4 at the time and had read the default version of gcc had changed, so wanted to make sure they were all built with the same version?

My 0.9.x installation was built with XCode-2.3 and that had the same SQLite error.

Cheers

Russell

[anonymous]

Replying to Russell Hind <rhind@mac.com>:

I can't help with the error message being different, but out of interest, what version of XCode do you have installed to build the ports? I did a force re-build with darwinports because I also updated to XCode 2.4 at the time and had read the default version of gcc had changed, so wanted to make sure they were all built with the same version?

I'd updated to 2.4 when it came out.

Another clue could be This Message I found, which states that this error can happen if SQLite is compiled with --enable-threadsafe.

[anonymous]

Check this - /usr/lib/libsqlite*

If you move them out somewhere and just keep the darwinports libraries, it seems to help.

[anonymous]

Mac seems to have installed old sqlite libraries by default in /usr/lib, which are linked in spite of the fact, that the shared object may explicitely point directly to the different location of libraries.

The solution to the problem seems to be replacing those libraries in /usr/lib with the newer version, i.e., the latest one that comes with darwinports, or just kicking them out (though that might be not a good idea, since something else in MacOSX may depend on it).

[slianoglou@…]

So I spent more hours than I would have liked to attacking this problem.

I now have trac 0.10.3 working (the 0.11.dev branch worked too) on my PowerPC Mac w/ a mostly MacPorts install base.

The kicker that seemed to do it for me was to install pysqlite from source and have it use the sqlite3 that's installed by OS X (not macports).

So, in short, to get trac working w/ macports on PowerPC via apache2/mod_python, I think this is the magic formula:

• Install via macports:
  • Python
  • subversion
  • subvesrion-pythonbindings
  • clearsilver
  • apache2
  • mod_python
  • trac (if you are ok using 0.10.3)
• Install by hand
  • trac (if you want to work out of svn/sandbox/whatever)
  • pysqlite
    • Don't change pysqlite's config to look in the macports /opt/local/* directories. Just let it pick up/use the Mac OS X sqlite3 installation

You should be in business.

Just to reiterate, I've only found this necessary to do on my PowerPC Mac (OS X.4.8). Trac and all of its dependencies installed perfectly on my MacBook? Pro.

Hope that helps.

[cboos]

I think we could close the ticket now that there's a well documented recipe to get it working (comment:30 above).