base_url considered harmful (trac accessed via two hostnames)

[garyo@…]

I'm running 0.10.4dev under Apache, and now that base_url seems to be becoming more a mandatory setting for various plug-ins and even the base code, my problem is becoming more severe. I run a trac that is accessed via two URLs:

http://host1/trac/... http://host2/trac/...

(one of them is via ssh tunnel from outside, the other is the local hostname inside the firewall). The problem is there is no possible absolute-URL setting of base_url that can work here.

Currently I have it set to "/trac" (a relative URL) which works in many cases. It satisfies all the plug-ins I currently use, e.g. RevTree?. But it breaks the redirects after posting changes (e.g. new tickets) which is really annoying.

Isn't there any good way to use the scheme and hostname of the current page as the scheme and hostname of the redirect? Seems to me like base_url shouldn't be needed in almost any case; I can hope anyway.

[eblot]

See also: http://groups.google.com/group/trac-dev/browse_frm/thread/a8d18e1d1ed98825

[slightly off-topic]
I need to check whether the Revtree could use href() calls rather than relying on base_url to build its own URLs.

[cboos]

I think we need an additional setting, in order to differentiate all the possible cases:

1. default_url: should be used when there's no possibility to reconstruct the absolute URL without additional information (i.e. when we don't have the req, or for self.env.abs_href).
2. base_url: should be used in preference to reconstructing the URL (see #2553)

If default_url is not set, base_url is used for 1.

When using proxies and https (#2553), base_url should be set.

For situations like the one discussed in this ticket, default_url should be set, but base_url must be unset.

[eblot]

One more setting than the ones in Apache config files?

Come on, this is going to be a administration nightmare (not even mentioning the duplicate tickets on installation issue). We need to find a way to reduce the configuration settings about this damn URL base, not to add an extra one ;-)

[cboos]

Well, specifying both settings will be superflous, it will be either one (default_url use it only for env.abs_href not for redirects -> addresses this ticket) or the other (base_url, use it for env.abs_href and for redirects -> addresses #2553).

Plus, this gives the possibility to specify neither of those, in which case this we could set the base needed for the env.abs_href on first use, by reconstructing the URL from the first req, and this would address the concern that base_url became mandatory.

[eblot]

Replying to cboos:

Plus, this gives the possibility to specify neither of those, in which case this we could set the base needed for the env.abs_href on first use, by reconstructing the URL from the first req, and this would address the concern that base_url became mandatory.

I would rather see a unique way to define the base URL once for all - always defined at the same location - ideally in the server configuration rather than in trac.ini file (the KISS way).

I'm not sure if this is possible at all but before adding another configuration directive, I really think we should try to find a better alternative. It really sounds to complex, confusing and error prone.

[cboos]

Replying to garyo@genarts.com:

.. I run a trac that is accessed via two URLs: http://host1/trac/... http://host2/trac/... (one of them is via ssh tunnel from outside, the other is the local hostname inside the firewall). The problem is there is no possible absolute-URL setting of base_url that can work here.

There might be another approach that can be tried to address this unusual setup. You could do a shallow copy of the Trac environment and arrange for the same database and repository settings to be used. But one environment will have a `base_url = http://host1/trac/... and the other a base_url = http://host2/trac/...` and of course you have to configure TracEnv accordingly.

The only trick is that you will have to share the attachment folder. See googlegroups:trac-users:ac2503231bd189c3 and answers.

There's probably a simpler way to handle all of this (taking #2553 into account), other ideas welcomed.

[John Hampton <pacopablo@…>]

Replying to cboos:

There might be another approach that can be tried to address this unusual setup. You could do a shallow copy of the Trac environment and arrange for the same database and repository settings to be used. But one environment will have a `base_url = http://host1/trac/... and the other a base_url = http://host2/trac/...` and of course you have to configure TracEnv accordingly.

OK, that is a VERY ugly hack. Ugly along the lines of: you were smoking crack when you wrote it, right?

I understand why we are using base_url for instances where there might not be a req object provided, such as automated email notification from the post-commit hook, but why are we using it when accessed via the web?

I should be able to run the same site from 10 different hostnames if I so choose. This is all stuff that should be able to be divined from the information that the webserver passes.

Can someone please enlighten me?

[cmlenz]

Replying to John Hampton <pacopablo@pacopablo.com>:

I should be able to run the same site from 10 different hostnames if I so choose. This is all stuff that should be able to be divined from the information that the webserver passes. Can someone please enlighten me?

I agree that we shouldn't be using base_url when we have an HTTP request to our disposal.

This whole mess seems to have been triggered by a problem with relying on the HTTPS environment variable, described here:

Trac learns that its using https by checking HTTPS env. Actually, it is set by mod_ssl, but AFTER trac already finished. So it sets scheme to http and uses it to make urls.

I don't have HTTPS setup in my test environment, but I really can't believe this doesn't work. We're definitely not the only ones relying on the HTTPS variable! If someone can dive in here and provide more information, that'd be very helpful.

[cmlenz]

What I actually intended to link to above was:

http://trac.edgewall.org/ticket/3208#comment:5

[eblot]

See also a previous discussion in trac-dev ML:
http://groups.google.com/group/trac-dev/browse_frm/thread/a8d18e1d1ed98825/aec0a17914345e91

[osimons <simon-code@…>]

Ticket #6150 closed as a duplicate.

However, this ticket highlights a change in behavior with 0.11dev from 0.10.4 that I am unsure about how or why happened. With 0.11dev (tested on r6049) the current status is that the base_url option has become a requirement even for sending ticket notifications in a request context. Not good as this is changed behavior for the worse...

I encourage those with insight and ability to bump the priority on this issue.

[cboos]

Replying to osimons <simon-code@bvnetwork.no>:

... the base_url option has become a requirement even for sending ticket notifications in a request context.

The idea is that we don't necessarily want to see the absolute URL corresponding to the req of the person who triggered the change in that notification mail. Imagine a developer accessing the server using a local name, or using an https connection: this is not what would be useful to the reporter getting that notification mail.

Not good as this is changed behavior for the worse...

Well, before, the above scenario could have happened, so this problem can be considered to be fixed, "accidentally" in 0.10.4 as we still have env.abs_ref = req.abs_href, correctly in trunk as we have env.abs_href set to Href(self.base_url).

But now of course, for most of the other uses, like the one described in this ticket, it would be preferable to keep the same absolute URL of the request instead of unexpectedly redirect to base_url which can be something totally different...

There's however no simple solution here, as trying to reconstruct the absolute URL from the req was failing (see comment:9). The patch on #2553 was a good workaround for all the people that have tried it, so I've finally committed it, see r5642 (which also highlighted the fact that it was only a workaround with its own drawbacks).

Bottom line: if we find a reliable way to reconstruct the absolute URL from the req in all cases, then r5642 wouldn't be needed. That could possibly only be a matter of properly documenting the setup of the server. Someone should verify that adding "SetEnv HTTPS 1" (see TracModPython#HTTPSissues) works.

[osimons <simon-code@…>]

Hmm...

If such a setting becomes required, then we should at least make some template variables available and make it possible to use a dynamic string so that at least advanced use cases can make this work - after the example of notification subject and similar.

The simple example would be $folder_name so that using parent_env setting it is possible to have one base_url in global trac.ini that expands each of the folders (projects) it covers:

base_url = http://www.myhost.org/trac/$folder_name

I would also very much like to make environment variables expandable, so that I in my various web server configs I can set a variable and resuse it in the setting. I have several parent_paths all sharing one global trac.ini, each with subtle differences.

# Apache
SetEnv PROJECT_TYPE p
# Trac.ini
base_url = https://www.coderesort.com/$PROJECT_TYPE/$folder_name
# Or just using the current 0.10.4 default behavior to fetch from req
base_url = $default

The $default could be the default behavior of the option so that 0.10.4 and 0.11 would not change behaviour. Let people with problems make the changes - don't break all the projects that do not currently experience any problems.

Could this be a way forward on the issue?

[osimons <simon-code@…>]

Nah, I reject my first idea - it is making things much too complicated... :-)

If we have to introduce new settings, then I would prefer new environment variables that work like their filesystem equivalents - it will provide much, much more flexibility: TRAC_BASE_URL (single environment) and TRAC_PARENT_BASE_URL (parent directory of projects).

The basic preference is in order:

1. TRAC_BASE_URL or TRAC_PARENT_BASE_URL available?
2. base_url option set (default empty)?
3. Reconstruct from req if available/possible.
4. if no environment variables, not base_url setting and no req there is nothing - abs_href will return the same as href.

[Marcus Lindblom <macke@…>]

Replying to cboos:

Someone should verify that adding "SetEnv HTTPS 1" (see TracModPython#HTTPSissues) works.

I can't get that to work with Trac 0.10.2 and Lighttpd.

[cboos]

Well, that was an Apache/mod_python setting, with Lighttpd you should use something else (setenv.add-environment {...}? - I don't know, never used Lighttpd myself, see http://trac.lighttpd.net/trac/wiki/Docs%3AModSetEnv).

[Marcus Lindblom <macke@…>]

Replying to cboos:

Well, that was an Apache/mod_python setting, with Lighttpd you should use something else (setenv.add-environment {...}? - I don't know, never used Lighttpd myself, see http://trac.lighttpd.net/trac/wiki/Docs%3AModSetEnv).

Of course. This is what I have:

 fastcgi.server = ("/trac" =>
                     ("trac" =>
                        ...
                        "bin-environment" =>   
                          ("TRAC_ENV_PARENT_DIR" => "/var/www/trac",
                           "HTTPS" => "yes")

I've tried with 1 and on too (as listed in my source). I've also verified that I am running this fcgi-server, since if I change trac-env-parent, it borks.

Perhaps I should move this to the mail-list.

[cboos]

Ok, here's another proposal to clean up this "mess":

1. revert to pre-r5642 behavior by default; that'll make the redirect work as expected in situations like the one described in this ticket and in general, for correctly configured servers as well (e.g. with the HTTPS environment variable set).
2. make the r5642 behavior optional for the people who really need it (like the many ones who have used that as a patch before). I'm thinking about a [trac] use_base_url_for_redirects = true setting (would be false by default). That's quite explicit and you'll immediately see that there will be a problem if you activate it in situations like the one described here.

[ThurnerRupert]

we would greatly appreciate if you could keep the 0.10.3 behaviour as default pls. we connect with http, and https to the same server, apache2, mod_python, no environment variable set, and everything takes the client browsers url as expected.

[cboos]

Replying to ThurnerRupert:

we would greatly appreciate if you could keep the 0.10.3 behaviour as default pls. we connect with http, and https to the same server, apache2, mod_python, no environment variable set, and everything takes the client browsers url as expected.

Thanks for the feedback, it's good to have confirmation that the 0.10.3 behavior was indeed working. By only hearing from people complaining that it doesn't work (re #2553 et al), you get the feeling that it works for no one :-) What would be ideal is to have the correct setup to get this working documented somewhere... I don't have the time to do it myself, but I suspect that the information given in TracModPython is maybe not complete (something to set on the mod_ssl side? SSLOptions +StdEnvVars ?).

[cboos]

Change (back?) the meaning of [trac] base_url to mean "use this as a reference URL", and don't use that for req.abs_href or req.redirects unless explicitly configured to do so, using the [trac] use_base_url_for_redirect.

[cboos]

Please try out the following attachment:use_base_url_for_redirect-r6199.diff

It worksforme, but I've not tested it in complex setups:

• for accessing an environment via multiple hosts, no configuration change is needed (with the new defaults, it should work)
• for accessing the environment via a proxy (#2553 et al), the base_url setting is not used anymore for the redirect and a use_base_url_for_redirect bool setting needs to be set to yes to get this behavior back.

[cboos]

Feedback on the above patch would be really appreciated before pushing that change in the upcoming 0.11beta1...

[garyo@…]

I'll test it tonight if I can apply it to r5923 cleanly. (I think can't run current trunk due to TracTags? plugin vs. context-refactoring changes.)

<goes away to try the patch...>

The trac/web/main.py part doesn't apply cleanly but I think I see where it should go. I'll let you know tonight.

I'm supposed to be able to completely remove base_url from my trac.ini now, right?

[osimons]

I tried this patch the other day, specifically looking for behavior when base_url was not set. I really wanted the 0.10 behavior back, whereby a no-problem configuration could format absolute links for notification even though base_url is not set.

Providing settings for those that have problems is a good thing, but forcing existing, working installations to change settings does not feel right to me.

[garyo@…]

osimons: what results did you get?

[osimons]

Replying to garyo@genarts.com:

osimons: what results did you get?

Empty base_url for notification means ticket notifications will look like this:

Comment(by simon):

with patch 6199, no base_url

-- 
Ticket URL: </ticket/1#comment:4>
testproject06 <>
This is the project description

0.10 behaviour is that if base_url is not set, we try to figure it out as best we can from the request - which I am sure worked well for nearly everyone.

[cboos]

Replying to osimons:

I tried this patch the other day, specifically looking for behavior when base_url was not set. I really wanted the 0.10 behavior back, whereby a no-problem configuration could format absolute links for notification even though base_url is not set.

That's a slightly different topic, as it concerns the more ancient change of r4086. The patch was focusing on the "trac accessed via two hostnames" use case this ticket is about.

Maybe a complementary patch addressing an autodetection of a reference URL can be proposed on #3965.

[osimons]

Just noticed that ticket myself now - but yes, there should be a unified idea of the use of base_url. I'll look into it, and move that part over onto #3965.

[cboos]

With [6240], Trac should again be accessible via two (or more) hostnames, whether the base_url is set or not, provided the use_base_url_for_redirect setting was not set to true.

[anonymous]

This patch + removing base_url from my trac.ini seems to work so far. I tried submitting a ticket change and a wiki change and both redirect properly using the original URL. Just a note though: TracTags? doesn't work perfectly without base_url; most stuff works but the Tags tab on the nav bar doesn't point to the proper URL. If my URL for the main trac is http://localhost:8080/trac/, the Tags tab points to http://localhost:8080/tags rather than .../trac/tags. Manually typing in the proper URL works though (shows the tag cloud). I'll submit as a TracTags? bug.

[osimons]

#6228 closed as a duplicate as it is closely related.

[cboos]

Backported to 0.10.5 in r7061.

Before the backport, we only had some 0.10.5dev (and possibly a few out-of-date 0.11dev ones) installations which used base_url for the redirects by default.

Now the situation is clearer:

• 0.10.4 Tracs don't use base_url for redirects (unless of course they're patched with #2253)
• 0.10.5 and onwards (0.11, 0.12dev) don't use base_url for redirects by default but can do so if they wish by setting use_base_url_for_redirects to a truth value

[cboos]

read #2553 in the above, sorry.

doc for TracIni updated in r7062.