Freedom not to skydive?

Porphyrogenitus writes in response to my letter (reproduced in full below).
I believe that his normally-concise reasoning is falling apart here, due to his antipathy toward government regulation.  (I share this antipathy in most things.  I first made the argument he received from Terrey Cobb well over ten years ago, and I still stand by it.  But that isn't an excuse to stop thinking.)
I believe in liberty, and one's freedom to take damnfool risks ought to extend just as far as one's willingness and ability (backed by insurance, if necessary) to bear the full cost of the results.  But that only extends to risks posed to one's own self.  If you pose a risk to others you give them every right to stop you (else why are we going after bomb-plotters in tribal Pakistan and nuclear proliferators in Iran and N. Korea?).  Choosing to consume nicotine, or skydive, is one thing; forcing others to do it with you, without their permission and contrary to their well-being, is another.
He makes a point about "public space", but what he really means is places of public accomodation.  It's settled law that a place of public accomodation (such as, say, a lunch counter) cannot bar access to people on the basis of skin color.  But what of people whose difference is not visible?  Should businesses be allowed to discriminate against athsmatics by allowing some patrons to maintain conditions harmful or deadly to them?
There are varying claims about smoking bans being bad or good for business.  These are of questionable relevance to arguments about freedom.  The issue Porphyrogenitus does not acknowledge is that the right to wave one's fist ends at another's nose, and while one has the right to consume nicotine (as repulsive as both of us find it), I believe that one does not have the right to heedlessly redistribute the effluent of one's consumption so as to endanger or even inconvenience other people.  Others' right to breathe trumps one's own right to smoke when and how one pleases.
The letter:

Date:  Wed, 10 Mar 2004 16:29:40 -0800 (PST)
From: "Engineer Poet" <engineerpoet38215@yahoo.com>
Subject: Banning, or restricting?

To: jruhlconob@sprynet.com

I'm normally a fan of your work (and I will regret the day that you
have to devote most of your blogging time to the Army), but I have a
bone to pick with your writing in
http://www.porphyrogenitus.net/archives/week_2004_03_07.html#002239.

First, you are mis-stating the point of the article you cite.  The
article talks about a ban on smoking in public places, not smoking
in general.  The difference is crucial, because many of the lives
saved (and much of the morbidity prevented) comes from non-smokers
who are currently exposed to smoke in public places.  The comparison
to sky-diving is inapt; skydiving is prohibited over populated areas,
and thus falling skydivers present a negligible threat to the public.

Smoking is a problematic habit because smoke does not respect
boundaries.  One cannot easily choose whether or not to inhale smoke,
and some people's health and wellbeing are threatened by even slight
exposure.  A ban on smoking in public spaces still leaves smokers
with many options, such as smoking in private spaces (smoking booths?),
consuming nicotine in the form of chewing tobacco or gum, or other
options not yet invented (entrepreneurial opportunities).  What it
will do is restrict exposure to people who actually choose it, much
as the threat from falling skydivers is restricted by the geographic
limitations on the sport.  As a libertarian, I believe this is a good
thing.

  ¶ 3/11/2004 11:50:00 PM