cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - If community subscription notifications are filling up your inbox you can set up a daily digest and get all your notifications in a single email. X

Gamma-Gamma Density Fit

schneidrax
1-Newbie

Gamma-Gamma Density Fit

I would like to fit a so-called gamma-gamma probability density function to a histogram. This density function involves a modified Bessel function of the second kind and (likely) fractional order. Although this Bessel function is not supported natively by my Mathcad 2001i numeric processor, I can access it through the symbolic processor.

I have been able to fit normal and log-normal probability density functions to histograms of the data using Minerr. I�ve tried (unsuccessfully) to fit the gamma-gamma probability density function to the histogram using Minerr, and so would like to try using Genfit. Unfortunately, I�ve been unable to create the �vector of derivatives�.

Can someone help me with this problem?
28 REPLIES 28

You just have a green typo.

jmG

On 3/31/2010 2:58:21 PM, jmG wrote:
>You just have a green typo.
>
>jmG
__________________________________

Not sure what to understand: the first fit is pretty good. The second is not going to start at 0 whereas you are multiplying by themselves two hyperbolic shapes !

jmG



On 3/31/2010 3:24:14 PM, jmG wrote:
>On 3/31/2010 2:58:21 PM, jmG wrote:
>>You just have a green typo.
>>
>>jmG
>__________________________________
>
>Not sure what to understand: the first
>fit is pretty good. The second is not
>going to start at 0 whereas you are
>multiplying by themselves two hyperbolic
>shapes !
>
>jmG
>__________________________________

The data set is not "white noise".
White noise is only on the Y's.
In that case, better get delta green and use Minerr [LM].
The first fit is then: like perfect !

jmG



Check the attached.

Regards. Alvaro.

PD: Rename N, I don't see that you use this.

Alvaro �

Using your series representation for non-integer orders of the Bessel function Kn, I was able to least-squares fit (albeit poorly) the gamma-gamma to the histogram [although I did find it necessary to modify slightly your program logic]. Thanks.

Unfortunately, that series representation did not help with Genfit.

P.S.:--My N represents the number of data points used in creating the histogram. The density functions must be multiplied by N for fitting.

Is it legitimate to wonder why so much difference between the "a" in your last sheet and the other ones if a function equals another one but different approximation ? What is a Gamma-Gamma distribution ? Nothing found on that.

jmG

On 4/1/2010 11:17:03 AM, jmG wrote:
>Is it legitimate to wonder why
>so much difference between the
>"a" in your last sheet and the
>other ones if a function
>equals another one but
>different approximation ? What
>is a Gamma-Gamma distribution
>? Nothing found on that.
>
Fluctuations in optical intensity due to atmospheric turbulence have long been described by gamma and log-normal probability density functions. More recently, the gamma-gamma probability density function has been proposed as a more general model.

It is indeed legitimate to ponder the significance of the a�s. For the log-normal model the a�s have a clear physical significance (e.g., average intensity). For the gamma and gamma-gamma model the physical significance of the a�s is not so direct (e.g., the a�s of the gamma-gamma model simply reflect the intensity of the turbulence).




On 4/1/2010 1:03:19 PM, schneidrax wrote:
>On 4/1/2010 11:17:03 AM, jmG wrote:
>>Is it legitimate to wonder why
>>so much difference between the
>>"a" in your last sheet and the
>>other ones if a function
>>equals another one but
>>different approximation ? What
>>is a Gamma-Gamma distribution
>>? Nothing found on that.
>>
>Fluctuations in optical intensity due to
>atmospheric turbulence have long been
>described by gamma and log-normal
>probability density functions. More
>recently, the gamma-gamma probability
>density function has been proposed as a
>more general model.
>
>It is indeed legitimate to ponder the
>significance of the a�s. For the
>log-normal model the a�s have a clear
>physical significance (e.g., average
>intensity). For the gamma and
>gamma-gamma model the physical
>significance of the a�s is not so direct
>(e.g., the a�s of the gamma-gamma model
>simply reflect the intensity of the
>turbulence).
> _________________________

Oh ! very dense !
I can take it, but the first term is not a "gamma distribution" [if there is a base gamma somewhere in there]. If the 2nd term, a Bessel term, is a modifying term [like a shape factor] ... then it has no effect. Your Gamma-Gamma looks typo from book. Do you have in hand my collection of PDF ?

That Gamma-Gamma, has it got something to do with Rayleigh ?

jmG

On 4/1/2010 10:29:01 PM, jmG wrote:

>I can take it, but the first term is not
>a "gamma distribution" [if there is a
>base gamma somewhere in there].

Check

http://books.google.com/books?id=tygEGtu1b7MC&pg=SA3-PA36&dq=%22Gamma-Gamma+Density+Function%22&hl=es&cd=1#v=onepage&q=%22Gamma-Gamma%20Density%20Function%22&f=false

Regards. Alvaro.

On 4/1/2010 11:48:59 PM, adiaz wrote:
>On 4/1/2010 10:29:01 PM, jmG wrote:
>
...
>
>Check
>
>http://books.google.com/books?id=tygEGtu
>1b7MC&pg=SA3-PA36&dq=%22Gamma-Gamma+Dens
>ity+Function%22&hl=es&cd=1#v=onepage&q=%
>22Gamma-Gamma%20Density%20Function%22&f=
>false
>
>Regards. Alvaro.
_________________________

What it says: Gamma-Gamma distribution
a formula not readable, w/o graph.
The data set from originator is not sourced, no abstract.

Can't help, Maybe Scheidrax wants to plot the given formula.

jmG



On 4/1/2010 8:39:15 AM, schneidrax wrote:

>Unfortunately, that series
>representation did not help
>with Genfit.

Fails in the Gutman's automation because the if clause, but there are also another problem, and it is that the symbolic engine can't found a closed form for the derivative of Kn.

In the attached a D-function which works numerically for using with Genfit.

Regards. Alvaro.

On 4/1/2010 6:03:11 PM, adiaz wrote:
>In the attached a D-function which works
>numerically for using with Genfit.
>
Alvaro:--I'm unable to compute the D-function numerically. Perhaps a limitation of Mathcad 2001i?


Mathcad 2001i not have a definition for Psi(x).

Mathcad 11 help says that as special symbolic function Digamma is

Psi(x) = diff(GAMMA(x),x),

but as I know it is

Psi(x) = diff(GAMMA(x),x)/GAMMA(x)

Digamma function is an example of user's dll in the userefi folder in mathcad 11, so it is not documented in the usual way.

So, hope that attached solve the problem in mc2001i. But I think that you need to download a very good library for special functions posted a lot time ago, but can't remember very well by who (Alex?).

Regards. Alvaro.

On 4/2/2010 2:48:34 PM, adiaz wrote:
>Mathcad 2001i not have a
>definition for Psi(x).
>
>Mathcad 11 help says that as
>special symbolic function
>Digamma is
>
>Psi(x) = diff(GAMMA(x),x),
>
>but as I know it is
>
>Psi(x) =
>diff(GAMMA(x),x)/GAMMA(x)
>
>Digamma function is an example
>of user's dll in the userefi
>folder in mathcad 11, so it is
>not documented in the usual
>way.
_______________________________

The MathLib for 2001i is from Giuseppe Borzi.
Revised and adapted by Xavier
Keyword for Giuseppe "Simplex", search PTC repository.

jmG
>
>So, hope that attached solve
>the problem in mc2001i. But I
>think that you need to
>download a very good library
>for special functions posted a
>lot time ago, but can't
>remember very well by who
>(Alex?).
>
>Regards. Alvaro.



On 4/2/2010 2:48:34 PM, adiaz wrote:
>Mathcad 2001i not have a
>definition for Psi(x).
>
>Mathcad 11 help says that as
>special symbolic function
>Digamma is
>
>Psi(x) = diff(GAMMA(x),x),
>
>but as I know it is
>
>Psi(x) =
>diff(GAMMA(x),x)/GAMMA(x)
>
Alvaro:--

The digamma function is correctly defined in Mathcad 2001i as Psi(x) = diff[ln(GAMMA(x)),x] and so agrees with your definition above.

I am pleased to report that I�m now able to fit a gamma-gamma probability density function to a histogram using genfit. Following your lead, I re-defined the modified Bessel function of the second kind � not using a series representation as you did, but using an integral representation from AMS-55. [I �validated� it using Maple�s BesselK function of the symbolic processor.]

I too abandoned both the Gutman and Ochkov approaches for creating the �vector of derivatives�. [See the Appendix of the attached file.] Although generated by the symbolic processor, algebraic simplification was done manually.

I also abandoned my original data set (which apparently is not well approximated by the gamma-gamma distribution) and created a new data set using a random number generator, the gamma-gamma probability distribution function (with known parameters), and a Mathcad solve block.

Thanks to you (and Jean) for your help.

On 4/4/2010 3:24:33 PM, schneidrax wrote:
...
>I am pleased to report that I�m now able
>to fit a gamma-gamma probability density
>function to a histogram using genfit.

==> OK but useless: genfit is an incomplete fitter also very slow vs Paul W. Minerr.

...
>I too abandoned both the Gutman and
>Ochkov approaches for creating the
>�vector of derivatives�.

==> Valery Genfit Diff works perfect for your case except it does not recognize the incomplete Gamma. So, you must copy/paste the genfit vector

[See the
>Appendix of the attached file.]
>Although generated by the symbolic
>processor, algebraic simplification was
>done manually.

==> Always be very careful when simplifying manually.
"simplifying" means simplifying but nor necessarily an equivalent form.

>I also abandoned my original data set
>(which apparently is not well
>approximated by the gamma-gamma
>distribution) and created a new data set
>using a random number generator, the
>gamma-gamma probability distribution
>function (with known parameters), and a
>Mathcad solve block.
>
>Thanks to you (and Jean) for your help.
>________________________

Though I have ignored genfit, I was able to recreate the function from the book, same as Alavaro. It does PW Minerr fine but the fits aren't the same at all. Gamma-Gamma is a "model fitter" but not a function as such to represent the physics of the project(s). It seems to help many of the projects described in the book but it fits NONE ! The book mentions Rayleigh but nothing about, hence my question if those "book projects" had connection with Rayleigh.

The bad news is that I can't pass the work sheet because IT CRASHED MATHCAD. Even if the part that did crash Mathcad is not in view in the first page and consequently I could disable the auto calculation, nothing to avail: the sheet crashes Mathcad and the PC.
Several collabs [developers like myself] have requested the capability to disable the autocalc from the file. No recollection it was logged as the primary feature to be implemented.
If you generate a random set of variates, you can test that Gamma-Gamma woks but can't prove that it fits the actual data. In my opinion, you should only work with collected data.

Jean

On 4/4/2010 5:00:42 PM, jmG wrote:
>
>==> OK but useless: genfit is an
>incomplete fitter also very slow vs Paul
>W. Minerr.
>
But according to Tom Gutman [15 Oct 2003]:

"I have worked a bit more with various sheets using minerr. They have all strengthened my previous conclusion that minerr is a poor choice that is to be avoided. I have yet to see any case where minerr is superior to all of the obvious alternatives (linfit, genfit, minimize). In several sheets I've replaced minerr with minimize (with TOL set to 1E-6) and gotten the same results in a fraction of the time. There is an old thread on minerr's (lack of) accuracy. My overall conclusion is that minerr is slow, inaccurate, and useless."

It was for this reason that I chose to work with genfit. Is Tom's conclusion no longer true?

>...
>>I too abandoned both the Gutman and
>>Ochkov approaches for creating the
>>�vector of derivatives�.
>
>==> Valery Genfit Diff works perfect for
>your case except it does not recognize
>the incomplete Gamma.

If it doesn't recognize the "incomplete Gamma" [I'm not sure why you're referencing this function. It doesn's appear in my formulation.], then it doesn't work "perfect".
>So, you must
>copy/paste the genfit vector.

That's my conclusion too.
>
>==> Always be very careful when
>simplifying manually.

Actually it should be avoided [Tom Gutman's Admonition and Refrain]. But in this case I wasn't able get the algebraic simplification I was looking for. The Symbolic processor can sometimes be very ornery.

>
>Though I have ignored genfit, I was able
>to recreate the function from the book,
>same as Alavaro.

To which book are you refering?

It does PW Minerr fine
>but the fits aren't the same at all.

So the operation was a success, but the patient died?
>
>Gamma-Gamma is a "model fitter" but not
>a function as such to represent the
>physics of the project(s). It seems to
>help many of the projects described in
>the book but it fits NONE ! The book
>mentions Rayleigh but nothing about,
>hence my question if those "book
>projects" had connection with Rayleigh.

I'm not sure I understand your point. My original data was experimental; the new data was contrived only to exercise the genfit algorithm.
>
>The bad news is that I can't pass the
>work sheet because IT CRASHED MATHCAD.
>Even if the part that did crash Mathcad
>is not in view in the first page and
>consequently I could disable the auto
>calculation, nothing to avail: the sheet
>crashes Mathcad and the PC.

My attached sheet was not saved with "Automatic Calculation." However, it only runs on my PC under the "Higher Speed Calculation" option.

>If you generate a random set of
>variates, you can test that Gamma-Gamma
>woks but can't prove that it fits the
>actual data.

That's quite true. Trying to fit the original (experimental) data using the gamma-gamma genfit program proved unsuccessful. But that may be because the program is not sufficiently robust to find a good fit.
>




But according to Tom Gutman [15 Oct 2003]:

"I have worked a bit more with various sheets using minerr. They have all strengthened my previous conclusion that minerr is a poor choice that is to be avoided. I have yet to see any case where minerr is superior to all of the obvious alternatives (linfit, genfit, minimize). In several sheets I've replaced minerr with minimize (with TOL set to 1E-6) and gotten the same results in a fraction of the time. There is an old thread on minerr's (lack of) accuracy. My overall conclusion is that minerr is slow, inaccurate, and useless."

It was for this reason that I chose to work with genfit. Is Tom's conclusion no longer true?
______________________

Ask the question directly.

Your quote above is before Paul W Minerr.
After that most essential feature of Minerr,
On 4/4/2010 3:40:37 PM, Tom_Gutman wrote:
>... Since I no longer consider genfit worth using , ...

Your project is interesting as long as it sticks to zillions of papers and as framed for purpose.

Conclusion ::

Your modified Bessel produces any kind of result depending upon the vector of initials. For the tentative fit around the data set you give, it produces a discontinuity. The original book Gamma-Gamma fits hyper quick PWMinerr, no discontinuity. Your modified Bessel looks "visual". What I'm saying after reading the book for few minutes, the Gamma-Gamma is a "tentative model fit" but not a fit as such. At least it should be respected wrt the application(s).
Note that Valery Genfit vector needs be interpreted.

You must have an experimental data set from source.

Sorry for the book, I can't read it back, like if it was a mistake it allowed me to read it !
It may be for purpose: read the all book but only once per search !

>That's quite true. Trying to fit the original (experimental) data using the gamma-gamma genfit program proved unsuccessful. But that may be because the program is not sufficiently robust to find a good fit.<<br> ==> The matter is not robustness, rather an insufficient model.
==> The proposed Gamma-Gamma does not fit any of the several project. They haven't even tried to ponderate the model. Weighting function(s) is easy to apply in Mathcad and is in fact the essential basis of the original Legendre least square method. Several hundred pages of material might resume in < � dozen Mathcad pages.


==============================
To which book are you referring ? [Schneidrax]

==> I don't know ! The book was apparently complete with a long "content", in blue, all the blue were linked to the corresponding pages page, that's how I got the Gamma-Gamma from source. Now, from my history visit [the last one in fact], the book is gone . After this posting , will try again from fresh start.

Jean


>>It was for this reason that I chose to work with genfit. Is Tom's conclusion no longer true?<<

It is no longer true. It was based on the misuse of minerr (some of it from Mathsoft provided examples) calculating an SSE and applying minerr to that. That doesn't work well at all.

As discussed in great detail in later posts the correct use of minerr is to simply calculate the predicted values and set the observed values equal to the predicted values in the minerr solve block. That is fast and accurate.
__________________
� � � � Tom Gutman

The paragraphs below summarize my understanding of the Topic to date:

On 31 March 2010 I wrote (above):--�I�ve tried (unsuccessfully) to fit the gamma-gamma probability density function to the histogram using Minerr, and so would like to try using Genfit.�

This seemed a reasonable strategy since:

On 15 Oct 2003 Tom Gutman stated:--"I have worked a bit more with various sheets using minerr. They have all strengthened my previous conclusion that minerr is a poor choice that is to be avoided. I have yet to see any case where minerr is superior to all of the obvious alternatives (linfit, genfit, minimize). In several sheets I've replaced minerr with minimize (with TOL set to 1E-6) and gotten the same results in a fraction of the time. There is an old thread on minerr's (lack of) accuracy. My overall conclusion is that minerr is slow, inaccurate, and useless."

On 4 April 2010 I wrote:--�It was for this reason that I chose to work with genfit. Is Tom's conclusion no longer true?�

On 4 April 2010 Tom replied:--�It is no longer true. It was based on the misuse of minerr (some of it from Mathsoft provided examples) calculating an SSE and applying minerr to that. That doesn't work well at all.�

�As discussed in great detail in later posts the correct use of minerr is to simply calculate the predicted values and set the observed values equal to the predicted values in the minerr solve block. That is fast and accurate.�

So, using the new pseudo-random data set generated from the gamma-gamma probability distribution function (with known parameters), I again tried using Minerr (see attached file). However, whereas Genfit converged to give a reasonable least-squares estimate of the density function, I was unable to achieve convergence with Minerr.

It seems that for this data set at least, Tom's earlier conclusion may be correct.

On 4/5/2010 1:07:14 PM, schneidrax wrote:
>The paragraphs below summarize
>my understanding of the Topic to date:
....................
===============================

OK, but nothing is correct. Your project starts from a blank sheet, abstracted of any previous matters and wrong concepts and interpretations as explained in the attached. Get the data from source and you will see PWMinerr wok fine.

Sorry for declaring your Gamma-Gamma at stage 0,
except for the formula from Larry C. Andrews.

Saved 2001i

jmG



I have no competence about Larry work, and no time and the Coliseum would be to small to content all the paper writers in reference [several 100's]. All what it says, "gamma-gamma" well known to be a better fit than lognorm ... So well known that nothing in Wiki !!!. Just a better fit, but looking at some graph, not yet a fit. A good way to get a headache for a long time is trying to fit "Gaussian like". The matter is always a good or the exact model. Exact model from unknown is pretty rare to find. Try to find the model for the 3 attached. No solver will find either because there are no coefficients involved, though the model is non-linear in essence.

Jean

>>It seems that for this data set at least, Tom's earlier conclusion may be correct.<<

Nope. Your problem has nothing to do with minerr. The formulation of the objective function for your genfit and minerr implementations are very different. It's not even clear that they represent the same function.

The actual problem, noticeable if you trace back the error, is that you are attempting to pass a vector argument to f, and f is not designed to support that. So the whole construct fails.

I stand by my assertion that genfit is unecessary and useless, and that minerr, when properly set up, is generally superior.

Also, in section 3, you are inverting function F (a slow process, as F is defined as an integral, and its integrand also involves an integral). Minerr is not appropriate here, the appropriate solve block is find. Usually the actual values will be the same, but find will report an error if a (reasonably) exact solution cannot be found, while minerr will happily return more or less random results, if that is all it can find.
__________________
� � � � Tom Gutman

I've managed to construct a Minerr implementation to complement my earlier Genfit implementation (see attached file). Their MSEs are about the same.

On 4/5/2010 6:32:21 PM, Tom_Gutman wrote:
>The formulation of the
>objective function for your
>genfit and minerr
>implementations are very
>different. It's not even
>clear that they represent the
>same function.

My Minerr objective function is defined explicitly. I'm not sure what the Genfit objective function is.
>
>The actual problem, noticeable
>if you trace back the error,
>is that you are attempting to
>pass a vector argument to f,
>and f is not designed to
>support that. So the whole
>construct fails.

My new Minerr objective function avoids this problem.
>
>I stand by my assertion that
>genfit is unnecessary and
>useless, and that minerr, when
>properly set up, is generally
>superior.

If my new Minerr implementation is proper, then Genfit is indeed unnecessary. Although Minerr seems only marginally superior wrt the MSE, it is much more convenient to set up (those vector derivatives needed by Genfit were tedious to derive).
>
>Also, in section 3, you are
>inverting function F (a slow
>process, as F is defined as an
>integral, and its integrand
>also involves an integral).
>Minerr is not appropriate
>here, the appropriate solve
>block is find. Usually the
>actual values will be the
>same, but find will report an
>error if a (reasonably) exact
>solution cannot be found,
>while minerr will happily
>return more or less random
>results, if that is all it can
>find.

Here, my choice of Minerr rather than Find was deliberate, considering: (a) the gamma-gamma probability distribution function is well-behaved, so I thought that Minerr will always return a (reasonably) approximate solution and so never report an error (as Find might); and (b) the ragged quality generally associated with the histograms means that exact solutions are unnecessary.

On 4/6/2010 2:23:42 PM, schneidrax wrote:
>I've managed to construct a
>Minerr implementation to
>complement my earlier Genfit
>implementation (see attached
>file). Their MSEs are about
>the same.
>............
______________________________

Let's try if it works !

I have created a "line trace" gamma-gamma,
from which you should be able to retrieve the two parameters .
Note that I scaled it, so, you may have to reduce.
It does not work well at all, because of the Bessel ???

jmG

On 4/7/2010 2:49:14 AM, jmG wrote:
>
>I have created a "line trace"
>gamma-gamma,

Jean:--Your {XY} data set is ill-defined because it is known a priori that any probability density function must be non-negative.

On 4/7/2010 10:53:20 AM, schneidrax wrote:
>On 4/7/2010 2:49:14 AM, jmG wrote:
>>
>>I have created a "line trace"
>>gamma-gamma,
>
>Jean:--Your {XY} data set is ill-defined
>because it is known a priori that any
>probability density function must be
>non-negative.
__________________________________________

That's where you are wrong ! The matter is not to fit a distribution, the matter is to fit a COLLECTION, a collection scintillates [�] around the yet unknown true value. That the fit is finally a distribution is the conclusion of the project, simply. Enjoy this gorgeous fit "too beautiful not to be true [Paul Dirac]" .

Note that this time Paul W. Minerr is in NULL style starting at index 1.

Jean



Top Tags