Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

2 Pages<12
Options
Go to last post Go to first unread
Offline Jean Giraud  
#21 Posted : 14 June 2021 02:31:15(UTC)
Jean Giraud


Rank: Advanced Member

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 5,430
Canada

Was thanked: 845 time(s) in 675 post(s)
Originally Posted by: anatolsen Go to Quoted Post
But I still need to calculate the Pearson correlation coefficients

There are 3 Pearson coefficients.
From there, up you go.

Pirson-0 [1].sm (41kb) downloaded 6 time(s).

Offline anatolsen  
#22 Posted : 14 June 2021 13:55:57(UTC)
anatolsen


Rank: Member

Groups: Registered
Joined: 08/06/2021(UTC)
Posts: 15
Location: Караганда

Dear overlord.

Thank you very much.
All problems, as I see, are solved.

How to use the loop for calculations (for loop and while), I, in general, imagined, but I did not guess to look in the sum functions.
The fact is that I use a non-English version, and the help in it does not work very well yet.
Thanks.

There is only one small problem left – is it possible to reduce the accuracy of calculating Sq, respectively Sqmin? As was seen in the calculations, they are calculated with an accuracy of up to 15 decimal places.
And this obviously affects the calculation time. Whereas 9 decimal places, maybe even 8 digits, would be enough for me.

As you rightly pointed out – the calculation time becomes huge.
But I now got out of the situation so that first I manually select an approximate solution (about 5-10 minutes). Then I set the iteration iteration step to 2 or 3 with a wide limit.
The calculation takes 40 minutes. After setting the iteration step to 1, simultaneously reducing the range of values. The calculation takes about 10 minutes.
Thus, the required accuracy is obtained – especially since the integration after the calculation confirms the accuracy of the calculation.
I'll also calculate the Pearson coefficient – in general, it will be fine, you can't ask for anything better. For one set of source data, it takes about 1 hour.
If we have to count three normal distributions instead of two, the time will increase, of course, but I think it will remain within reasonable limits.
Although, if it were possible to reduce the accuracy of the calculations, the calculation would become even easier.

Regards.

Offline anatolsen  
#23 Posted : 14 June 2021 14:04:03(UTC)
anatolsen


Rank: Member

Groups: Registered
Joined: 08/06/2021(UTC)
Posts: 15
Location: Караганда

Dear Martin Kraska

I tried to compose the solution as you suggested.
But for some reason, the program gives an error.

In addition, I have a question – you suggested a solution when the approximation is based on the sum of two normal distributions.
And in my source data, this does not always happen. It is often necessary to look for an approximation by the sum of the three distributions.
Will this increase the calculation time to inconvenient values?

Fit-1.sm (11kb) downloaded 3 time(s).

Regards.
Offline anatolsen  
#24 Posted : 14 June 2021 14:14:28(UTC)
anatolsen


Rank: Member

Groups: Registered
Joined: 08/06/2021(UTC)
Posts: 15
Location: Караганда

Originally Posted by: Jean Giraud Go to Quoted Post

There are 3 Pearson coefficients.
From there, up you go.



Thanks

Your solution record is certainly more compact.
However, it will be more difficult for me to modify it for my other cases compared to the Overlord entry.

Regards.
Offline overlord  
#25 Posted : 14 June 2021 14:47:14(UTC)
overlord


Rank: Advanced Member

Groups: Registered
Joined: 23/07/2013(UTC)
Posts: 365
Turkey

Was thanked: 100 time(s) in 74 post(s)
Originally Posted by: anatolsen Go to Quoted Post
There is only one small problem left – is it possible to reduce the accuracy of calculating Sq, respectively Sqmin?


I have an idea for it, for your original function. Don't know if it is applicable or I have the right idea or skills.

I will try to implement when I am available.

Regards

Offline Jean Giraud  
#26 Posted : 14 June 2021 15:05:40(UTC)
Jean Giraud


Rank: Advanced Member

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 5,430
Canada

Was thanked: 845 time(s) in 675 post(s)
Originally Posted by: anatolsen Go to Quoted Post
Your solution record is certainly more compact.
However, it will be more difficult for me to modify it for my other cases

Problem is that we don't know what you are looking for.
Are you looking for the best fit to your data set ???
If so, minute job.
Please, don't hesitate ... Jean

Pearson UNK.sm (41kb) downloaded 6 time(s).
Offline anatolsen  
#27 Posted : 14 June 2021 17:49:53(UTC)
anatolsen


Rank: Member

Groups: Registered
Joined: 08/06/2021(UTC)
Posts: 15
Location: Караганда

Originally Posted by: overlord Go to Quoted Post

I have an idea for it, for your original function. Don't know if it is applicable or I have the right idea or skills.

I will try to implement when I am available.

Regards



Thanks.
Here is the entire calculation from start to finish as it currently stands

Pirsonend.sm (37kb) downloaded 6 time(s).

Regards
Offline anatolsen  
#28 Posted : 14 June 2021 18:16:14(UTC)
anatolsen


Rank: Member

Groups: Registered
Joined: 08/06/2021(UTC)
Posts: 15
Location: Караганда

Originally Posted by: Jean Giraud Go to Quoted Post

Problem is that we don't know what you are looking for.
Are you looking for the best fit to your data set ???
If so, minute job.
Please, don't hesitate ... Jean



So I wrote down what I needed at the very beginning.
There is a data set (two rows of numbers).
Basically, they should be random sets.
However, in reality, two or even three different data sets are mixed there.
I need to separate them - to identify those normal distributions that correspond to pure data sets (not mixed data sets).
In the previous replica of for overlord, I posted the full calculation method.

Now only one question remains. Iterating through 9 parameters requires a lot of time. And the calculation is carried out with an accuracy that is superfluous for my purposes.
Sq and Sqmin are calculated with an accuracy of 15 decimal places. And I would have more than enough 8 or 9 digits after the decimal point.
This would significantly reduce the calculation time.
And here is the Pearson correlation coefficient I have 0.9999, whereas I would have more than enough 0.9

On the other hand, perhaps the Pearson coefficient should not be worsened.
Here in that calculation (in the previous replica) I had an error after integrating 1.5%, and I slightly knocked down the parameters, and it increased to 2.4%.
And this is more important than the Pearson coefficient. I would not count it at all, but I will definitely be asked a question about this coefficient.

There the definite integral is taken WW(x). And it is equal to 0.9658.
And initially I have a histogram g2(x), and if you take the integral of it, it will be equal to 0.99.
And in my case, this is more important than the Pearson coefficient.
Offline Jean Giraud  
#29 Posted : 14 June 2021 20:28:25(UTC)
Jean Giraud


Rank: Advanced Member

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 5,430
Canada

Was thanked: 845 time(s) in 675 post(s)
Originally Posted by: anatolsen Go to Quoted Post
So I wrote down what I needed at the very beginning.

I need to sum three normal distributions.
Normal PDF almost don't exist in physics unless intentional
like shooting 100 times the target with the same gun.
Under the hood there may be Logistic, Weibull noise.
If you have these 3 data sets from raw experiment,
please: attach as collected, absolutely untouched.
Cheers ... Jean
Offline anatolsen  
#30 Posted : 15 June 2021 15:19:52(UTC)
anatolsen


Rank: Member

Groups: Registered
Joined: 08/06/2021(UTC)
Posts: 15
Location: Караганда

Originally Posted by: Jean Giraud Go to Quoted Post

Normal PDF almost don't exist in physics unless intentional
like shooting 100 times the target with the same gun.
Under the hood there may be Logistic, Weibull noise.
If you have these 3 data sets from raw experiment,
please: attach as collected, absolutely untouched.
Cheers ... Jean


So here it is. Two cases. I counted the first one in the very first examples. There, the three normal distributions are summed up. In the last example, the other case is that two normal distributions are summed up.

But, actually, the problem is solved.
Only the time for calculating each case is too long.
Offline Jean Giraud  
#31 Posted : 15 June 2021 16:27:50(UTC)
Jean Giraud


Rank: Advanced Member

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 5,430
Canada

Was thanked: 845 time(s) in 675 post(s)
Originally Posted by: anatolsen Go to Quoted Post
So here it is.

You forgot to attach the untouched experimental data set.
That's the only thing I need in statistical work.
Offline anatolsen  
#32 Posted : 15 June 2021 18:19:52(UTC)
anatolsen


Rank: Member

Groups: Registered
Joined: 08/06/2021(UTC)
Posts: 15
Location: Караганда

Originally Posted by: Jean Giraud Go to Quoted Post
You forgot to attach the untouched experimental data set.
That's the only thing I need in statistical work.


Why did I forget? Both sets of data are there.

Here with two normal distributions (the third one is not visible).
2.png

Here with three normal distributions.
3.png

In principle, it is still necessary to take into account that these are histograms.
This is given as X the middle of the column.
Initially, the column width is 20 units, then 40 is for X=180, then in the first case the width is 100, and the last column is 50.
In the second case, the last three columns are 50 each.
Offline Jean Giraud  
#33 Posted : 15 June 2021 20:24:55(UTC)
Jean Giraud


Rank: Advanced Member

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 5,430
Canada

Was thanked: 845 time(s) in 675 post(s)
Originally Posted by: anatolsen Go to Quoted Post
In principle, it is still necessary to take into account that these are histograms.

This is what I reject: freaked experimental data.
An histogram is the first freak in statistical work.
I will make my own from a long time ago Smath utility.
Please, attach the Smath document that contents the two
sets of raw data as collected.
At this point, you have no project for collaboration.
Alternately, let me know and I will delete myself from this thread.
Cheers ... Jean.
Offline overlord  
#34 Posted : 15 June 2021 22:12:14(UTC)
overlord


Rank: Advanced Member

Groups: Registered
Joined: 23/07/2013(UTC)
Posts: 365
Turkey

Was thanked: 100 time(s) in 74 post(s)
Originally Posted by: anatolsen Go to Quoted Post
Originally Posted by: Jean Giraud Go to Quoted Post

Problem is that we don't know what you are looking for.
Are you looking for the best fit to your data set ???
If so, minute job.
Please, don't hesitate ... Jean


Now only one question remains. Iterating through 9 parameters requires a lot of time. And the calculation is carried out with an accuracy that is superfluous for my purposes.
Sq and Sqmin are calculated with an accuracy of 15 decimal places. And I would have more than enough 8 or 9 digits after the decimal point.
This would significantly reduce the calculation time.
And here is the Pearson correlation coefficient I have 0.9999, whereas I would have more than enough 0.9

Here is your Sq and Sqmin calculations.
I have noticed Sqmin make a peak to lowest and rise again at some point.
I coded to find those points range within first iteration.
This is done by evaluating variables one by one, not all at one.
All iterations are taken under 5 seconds on my PC.
Please inform if this results are enough for you.

Regards

eng-min_rev_fast.pdf (618kb) downloaded 2 time(s).
eng-min_rev_fast.sm (41kb) downloaded 3 time(s).

Edited by user 15 June 2021 23:49:52(UTC)  | Reason: Not specified

Offline Jean Giraud  
#35 Posted : 16 June 2021 00:37:16(UTC)
Jean Giraud


Rank: Advanced Member

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 5,430
Canada

Was thanked: 845 time(s) in 675 post(s)
Originally Posted by: Jean Giraud Go to Quoted Post
This is what I reject: freaked experimental data.

If I understand what you said: you have two data sets.
No problem to approximate both and average.
Two data sets for the same experiment ... NON-sense.
If the underlying phenomenon is same, collect more data.
Read carefully this document.
The Lorentz is pure brute force from well collected data.
Experimental data set represents something to me.
R is a freeware advanced statistical software, try it.
Cheers ... Jean.

UnknownData.sm (65kb) downloaded 4 time(s).



Offline overlord  
#36 Posted : 16 June 2021 01:13:29(UTC)
overlord


Rank: Advanced Member

Groups: Registered
Joined: 23/07/2013(UTC)
Posts: 365
Turkey

Was thanked: 100 time(s) in 74 post(s)
A revised Sq and Sqmin calculations of previous.
All iterations are taken under 25 seconds on my PC.
Sqmin is dropped to 6x10^-6. And now I am out of ideas.
Please inform if this results are enough for you.

Regards

eng-min_rev_fast.pdf (672kb) downloaded 7 time(s).
eng-min_rev_fast.sm (48kb) downloaded 8 time(s).

Offline Jean Giraud  
#37 Posted : 16 June 2021 15:09:45(UTC)
Jean Giraud


Rank: Advanced Member

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 5,430
Canada

Was thanked: 845 time(s) in 675 post(s)
Originally Posted by: overlord Go to Quoted Post
Please inform if this results are enough for you.

You can plot the BarSize like in post # 12.
Users browsing this topic
Similar Topics
Approximate solution of equation system? (Questions)
by Ushwood 14/08/2018 13:43:15(UTC)
2 Pages<12
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.