mcmc_equilibrate - Segmentation fault (core dumped)

classic Classic list List threaded Threaded
9 messages Options
P-M
Reply | Threaded
Open this post in threaded view
|

mcmc_equilibrate - Segmentation fault (core dumped)

P-M
I am trying to run the following snippet of code:

    import graph_tool.all as gt
    import timeit, os
    import cPickle as pickle
    import numpy as np

    g = gt.load_graph('graph_no_multi_reac_type.gt')
    gt.remove_parallel_edges(g)

    print 'minimising state'
    start = timeit.default_timer()
    state = gt.minimize_nested_blockmodel_dl(g, deg_corr=True,verbose=False)
    duration = timeit.default_timer()-start
    print 'duration: ', duration

    state = state.copy(sampling=True)
    with open('state_n_dc.pkl','wb') as state_pkl:
        pickle.dump(state,state_pkl,-1)
    print 'equilibrating Markov chain'
    start = timeit.default_timer()
    mcmc_args = {}
    mcmc_args=dict(parallel = True,niter=10)
    history = gt.mcmc_equilibrate(state, wait=1000, history=True,mcmc_args=mcmc_args)
    duration = timeit.default_timer()-start
    print 'duration: ', duration
    with open('history1.pkl','wb') as his1_pkl:
        pickle.dump(history,his1_pkl,-1)

On the attached graph (graph_no_multi_reac_type.gt ). I, however, get an error message stating "Segmentation fault (core dumped)". To be precise my screen output is:

    minimising state
    duration:  991.349531889
    equilibrating Markov chain
    Segmentation fault (core dumped)

So the error seems to arise during `history = gt.mcmc_equilibrate(state, wait=1000, history=True,mcmc_args=mcmc_args)`. Why might that be?

I have observed the issue both with  `2.20dev (commit 01432d60, Sun Jan 15 19:54:52 2017 +0000)` and `2.20 (commit f6ef9990, Fri Jan 27 16:40:08 2017 +0000)`.

Best,

Philipp
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

Tiago Peixoto
Administrator
On 14.02.2017 17:20, P-M wrote:

> I am trying to run the following snippet of code:
>
>     import graph_tool.all as gt
>     import timeit, os
>     import cPickle as pickle
>     import numpy as np
>
>     g = gt.load_graph('graph_no_multi_reac_type.gt')
>     gt.remove_parallel_edges(g)
>
>     print 'minimising state'
>     start = timeit.default_timer()
>     state = gt.minimize_nested_blockmodel_dl(g, deg_corr=True,verbose=False)
>     duration = timeit.default_timer()-start
>     print 'duration: ', duration
>
>     state = state.copy(sampling=True)
>     with open('state_n_dc.pkl','wb') as state_pkl:
>         pickle.dump(state,state_pkl,-1)
>     print 'equilibrating Markov chain'
>     start = timeit.default_timer()
>     mcmc_args = {}
>     mcmc_args=dict(parallel = True,niter=10)
>     history = gt.mcmc_equilibrate(state, wait=1000,
> history=True,mcmc_args=mcmc_args)
>     duration = timeit.default_timer()-start
>     print 'duration: ', duration
>     with open('history1.pkl','wb') as his1_pkl:
>         pickle.dump(history,his1_pkl,-1)
>
> On the attached graph ( graph_no_multi_reac_type.gt
> <http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027034/graph_no_multi_reac_type.gt>  
> ). I, however, get an error message stating "Segmentation fault (core
> dumped)". To be precise my screen output is:
>
>     minimising state
>     duration:  991.349531889
>     equilibrating Markov chain
>     Segmentation fault (core dumped)
>
> So the error seems to arise during `history = gt.mcmc_equilibrate(state,
> wait=1000, history=True,mcmc_args=mcmc_args)`. Why might that be?
>
> I have observed the issue both with  `2.20dev (commit 01432d60, Sun Jan 15
> 19:54:52 2017 +0000)` and `2.20 (commit f6ef9990, Fri Jan 27 16:40:08 2017
> +0000)`.
>
> Best,
>
> Philipp
Please supply the actual graph you used, so this can be investigated.

The git version you tried is more than a month old; do you see this problem
with a current git version?

Since you tried with 2.20dev, I assume you compiled it yourself. What OS are
you using, what GCC version do you have, and are you using sparsehash or not?

Best,
Tiago

--
Tiago de Paula Peixoto <[hidden email]>


_______________________________________________
graph-tool mailing list
[hidden email]
https://lists.skewed.de/mailman/listinfo/graph-tool

signature.asc (849 bytes) Download Attachment
--
Tiago de Paula Peixoto <tiago@skewed.de>
P-M
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

P-M
Hi Tiago,

Tiago Peixoto wrote
Please supply the actual graph you used, so this can be investigated.
The graph should be online (at least it shows up via the Nabble interface). Or what do you mean by "actual"?

Tiago Peixoto wrote
The git version you tried is more than a month old; do you see this problem
with a current git version?
Good question, I am not sure, shall check that as soon as I can.

Tiago Peixoto wrote
Since you tried with 2.20dev, I assume you compiled it yourself. What OS are
you using, what GCC version do you have, and are you using sparsehash or not?
I compiled it myself indeed. I am using Ubuntu 16.04 LTS and "gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)". `libsparsehash-dev` is installed on the machine. I have not provided any specific compiler flags for it but I would expect that it will have picked it up automatically.

Best,

Philipp
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

Tiago Peixoto
Administrator
On 14.02.2017 20:20, P-M wrote:
> The graph should be online (at least it shows up via the Nabble interface).
> Or what do you mean by "actual"?

I'm sorry, I missed this in the original email.

>> The git version you tried is more than a month old; do you see this
>> problem
>> with a current git version?
>
> Good question, I am not sure, shall check that as soon as I can.

I was able to reproduce the bug with the newest git. It is caused by using
mcmc_args=dict(parallel=True). I will fix this soon, but in the meantime you
can just use parallel=False. Note that, as the documentation warns,
parallel=True triggers an approximated MCMC which will _not_ lead to the
correct posterior distribution.

Best,
Tiago

--
Tiago de Paula Peixoto <[hidden email]>
_______________________________________________
graph-tool mailing list
[hidden email]
https://lists.skewed.de/mailman/listinfo/graph-tool
--
Tiago de Paula Peixoto <tiago@skewed.de>
P-M
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

P-M
Tiago Peixoto wrote
Note that, as the documentation warns,
parallel=True triggers an approximated MCMC which will _not_ lead to the
correct posterior distribution.
I know, the documentation states that assymptotic exactness cannot be guaranteed if using parallel=True. What I wasn't quite clear on is how severe the consequences of this are. Is this a case of potentially introducing a small error or is this a case of "we have no idea what the answer will say"? If the latter is the case what would a sensible use case be for switching parallel to True? (I am trying to use it to calculate missing edges which even with parallel takes very long for the network so I fear without it it would just not be feasible.)

Best,

Philipp
P-M
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

P-M
Hi Tiago,

Do you have any thoughts on this?

Best,

Philipp
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

Tiago Peixoto
Administrator
In reply to this post by P-M
On 15.02.2017 11:36, P-M wrote:

> Tiago Peixoto wrote
>> Note that, as the documentation warns,
>> parallel=True triggers an approximated MCMC which will _not_ lead to the
>> correct posterior distribution.
>
> I know, the documentation states that assymptotic exactness cannot be
> guaranteed if using parallel=True. What I wasn't quite clear on is how
> severe the consequences of this are. Is this a case of potentially
> introducing a small error or is this a case of "we have no idea what the
> answer will say"? If the latter is the case what would a sensible use case
> be for switching parallel to True? (I am trying to use it to calculate
> missing edges which even with parallel takes very long for the network so I
> fear without it it would just not be feasible.)
The posterior will be wrong, but it is not possible to say how wrong in
general. This is useful only when, e.g. using it is a heuristic to find the
minimum description length, since the shape of the posterior does not
matter, only its maximum.

--
Tiago de Paula Peixoto <[hidden email]>


_______________________________________________
graph-tool mailing list
[hidden email]
https://lists.skewed.de/mailman/listinfo/graph-tool

signature.asc (849 bytes) Download Attachment
--
Tiago de Paula Peixoto <tiago@skewed.de>
P-M
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

P-M
Thank you for that clarification. I presume that I should not be using it to calculate the likelihood of missing edges then?

From what I gather implementing parallel MCDM is not straightforward (though my knowledge of this is hazy). Is it on your list of future developments at all? Or will this section of graph-tool stay large non-parallel for now?

Best,

Philipp
Reply | Threaded
Open this post in threaded view
|

Re: mcmc_equilibrate - Segmentation fault (core dumped)

Tiago Peixoto
Administrator
On 21.02.2017 10:54, P-M wrote:
> Thank you for that clarification. I presume that I should not be using it to
> calculate the likelihood of missing edges then?

Not if you want the probabilities to be correct; though they might still be
approximately correct.

> From what I gather implementing parallel MCDM is not straightforward (though
> my knowledge of this is hazy). Is it on your list of future developments at
> all? Or will this section of graph-tool stay large non-parallel for now?

I don't think it is trivial to write a fully parallel code, since the model
likelihood depends on the global state. So, threads that read it and modify
it need to synchronize.

--
Tiago de Paula Peixoto <[hidden email]>


_______________________________________________
graph-tool mailing list
[hidden email]
https://lists.skewed.de/mailman/listinfo/graph-tool

signature.asc (849 bytes) Download Attachment
--
Tiago de Paula Peixoto <tiago@skewed.de>