Hi, I want to stop, save and restart a sample_smc process with ABC kernel. For example I would like to save the generated data every 2000 draws, so if there is a power outage, I could restart the model from the last saved instance instead of starting it from the beginning. The reason I need that is that the mathematical model is computationally heavy and it takes 4 minutes between start of one and start of the next draw. The total process may take 2 months.
I attach my code below, after simplifying it a lot:
def simulator(xs1, xs2): xs = [xs1, xs2] #searched model parameters generate_input(xs) #generate input for the external software os.system("ansys input.inp") #run the input using external software result = read_result("output.out") # read the result return result bay_scale_model = pm.Model() with bay_scale_model: # Priors for simulator model parameters BoundedNormal = pm.Bound(pm.Normal, lower = 0.0, upper = 2.4) xs1 = BoundedNormal("xs1", mu=1, sd=0.227689) xs2 = BoundedNormal("xs2", mu=1, sd=0.257015) sim = pm.Simulator("sim", simulator, params=(xs1, xs2), sum_stat="sort", epsilon=0.003, observed=observed_data) trace, sim_data = pm.sample_smc(2000, kernel="ABC", n_steps=9, parallel=False, save_sim_data=True, cores = 1, chains = 1, threshold=0.01) idata = az.from_pymc3(trace, posterior_predictive=sim_data)
So is there any way to stop and save such a process and restart it later?