Questions: > How can we deal with interstellar extinction best? > An apparent magnitude prior seems like a good shout/maybe the NN will work this out Stuff I could do: > Save a copy of the graph of the network on exit and save the validation data results to the class/elsewhere > Add MAP value estimation to MixtureDensityNetwork and also add some plotting tools to show distributions at a specific data point (will be very useful for redshifts) > Add priors to the loss functions > Deal with interstellar extinction better, somehow > Implement TensorBoard support ---- ---- ---- ---- ---- 24/12/18 - TAC stuff is coming along nicely - Running with pc still on was a mistake. Also it doesn't take long to run at all. Re-running with 23/12/18 - Tried doing some scipy minimisation of a 4 parameter perturbation law but it's taking hecking forever! It came up with [ 2.06914063e-02 8.49609374e-05 -2.40234375e-04 1.53320312e-04] with nmad=0.024187255520895155, maxres=0.002419339924292079 though - Using median instead of not improves test data NMAD ever so slightly (presumably it fixes some of the crazier PDFs) - Use a big ass expanded dataset! It makes life betterrrrr 11/12/18 - Using asinh magnitudes helps! No more worrying about stupid fucking negative fluxes. YAY. - Got the best NMAD yet with this! And all while still including data. woop. - Working out b from median standard deviations seems a lot more robust than using means. - Added a new error inferrer that can set missing errors to x-sigma times the median of that column (again - medians help here) - It's weird that medians for errors but means for fluxes is the best. Weirdweird. - Can now toggle flux and error clipping in the dataset expander (can turn off flux clipping when using log fluxes.) - We do absolutely shit on the test dataset vs EAZY. Some galaxy types not being in the training sample is probably the issue. 10/12/18 - The error largener thing actually works! Trained on uniform dist perturbed data it did a bit shit, but exponential perturbation helped the network learn that bigger signal to noise ratios mean worse data. Well done, network! ~~ Negative fluxes investigation ~~ After removing anything without zspec: - About 1750 of 96681 fluxes are negative. - 21 errors are negative. They are an absolute pain in the backside, as they make it very difficult sometimes to work with bits and bobs. pd.DataFrame([preprocessor.maximum_flux, preprocessor.minimum_flux], index=['max', 'min']).transpose() max min f_b 192.455816 -5.454247 f_f125w 1922.904468 -0.125443 f_f140w 2158.551050 -13.645748 f_f160w 2297.888236 0.000000 f_f435w 741.686021 -5.499743 f_f606w 1525.043733 -10.359790 f_f775w 3097.387219 -17.410079 f_f814wcand 700.550732 -19.114537 f_f850lp 4579.926452 -22.758663 f_f850lpcand 4668.246252 -40.847501 f_h 4519.814336 -79.872263 f_i 1183.570720 -113.487723 f_ia427 120.785593 -25.212292 f_ia445 150.616354 -88.206533 f_ia505 375.948210 -18.756212 f_ia527 432.810959 -3.460642 f_ia550 503.529825 -23.180425 f_ia574 519.079493 -28.480003 f_ia598 583.261365 -8.272130 f_ia624 627.088048 -13.978805 f_ia651 688.794049 -8.197568 f_ia679 743.296135 -17.362523 f_ia738 851.550280 -9.627482 f_ia767 937.773423 -37.578173 f_ia797 950.475167 -65.736083 f_ia856 1175.615816 -121.805712 f_irac1 6475.353971 -694.746795 f_irac2 4285.545098 -543.222472 f_irac3 5643.858541 -3216.777607 f_irac4 23540.692575 -1661.184264 f_j 8823.018199 -40.905936 f_ks 11550.976187 -68.320645 f_r 668.007362 -9.182675 f_rc 674.859432 -19.406909 f_tenisj 1850.500824 -30.987356 f_tenisk 2374.365191 -7.334986 f_u 88.152431 -9.407879 f_u38 49.994829 -28.131357 f_v 434.515190 -9.416636 pd.DataFrame([preprocessor.maximum_signal_to_noise, preprocessor.minimum_signal_to_noise], index=['max e', 'min e']).transpose() max e min e e_b 1813.072342 -4.585194 e_f125w 11577.271019 -0.368527 e_f140w 6021.499607 -3.153202 e_f160w 3395.355371 0.000000 e_f435w 1766.453150 -3.747137 e_f606w 4845.928824 -2.558229 e_f775w 4963.754806 -2.503090 e_f814wcand 5282.094422 -10.894921 e_f850lp 5350.627595 -2.677730 e_f850lpcand 1835.628232 -33.325042 e_h 786.775840 -10.821966 e_i 1680.158310 -5.208479 e_ia427 394.792539 -8.153162 e_ia445 658.069671 -5.815636 e_ia505 1348.117611 -13.919531 e_ia527 2565.094494 -4.572490 e_ia550 2542.027166 -4.591347 e_ia574 1640.647350 -4.712590 e_ia598 3845.189174 -5.828010 e_ia624 3576.493407 -4.439121 e_ia651 5341.103504 -4.477615 e_ia679 4697.300507 -4.536154 e_ia738 3889.245398 -4.614795 e_ia767 1987.856778 -4.684291 e_ia797 1721.837871 -4.487017 e_ia856 1705.155162 -10.760169 e_irac1 1611.137636 -152.873279 e_irac2 1131.789542 -82.343698 e_irac3 264.523334 -13.081258 e_irac4 249.965349 -48.047740 e_j 912.519472 -5.232934 e_ks 775.988918 -5.634080 e_r 12667.488945 -19.659572 e_rc 4586.886466 -4.054433 e_tenisj 2756.580596 -4.071561 e_tenisk 2130.908854 -3.462072 e_u 1693.871190 -5.172567 e_u38 186.074748 -7.142380 e_v 3153.665527 -4.341287 06/12/18 - THE PERTURBATION COEFFICIENT IS THE HERO I NEEDED - MDNs are designed inherently to predict *generative* data models. Therefore, the training data must itself be generative. - This has produced the best results yet! It actually slightly lowers the NMAD (probably a bit like a form of regularisation) and is getting results that really aren't far from EAZY. In addition, the 5 sigma outlier level is just *better.* This method causes more than half as few outliers, which is AWESOME! - I should try using a perturbation function. It seems loosely analogous to the NMAD, so logically, each zspec will want to use a slightly different amount of perturbation (probably, for instance, as a function of increasing zspec.) 05/12/18 - Still lots of issues with the CDFs, even when normalising them to the allowed range. It just still doesn't really play ball and is a bit unstable. 04/12/18 - Ratio normalised column means are the new winner! Thankyou Stijn <3 - This network is a fucking cheat - By putting distributions all over the place, including outside the redshift range, it can minimise the Wittman condition still - Is it really a pdf if the area inside the allowed region doesn't sum to unity? << THERE'S YOUR PROBLEM - Binary activation function is sorta helpful, but not massively - YOU COULD NORMALISE THE CDFS AGAINST THE AREA ACTUALLY WITHIN THE ALLOWED RANGE!!! FUCK YES this would work :D 03/12/18 - Lots of testing and trying to tune the CDF loss, now with PDFs normalised with an estimate of their maximums ~~ Info on CDF loss latest testing round ~~ - Log cosh loss everywhere seems to be good for stability and the like - The cdf test is a bit unstable and I would love if it was better - Test 11 seems to be the settings winner so far (NMAD ~0.04) ---- Week X ---- 30/11/18 - Added convolutional layers in the beginning of a fun time - Began re-investigating the CDF lossfunc ~~ Info on different conv tests ~~ - Tanh activation seems better than relu still - Conv layers offer a *marginal* improvement (that might just be from longer runtime tbh...) - Test 6 (filter size 8, stride 4) worked the best! It beat its control by 15%. - Hilariously, normal distributions are better still! (See test 11) yet without a convolutional layer, are a decent bit (25%) worse than beta distributions. ~~ Info on CDF tests ~~ - CDF lossfunc seems to vaaaguely help - But ultimately, it becomes degenerate with a reduction in MAP accuracy before being helpful enough to improve PDF validity (test 17) - Test 18 & 19 refactored to have the multiplier inside the log, but this seems to be a bad idea?! Doesn't seem to actually give enough onus on learning (duuuuhhh, since alog(b) == log(b^a) but log(a*b) == log(a) + log(b), aka the second one is just an offset) 29/11/18 ~~ Info on different tuning tests ~~ - Removing outliers is a bad idea - Linear interpolation between points doesn't help as much as you might hope - it presumably removes too many features - Removing nan values from the mean seemed to make the column means estimate... worse?! (adds more outliers though) - Normalised column means don't help at all vs column means it seems - Robust scaling is indeedy a bit shite, and causes horizontal lines - Not removing any columns at all is a Good Idea - Normalised column mean has a marginally higher NMAD but marginally fewer outliers - test 14 - Log fluxes improves things - Running without any errors at all is better... test 23 - Best missing data handling in order is: 1. Row-normalised column means 2. Linear interpolation 3. Column means 4. Row means 5. Removing galaxies with missing data ---- Week IX ---- -------------------------- 02/11/18 - Added some pdf plotting features, still need some bugfixing though - Hit some mad bugs with nans appearing for no apparent reason - Fixed temporarily with 64 bit precision, but this needs fixing long-term (probs helped by TensorBoard) in a better, less computationally intensive way 01/11/18 - Had a bit of a sad. It's ok though :) 31/10/18 - Running successfully on blog example stuff (yay!) - Now using a couple of data scaling systems from sklearn, which are implemented in MixtureDensityNetwork (this needs better testing, and a way to spec scaling options on the minmax scaler) - Looked into how to use TensorBoard. It looks like it'd be really awesome to add for diagnostics! 30/10/18 - MAP value calculator is made - Looked at some ways of working out good hidden layer sizes - Got a bit stressed about life =( - Couldn't see how to add HDI and ETI error bound estimation without a total faff. Needs more faffing with to un-faff the situation. 29/10/18 - Began work on a MAP value calculator. - Getting very tempted to add priors ---- Week V ---- 26/10/18 - Changed loss functions to now use a class system (lol) - Added regularisation to MixtureDensityNetwork - MixtureDensityNetwork.validate() now returns output in a dictionary, with keys that are loss-function specific - Regularisation improves the code on the test example - The MDN code I have is now very ready for action, yay! 25/10/18 - Finished the barebones and testing of the MDN class! - Added some gorgeous running helping (mostly modified from BATDOG), including ETAs and reporting only every few seconds - This class is a total fucking babe because you can easily change parameters in the network like number of layers, size etc - Some basic analysis has been added, like a loss function evolution plotter. Could do with adding MAP estimation. - Also it only supports one lossfunc type (with no regularisation) atm and that sucks lol 24/10/18 - Started work on an MDN class to make the code a lot nicer 23/10/18 - Got an MDN example working!!! YAY (maybe) - I had bigtime issues with tf.subtract() broadcasting not working. I *hope* that the cheeky fix I've done works. - I'm not currently reproducing loss functions as low as in the Edward example, so fingers crossed it's even actually working. 22/10/18 - Got the blog example running on my computer - Started investigating how MDNs work in tensorflow - Found quite a few useful packages but struggled to get inter-compatibility - Decided to plow on with using only tensorflow. It will be a bit harder but will save a hell of a lot of compatibility issues. - Started on an example mdn! ---- Week IV ---- - had the fucking presentation ---- Week III 10/10/18 - Fuck BUCS lol, moving my shit!!1! - Fixed the above - Finished reading Brammer. Lots of useful stuff has come from reading it methinks. - Started reading Quadri 09/10/18 - Worked on IMI poster again (u lazy aha) - 08/10/18 - More of Brammer - Worked on my IMI poster instead ---- Week II ---- 05/10/18 - Sorted Npairs vs deltaZ - Now also overplotting a Gaussian - Started reading Brammer, Quadri 04/10/18 - Done the zphot vs zspec plot - Almost there on Npairs vs deltaZ, just need to make sure no invalid redshifts get added 03/10/18 - Read upto chapter 4 of Raschke 02/10/18 - XDrive mounted - Started reading some Raschke stuff 01/10/18 - Got some of the ssh stuff working ---- Week I ----