Something went wrong. Try again.
[READ-ONLY] Mirror of https://github.com/emilyhunt/masters_project. My Master's project code, using a mixture density network to infer photometric redshifts to galaxies with spectroscopic redshifts as training data.
Something went wrong. Try again.
masters_project stuff_ive_done
13 kB · 294 lines
at master
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295Questions:> How can we deal with interstellar extinction best?> An apparent magnitude prior seems like a good shout/maybe the NN will work this outStuff I could do:> Save a copy of the graph of the network on exit and save the validation data results to the class/elsewhere> Add MAP value estimation to MixtureDensityNetwork and also add some plotting tools to show distributions at a specific data point (will be very useful for redshifts)> Add priors to the loss functions> Deal with interstellar extinction better, somehow> Implement TensorBoard support---- ---- ---- ---- ----24/12/18- TAC stuff is coming along nicely- Running with pc still on was a mistake. Also it doesn't take long to run at all. Re-running with 23/12/18- Tried doing some scipy minimisation of a 4 parameter perturbation law but it's taking hecking forever! It came up with [ 2.06914063e-02 8.49609374e-05 -2.40234375e-04 1.53320312e-04] with nmad=0.024187255520895155, maxres=0.002419339924292079 though- Using median instead of not improves test data NMAD ever so slightly (presumably it fixes some of the crazier PDFs)- Use a big ass expanded dataset! It makes life betterrrrr11/12/18- Using asinh magnitudes helps! No more worrying about stupid fucking negative fluxes. YAY.- Got the best NMAD yet with this! And all while still including data. woop.- Working out b from median standard deviations seems a lot more robust than using means.- Added a new error inferrer that can set missing errors to x-sigma times the median of that column (again - medians help here)- It's weird that medians for errors but means for fluxes is the best. Weirdweird.- Can now toggle flux and error clipping in the dataset expander (can turn off flux clipping when using log fluxes.)- We do absolutely shit on the test dataset vs EAZY. Some galaxy types not being in the training sample is probably the issue.10/12/18- The error largener thing actually works! Trained on uniform dist perturbed data it did a bit shit, but exponential perturbation helped the network learn that bigger signal to noise ratios mean worse data. Well done, network!~~ Negative fluxes investigation ~~After removing anything without zspec:- About 1750 of 96681 fluxes are negative.- 21 errors are negative. They are an absolute pain in the backside, as they make it very difficult sometimes to work with bits and bobs.pd.DataFrame([preprocessor.maximum_flux, preprocessor.minimum_flux], index=['max', 'min']).transpose() max minf_b 192.455816 -5.454247f_f125w 1922.904468 -0.125443f_f140w 2158.551050 -13.645748f_f160w 2297.888236 0.000000f_f435w 741.686021 -5.499743f_f606w 1525.043733 -10.359790f_f775w 3097.387219 -17.410079f_f814wcand 700.550732 -19.114537f_f850lp 4579.926452 -22.758663f_f850lpcand 4668.246252 -40.847501f_h 4519.814336 -79.872263f_i 1183.570720 -113.487723f_ia427 120.785593 -25.212292f_ia445 150.616354 -88.206533f_ia505 375.948210 -18.756212f_ia527 432.810959 -3.460642f_ia550 503.529825 -23.180425f_ia574 519.079493 -28.480003f_ia598 583.261365 -8.272130f_ia624 627.088048 -13.978805f_ia651 688.794049 -8.197568f_ia679 743.296135 -17.362523f_ia738 851.550280 -9.627482f_ia767 937.773423 -37.578173f_ia797 950.475167 -65.736083f_ia856 1175.615816 -121.805712f_irac1 6475.353971 -694.746795f_irac2 4285.545098 -543.222472f_irac3 5643.858541 -3216.777607f_irac4 23540.692575 -1661.184264f_j 8823.018199 -40.905936f_ks 11550.976187 -68.320645f_r 668.007362 -9.182675f_rc 674.859432 -19.406909f_tenisj 1850.500824 -30.987356f_tenisk 2374.365191 -7.334986f_u 88.152431 -9.407879f_u38 49.994829 -28.131357f_v 434.515190 -9.416636pd.DataFrame([preprocessor.maximum_signal_to_noise, preprocessor.minimum_signal_to_noise], index=['max e', 'min e']).transpose() max e min ee_b 1813.072342 -4.585194e_f125w 11577.271019 -0.368527e_f140w 6021.499607 -3.153202e_f160w 3395.355371 0.000000e_f435w 1766.453150 -3.747137e_f606w 4845.928824 -2.558229e_f775w 4963.754806 -2.503090e_f814wcand 5282.094422 -10.894921e_f850lp 5350.627595 -2.677730e_f850lpcand 1835.628232 -33.325042e_h 786.775840 -10.821966e_i 1680.158310 -5.208479e_ia427 394.792539 -8.153162e_ia445 658.069671 -5.815636e_ia505 1348.117611 -13.919531e_ia527 2565.094494 -4.572490e_ia550 2542.027166 -4.591347e_ia574 1640.647350 -4.712590e_ia598 3845.189174 -5.828010e_ia624 3576.493407 -4.439121e_ia651 5341.103504 -4.477615e_ia679 4697.300507 -4.536154e_ia738 3889.245398 -4.614795e_ia767 1987.856778 -4.684291e_ia797 1721.837871 -4.487017e_ia856 1705.155162 -10.760169e_irac1 1611.137636 -152.873279e_irac2 1131.789542 -82.343698e_irac3 264.523334 -13.081258e_irac4 249.965349 -48.047740e_j 912.519472 -5.232934e_ks 775.988918 -5.634080e_r 12667.488945 -19.659572e_rc 4586.886466 -4.054433e_tenisj 2756.580596 -4.071561e_tenisk 2130.908854 -3.462072e_u 1693.871190 -5.172567e_u38 186.074748 -7.142380e_v 3153.665527 -4.34128706/12/18- THE PERTURBATION COEFFICIENT IS THE HERO I NEEDED- MDNs are designed inherently to predict *generative* data models. Therefore, the training data must itself be generative.- This has produced the best results yet! It actually slightly lowers the NMAD (probably a bit like a form of regularisation) and is getting results that really aren't far from EAZY. In addition, the 5 sigma outlier level is just *better.* This method causes more than half as few outliers, which is AWESOME!- I should try using a perturbation function. It seems loosely analogous to the NMAD, so logically, each zspec will want to use a slightly different amount of perturbation (probably, for instance, as a function of increasing zspec.)05/12/18- Still lots of issues with the CDFs, even when normalising them to the allowed range. It just still doesn't really play ball and is a bit unstable.04/12/18- Ratio normalised column means are the new winner! Thankyou Stijn <3- This network is a fucking cheat- By putting distributions all over the place, including outside the redshift range, it can minimise the Wittman condition still- Is it really a pdf if the area inside the allowed region doesn't sum to unity? << THERE'S YOUR PROBLEM- Binary activation function is sorta helpful, but not massively- YOU COULD NORMALISE THE CDFS AGAINST THE AREA ACTUALLY WITHIN THE ALLOWED RANGE!!! FUCK YES this would work :D03/12/18- Lots of testing and trying to tune the CDF loss, now with PDFs normalised with an estimate of their maximums~~ Info on CDF loss latest testing round ~~- Log cosh loss everywhere seems to be good for stability and the like- The cdf test is a bit unstable and I would love if it was better- Test 11 seems to be the settings winner so far (NMAD ~0.04)---- Week X ----30/11/18- Added convolutional layers in the beginning of a fun time- Began re-investigating the CDF lossfunc~~ Info on different conv tests ~~- Tanh activation seems better than relu still- Conv layers offer a *marginal* improvement (that might just be from longer runtime tbh...)- Test 6 (filter size 8, stride 4) worked the best! It beat its control by 15%.- Hilariously, normal distributions are better still! (See test 11) yet without a convolutional layer, are a decent bit (25%) worse than beta distributions.~~ Info on CDF tests ~~- CDF lossfunc seems to vaaaguely help- But ultimately, it becomes degenerate with a reduction in MAP accuracy before being helpful enough to improve PDF validity (test 17)- Test 18 & 19 refactored to have the multiplier inside the log, but this seems to be a bad idea?! Doesn't seem to actually give enough onus on learning (duuuuhhh, since alog(b) == log(b^a) but log(a*b) == log(a) + log(b), aka the second one is just an offset)29/11/18~~ Info on different tuning tests ~~- Removing outliers is a bad idea- Linear interpolation between points doesn't help as much as you might hope - it presumably removes too many features- Removing nan values from the mean seemed to make the column means estimate... worse?! (adds more outliers though)- Normalised column means don't help at all vs column means it seems- Robust scaling is indeedy a bit shite, and causes horizontal lines- Not removing any columns at all is a Good Idea- Normalised column mean has a marginally higher NMAD but marginally fewer outliers - test 14- Log fluxes improves things- Running without any errors at all is better... test 23- Best missing data handling in order is: 1. Row-normalised column means 2. Linear interpolation 3. Column means 4. Row means 5. Removing galaxies with missing data---- Week IX ------------------------------02/11/18- Added some pdf plotting features, still need some bugfixing though- Hit some mad bugs with nans appearing for no apparent reason- Fixed temporarily with 64 bit precision, but this needs fixing long-term (probs helped by TensorBoard) in a better, less computationally intensive way01/11/18- Had a bit of a sad. It's ok though :)31/10/18- Running successfully on blog example stuff (yay!)- Now using a couple of data scaling systems from sklearn, which are implemented in MixtureDensityNetwork (this needs better testing, and a way to spec scaling options on the minmax scaler)- Looked into how to use TensorBoard. It looks like it'd be really awesome to add for diagnostics!30/10/18- MAP value calculator is made- Looked at some ways of working out good hidden layer sizes- Got a bit stressed about life =(- Couldn't see how to add HDI and ETI error bound estimation without a total faff. Needs more faffing with to un-faff the situation.29/10/18- Began work on a MAP value calculator.- Getting very tempted to add priors---- Week V ----26/10/18- Changed loss functions to now use a class system (lol)- Added regularisation to MixtureDensityNetwork- MixtureDensityNetwork.validate() now returns output in a dictionary, with keys that are loss-function specific- Regularisation improves the code on the test example- The MDN code I have is now very ready for action, yay!25/10/18- Finished the barebones and testing of the MDN class!- Added some gorgeous running helping (mostly modified from BATDOG), including ETAs and reporting only every few seconds- This class is a total fucking babe because you can easily change parameters in the network like number of layers, size etc- Some basic analysis has been added, like a loss function evolution plotter. Could do with adding MAP estimation.- Also it only supports one lossfunc type (with no regularisation) atm and that sucks lol24/10/18- Started work on an MDN class to make the code a lot nicer23/10/18- Got an MDN example working!!! YAY (maybe)- I had bigtime issues with tf.subtract() broadcasting not working. I *hope* that the cheeky fix I've done works.- I'm not currently reproducing loss functions as low as in the Edward example, so fingers crossed it's even actually working.22/10/18- Got the blog example running on my computer- Started investigating how MDNs work in tensorflow- Found quite a few useful packages but struggled to get inter-compatibility- Decided to plow on with using only tensorflow. It will be a bit harder but will save a hell of a lot of compatibility issues.- Started on an example mdn!---- Week IV ----- had the fucking presentation---- Week III10/10/18- Fuck BUCS lol, moving my shit!!1!- Fixed the above- Finished reading Brammer. Lots of useful stuff has come from reading it methinks.- Started reading Quadri09/10/18- Worked on IMI poster again (u lazy aha)- 08/10/18- More of Brammer- Worked on my IMI poster instead---- Week II ----05/10/18- Sorted Npairs vs deltaZ- Now also overplotting a Gaussian- Started reading Brammer, Quadri04/10/18- Done the zphot vs zspec plot- Almost there on Npairs vs deltaZ, just need to make sure no invalid redshifts get added03/10/18- Read upto chapter 4 of Raschke02/10/18- XDrive mounted- Started reading some Raschke stuff01/10/18- Got some of the ssh stuff working---- Week I ----