Something went wrong. Try again.
[READ-ONLY] Mirror of https://github.com/emilyhunt/masters_project. My Master's project code, using a mixture density network to infer photometric redshifts to galaxies with spectroscopic redshifts as training data.
Something went wrong. Try again.
Commits on master
emilyhunt committed
emilyhunt 450cd1f8 Added some new ways to run the code.
- Can now run on a grid ot optimise perturbation!
- Incidentally, also played around with some different perturbation settings and got nowhere (e.g. having a polynomial of co-efficients.)
- Minor plotting improvements and updates. (When isn't there?)
emilyhunt committed
emilyhunt 23b79077 Density plotting now online!
- z_plot.density_plot() allows for making arbitrary density plots of some particular kind of data. It's pretty cool :)
- Now importing EAZY-derived SFR and mass information for diagnostic purposes
- It looks like the training sample misses out on lots of smaller, redder galaxies, which are then the outliers in the plot
- This seems to be the cause of the mad issues when comparing network to EAZY redshifts.
- This could probably only be fixed with a more representative (or inferred to be more representative) galaxy set
- Only thing I haven't tried is running with actual low SNR values (especially at low z). This ought to help a bit more.
emilyhunt committed
emilyhunt d213b391 More fixy attempts.
- Added some cool new adhoc plots to run3dhst.py (at the end)
- Changed util.where_valid_redshifts to allow None validity conditions so it can be ignored in code
- Added a "zeroed magnitude" mode to the preprocessing util so that it can try using shapes only, and not brightnesses
emilyhunt committed
emilyhunt c9ac3764 More work on the dataset enlarger.
- Now invulnerable to errors causing log value fails
- Stopped accidentally polluting the validation set with training set galaxies by re-ordering some code
- Haven't yet found a good relationship between signal to noise at the input changing and this actually resulting in greater errors at the output. Come on network! You can do eit
emilyhunt committed
emilyhunt 2d334313 Fixed a bug that was causing only half as many fluxes to be entered into the code and added a dataset enlarger to preprocessing.py.
- Stupid, stupid bug
- Don't delete parts of a list you're iterating over, else you might accidentally halve your dataset...
- Dataset enlarger is working well! However, it needs to become more of a part of the PhotometryScaler and account for minimum fluxes correctly.
emilyhunt committed
emilyhunt 6631a447 Refactored preprocessing and file handling systems.
- Now have a preprocessor class to handle refactoring, keeping tally of minimums across all data so log transforms work
- However, I found that taking all negative flux values as wrong and inferring new values helped more
- Need to check with Stijn about this
- I also moved file handling functions to their own file as util was getting crowded.
emilyhunt committed
emilyhunt c7a9c962 THE PERTURBATION UPDATE
- Now returned to using NormalPDFLoss, but with a new, exciting addition...
- A single "perturbation parameter" specifying a deviation to apply to spectroscopic redshifts in the training sample is applied
- It has a very similar value to the NMAD, in fact! THis is by design!
- MDNs are supposed to work from a generative data set... yet using spectroscopic redshifts as given single points isn't exactly very generative
- This has given the absolute best results so far, with a set NMAD of 0.0334 and *near-perfect* PDFs.
- The perturbation even lowers the NMAD! And while running without errors is still better, this shouldn't be for too long...
emilyhunt committed
emilyhunt af387696 Lots of messing around with the loss function
- Can now specify the strength of normalisation - this may be useful
- The method is cheating by putting distributions outside of the allowed redshift range :@
- This could be fixed by either normalising the allowed window or using a binary activation function
- I think normalising the allowed window is the best shout!
- This will be a pain to implement
emilyhunt committed
emilyhunt 9661987f More work on improving NormalCDFLoss().
- Added a standard deviation weighting parameter which pushes distributions to be narrower
- Added pdf normalisation estimates using the most heavily weighted mixture. However, this can be quite inaccurate for some PDFs. (The peak could be narrow, or formed of multiple distributions.)
- Trying to work on a grid system for finding maximums. Wish me luck? ...
- Needs more work with dimensionality and should maybe be parameterised with a dictionary opposed to loads of kwargs to make life easier.
- Will probably be slow as fuck. I WISH there was an easy, fast way to estimate maxima of a mixture of probability distributions. :(
emilyhunt committed
emilyhunt 52bb4a1c Completed some hyperparameter tuning and added various new log data functionality.
- Column means or normalised column means seem to be the best!
- Ignoring columns is a pretty bad idea. The network may learn to use them less on its own, it seems :)
- Added log error and flux utilities to improve distributions being fed into the network. Having bugs with 3dhst coverage though...
- Values seem to get scrambled by normalised_column_mean flux key handling? And NaNs appear!
- Something might be being set on an actual part of a data frame, not a slice :s that, or re-indexing is going woefully wrong somewhere
- Oh pandas... how I love you
- >:-@
emilyhunt committed
emilyhunt 8a35a3d6 Re-wrote util.make_3dhst_table and added more config options.
- make_3dhst_table refactored to read_fits, and now behaves similar to read_save (specify the location of the file and it does the rest for you.)
- New config options added from Skelton+2014, with info on filters in a csv and wavelength-ordered data keys in the goodss_config.py file.
emilyhunt committed
emilyhunt c322ee44 Made the code SO MUCH FASTER
- I DON'T ENTIRELY KNOW HOW THAT WORKED
- WHAT THE FUCK IT'S 300x QUICKER LMAO
- Greatly reducing the amount MDN.train() calls have to do has been very, very helpful
- The summary writing was probably taking up a lot of resources
- Played around with CDF stuff more. Can't really get it to work super well yet as a loss function - will need a lot of hyperparameter tuning or similar to get there.
- It's quite technically challenging to balance the effect of both the pdf loss and the cdf loss all at the same time. And... honestly maybe a bit too hard.
- Next step needs to be levelling up the pre-processing and seeing if that makes the MDN a bit happier for now.
emilyhunt committed
emilyhunt 357607db Attempts at re-defining the loss function to use a cumulative distribution function and a Wittman+2015 method added.
- It isn't working yet, but some code is there. Yay!
- A residual method is working somewhat, but the half working with CDFs etc still needs some testing. It isn't stable and solutions don't happen.
emilyhunt committed
emilyhunt f90fa995 Various fiddling with run_3dhst.py to improve its performance.
- Now able to use column means as the missing data handling method.
- Standard scaling (that doesn't kill outliers) now added to MixtureDensityNetwork.
- Getting progressively better results with the network by messing around with lots of different settings.
- Struggling with a lack of spectroscopy (could be solved with grism redshifts)
- Tried adding a 1D convolutional layer at the start but couldn't get it working. Would be worth trying, along with using S/N as the error input instead of error itslf
- Should definitely try a 100% coverage run.
emilyhunt committed
emilyhunt f3d4c669 Updates to preprocessing systems.
- Now reading in but without causing stupid fucking pandas errors because I wans't indexing correctly.
- Reading in (I've forgotten the name of the function... it's in util at the bottom) now splits into a dataset with spectroscopic redshifts and a dataset without. This means the one without can be used as a validation set using the pair algorithm.
emilyhunt committed
emilyhunt 455edd33 Added new error analysis tool and completed z_plot refactoring.
- New function error_evaluator is able to see if quoted errors are realistic.
- Might be better for it to work with PDFs eventually but oh well lel, this way is much simpler
- z_plot refactoring completed to now be neater and have module-common arguments.
emilyhunt committed
emilyhunt b585fba3 Now ready to go on running on REAL 3dhst data!
- Utilities for processing 3dhst data added to util, including a photometric table maker and a function that checks the coverage of the photometry and suggests bands to remove/fix
- zphot_vs_zspec refactoring completed: it now makes prettier plots!
- Bugfix when calculating inter-quartile ranges (since I was quoting decimals, not percents, as the quartiles)
- Ran and quickly tested a script that runs on 3dhst data. Now to run it for realsies for an extended period of time and see what happens.
- Still need to supplement quite a lot of function strings with more info.
emilyhunt committed
emilyhunt 99c3738f Various stability updates.
- Network now has and returns self.network.training_success, a boolean that dictates whether or not training was successful (without, for instance, a NaN value happening)
- run_fake_data.py now uses while loops to cycle over configurations, which allows for restarts at a user-specified point
- Code has now been ran on a number of different systems successfully, showing that its compatibility is pretty solid!
- Previous memory leak issues are a problem with my laptop only. The code itself seems absolutely fine being on other computers. This could be because conda installed a version of tensorflow that isn't super well compiled for my laptop. Pip seems to do a better job of installing the correct tensorflow version?
emilyhunt committed
emilyhunt f21910f0 Fixed the Adam optimizer, also fixed regularisation!
- Adam optimizer now working a lot better. I didn't set the activation functions right... oops
- Regularisation wasn't implemented right as the get_regularization_loss() method was before the layers (not after.) Regularisation is now working!
- Twitter support is working beautifully!
- Now ready to do a big play with hyperparameters this evening on a long run of the code.
emilyhunt committed
emilyhunt 6435a8a8 Various feature additions and testing to find the Adam death bug.
- Added Twitter support by copying stuff over from emilyutil like a lazy human
- Implemented Tensorboard support including recording of *most* variables. Should next add gradient tracking to the optimizer and fix the regularization tracking.
- Messed around with trying to find why the Adam optimizer fails. Adagrad works fine (but sucks.) Adam seems to hit a sudden issue when changing learning rates maybe?! Or similar. It seems to be from the gradient somehow.
emilyhunt committed
emilyhunt 80bf53cc Continued work on adding support for pdf viewing features.
- A handful of bugs left until pdf plotting is working.
- MAP value estimation now uses a quick evaluation of a few different points to test for where the global minimum is.
- Changed temporarily to 64 bit precision as division by zero errors were occuring. However, this is slow! It would be good to change back again.
emilyhunt committed
emilyhunt 9c857cab Now running on blog data and added basic input scaling to MixtureDensityNetwork.
- MDN can now use RobustScaler and MinMaxScaler from sklearn.preprocessing both for tidying up inout features and for re-scaling output (which is extremely useful for beta distributions!)
- run_fake_data.py now implements a network and runs it on the blog example data. Yay!
- I'm so happy with MDN, as an API it works beautifully at making the code so much damn nicer to use
- Looking to add TensorBoard support imminently because regularisation on the blog data causes nans after ~10 iterations. It will seriously help with debugging.
emilyhunt committed
emilyhunt 5087de9b Began writing a MAP value calculator for MixtureDensityNetwork.
- Have added a new .pdf() method to NormalDistribution that evaluates pdf values quickly with numpy. (Not with none of that tensor milarkey.)
- Began implementing calculate_map, which uses scipy and a soon-to-be-made-sensible starting guess to find MAP values.
emilyhunt committed
emilyhunt 2be74494 Loss functions now using a class system (lol)
- Classes should all have 'evaluate' methods
- Various stuff in their __init__ is common too
- mdn.py now supports different loss functions
- Regularisation was also added, and can be toggled at initialisation of an MDN.
- Output from MixtureDensityNetwork.validate() changed to be a dictionary, with keys named depending on the kind of distribution used. This makes more sense to the user when using different loss functions.
emilyhunt committed
emilyhunt 8e2e77eb MixtureDensityNetwork class and corresponding unit tests now in a workable position!
- The class makes using mdns soooo much easier!
- Still a couple more features to add, most notably support for different loss functions and regularisation
- Feeling very close to running this on actual redshift data!
emilyhunt committed
emilyhunt f2d0beaf Continuing work on mdn.py
- Began adding some helpful function annotations
- Introduced type hinting in the MixtureDensityNetwork methods
- About to move away (after this commit) from having a master dictionary of network feed data, instead only having training and test data (this will allow for more simplicity and will instead let me use train, test etc member functions to progress the neural network (opposed to just trying to do everything with a single overused 'run' method.)
emilyhunt committed
emilyhunt cd4316a1 toy_mdn_emily.py is now running!
- Closely follows a couple of examples, but is different by using the layers api
- Had various issues with tf.subtract(). I need to implement some criticism methods to allow me to see if the output is actually working.
- For now though, it actually runs and the loss func evolves, which is great!
emilyhunt committed
emilyhunt f192f547 Playing with various examples.
- Various examples are a pain in the backside!
- Everything uses different tensorflow versions, and helpful modules are often not compatible...
- I want to make a basic MDN example on my own to learn tensorflow and go from there.
- Have started this in examples > toy_mdn_emily.
emilyhunt committed
emilyhunt 37be6973 Now perturbing galaxies by a random amount!
- Plots are looking much nicer!
- The perturbations are at least larger than the minimum pair separation, meaning that physical pairs still in the ra/dec catalogue are broken up but without losing the shape of the original catalogue
- It could be improved yet further by assigning redshifts based on the original redshift distribution for each sky position.
emilyhunt committed
emilyhunt aafa6d00 Work continues on debugging the pair comparison code.
- It probably doesn't need normalising! See my little proof
- Just shuffling the co-ordinates retains the large scale structure already encoded in them
- I should instead try perturbing all co-ordinates with deviates - retains most of the overall structure and densities, but if done by enough it should remove the intrinsic physical pairings.
emilyhunt committed
emilyhunt a2abbb8c Lots of attempts to bugfix the sky pair plotting, and addition of Gaussian fitting.
- Plotting seems to be working well now, after a string of fixes caused by silly issues.
- Redshift range restriction is still a bit iffy - I'm not confident that I'm restricting the range of random redshifts in a remotely good way.
- Normalisation of the random distribution probably isn't correct but I'm not totally sure about the best way to go about fixing it. (Quadri is like hieroglyphs...)
- Now fitting single and double Gaussians to the distribution, yay!
emilyhunt committed
emilyhunt 98fed583 Pairs now plotting, issue exists in random distribution subtraction.
- Pair plotting is now at least run-able, yay!
- Normalising the distribution unfortunately doesn't seem to help. (May not even be valid what I've done)
- There may be a systematic or random error at play, as the distribution has pretty icky tails.
- Gonna need to look at this again next week, and look for errors/ways I can improve the code.
- I wouldn't be surprised if there's a big error somewhere.
- Alternatively, the photometric redshifts might just be a bit shit. It would be good to look at the number distribution of zphots in the data set.
emilyhunt committed
emilyhunt c6116280 Beginning work on fixing sky pair algorithm & modularising files.
- Re-wrote find_pairs_on_sky to be wrapped by store_pairs_on_sky, which (untested!) should store found pairs to files.
- Had issues with not being able to import from local directories so I'm going home now instead. aha.
- Also split up the code, now having z_plot for plot functions and z_util for processing, as I was starting to get a hell of a lot of calculation things all over the place.
emilyhunt committed
emilyhunt 55f1f4b3 Repo initialised. Current status:
- redshift_evaluator.py allows for comparison of spec and photometric redshifts, and also performs an N_pairs vs difference in redshift comparison.
- Scripts stored in the aptly named /scripts/ directory
- Currently nothing is rough enough on my CPU to need running on the server
- Coding is being done as best as possible with PEP8 compliance and making sure anything I write is accessible as a function by other scripts :)
emilyhunt committed