aurelien-goutsmedt.com / building_elsevier_database
Scripts to use scopus API to download economics journals and articles metadata + use of Elsevier API for full-text
Star
Fork 0
Watch

Commits on master

Delete claude settings master
Aurélien Goutsmedt committed
refactor: unify XML path variable and store full texts in DuckDB
Aurélien Goutsmedt committed
Merge branch 'master' of https://codeberg.org/agoutsmedt/building_elsevier_database
tdelcey committed
update paths and tmp scripts
tdelcey committed
refactor: consolidate db/path setup and use duckdb for downloading .xml
Aurélien Goutsmedt committed
`feat: load articles from DuckDB scopus.duckdb filtered by Elsevier journal ISSNs`
Aurélien Goutsmedt committed
update name duck db metadata
tdelcey committed
update paths
tdelcey committed
update paths
tdelcey committed
update depreciated files going to main
tdelcey committed
Replace RDS storage with DuckDB for Scopus metadata; deduplicate incrementally and articles affiliations, and links as tables
Aurélien Goutsmedt committed
Cross-filter Scopus econ journals with EconLit using normalized title matching Save scopus metadata in duckdb database (next step, fetch the database progressively)
Aurélien Goutsmedt committed
Expand journal filtering to all Scopus econ journals and add incremental fetching of article metadata
Aurélien Goutsmedt committed
update codes to remove depreciated (istex) components
tdelcey committed
Move outdated scripts to depreciated directory
Aurélien Goutsmedt committed
Adding script to match scopus and econlit
Aurélien Goutsmedt committed
renaming files
Aurélien Goutsmedt committed
Corrections to make work the matching between Scopus and Wos
Aurélien Goutsmedt committed
update cleaning texts to remove introduction at the start update saving scripts for better integration with python vectorizers
tdelcey committed
Merge branch 'master' of https://github.com/agoutsmedt/elsevier_database
tdelcey committed
add a .parquet file for scopus metadata for python usage
tdelcey committed
adjusting packages in paths and packages
Aurélien Goutsmedt committed
update project with istex fulltexts
tdelcey committed
Adjusting file names and variables for consistency: elsevier becomes scopus when necessary. Small adjustement in the metadata script to extract the new metadata.
Aurélien Goutsmedt committed
Finding new journals to collect via istex
Aurélien Goutsmedt committed
Reorganising file names for clarity
Aurélien Goutsmedt committed
saving missing journals in elsevier from wos
Aurélien Goutsmedt committed
minor changes on functions
Aurélien Goutsmedt committed
Matching elsevier metadata with WoS metadata
Aurélien Goutsmedt committed
Finishing cleaning the script to extract text from xml
Aurélien Goutsmedt committed
Decomposing `extracting_elsevier_data.R` in 3 scripts for extracting economics journals, extracting metadata of these journals, extracting full text xml files from these metadata.
Aurélien Goutsmedt committed
Same for path_and_packages after merging conflicts
Aurélien Goutsmedt committed
Finishing merging the code conflicts from the previous step.
Aurélien Goutsmedt committed
Change again on the way to introduce the API
Aurélien Goutsmedt committed
Change the way to ask for elsevier api key
Aurélien Goutsmedt committed
Reorganising files and first steps of the script to clean .xml files
Aurélien Goutsmedt committed
updates path
tdelcey_w committed
Improving the script to take care of file size.
Aurélien Goutsmedt committed
First commit
Aurélien Goutsmedt committed